Better handle Joomal Upgrades when "crazy" webhosts restrict /logs folders (or others)

[PhilETaylor]

Certain web hosts (Famously 1&1) maintain their own folders within a customers webspace - for 1&1 the example is /logs which is owned by the root user and is unwritable by their paying customer.

This causes issue when upgrading Joomla from within Joomla (e.g. 3.3.3 to 3.4.3) using the com_joomlaupdate and errors half way through the writing files directly with an unwritable with /logs/index.html

These leaves the upgrade in a half completed state - after upgrading the version number - and so most Joomla admins just assume all is ok as Joomla is now reporting the new version number - but most of their site is still running old version code, left ignored, this can lead to issues in the future of their site.

@nikosdion has already handled this in Akeeba Kickstart but this change has to be made to Joomla core also.

Ideally the upgrade should continue and report at the end, any files/folders it was unable to overwrite/write to - with some helpful explanation that the upgrade failed in part due to these.

At the moment the 1&1 /logs folder issue is the only one I can give you to explain this issue - but I know @nikosdion probably has other examples. Basically any non-writable file/folder in a webspace, that is also the name of a Joomla core file/folder will cause an upgrade to fail when using the internal core com_joomlaupdate

Thanks to a paying myJoomla.com customer for bringing this to the forefront again today after they came across this issue and left their site half upgraded and half not, and for @nikosdion for agreeing to look at this once the issue was raised here.

At the moment - upgrading Joomla from within Joomla - on any 1&1 hosting is impossible - making this quite an important fix for those customers.

[brianteeman]

Ideally the upgrade should continue and report at the end, any files/folders it was unable to overwrite/write to - with some helpful explanation that the upgrade failed in part due to these.

That's the problem Joomla does keep a record of this but its in the logs folder

[PhilETaylor]

Well for Joomla to work on this particular webhost, the configured log folder (as per configuration.php $log_path) is not /logs but something else (kickstart/akeeba makes it /log) and so if Joomla is correctly configured before the upgrade, and the log file is being written to where ever the configured Joomla log folder is (as per configuration.php $log_path) then that data should be readable to report it then :-)

The problem at the moment is you just get a JS popup that states /logs/index.html is unwritable and then it stalls back to a progress bar thats not moving anywhere... No other feedback

[brianteeman]

So the problem you are reporting is that Joomla is not using the $log_path during the update and is hard coded to /logs or that the JS popup needs to be worded better?

---

_{This comment was created with the J!Tracker Application at issues.joomla.org/joomla-cms/7603.}

[PhilETaylor]

No. Not at all.

The problem is that the Joomla core Stable and Upgrade files can, and do, contain and distribute https://github.com/joomla/joomla-cms/blob/staging/logs/index.html file and when com_joomlaupdate extracts the ZIP file it extracts /logs/index.html but then cannot write it to the /logs/index.html file on the hard disk as the permissions of the logs folder on some hosts is too restrictive - then gives JS popup and once the dialog is dismissed nothing happens.

The $log_path is irrelevant.

@nikosdion is fully aware of the issues involved and will be along shortly to explain why and what needs to be done - we have discussed this issue many times over the years and he already handles the problem correctly in Kickstart.

Although this is a known issue and reproducible on 1&1 with the /logs/index.html folder - the same error handling should be applied to ANY file that cannot be extracted from the update zip and overwritten on the hard drive. I'm guessing one could replicate this kind of issue by changing the permissions/ownership on any core Joomla folder/file and attempting an update.

[PhilETaylor]

Start an upgrade:

Error JS dialog pops up:

Dismiss the dialog and then get left with this:

[brianteeman]

Ah I understand you now - it wasnt clear (to me) exactly what you were referring to

On 31 July 2015 at 17:52, Phil Taylor notifications@github.com wrote:

No. Not at all.

The problem is that the Joomla core Stable and Upgrade files can, and do, contain and distribute https://github.com/joomla/joomla-cms/blob/staging/logs/index.html file and when com_joomlaupdate extracts the ZIP file it extracts /logs/index.html but then cannot write it to the /logs/index.html file on the hard disk as the permissions of the logs folder on some hosts is too restrictive - then gives JS popup and once the dialog is dismissed nothing happens.

The $log_path is irrelevant.

@nikosdion https://github.com/nikosdion is fully aware of the issues involved and will be along shortly to explain why and what needs to be done

• we have discussed this issue many times over the years and he already handles the problem correctly in Kickstart.

Although this is a known issue and reproducible on 1&1 with the /logs/index.html folder - the same error handling should be applied to ANY file that cannot be extracted from the update zip and overwritten on the hard drive. I'm guessing one could replicate this kind of issue by changing the permissions/ownership on any core Joomla folder/file and attempting an update.

— Reply to this email directly or view it on GitHub https://github.com/joomla/joomla-cms/issues/7603#issuecomment-126750416.

Brian Teeman Co-founder Joomla! and OpenSourceMatters Inc. http://brian.teeman.net/

[PhilETaylor]

Well communication has never been my strong point haha

[brianteeman]

So addressing the issue.

1. There really is no reason at all for /logs/index.html to be in the update pack.
2. We need a better error message in the JS popup
3. On closing the popup we need a better process

So perhaps

1. Remove /logs/index.html from the update package
2. Change the error message to "The update could not be completed because %s was not writeable. Please resolve this issue and perform the update again.
3. Dont know

On 31 July 2015 at 17:54, Phil Taylor notifications@github.com wrote:

Start an upgrade:

[image: screen shot 2015-07-31 at 17 53 28] https://cloud.githubusercontent.com/assets/400092/9012479/2591a186-37ad-11e5-8497-478641676bc4.png

Error JS dialog pops up:

[image: screen shot 2015-07-31 at 17 53 45] https://cloud.githubusercontent.com/assets/400092/9012483/2dbc4fb4-37ad-11e5-9c8f-193bee5c8c8b.png

Dismiss the dialog and then get left with this:

[image: screen shot 2015-07-31 at 17 53 51] https://cloud.githubusercontent.com/assets/400092/9012486/37b1e272-37ad-11e5-8ddd-bbe2c40a72b4.png

— Reply to this email directly or view it on GitHub https://github.com/joomla/joomla-cms/issues/7603#issuecomment-126750931.

Brian Teeman Co-founder Joomla! and OpenSourceMatters Inc. http://brian.teeman.net/

[mbabker]

Since Joomla is updated as a files extension, there are certain packaging requirements based on that and the extension manifest. One of them is that we package something in each top-level folder we ship (typically the index.html file). I don't know if we can remove logs/index.html (and inherently others like cache/index.html or tmp/index.html) from the update processes without negative repercussions; that would have probably as well tested as the update component was before it got merged.

[PhilETaylor]

Like I said before, dont restrict this issue/fix to this specific issue of logs/index.html - as there could be any number of reasons an update can fail for this general reason - for example, some hosts, on finding a file is hacked, set its permissions to 000 - so if you try to upgrade a site like that then the same issue could occur on any other Joomla core file or folder where the update process cannot write to that file or folder.

The correct "fix" is to run as much of the upgrade as possible and to document any files that the update was unable to overwrite/write to the hard disk. Much the same way that akeeba backup reports unreadable files (hacked normally) when backing up a site. like this:

[PhilETaylor]

and yes - for the record this site is hacked and those files on a 1&1 host are all set to perms "200" by the webhost :-(

[mbabker]

I was only answering the part about changing the packaging. The rest of your notes (which I haven't fully consumed but seem to make sense) are all valid points for consideration.

[brianteeman]

@mbabker What issue can there be removing the empty tmp, cache, and logs folders from the update. They only exist with the containing index.html so that we can force creating them which obviously does not need to be done on an update AND if the user has changed them to some other location then we dont need to create them

On 31 July 2015 at 18:05, Phil Taylor notifications@github.com wrote:

and yes - for the record this site is hacked and those files on a 1&1 host are all set to perms "200" by the webhost :-(

— Reply to this email directly or view it on GitHub https://github.com/joomla/joomla-cms/issues/7603#issuecomment-126753267.

Brian Teeman Co-founder Joomla! and OpenSourceMatters Inc. http://brian.teeman.net/

[PhilETaylor]

IIRC and I might not remember correctly, the reason we "had" to have a index.html in a logs/tmp folder is that SVN could not have a empty folder and so to force a folder creation we had to have a file in the folder.

Git allows for empty folders :)

[mbabker]

It has nothing to do with the VCS. I haven't dug into it in a while, but IIRC the way Joomla updates using the files extension type, we have to include all the top-level folders in the manifest and the package otherwise something misbehaved. I know Mark dealt with that a lot in earlier releases, I don't know if it is still valid today.

[brianteeman]

Are his issues documented anywhere? Otherwise we could just remove them from an update and try to see what problem exists in an update. Cant for the life of me imagine what could be the issue with those three specific folders.

On 31 July 2015 at 18:10, Michael Babker notifications@github.com wrote:

It has nothing to do with the VCS. I haven't dug into it in a while, but IIRC the way Joomla updates using the files extension type, we have to include all the top-level folders in the manifest and the package otherwise something misbehaved. I know Mark dealt with that a lot in earlier releases, I don't know if it is still valid today.

— Reply to this email directly or view it on GitHub https://github.com/joomla/joomla-cms/issues/7603#issuecomment-126754954.

Brian Teeman Co-founder Joomla! and OpenSourceMatters Inc. http://brian.teeman.net/

[mbabker]

You'd have to search the JC archive to try and find it at this point or follow the revision history in certain files (I wouldn't even know where to start off the top of my head). I think one of the cases he hit was that since the manifest declares all of the top level folders, they have to be in the package otherwise Joomla throws an error about a missing folder (so we'd have to programmatically alter the manifest at packaging to deal with that).

[PhilETaylor]

@brianteeman This is not just an issue with the Update packages. Think about a Joomla to Joomla 3.4.3 update using the internal Joomla com_joomlaupdate, this would use the Stable zip for the 3.4.3 version - which again would have this same issue if the /logs/index.html was present. ( I think Im right in this)

Also - and I say this again - this issue is not just about /logs/index.html - its about ANY unwritable file in the webspace that com_joomlaupdate cannot overwrite at the time of update - at the moment it bombs out leaving a site in an unstable state, partially upgraded, with no feedback about what was already upgraded, what still needs upgrading and what files caused issue...

[brianteeman]

@PhilETaylor why dont you start again with a NEW issue explaining correctly exactly the issue. It will stop people having to read through all the clarifiying posts to understand the actual issue

[PhilETaylor]

The first post in this github issue accurately describes the problem, and at the request of @nikosdion this issue was raised - I believe he is going to write the code to close this issue with a PR once people agree its an issue that needs to be fixed - which it is.

[nikosdion]

I can help you with that since I've bumped into this issue with Kickstart and Akeeba CMS Update.

Joomla! includes the logs/index.html file in the update package. This file is not required for the update packages. If someone is updating (instead of installing) Joomla! on a host with a user-unwritable logs folder (e.g. all Plesk-powered hosts) then they have already specified a different log directory in their Global Configuration or Joomla! cannot work.

But since this file is present com_joomlaupdate tries to write / upload it into the unwriteable logs directory and fails.

This problem is much deeper than the updater. Since the log directory MUST exist and MUST be writeable for Joomla! to work out of the box -at least with Debug Site enabled- it means that us using the directory name "logs" instead of "log" effectively forbids our users to use Joomla! on such hosts unless they manually edit configuration.php. The main reason people don't report that as an issue is that Akeeba Backup is smart enough to detect this issue and transparently work around it when you are restoring a site. Since the people who could report this issue are very likely using Akeeba Backup to transfer their sites you get no report about the issue. It's my "fault" :p

The proper fix would be renaming the directory "logs" to "log" and change the default configuration.php to use the new directory name. One directory rename and one (maybe two) line change.

[brianteeman]

While I agree with you @nikosdion and that's a quick and easy fix.

But Phil is also talking about issues when any files is unwritable.

[nikosdion]

When files are unwritable the update is supposed to fail. If you skip the unwritable files you end up with a broken site. If you stop the update you end up with a broken site. If you try displaying an error you end up with a broken site. Basically the problem is that we can't have an atomic operation with rollback should something go wrong. To the best of my knowledge you can't solve that issue in a web application that has to run on slow shared hosting which requires the FTP layer to write to files. At best we could tell the users to backup their sites before updating but wouldn't that be an indirect advertisement to the very few of us writing backup components? Anyway I look at it it's problematic.

[Fedik]

fix very simple:

• I would write this hosting support
• then I would write again, but with couple "tender words"
• and then change hosting, there a lot alternatives :smile:

why I should pay for this :hankey:

[nikosdion]

Plesk has been using the logs directory for access log digests long before Joomla or Mambo were conceived. So it is not the host's fault at all. It is our fault.

[brianteeman]

So lets bite the bullet and change the name in Joomla

[Kubik-Rubik]

@PhilETaylor This is a system folder that is generated automatically as @nikosdion said. You should not install Joomla! into the root or change the logs folder in Joomla!. The system logs folder will not be removed in the near future, it's our task to change in within Joomla!.

[PhilETaylor]

Well after 10 Joomla years and many Mambo years before - maybe it is time to change Joomla's core folders :8ball: good luck with that - its not been done all that time so fingers crossed it gets done... eventually :+1:

[PhilETaylor]

Forked into #7611 and #7612