Hackwar
commented
3 years ago I have spotted a few issues with the way the Joomla extensions update works. The Joomla project is already aware of these issues. I had privately discussed them in person with @wilsonge several years ago and emailed @zero-24 about them several months ago. No action has been taken. Just like when I had reported the same issues to your predecessors between 2013 and 2016. No action taken whatsoever.
Since the JED requires developers to use this subpar updater to be listed in the JED, the Joomla project needs to either fix these issues or allow developers to implement THEIR OWN updater code. It's not reasonable demanding that 3PDs use your updater and bear the temporal, financial and repetitional cost of its inadequacies.
Seeing how certain core contributors abused the discretion expressed in my previous disclosures of these issues in private to make unsubstantiated public claims I feel the urge to report them publicly. You can no longer pretend you have never heard of these issues I have reported to the Joomla project the last 8 years. You can no longer pretend that the JED rule makes perfect sense and somehow serves the 3PDs best interests when all it does is hinder us in delivering updates reliably to our users while bearing the costs of a decision imposed upon us by OpenSourceMatters Inc.
You cannot update the
extra_queryafter an update has been discoveredJoomla is using a weird update process. The update site has the
extra_querycolumn where the developer (or the user, as of Joomla 4) can enter an authentication query to the server delivering the update. Joomla retrieves the update information, populates the#__updatetable and copies over theextra_querycolumn. I know how this part works because I contributed it for Joomla 3.2.Before you start saying things like that I am reporting a bug I created please be advised that the fact it is implemented the way it is was NOT my choice. It was something discussed with Mark Dexter who explicitly framed my proposed contribution as something that should not change the way Joomla's extensions updates work. A few months after this contribution I had written a white paper on creating a better extensions updater that the Joomla project refused to consider. This was 8 years ago.
In the original version of the code updating the
extra_querycolumn in#__update_siteswould simply require to Clear Cache on the Update page and then use Fetch again to retrieve the new updates, copying over the updatedextra_querycolumn. Further to that, Joomla would display the server response which would be used by the developer to communicate that the user needs to enter the correct download key and refresh the updates. The latter was removed for security reasons but it's largely irrelevant since Joomla 4 does notify the user when an update key is missing.The problem is that it's no longer adequate to Clear Cache and fetch the updates again. The reason is that the queries are cached. There is a very delicate and frustrating 14 step dance you have to do to ACTUALLY be able to use a new download key in Joomla 4 when updates have already been found.
This has been a recurring issue which is costing us a great deal of support time and client goodwill on each and every update we publish. Every time we publish an update we know that we have to waste two days telling people to follow the 14 to 18 step process. This has happened more frequently ever since the rebuild button was added to the Update Sites management page, a feature which removes the
extra_querycolumn without the user ever knowing about it.This issue is largely fixed in Joomla 4 EXCEPT for the part concerning the
extra_querynot being used unless you also clear the administrator cache.Proposed solution: clear the query cache when you clear the update cache.
Files are not copied on update
This is a problem we have experienced since 2013, when we first started using the Joomla extensions updater instead of our own updater.
When installing an update to an extension Joomla will sometimes not replace all of the files included in the extension package.
This does not happen every time. It happens to about 3% to 5% of updates or at least that's the number we can infer based on support tickets. It might be substantially higher, just people not having realised because the file / folder not copied was either unchanged between versions, was about a feature they are not using or didn't cause a visible issue the user would report.
Our solution was to tell users to install the extension manually, twice in a row, without uninstalling it before or in between as you can see in this ticket from 2013. For some reason, the second installation attempt seemed to work. The only difference we could discern is that the version number was the same, therefore the cached XML manifest was identical between the version supposedly installed on the site and version being installed as an update.
In an effort to find a solution to this issue which doesn't require our clients to do a double installation — a completely unrealistic approach when you have dozens to hundreds of sites, especially since the advent of site management services — we have tried the following solutions over the years:
* Listing each and every file in the XML manifest instead of the top level files and folders. It had no effect and was quickly abandoned. * Shipping a list of files with their file sizes and CRC32 sums and check the consistency of the files when the client tried to use the extension. This was problematic if an FTP transfer in ASCII mode was involved at any time. Furthermore, the list of files and checksums was also subject to Joomla's bug of not copying files, rendering the whole exercise a moot point. It was quickly abandoned. * Rig our manifest script's `postflight()` method to go through the files shipped in the extension package (using the `$parent` adapter passed to it by Joomla) and comparing EACH AND EVERY ONE of the files' sizes. We didn't use a checksum because on larger extensions it can cause a timeout error in PHP or even the web server itself. If the file seems to be different we copy it manually. **THIS WORKED**. This is the only thing that made 7 years of supports requests about this issue come to an abrupt halt. FEF was the only package we ship that lacked this code, hence it bumping into that problem two days ago.Observing the results it looks like Joomla definitely has an issue here. I suspect it has to do with how it handles the XML manifest file before using it to determine which files and folders to copy on update.
This issue did happen with all of the updates we published on March 2nd because one package (FEF) did not have the workaround code.
It is, however, unreasonable to expect 3PDs to essentially implement an updater inside the updater to work around your updater being broken. It is equally unreasonable expecting users to use Upload and Install or Install from URL instead of the Update page to install updates, especially when there are web design companies with entire fleets of dozens to hundreds of sites.
Proposed solution: fix the installer so that it always uses the manifest information present INSIDE THE PACKAGE being installed. Alternatively, let 3PDs implement their own updater.
OPcache
Joomla's extensions updater essentially replaces parts of the site itself, including executable code (.php files).
Unlike what was the case in 2010, it's not guaranteed that writing a file to the server's storage will result in that file being loaded. Modern PHP-based sites are very likely to use OPcache to accelerate the site by caching the parsed opcode of frequently used .php files. Given the way Joomla works, when installing updates this definitely includes the files of
systemandinstallerplugins and any code they loaded during the request which initiated the update and the request before it (this can include components' and other extensions' .php files).This is highly problematic for two reasons:
* If the post-installation code in the XML manifest script relies on the newly installed files being loaded it's very likely that they are already in OPcache and that OPcache has `opcache.revalidate_freq` set to a value higher than the time it takes for the extensions updater to download and install an update. This also applies when doing a manual update using Upload and Install or Install from URL. This means that the post-installation code is being served older versions of these files which might not be compatible with type hinting in its methods or otherwise cause fatal errors due to the disparity between the expected and the installed version. I was bitten by this exact issue with the Akeeba Backup 8.0.0 and Admin Tools 6.0.0 updates two days ago. * After the update is complete it's possible that the server uses the old code of system plugins but loads the new code of a component or other extension these plugins are dependent on. Since the check to see if the extension they depend on is installed and activated goes through Joomla's ComponentHelper et al is uses database data. Checking files for existence or even their content is also not reliable because the files do exist, their content is what it should be but PHP is using an older version of the file which is no longer there. I was also bitten by this issue in the same versions of my software.I thought I had solved this issue by using opcache_reset() in the beginning of my
postflight()method. This is the same trick I am using in the Joomla Update code I contributed to Joomla in 2.5.1 for the exact same reason. However, the PHP documentation is wrong or, at the very least, incomplete.opcache_reset()does NOT reset the opcode cache immediately, even when it returns true. What it really does is that it WILL reset the opcache when PHP is exiting. This solves the second issue but not the first. I was bitten by this in the Akeeba Backup 8.0.1 and Admin Tools 6.0.2 updates.The solution was to use
opcache_invalidate()on each .php file being installed which does work as documented, i.e. it immediately invalidates the opcache for the given file. Akeeba Backup 8.0.2 and Admin Tools 6.0.2 prove beyond any doubt that this approach does work. This is critical for the proposed solution.Proposed solution: since Joomla already has code to copy each and every file in the extension package it should check if the file being copied has a .php extension and invalidate the OPcache for the file's final location. This should happen conditionally, so Joomla doesn't break if OPcache is not installed.
function_exists()is your friend.
Quoting this here just to prevent whitewashing this report here later on. 😘
nikosdion
blueforce
infograf768
I have spotted a few issues with the way the Joomla extensions update works. The Joomla project is already aware of these issues. I had privately discussed them in person with @wilsonge several years ago and emailed @zero-24 about them several months ago. No action has been taken. Just like when I had reported the same issues to your predecessors between 2013 and 2016. No action taken whatsoever.
Since the JED requires developers to use this subpar updater to be listed in the JED, the Joomla project needs to either fix these issues or allow developers to implement THEIR OWN updater code. It's not reasonable demanding that 3PDs use your updater and bear the temporal, financial and repetitional cost of its inadequacies.
Seeing how certain core contributors abused the discretion expressed in my previous disclosures of these issues in private to make unsubstantiated public claims I feel the urge to report them publicly. You can no longer pretend you have never heard of these issues I have reported to the Joomla project the last 8 years. You can no longer pretend that the JED rule makes perfect sense and somehow serves the 3PDs best interests when all it does is hinder us in delivering updates reliably to our users while bearing the costs of a decision imposed upon us by OpenSourceMatters Inc.
You cannot update the
extra_queryafter an update has been discoveredJoomla is using a weird update process. The update site has the
extra_querycolumn where the developer (or the user, as of Joomla 4) can enter an authentication query to the server delivering the update. Joomla retrieves the update information, populates the#__updatetable and copies over theextra_querycolumn. I know how this part works because I contributed it for Joomla 3.2.Before you start saying things like that I am reporting a bug I created please be advised that the fact it is implemented the way it is was NOT my choice. It was something discussed with Mark Dexter who explicitly framed my proposed contribution as something that should not change the way Joomla's extensions updates work. A few months after this contribution I had written a white paper on creating a better extensions updater that the Joomla project refused to consider. This was 8 years ago.
In the original version of the code updating the
extra_querycolumn in#__update_siteswould simply require to Clear Cache on the Update page and then use Fetch again to retrieve the new updates, copying over the updatedextra_querycolumn. Further to that, Joomla would display the server response which would be used by the developer to communicate that the user needs to enter the correct download key and refresh the updates. The latter was removed for security reasons but it's largely irrelevant since Joomla 4 does notify the user when an update key is missing.The problem is that it's no longer adequate to Clear Cache and fetch the updates again. The reason is that the queries are cached. There is a very delicate and frustrating 14 step dance you have to do to ACTUALLY be able to use a new download key in Joomla 4 when updates have already been found.
This has been a recurring issue which is costing us a great deal of support time and client goodwill on each and every update we publish. Every time we publish an update we know that we have to waste two days telling people to follow the 14 to 18 step process. This has happened more frequently ever since the rebuild button was added to the Update Sites management page, a feature which removes the
extra_querycolumn without the user ever knowing about it.This issue is largely fixed in Joomla 4 EXCEPT for the part concerning the
extra_querynot being used unless you also clear the administrator cache.Proposed solution: clear the query cache when you clear the update cache.
Files are not copied on update
This is a problem we have experienced since 2013, when we first started using the Joomla extensions updater instead of our own updater.
When installing an update to an extension Joomla will sometimes not replace all of the files included in the extension package.
This does not happen every time. It happens to about 3% to 5% of updates or at least that's the number we can infer based on support tickets. It might be substantially higher, just people not having realised because the file / folder not copied was either unchanged between versions, was about a feature they are not using or didn't cause a visible issue the user would report.
Our solution was to tell users to install the extension manually, twice in a row, without uninstalling it before or in between as you can see in this ticket from 2013. For some reason, the second installation attempt seemed to work. The only difference we could discern is that the version number was the same, therefore the cached XML manifest was identical between the version supposedly installed on the site and version being installed as an update.
In an effort to find a solution to this issue which doesn't require our clients to do a double installation — a completely unrealistic approach when you have dozens to hundreds of sites, especially since the advent of site management services — we have tried the following solutions over the years:
postflight()method to go through the files shipped in the extension package (using the$parentadapter passed to it by Joomla) and comparing EACH AND EVERY ONE of the files' sizes. We didn't use a checksum because on larger extensions it can cause a timeout error in PHP or even the web server itself. If the file seems to be different we copy it manually. THIS WORKED. This is the only thing that made 7 years of supports requests about this issue come to an abrupt halt. FEF was the only package we ship that lacked this code, hence it bumping into that problem two days ago.Observing the results it looks like Joomla definitely has an issue here. I suspect it has to do with how it handles the XML manifest file before using it to determine which files and folders to copy on update.
This issue did happen with all of the updates we published on March 2nd because one package (FEF) did not have the workaround code.
It is, however, unreasonable to expect 3PDs to essentially implement an updater inside the updater to work around your updater being broken. It is equally unreasonable expecting users to use Upload and Install or Install from URL instead of the Update page to install updates, especially when there are web design companies with entire fleets of dozens to hundreds of sites.
Proposed solution: fix the installer so that it always uses the manifest information present INSIDE THE PACKAGE being installed. Alternatively, let 3PDs implement their own updater.
OPcache
Joomla's extensions updater essentially replaces parts of the site itself, including executable code (.php files).
Unlike what was the case in 2010, it's not guaranteed that writing a file to the server's storage will result in that file being loaded. Modern PHP-based sites are very likely to use OPcache to accelerate the site by caching the parsed opcode of frequently used .php files. Given the way Joomla works, when installing updates this definitely includes the files of
systemandinstallerplugins and any code they loaded during the request which initiated the update and the request before it (this can include components' and other extensions' .php files).This is highly problematic for two reasons:
opcache.revalidate_freqset to a value higher than the time it takes for the extensions updater to download and install an update. This also applies when doing a manual update using Upload and Install or Install from URL. This means that the post-installation code is being served older versions of these files which might not be compatible with type hinting in its methods or otherwise cause fatal errors due to the disparity between the expected and the installed version. I was bitten by this exact issue with the Akeeba Backup 8.0.0 and Admin Tools 6.0.0 updates two days ago.I thought I had solved this issue by using opcache_reset() in the beginning of my
postflight()method. This is the same trick I am using in the Joomla Update code I contributed to Joomla in 2.5.1 for the exact same reason. However, the PHP documentation is wrong or, at the very least, incomplete.opcache_reset()does NOT reset the opcode cache immediately, even when it returns true. What it really does is that it WILL reset the opcache when PHP is exiting. This solves the second issue but not the first. I was bitten by this in the Akeeba Backup 8.0.1 and Admin Tools 6.0.2 updates.The solution was to use
opcache_invalidate()on each .php file being installed which does work as documented, i.e. it immediately invalidates the opcache for the given file. Akeeba Backup 8.0.2 and Admin Tools 6.0.2 prove beyond any doubt that this approach does work. This is critical for the proposed solution.Proposed solution: since Joomla already has code to copy each and every file in the extension package it should check if the file being copied has a .php extension and invalidate the OPcache for the file's final location. This should happen conditionally, so Joomla doesn't break if OPcache is not installed.
function_exists()is your friend.