Closed smxi closed 6 years ago
smxi
commented
6 years ago I double checked, using a trick to find newest files
Inside of the 'core' the only files that were newer than 2017-01-01 were in: packages, cache, and components.
Nothing else was newer, so 'core' definitely did not upgrade.
Note again, the actual name of our 'core' is not 'core', and it's below document root.
BobRay
commented
6 years ago Does the config.core.php file in the MODX root contain the correct path to your core? That's what UGM uses to place the core files. Does the original core directory still exist?
smxi
commented
6 years ago The files
/config.core.php /[manager]/config.core.php /[connectors]/config.core.php
All contain the correct path to the [core] that is below document root. I tried it again, and noticed this 'warning' which wasn't open but is there, probably means nothing. What I find odd/worrying is that modx itself seems quite convinced that it has upgraded itself, yet it has not, which suggests that the installer needs some more checks, since obviously if nothing got updated, the error handlers in the installer are not working at all in this regard.
I tried it again, and it's the same result, nothing in 'core' was updated, the installer believes it all was updated, but the only thing that actually got updated was 'manager'
I'm not sure what you mean re does the original core directory still exist, the site's 'core' directory exists, it's what runs the site, I actually tried moving it but then of course the manager page crashes since it has no core files to work from.
Config file successfully written. Config file permissions were not updated. You may want to change the permissions on your config file to secure the file from tampering. Error updating table for class modUser Array ( [0] => 00000 [1] => [2] => ) Legacy file clean up complete. Removed 9 file(s) and 1 folder(s). Successfully created table for class modAccessNamespace
smxi
commented
6 years ago Note that normally when I upgrade modx, I download the latest -pl-advanced version, extract it to the setup folder, move the existing modx core folder out of the way, then let the installer rebuild the stuff, but this has to be done manually. Then after the install I copy over the packages/ and components/ from the backup core to the new core. This is roughly how I upgrade wordpress too by the way, and that's proven to be fairly reliable since it gets rid of the problem of modx not checking for write permissions in core and let's it simply recreate it. But this manual process is what I'm hoping to get around.
smxi
commented
6 years ago Just as an aside, I've done various widely used free software projects for years, more than a few, and I can assure you, it's trivially easy to make something work on your own test systems, I've gotten complex features running in a few hours that worked fine for my stuff, then I started getting user bug / issue reports, and it took me literally 1 month to get something that would usually work for most users. That process has been the norm, not the exception, but it's what makes the difference between robust software and stuff that kind of works sometimes if everything is just so. It's handling when things are not just so that proves the reliability of the tools. I suspect that both issue 32 and my issue may expose some assumptions made about general states of installs that are not globally valid, and thus, lead to failures.
Of all the things modx needs, it needs your installer to make upgrades quick and safe and reliable and easy, that's been my top complaint about modx since I started using it, really my only one.
For example, the wordpress upgrade, using in place directory replacement, usually takes me about 2 minutes total, literally (5 or 6 if you include my initial local dev tests to confirm no issues, that is, 5 or 6 minutes for 2 full upgrades). that's because wordpress avoided the lethal error modx committed, having variable user added content, like plugins, themes, etc, mixed in with the directories that require updating, wordpress corrected this problem by having a 3rd directory, that contains only user added materials, which you don't have to touch during upgrades. Note that the wordpress 1 click upgrader I found lethal and would never use, but it's so easy just uploading the new stuff that I find it almost risk free.
If you can figure out the issues, which I believe would lie in creating a far more robust file/directory permission pre install set of tests, you might be able to fix in one fell swoop the main problem with running modx, which would be an amazing achievement in my opinion, it's really I think the primary stumbling block modx has always faced re expanding its user base. Tests like that would not I suspect require more than a few hundred new lines of code to fix most issues permanently for most users.
If I had some idea of where to look for the failure, I'd write you a patch for your upgrader tool, but since it never knows it failed, I doubt any failures were logged.
BobRay
commented
6 years ago I agree with everything you say, but I've had no luck duplicating the problem, and no one else has reported it. I've tried to find something that would cause it, with no luck. It's very unlikely that it is related to MODX's directory structure.
If you can figure out what's causing it on your platform, I can probably fix it, or tell you how to work around it.
The most likely scenario, imo is that UGM is downloading the correct files, but putting them in the wrong place, or is unable to overwrite the existing files due to permission or ownership problems. so when UGM launches setup, setup is installing the existing files.
If you comment out line 394 of the UpgradeMODXSnippetScriptSource chunk:
MODXInstaller::removeFolder($tempDir, true);
change it to this:
// MODXInstaller::removeFolder($tempDir, true);
The uznipped, downloaded files should remain in the in the /ugemtemp directory. The top folder there should reflect the name of the MODX version that was downloaded. That would tell us if the problem comes before or after the download.
[Update] Here's something else that's really simple to try. Launch UGM by clicking on the Upgrade MODX Button, and do a "View Source" on the page when the form appears. You could see code like this:
<label><input type="radio" name="modx" value="2.5.7-pl"> <span>MODX Revolution 2.5.7-pl</span></label><br>
--
| <label><input type="radio" name="modx" value="2.5.6-pl"><span>MODX Revolution 2.5.6-pl</span></label><br>See if the versions numbers on the left match those on the right.
smxi
commented
6 years ago <label><input type="radio" name="modx" value="2.5.7-pl"> <span>MODX Revolution 2.5.7-pl</span></label><br>
<label><input type="radio" name="modx" value="2.5.6-pl"> <span>MODX Revolution 2.5.6-pl</span></label><br>
<label><input type="radio" name="modx" value="2.5.5-pl"> <span>MODX Revolution 2.5.5-pl</span></label><br>
<label><input type="radio" name="modx" value="2.5.4-pl"> <span>MODX Revolution 2.5.4-pl</span></label><br>
<label><input type="radio" name="modx" value="2.5.2-pl"> <span>MODX Revolution 2.5.2-pl</span></label><br></div>
<input type="hidden" name="userId" value="1">As you can see, no issues. Also, as I noted, I checked the modified dates of the [manager] and [core] after doing the failed update, and the [manager] files were updated, the [core] files were not.
Since the files that updated manager came from the downloaded gz package, that package clearly downloaded, and contained files that were newer than the previous ones.
The way this is tested is fairly simple:
go inside the directory you want to check the last modified dates of:
touch -t 201701010000 touchtime
find * -type f -name *.php -newer touchtimefor instance, will show all newer .php files.
What I'd need to do here I believe is add in some permission checks to your updater module, which would work around the installer failure to actually check the permissions of the directories it needs to write to.
This isn't difficult, it's just difficult to know where to inject the tests.
Once verified, with better output on failure, then these checks could be moved upstream to the actual installer, which should have been doing this job all along, though it never has.
I'll run the other test and report back, but technically I already know the answer, since when gz files are extracted and moved to a directory, they will have the last modified date of the files they were made out of, which tells you when roughly the compressed file was created.
smxi
commented
6 years ago Testing the upgrade, I noticed this has happened:
in /[manager]/config.core.php
as soon as I start the installer, this file is recreated in [manager], it has the WRONG path in it.
I don't recall this being an issue in the past.
Note that there's another actual bug in the installer, which is that it's failing to read the existing /site/config.core.php mod core path, and always asks for the file location, this is obviously a bug, since if the file exists, it should have used it as the default value, and simply confirmed that the path there is the one you wanted.
/*
* This file is managed by the installation process. Any modifications to it may get overwritten.
* Add customizations to the $config_options array in `core/config/config.inc.php`.
*
*/
define('MODX_CORE_PATH', dirname(dirname(__FILE__)) . '/core/');
define('MODX_CONFIG_KEY', 'config');I tried it again, but I do not at any point see a directory anywhere called /ugemtemp is that supposed to be in site document root?
I do however see in [core]/packages/core new files, including a manifest that is updated today.
All the directories in that [core]/packages/core show a last modified date of 2017-08-29, so clearly they got written to.
this is clearly a failure to check some critical permissions, but, far worse in my opinion, after failing to check them, not realizing that the files are actually not in place, and are in fact not the files of the new install. And, even worse, not checking the actual version number to verify that the install actually happened.
I can supply some programming help to fix this. The failure itself is a crystal clear example of total failure to check anything at all in the installer process, beyond those initial installer checks it runs and gives a green text ok to, those are obviously inadequate and incomplete, as are the success checks, which don't appear to be checking anything at all, or the installer should have reported failure, since it did fail.
I think if I were to try to submit a patch, the first place I'd start is having the installer actually LOOK for the paths in /config.core.php since if that file exists, the values in it obviously are what should be used. It's kind of silly that in 2017, a trivially easy test: does file exist? does value differ from default? if so, use value isn't being run there. Or, even more simple: use value from that file. If that path exists, then use it.
My feeling is it would be easier to test out patches for enhanced tests on your extension, since I have direct access to that and it doesn't get updated.
BobRay
commented
6 years ago Thanks again for all the testing. I really don't think this has anything to do with file permissions any more. Here's what I think is happening based on your symptoms.
First, UGM and setup are two separate and independent programs. UGM puts the files in place (and it looks like it's doing so). Then it launches setup, which for some reason is confused about where your core is, so it copies its own source files to where it thinks the core is, which of course has no effect on the site.
When Setup can't find the core in its default location, it asks you where it is, and I think this is where things are going wrong. I think Setup is getting the path to the core wrong. Is it possible that you're not giving it the correct, full path (not a URL) to your actual core directory? Or is there maybe a proxy or rewrite rule that would change the path you're giving it?
I'm not sure what you mean re does the original core directory still exist, the site's 'core' directory exists, it's what runs the site, I actually tried moving it but then of course the manager page crashes since it has no core files to work from.
Sorry, I didn't ask that clearly. By default, the core file is called 'core' and is directly under the MODX root. And directly under that is the core/config/config.inc.php file. If Setup can't find that file (and obviously it can't, since it's asking you where it is) it would normally mean that you've moved or renamed the core directory or the config directory. I was asking if that was the case.
If the core has its default name and is in the default location (under the MODX root directory along with the manager, connectors, and setup directories), this problem has nothing to do with UpgradeMODX, since it never touches the config files or their directories. Something about your configuration is confusing Setup.
If I'm reading you right, (and I might not be) UGM is downloading the files and putting them in the correct place. That would mean the problem is with the MODX Setup script, which is used successfully by many thousands of people. Setup does pretty good tests of permissions, so this suggests that either the path you're supplying is not correct, or there's something pretty unusual about your system.
I hope this all makes sense.
smxi
commented
6 years ago I'm not ignoring this, i just have been busy, by the way. I'm deciding whether to actually allocate time to fix the core installer bug, or to see if I can add a hack to your upgrading extension. Or to just give up on it, and do the manual installs.
BobRay
commented
6 years ago I don't think there's anything you can to to UGM that will help. If I'm understanding the situation, the problem is caused by a faulty interaction between your environment and the MODX Setup script.
Doing the upgrade the old-fashioned way by copying the files may give you the same result.
smxi
commented
6 years ago I'm not convinced of that, but testing takes time. I can assure you, I NEVER get this result doing it manually, core goes to where I tell it to go. Otherwise I would never have been able to upgrade modx, afterall.
There's a few things I can test once I have time to do it, the tests are tedious to do, but I have a guess I'm going to check on my next test.
The problem here is that I can't really easily hack the modx installer itself, but the fact that nothing even knows or checks to see if core gets placed where it's supposed to go, is actually kind of amazing, that's a pretty basic check to make, and trivially easy to do, so it's hard to understand how an oversight like this could even happen in the first place.
One thing I know is that when you use the advanced pl installer, the only thing you upload is the setup directory, it creates manager and core upgrades by itself at that point. modxUpgrade won't support that because the extension needs the core files to run in the first place. That is, core is recreated by the setup zip, it's not coming from anywhere else.
I'm wondering if the pl download vs the pl-advanced download is different in some fundamental way that hasn't been considered, if that's the case, then the issue might actually be with the upgradeModx tool in this scenario.
BobRay
commented
6 years ago You may be right, but the UGM script doesn't use MODX at all. The widget does, but only to find out if there's an upgrade available and write the versions file. It's the script in the MODX chunk that actually does the file copying and it never instantiates MODX.
A good start might be to comment out the code that deletes the files and see if the new core files go to the place you're telling Setup to find them.
smxi
commented
6 years ago I'm going to go ahead and close this issue, but to do so, I ended up writing my own updater. There's a few probably critical differences, one, I'm using modx-version-pl-advanced, which is different from the regular modx-version-pl download. I found the direct download, so it works sort of like your thing, except all from command line.
I took advantage of the new modx CLI installer, which works fine after I got the issues ironed out, but I think my conclusion would be, don't try using upgrademodx for the pl-advanced install, I debated spending time trying to debug the modx installer failure vs writing my own, and decided the time would be pretty similar, except I'd end up with something that maybe works ok if I did it myself.
The use of the config.xml file is useful, finally on an install the things found the paths correctly, no issues.
BobRay
commented
6 years ago Sorry, I've been kind of dense in responding to this. I somehow missed the fact that you were upgrading an Advanced distribution install with UGM (even though it's clearly stated in the OP). Since UGM always downloads the Traditional Distribution, I suspect that you're right that this is a problem with UGM handling Advanced installs. If I can find some time, I'll look into it.
smxi
commented
6 years ago It took about 300 lines of code to fully do it (bash in my case, though i was tempted to use Perl), but I added some bells and whistles that aren't strictly necessary, like logging upgrades in a text file so we'll know what the last version was, when the last upgrade was, and what the present version is. I wouldn't try doing this without the CLI installer option, which fully supports the various type options, with their config.xml files. But I did get it all working, download, sync local db to live db, basically everything I needed and used to do manually. I think it was probably better that I bit the bullet and scripted it myself since then I could make it do everything I used to do manually.
I don't believe the standard pl will work with pl-advanced, and I can tell you that when you pass the CLI installer the right arguments, all those annoying old failures to get the right path for core, manager, etc, all went away.
I have to give modx kudos for making the CLI installer option, it's much cleaner in my opinion. And, unlike everything I've tried in the past, it works. After download, the upgrade on the live server took 2.6 seconds, heh, including setting all the permissions etc, maybe 3 total. Basically zero downtime.
BobRay
commented
6 years ago I don't believe the standard pl will work with pl-advanced
In some ways it's odd that it doesn't. All the MODX files are in the Traditional Distro and UGM puts them where they belong based on your config.core.php and config.inc.php files, then it launches setup, which asks you where the core is and finds out where the other directories are from the config.inc.php file.. Everything really ought to work for there on.
UGM works fine on an install with a moved and renamed core and a renamed manager. It shouldn't really matter that the original install came from the Advanced distro, but apparently it does. Setup consults the core/docs/version.inc.php file to tell which distro is being used, but UGM should overwrite that file so setup thinks it's the Traditional distro.
smxi
commented
6 years ago I have never, not once, had the setup installer work as it should, it never reads the config.xx.php files, it always overwrites them, then I have to replace the overwritten files, and let the installer test again.
It's useful to look at the differences between the 3 config.xml files in the setup directory:
Basic 'dist.upgrade'
<modx>
<!-- Set this to 1 if you are using MODX from Git or extracted it from the full MODX package to the server prior
to installation. -->
<inplace>0</inplace>
<!-- Set this to 1 if you have manually extracted the core package from the file core/packages/core.transport.zip.
This will reduce the time it takes for the installation process on systems that do not allow the PHP time_limit
and Apache script execution time settings to be altered. -->
<unpacked>0</unpacked>
<!-- The language to upgrade MODX for. This will set the default manager language to this. Use IANA codes. -->
<language>en</language>
<!-- Paths for your MODX core directory -->
<core_path>/www/modx/core/</core_path>
<!-- Whether or not to remove the setup/ directory after installation. -->
<remove_setup_directory>1</remove_setup_directory>
</modx>Vs dist.upgrade-advanced:
<modx>
<database_type>mysql</database_type>
<database_server>localhost</database_server>
<database>modx_modx</database>
<database_user>db_username</database_user>
<database_password>db_password</database_password>
<database_connection_charset>utf8</database_connection_charset>
<database_charset>utf8</database_charset>
<database_collation>utf8_general_ci</database_collation>
<table_prefix>modx_</table_prefix>
<https_port>443</https_port>
<http_host>localhost</http_host>
<cache_disabled>0</cache_disabled>
<!-- Set this to 1 if you are using MODX from Git or extracted it from the full MODX package to the server prior
to installation. -->
<inplace>0</inplace>
<!-- Set this to 1 if you have manually extracted the core package from the file core/packages/core.transport.zip.
This will reduce the time it takes for the installation process on systems that do not allow the PHP time_limit
and Apache script execution time settings to be altered. -->
<unpacked>0</unpacked>
<!-- The language to install MODX for. This will set the default manager language to this. Use IANA codes. -->
<language>en</language>
<!-- Paths for your MODX core directory -->
<core_path>/www/modx/core/</core_path>
<!-- Paths for the default contexts that are installed. -->
<context_mgr_path>/www/modx/manager/</context_mgr_path>
<context_mgr_url>/modx/manager/</context_mgr_url>
<context_connectors_path>/www/modx/connectors/</context_connectors_path>
<context_connectors_url>/modx/connectors/</context_connectors_url>
<context_web_path>/www/modx/</context_web_path>
<context_web_url>/modx/</context_web_url>
<!-- Whether or not to remove the setup/ directory after installation. -->
<remove_setup_directory>1</remove_setup_directory>
</modx>Basically dist.upgrade-advanced has all the information a fresh install config.xml has.
What had always happened in the past, until using this CLI installer, was that I'd extract setup/core, move core to replace the existing xxx-core location, move setup to site document root, run setup via browser, and it would always, without exception ,fail to read the config files, it would overwrite the main config file in /, which I'd then replace with the real one, and refresh the installer page, at which point it would proceed.
php $SITE_PATH/public_html/setup/index.php --core_path=$SITE_PATH/core-path/ --config=$SITE_PATH/config-files/config.xml --installmode=upgrade-advanced
While I have a ton of code wrapping around this, this is what works. $SITE_PATH is the path to the main site directory, below site document root, where modx etc live.
UpgradeModx when it ran setup using modx-pl, would totally ignore the config files (I assume the pl installer ignored them, that is, don't know), installed core to nowhere, had no idea it had not actually installed anything, but did install some stuff to manager for some reason, if I remember right. I assume this is the difference between running the -pl and -pl-advanced version of the installer, don't know.
What worried me there was the absolute and total absence of any error handling or checking in the installer, which means I really can't trust it anyway, if it thinks it wrote core to somewhere but it didn't write it to anywhere, that's worrisome, heh.
Note that 'core' is named something else, and is below document root, manager, connectors, are named something else as well.
I wish I'd learned about the CLI installer stuff earlier, oh well.
From now on I'm always going to skip the gui installers and use the config file and command line installer, that's predictable and way easier too.
BobRay
commented
6 years ago I have never, not once, had the setup installer work as it should
In that case, I suspect that there's something odd about the way your site is set up. Tens of thousands of people have used Setup successfully. I've probably used it a couple hundred times myself, almost always on sites with a renamed manager and moved and renamed core.
The CLI method is definitely better, but many people either don't have command-line access or wouldn't be comfortable doing it that way.
UpgradeModx when it ran setup using modx-pl, would totally ignore the config files
If you mean the config.core.php file in the root and config.inc.php, that really can't happen. If UGM can't find them, it won't run at all.
`if (file_exists('config.core.php')) { @include 'config.core.php'; } if (!defined('MODX_CORE_PATH')) { MODXInstaller::quit('Could not read config.core.php'); }
@include MODX_CORE_PATH . 'config/' . MODX_CONFIG_KEY . '.inc.php';
if (!defined('MODX_CONNECTORS_PATH')) {
MODXInstaller::quit ('Could not read main config file');
}`In any event, I'm glad you got things sorted.
smxi
commented
6 years ago if (!defined('MODX_CORE_PATH')) {
That''s the problem, the default file defines it, and if it overwrites the modified version, the path is still defined, but wrong. That's an example of a totally inadequate series of probes by the installer, and inadequate checks, and inadequate error handling. That's why I wrote my own.
That's what happened in my case I believe, for example, running the pl-advanced setup with UpgradeMODX -> modx setup.
When I would check that file, where that is defined, it had been overwritten by the setup routine, which resulted in incorrect core location. However, note, your upgrademodx downloads -pl, not pl-advanced, and I've never tried pl, don't know the differences, but judging from the config files, the differences are very significant, particularly for paths.
https://modx.com/download/other-downloads
Current Version – 2.6.0 Advanced Distribution
2,819 total direct downloads since Nov 1, 2017 631 downloads and counting…
This suggests to me most pl-advanced users are doing what I wish I'd discovered years ago, using the CLI installer, and grabbing the file with curl or wget or something like that.
Again, if you read the above observations, I tested this pretty carefullly before deciding to give up on the gui method and script it, the /config.core.php file got overwritten by the setup, alawys, it did not read it, and because it was overwritten, the location of the core was dropped to default, which is defined, but wrong. That's the bug. Or rather, it's one of the bugs.
For example, on those counts, I assume the smaller number is gui downloads via the advanced download page, and a download, for example, I did one, does not equal an install or upgrade, it just means a download. Then, there is truly no way to know of those numbers how many are new installs and how many are upgrades, so we could be looking at relatively miniscule gui installs/upgrades of pl-advanced, and of those, it's not possible to know how many put core in a non default location.
These numbers are certainly small enough to cover up existing bugs in certain scenarios.
I was faced with the prospect of spending client time/money on A: tracking down the modx installer bug for pl-advanced, which is not fun, and very hard to actually test for and locate, and which could have resulted in either finding the bug, filing a bug report with patch, and having it fixed, or failing to find it, or finding it and not having it fixed (I filed a bug report to the dev of a code editor maybe 4 years ago, and it was only fixed this last year, even though he submitted the patch live as I was talking to him 4 years ago, for example).
Or, B: At that point, I couldn't really justify billing the client for time that might have resulted in no fix, so I'm glad I waited, and wrote the full installer logic myself, now I can easily trace down in particularly permissions issues.
For example, the event of writing a non existent core and not reporting an error and exit can't happen with my scripting, because it verifies every single step of the way, and has error handlers for each step, so ti can exit without breaking anything too seriously.
A few years back, I was almost going to submit my steps to do a reasonably safe upgrade, but my notes were so messy that it would have take me too long to clean them up and abstract them for regular installs. As part of this process, I did finally cleanup that documentation, and make it somewhat readable, but it's still a mess and I wouldn't trust it for general use.
I could actually do that more easily with my updated installer logic, since it's programming, without any question at all, I'd use that for all commercial modx installs since it's just 2 files, the installer, which would need to be updated re paths and db and passwords etc, and the config file, same, paths, passwords, db names, etc.
I'm actually now far more likely to do more modx installs now that I have a scriptable install/upgrade solution, I'd avoided it in the past.
The problem however is with relying on only ones own setups to determine if things work, because in general, we tend as developers to do things in the same way, so it's unlikely that a bug would exist that is triggered by the way we do things, otherwise we would find and fix it. that's the value of bug reports from other people, for example, you had now two reports of failure from people who are not you, both are technically unresolved, which means the bug or bugs exists, but is not resolved.
On my free software projects, stuff always works for my setups, since any bugs that are triggered by my stuff are fixed by me since I find them during testing and use, the more interesting bugs are found by other users, for example, one of my programs ended up triggering a bug in a common downloader tool, and that bug today was just sent upstream because in fact, we were the ones to discover it. It did NOT trigger on my setups, note, but it did for others, who had a different version of the downloader installed, the one with the bug.
Of all my long standing issues for a major client, modx upgrades were front and center because they were so scary, lol.
smxi
commented
6 years ago By the way, the actual problem with the modx installer is an architecture issue, while I would never point to wordpress as being good for anything but a bloated blog platform, the one thing they have always done correctly, from the start, was to totally separate user installed things that can vary in content from what are basically the two directories that are core/manager equivalents.
Also, having multiple config files is an error, that should never have existed, it's a pain to deal with.
To be clear, a wordpress install is like this:
/configfile.php /wp-admin /wp-content /wp-includes /... a bunch of root .php files.
ALL user modifiable content, including extensions, themes, etc, go in /wp-content. You never need to change that, in fact, I haven't done more than modify my themes and upload new extension versions to that directory in 10 years.
The config file remains untouched, wordpress never touches it, it does not try to be too clever for its own good, therefore, the issue with accidental overwrites simply never happens.
wp-admin, which corresponds to /manager in modx, is simply deleted, and replaced with the new one, period. No permissions issues, no failures to overwrite, no lingering old fiiles from legacy releases lingering forever. wp-includes is basically core, and it's the same, you delete it, and replace it with the new one, and there is zero way it can fail.
With one caveat: I never use the wordpress automated upgrade option, though I'm sure that is by now working very well, last time I tried it, wordpress blew up, but they have so many users I'm sure it works solidly by now.
Modx made one critical error in its initial design, well, a series of them, one was having too many locations with config files, I believe it's 3, or is it 4? one in setup, one in manager, one in doc root, and one in core. That was a serious error. The second serious error was allowing user content which can change, like extension installs, to be in core, and, what's even worse to have TWO places it can be, one in core, one in /connectors. connectors shouldn't exist at all, in fact, when I've tried syncing modx installs, I always forget that some extensions may be located in EITHER connectors OR in the packages in core, again, that's a serious error because it introduces unpredictable ambiguity to the locations.
These errors are what have caused all my install issues, and the sad thing is, none of them should have existed at all.
If I were modx, I'd fix this, but as long as the cli installer works and takes data from the config.xml file, I believe that resolves some but not all of these issues. I still for example have the script mv core to a new location for backup, then write core to old location, and copy over directories to core, like cache, which is NOT in the default, and the various writable directories for extentions.
In the old days, I would have joined the project,and tried to correct these errors, but those days are behind me, now I'm just too impatient to spend that much time on stuff without getting paid for it.
BobRay
commented
6 years ago That''s the problem, the default file defines it, and if it overwrites the modified version, the path is still defined, but wrong.
If that were the case this code would abort the program:
if (!defined('MODX_CONNECTORS_PATH')) {
MODXInstaller::quit ('Could not read main config file');
}It's finding the config.inc.php file (the only place the MODX_CONNECTORS_PATH is defined), which can't have been overwritten since it's not in the download package. The only explanation I can think of is that you have another core directory at the location specified in the default config.core.php file with its own config.inc.php file pointing to that "other" core.
smxi
commented
6 years ago connectors and manager always were found, though permission checks in manager are not done, an issue that has created problems for me in the past.
Again, all the paths are defined, so none of those error cases will ever trigger.
I believe that is maybe the cause of at least some of the pl-advanced issues, relying on simple tests for defined when the actual test needs to be much more robust, that is, does file exist, if so, use it (which means, overwriting could simply not happen, ever). Then, if file exists, and if defined, and if not exists location, then... (that test is missing).
I'm glad I bit the bullet, which made me discover the cli option, and the config.xml option, these two seem to simply bypass the bugs in the setup routine, and force all the correct values to be used, which solves the issues, most of them anyway.
smxi
commented
6 years ago if ( ( !defined('MODX_CONNECTORS_PATH') ) || ! file_exists(MODX_CONNECTORS_PATH) ) {
for instance. But connectors has never been an issue, that comes later in the process, the issue has always been reading core location, respecting and not overwriting the core definition files, then reading those values, checking them, etc. Well, and not actually checking internal directory permissions, that's been a bug that has bitten me in the past, but I can now fix that too in my installer script by simply adding necessary permission checks.
What strikes me is that in fact, when they made the CLI installer, they corrected all these bugs without realizing it (or, more accurate, they actually bypassed the bugs, but left them in place, unfixed), but because they are corrected in cli install code, they have not been fixed in the gui version. And that's very easy to understand the reason for,, as a programmer, when you receive the data in the xml file, you do not try to be clever and have the programming discover the data for itself, you simply read the xml file and use those values, and presto, all the bugs vanish.
And you can kind of tell too the difficulty there was in setting core paths, because the cli installer itself allows you, redundantly, to set the core path in the arguments as well as in the config.xml file. I also had my script set the path by overwriting the setup/ config file as well, with the correct values, so there are actually 3 places where the core path is now set in my version, and, for the first time, the core path was located and used without any further issue.
And, as a further aside, that also shows fairly clearly that the bug is in the setup installer, where they set these values, NOT in my specific setup, since if the bug was in my setup, the cli installer would fail just like the gui version does. Which means, the bug is in the steps in the setup logic before the data is loaded from the xml file.
BobRay
commented
6 years ago For others who read this: I installed the Advanced Distribution of MODX on a couple different servers, renaming the directories. I then upgraded those installs several times with UpgradeMODX with no trouble.
One minor surprise was that Upgrade MODX converted the Advanced Distribution to the Traditional Distribution -- as it should, since it's downloading the Traditional distro.
As a side note: there's no difference at all between the Advanced and Traditional distributions once they're installed, other than the string in the version file, which MODX never looks at except when running setup.
smxi
commented
6 years ago As an aside, you're still making the same error, which it took me a long time to figure out.
Over time and various moves by the site, developer stuff, etc, permissions on the modx files can start to morph.
It took me quite a while to finally resolve the issues, and I may or may not put out as Free Software my scripted solution, but given it assumes basic technical competence, and I certainly don't want to be put in a position of supporting users who don't have that competence when they do it wrong, I will probably end up not doing it.
This assumes several things:
Here's the steps I found were required to get a rock solid upgrade, and I realized, basically what I'm doing in this logic is emulating the same procedure I used for flawless wordpress upgrades, which by the way, also blew up using the gui tool, and I now know why that happens.
But before anything else, the fact remains, much as I like your other work, the upgrader did NOT work, period. No ifs ands or buts, it did not work. So I suspect your installs are all following the same methods, and your upgrader works within that context, but it is not reliable and does not work outside of those contexts. Our sites, and my dev sites, are all outside of that context.
here's the basic pseudo code I did, I used Bash, running on Linux in my case. Note that most of these steps were taken from my own documentation on how to do upgrades, but during testing and development of this new upgrade tool, I found several areas where modx has no docs at all about certain issues (permission changes in manager for sure, for example, never once mentioned), so I had to significantly tighten all tests and pre install methods and error handlers, and add many permissions tests and changes during all steps.
All of these steps are automated except for clearing cache, which I could and probably should automate too, and checking that the site works, manager, pages.
clear cache, logout
download using rsync a gz'ed version of our modx db
update the local dev server db, change the paths for the site in the db, using cli mysql
check manager and site pages for errors
download the advanced installer zip file
extract it into a modx staging directory, which also contains the CLI xml upgrade files, and a copy of the core config php file
test all relevant locations in the modx install for read/write permissions (this is where yours fails I believe).
This is important, every step above will exit on error, so nothing bad will happen at all if any of these steps fails. As I tested, and I tested a lot, I hit more and more corner failure cases, and added more and more error handlers, so it would exit always before any damage was done to the live site.
move, mv, the core and manager directories to a temporary location. mv is the only way to handle the permission conflicts, because rm cannot handle permissions not made by the file owner, and if any files were created by modx, those are owned by www apache user, and the regular login ssh user CANNOT remove those files. Unless your server is seriously misconfigured, or unless you have root on the server. However, note, my goal was to never have to think about root, because we use pro managed servers, and part of that management agreement is that we don't get root. Tradeoff, worth it for us.
then move setup directory to site root, and core directory to the old core directory location, renamed if needed.
at this point, and this is an issue I had for years, permissions in manager made upgrades fail, which led to obsolete files in manager not getting upgraded, until manager became so degraded it would start corrupting. I was very fortunate to have a live and dev version of a site that had not been upgraded since I think modx 2.2 to do my rough testing and dev on, so I was able to trigger all the failure points, this proved to be a real blessing to my efforts since basically the entire manager exploded on impact, and failed totally.
That was the point where I realized that manager could never safely be upgraded with files in place, it was the same as core, you have to totally remove its contents, period, which means of course, you cannot operate the upgrade from inside of manager, which is why the gui method would never work reliably. I think I may have restored from backup a few times during testing, I can't remember, but I had the proper permissions issues triggered to where I was able to figure out the only safe way to do any upgrade.
Once I realized that there was the CLI upgrade method, with xml configs, which means, I no longer had to depend on a buggy system that would fail to find the paths etc, fail to read the configs that exist, and would simply bypass these steps and read the config xml, which is what the installer should be doing anyway, I was able to start playing with the options until I found the way to get the manager files replace, after removing them to another directory. It took me a bit of trial and error to realize that both had to be set to 0 to fully replace the manager contents, otherwise only core got replaced.
<inplace>0</inplace>
<unpacked>0</unpacked>Now that it's all solid, I take the backup directory core/packages file, recreate the cache directory, which for some inexplicable reason is not created by the installer, reset all the permissions in core (and that took a lot more trial and error, because I would set them, then the installer would actually reset them, so I had to reset the permissions again). Also note: when you extract from zip the core/setup directories, they come out with NO READ PERMISSIONS for anyone but site owner. This is an error by the people who make modx installer zip files, you can change those permissions before zipping it, but that caught me off guard, because the first test I did gave me the dreaded no permissions which luckily for me was solved by a sys admin friend of mine who I happened to be on the phone with, who said, that's probably directory permissions, and sure enough, it was, no read/execute access in those directories to anyone but the user who extracted them (that's: 700). That was easily resolved by setting the permissions on them before moving them to / and core location.
Now I was close, but I still had the issue of getting rid of the directories and fiies that were owned by hosting account user, not root, and by apache/www user, one method cannot remove both, but I already knew this from previous attempts to delete the directories once they had been moved using only a php apache command. So the solution, and it again, took quite a bit of testing, involved running through the directory that now contained the old stuff, minus the packages and components directories, which were moved back into the new core. To solve the deletion issue, which has always plagued me with modx, I created a bash routine AND php page on the site, first I had bash remove all the files and directories it can from core and manager backups, THEN I run the php apache page, which, because it's running as apache www user, nobody in this case, that's the name,nobody, can delete the files that were owned by apache. I suppress errors on these first two deletions because they don't matter, and then I have to run both again, I do not know why, and in those, I do not suppress errors, because I want to and need to see if any files remain that are not deleted. The second run through always works, don't know why two of both are required, bash deletion, apache deletion, bash deletion, apache deletion. The bash script calls the php page by lynx, thus transforming the acting permissions to apache www user nobody, who can delete apache owned files.
Ok, that was the dev server. You have now verified that all upgrades worked, pages work, manager works, logins work, everything works. This process takes a few minutes. I follow a documented list of steps re which command line options to run in which sequence to avoid errors. Once verified that the dev server upgrade worked, run the same exact steps, except of course for syncing the database from the live server, on the live server. Note that the actual time that the site is offline is under 3 seconds max in my tests, and may be much less, since there is no pause for commands to be typed in once the actual upgrade component is run, that's very very fast. I consider it basically as zero downtime on the live site.
this is what is required to do a 100% rock solid modx upgrade, Manager MUST be cleared of all contents, period, if it is not, over time you will end up with permissions conflicts, which have always happened to me with modx installs.
Because I am using bash now to do the initial permissions checks, and resets of the pages modx wants to touch, I no longer have to do that absolutely terrible modx recommendation to set / to 777 permissions, which is crazy, and was always installer instruction bug number one.
Now I have the script set the file permissions where required, then after the install succeeds, it resets them to the correct, and I never have to deal with the wrong advice to set / to 777, which by the way doesn't even work since once apache has created files in manager, you can no longer reset their permissions, which starts the entire merry go round again.
It took about 400 lines of code in the basic script, plus the php cleaner file, and a few days of testing and tweaks and finding new ways to make it fail, to get this to the point where I can upgrade with a series of simple commands, reliably, and in my opinion, unless a new modx installer issue appears, in a way where I can finally trust the process completely. I could in theory run it as one action, but that removes the ability to stop and check everything is working after each main step.
I believe the options from the script will be enough guideline for anyone who wants and needs to permanently fix upgrade issues, and the ensuing terror I have always felt at various upgrade problems.:
modxu -h
These are the options:
---------------------------------------------------------------------------
-a - Remove site/manager*/ directory contents.
-b - Remove modx-core-bu directory after successful install.
-c - Remove modx-core/cache/ directory contents.
-d - [ version: like: -d 2.6.0 ] Download current modx pl-advanced.
-f - Full directory clean: cache, manager, and core-bu.
-h - This help menu
-i - Install Modx, update permissions, etc
-m - Download live modx db file, install to local system
-p - Print upgrade data, last uprade date, version, and previous upgrades.
-u - [ version: like: -u 2.6.0 ] The zip file must be located in the local
modx-staging directory, and must be modx-2.6.0-pl-advanced.zip syntax.
This option is not required usually since -d can do it on live and local.
-y - Clean staging data. Like -z, except leaves modx .zip file(s).
-z - Clean staging data. Deletes staging/setup|core|*.zip, site/setup filesAs I said, I was greatly fortunate to have an old install, two, one live, and one dev, same site, that let me find all the possible failure points, particularly on old installs, so this is very very robust.
Note that the options are not for other people to understand, they are for me, to be more clear they would need more words to each option, or a README for it, at least. So they aren't all super clear. -u for example updates using the downloaded zip installer file by moving it with sftp to the live server, then running the upgrader remotely, but that is not a feature I used except in very early initial testing, and I wouldn't use it now because it's faster to upgrade the live site directly. The various cleaner options are for installs that have very badly stuck permission issues, and all were added based on real world real situations that I saw firsthand.
Also note, the xml config installer file for advanced has all the options, but the basic installer xml file only has a subset of the options. You need all the options in the xml file.
After I got this roughly stable, I added logging of all updates, the log file goes into the modx-staging directory, and is not part of modx, which then allows a convenient reference for when the last upgrade ran, it also reads the existing full modx version files from the new core and the old core as well, so that it logs what it was upgraded from, and to what version, including advanced or not advanced.
After I tested this by up, and downgrading, modx, over and over, on all the dev sites and live sites, I felt finally, for the first time ever, comfortable with the process of upgrading modx. I tip my hate to whoever had the wisdom to create the modx cli upgrade // install tool, it is my conclusion that an upgrade that runs from inside the gui can never be safe, that is what I found with wordpress too, by the way, and why I evolved a very similar, though much easier, with fewer steps, method for upgrading wordpress.
In my opinion, after heavy testing of all these tests etc, and having for years had buggy and glitchy modx upgrades, not one of these steps can be safely left out if you run a real production server with a commercial site, and in fact, now that I finally have the modx upgrade issue totally resolved, I would never use the gui installer again, nor would I ever use the gui upgrader again. I am also now far more likely to use modx more than I did in the past, since the upgrades now do not cause me any stress, and take a few minutes per install.
BobRay
commented
6 years ago If I'm understanding you, you changed the permissions on MODX files, for what reason I can't imagine, and rather than just changing them back with a simple shell command, you blamed UGM and Setup and wrote a script that did it for you in a rather inefficient manner. MODX never changes file permissions, nor do any MODX extras that I'm aware of.
BTW, The part of UGM that failed for you does not, and never did, run in a GUI. Setup does, but I strongly suspect that it would not have had any trouble if the permissions had not been altered.
That said, both Setup and UGM might do a better job of testing file permissions, though that could increase the risk of PHP timeouts.
It's unreasonable to expect all MODX or WordPress users to use shell commands to upgrade, and millions of them use the built-in upgrade capability with no trouble.
smxi
commented
6 years ago You are misunderstanding, yes, that's correct.
I'm not using an inefficient manner, I'm using the first thing that has ever worked, rock solid, and that solves all the problems. Every solution I scripted was based on reality, not an opinion of what reality might be, the failures were there and not inventions.
Apache owns all web generated files. The modx docs clearly and explicitly state to set permissions on the command line, this may not mean much to windows users, I don't know, can't say, but the errors I experienced were repeated.
To say they might do a better job of testing for file permissions is a significant understatement, they don't do that at all beyond a very superficial level. They never have either, that's why I have experienced breaks of manager consistently.
The reason I noted that I would not release this script as free software is specifically because I do not want to support windows users, or people who don't understand the command line, or whatever else can be done wrong, but the reason I would think of releasing it would be simply as a service for admins who know what I'm talking about, and are tired of the glitches over time.
There aren't millions of modx users, lol. Not of the advanced install.
from today: Advanced Distribution 2,062 downloads and counting…
And remember, it's unlikely those are all installs, some may be just kicking tires.
I think you are massively underestimating the variety of installer scenarios and site scenarios for users of the expert advanced install, but I also suspect most serious modx admiins have already done roughly what I am doing, and are probably not even grabbing the installer from the website gui, but getting it the same way I am.
So again, we're not talking about millions of cases, we're talking about hundreds, a few thousand max.
That is why the issues exist, it's obvious to me, too few cases, too varied installs, many end up doing what I did, do it all by command line.
The millions are not using pl-advanced, they are using the default one, I don't know those counts, but I suspect it's well below the millions. Plus the modx managed services.
note that I posted the full logic not to promote a debate or defensive reaction, I posted it because the upgrader failed, and I found a rock solid solution that if I post the logic to, is enough for most real modx admins to figure out how to code it.
I also found out why these things fail, and why the wordpress gui fails, and why doing it the way I am doing it is by far, far far, the safest and most reliable for advanced pl installs.
Note that I've given you 2, actually 4, failure cases, someone else posted here 1, so that's 5 who took the time to post, and that's a significant percentage of those listed 2000 pl-advanced downloads.
My strong, strong, suspicion, is that all serious modx admins, or almost all, long since moved to the cli installer/upgrader method, and you won't even see those in any stats unless the downloader tool in the modx website counts raw wget requests, or curl requests, etc, for the pl-advanced tool. If I'd been aware of the cli installer/upgrader tool, I would have moved to it the day I found it, for example. I was lucky to run across it during some search while developing this, by the way.
Your sample size is simply too small to make any conclusions, and I doubt you run this stuff the way I do, either as dev or live sites, so it's not about my doing it wrong, it's about my triggering failure cases that are not handled by the gui installer tools.
Note that they can be handled, but not completely, which is why I realized cli all the way, with a small php page deletion boost, was the way to go.
This isn't excessive, it's simply smart, I have now developed the ability to run upgrades in a few minutes without worry, previously they were so stressful that I simply did not do them, plus each one cost my client about 300 dollars, which is not cool, now they are barely billable, so that's a massive upgrade.
400 lines of code is not very much to anyone who can program, and probably 100 of those are for things I ended up not really needing except for one time first time cleanups of corrupted installs, so we're really talking about 300 lines to permanently fix a serious issue.
Regular gui users, as I noted, don't interest me, nor would I make any attempts to support them, that's just a headache.
I may end up releasing this, or I may not, but my strong feeling is that most devs who do it for professional clients already evolved something very much along these lines.
I really appreciate you and all your support etc, as you know, but you are simply not right in this case, I've done enough hard core programming to know when the time has come to fix it myself. My only real choice was whether to fix the installer in modx, or to program my own fix. After realizing what is needed, I'm glad I opted for the second option, since the first isn't actually fixable.
Shell/command line is usually the best solution for all real sys admin problems I find, and the time spent on crafting those solutions is not a waste, it's usually a permanent fix, as was the case this time, I now view the modx upgrade issue, which was the top issue by far I had with modx, as totally solved. And again, I am very very grateful to the wiz kid who decided to bypass all the internal gui installer stuff, and create the xml config file fueled cli upgrader, hats off, many times, that person understands the problem, and the solution.
BobRay
commented
6 years ago I don't mean to sound defensive, but we disagree about why you had difficulty with UGM, I was clearly referring to "MODX or WordPress" users combined.
I'm not clear on this: are you saying that you corrected the file permissions and things still didn't work? Other than the permissions issue, I don't see an explanation of what caused your problems.
The advanced distribution and the traditional distribution are essentially identical except for the way the files are put in place. Moving and renaming the directories is possible and common with either one. Neither distribution will be installable without some chicanery if the file permissions have been altered.
The reason there's little or no checking of file permissions is that most people don't alter them.
smxi
commented
6 years ago regular ssh user cannot change apache www nobody owned file permissions, and apache www nobody cannot change hosting user account, that's what you login with via ssh, owned files.
That is and always has been the problem with my upgrades.
The modx install instructions clearly say to change the permissions, but note, I've found that more than once I've hit issues by actually fully reading the directions, and following them literally. However, those instructions have always been flawed, but by necessity, primarily because you cannot get permission to create index.php from the setup installer without having / be at least 707. Note that even there, the modx instructions are both unclear and wrong, you don't actually need 777, usually, almost always, 707 is enough. However, on reflection, I think even that issue can be resolved simply by creating an empty index.php file, then giving it rw global permissions pre install, then letting the upgrader run and replace it. The installer doesn't actually need / 707, what it needs is index.php 606, config file 606, manager 707, and assets, connectors at 707, that is, you can just create all those files and directories, and make them writable for the installer, and never touch your / permissions at all, I think. that's using the cli installer, and my updater script, which sets and resets all those permissions.
Note also the following: if you upload to site via ftp or download from site, the zip installer, then extract it, by definition, those files are then owned by user, NOT by apache. Then, when apache creates new files, say, cache files, for example, it owns those, and the user cannot delete them via cli.
What I discovered in what I consider my best case development scenarios, very old installs, some moved, etc, some with permissions issues, was how to make a rock solid, will always work, upgrade path. The solution there was the same as for wordpress, mv the two core directories, core/manager in modx, wp-admin/wp-includes in wordpress, but in wordpress, you also remove all the other files in / except for the config file, then simply move all the directories and files from the staging directory to the / directory.
With wordpress, since I didn't script this, I get a roughly 15 second downtime, but I do move all the files after ssh'ing into the server, but if I scripted that process, it would be the same as modx, about 2 or 3 seconds down, plus running the upgrade.php in wordpress, but again, that can be done using a cli lynx request to the page, no need for being in gui at all.
It's because I learned this upgrade method for wordpress years ago that I finally realized that modx and wordpress are almost the same, with the one exception of wordpress never storing user modified content in wp-admin or wp-includes, modx does store stuff in core, so I had to add the step of moving those directories back into core from the old version that is backed up until deleted.
Also, note, NEVER has modx installer ever seen the real path to core, not one time, no matter what I tried, I always had to enter the path, have it fail, then edit the paths in the files again anyway. the CLI method is so robust that it supplies you with a redundant way of specifying the core path, one on the cli arguments, and one in the config file. I used both, since I was sick of the problem, lol.
What I wanted, and what I now have, the same as with wordpress, is a totally error free and flawless and regardless of permission issues way to upgrade. Rock solid, able to handle any permission issues, really anything at all, with one exception, if root owns files, that would be a case it can't handle unless you ran it as root, but that's a bad idea in my opinion in all cases.
I'm quite pleased with the solution, for the first time, I was hitting a few commands, upgrading seamlessly, live, remote, updating local dev db, everything I do, and all the headaches were gone. Took headaches to get to that point, of course, but I considered a good long term time investment to permanently solve this problem. It was.
If I were to become a modx installer by profession, I'd just add a few first time install tweaks to the script, and never touch the gui again. Each new site would use pl-advanced, with xml advanced config file, which I would simply type up for the new site, local and dev versions, then install, would take a few minutes per install, maybe make dummy first time files etc like index.php to avoid permissions resets on global /, which the cli does not, thank god, check for.
Every upgrade would be done the same way, cli, using a staging directory, predictable xml config files, and the cli installer. Technically with some more work on this script, I could literally become a modx admin/installer for work, if I wanted, but I don't want, lol, but it is now an option for me should I chose to do it.
smxi
commented
6 years ago By the way, my one experience with the wordpress gui updater, I did not make that install, a client made it, so I figured, what the heck, I'll try it. However, one thing really stopped me from ever using that again, aside from the fact it failed, and that was its request for my site's ftp connection information, which is NOT NECESSARY!!!
I would NEVER trust wordpress or any other site with that information, that's insane.
However, since I knew I was going to nuke that wordpress install anyway, I decided, hey, what the heck, let's try their gui upgrader. Note: I had not touched this site, it was a default install, so forget any notion of my changing permissions or anything. So I tried it. It failed, and that was it, my last attempt to use the wordpress gui upgrade tool. Never even thought about touching that again.
Adopted the flawless, never fails method, have been running the same wordpress install now for over 10 years. Always do the same thing, backup dev site, download new wp zip, extract, delete admin/includes, delete all/ files, move new files/folders to dev site, run upgrade. Confirm works, then do the same for live site. That's how I knew how to solve the modx issue, though it took me a bit to realize it was the same problem.
I may now however script a wordpress upgrader tool to skip the manual steps and be like my modx upgrader, since that drops site down time to about 1 second I think.
And now, lol, I really have to get back to work, because for me, modx is our cms, it's not something I want to spend time doing admin on, etc, that's not my real job, it's just a thing I have to do in order to keep it updated, my job is writing code. The upgrade issues had simply created a very bad situation where I dreaded doing them so much due to things always gliitching that I didn't do them, which meant, I was not doing timely security updates. With this new methods, because it's basically just a few minutes per install, most of those are checking for errors on site, I can do:
So 3 things would have to pass and a fluke on the live real site would have to exist for any real failure, most likely is one of the 3 previous tests would show failure first, at which point, I'd stop, and see if I can figure it out, do the modx forums, look for solutions, once found, run the final one.
For example, the recent failure caused by the stuck files from the manager code editor, you, as usual, gave the solution to that on the forums, so I found it on the test sites, and knew what to fix on the live site before even running the upgrade, so no issues.
BobRay
commented
6 years ago I will absolutely agree that Setup is brain-dead about handling a moved/renamed core, especially since config.core.php is right there in the MODX root and it's recommended to move and rename the core. UGM uses config.core.php and config.inc.php to place the files.
How did the permissions get changed in the first place? Would it not have been possible to change them as a sudo user?
I'm still not clear on the answer to this question: "you corrected the file permissions and things still didn't work?
smxi
commented
6 years ago As noted, on our live sites, we opted for managed servers, that's because I can't run servers safely, so I want the hosting company to do it for us. they are very good at it, but one reason they are good at it is they do not allow their managed servers users to have root access. sudo - is the same as allowing root access, so that was never an option.
Note also, because of the majjor upgrade headaches, one install, both my local and live version, had been dorment since I believe 2012, version 2.2.x, so I have no memory of that one at all. All I know is that my initial test on upgrading it failed totally, which is what made me realize I needed a far more robust solution. To the great credit of modx (who do some things SO well, but others, like the installer, not so much), the upgrade actually worked fine, 2.2 -> 2.6, no real issues, though I also downgraded to 2.5 for a bit to see if that was the cause of the issues. When I say it worked fine, I mean, it installed, the db updated, but the site itself was totally blown, lol, no pages worked.
Note that once you get into trying to change permissions on manager or core as a whole, you are entering into a deep dark rabit hole, with no documentation at all to help you, and the installers do not check or test anything inside those directories.
If I were to guess I'd guess I probably ran chmod -R 707 on the directory as regular user to try to solve an upgrade failure, or something like that, but I really can't say, I hadn't upgraded one since 2012, and the other since 2015, so about 5 years and 2 years respectively.
The reason for that was my cheat sheet of upgrade safely rules, which are now fully automated, plus having been seriously tested bit by bit to find out what is needed and what is not needed, had grown so messy, and the procedure so messy, that I simply put it off, since failure meant our site was down, with very few options to fix it available. So I just put it off. That's unwise, to put it mildly, for web facing software, but that's what happened.
I'm not sure about your last question, your upgrader never worked, so that was never in question, it simply did not work, no core was written to the real core, no manager was updated. It however did not know this, or should I say to be more correct, the modx installer did not know this, which is what led me to conclude that I'd be much better off writing a tool that would in fact know when things failed, and exit gracefully with clear error messages in the process.
The failure to read alternate core locations to me just points to some fundamental failures in the modx installer, since that is really absurd, I mean, find file, open file, read file, that is not very advanced programming, so I did not have great hopes for actually debugging the actual modx installer and since I had to assign priority and billing hours, I picked the course of action with the highest likelihood of success. I was not mistaken in this, though discovering the modx cli installer was a true blessing since I wasn't even aware of that option when I started hacking at it.
root would also not have solved the permissions issues in manager, which really needs to be fully replaced every update anyway, I mean, over time, all kinds of junk builds up in it, the installer does not appear to really check for stuff to remove, and if the permissions are glitched, most stuff fails to update, until the manager blows up. So there's really no reason to keep the manager code in the upgrade, that's what I learned, just dump it, just like I already dump wp-admin as well.
To me, all of those weak spots were known by at least one person, which I believe was the person who did the cli installer method. He was obviously aware of the issues, since he perfectly programmed around them, bypassing the problem area, and there's no way in my opinion that was an accident, since his solution fixes the issues totally, by bypassing that entire logic internally I assume, then reconnecting to the solid working part of the system. That's exactly what I would do as a programmer if I felt I could not fix due to technical or internal political issues a real problem, I'd just hack my way around it, as in fact I just did, that's what he did to in my opinion.
I actually chose that site to test because nobody really cares about it anymore, we don't use it, it's been dormant, so I knew, if I was going to make a site blow up, and fail, that was the site to do it on. And boy did it blow up, lol. At least until I solved the upgrade failures, which took a few hours of serious testing and debugging.
I've always, since day one, found the modx problem #1 was their upgrades, nothing else was a problem at all, in every other way I think it's a great, great cms, made for developers, well designed, well architected, with that one exception, and that series of rough spots manifests by things like putting user modified content in core, not creating a new cache directory by default on install, and a bunch of other oversights in certain install modes.
Ideally you should be able to nuke core, dump it, replace it, and have the user data, like extensions etc installed, live in another directory, that's how wordpress does it, and I'm hard pressed to find many good things to say about wordpress, particularly as a cms, but it's slow and bloated for blogging too, but this one part they got right, the upgrades simply do not fail (unless you risk their gui updater, lol)
However do not in any way misunderstand me, I have tremendous respect and appreciation for all your help, both for me, your plugins in general, and your support on the forums. Without you, modx community would be in serious trouble, lol.
smxi
commented
6 years ago By the way, there is NO documentation at all on the manager issue, cruft build up, failure to update key files, etc. Nor is there any documenation anywhere that I saw that observes that by simply checking that one xml config option, and nuking the contents of manager, it all rebuilds from scratch, and totally solves the manager update issues.
that's significant in my opinion, for my case, I only tried that as a desperate last guess, then guessed some more at the xml config file options, which when correctly set, replace all the files, not caring if they are there nor not. Again, without that option, the solution would not have been fixable, and certainly only pure guesswork got me to that point, plus being willing to destroy the install while testing it, which I did repeatedly, though never of course the actual db.
I doubt most users or developers would be willing to take those chances on a real site, which could be why there is this big knowledge gap in the docs about these types of failures. I wasn't willing to take those chances on my dev site for example, nor on the live real site, but since I had a local and live version of the abandoned site, it was ok to break it if necessary.
I believe that almost none of the solutions I found are documented on modx anywhere. The cli installer has many hints, and the xml config files give you searchable values to really start to understand what the various options do, and I think that was the reason I could solve the problem.
The difficulty in testing and debugging, and the total and absolute lack of documentation in key areas, and seriously incomplete docs in others, is by the way why I would consider releasing the upgrade cli tool as free/open source software, the problems are difficult to understand and solve, and require extensive testing to develop, and of course, you need systems that fail, so you can create all the error handlers and fixes to handle all cases, not just a few best case scenarios.
However, we all know that no matter how much upper case warning I put on the repo source, newbie users would do it wrong, or ask stupid questions, or want their hands held, and that's the part I'd refuse to do, plus I would not want to subject modx forum people to having to try to support such users. Note that all advanced admin types would never fail since it's very easy to setup and understand, but that assumes cli understanding, and configuration understanding, etc.
BobRay
commented
6 years ago It's not a bad strategy, and I'm glad it worked for you, but it would nuke any custom Manager templates a user had installed in the Manager folder, which (though it shouldn't) will make the Manager crash hard.
I would never put anything in the Manager folder, but some people do.
I am curious about how, if the Manager files were not writable, you had permission to remove them.
smxi
commented
6 years ago As I said, the solution, and note I'd already been using a simplified form of this previously, but not adequate, was to create a web page that uses php exec() or system() to delete the files owned by apache.
However, because I'd never actually sat down and really seriously tested this logic before I fully redid it, I'd always noticed failures, that is, apache could not remove all the files, and bash could not remove all the files, but for some reason, running both twice, bash, apache, bash, apache, does work.
The method, and parts of this I discovered years ago since there was no other way to do it:
mv core backup/
mv manager/* backup/manager
... do upgrade, on success
mv backup/core/components core/
mv backup/core/packages core/
mkdir core/cache
# and a few other directories.
chmod -R 707 core/components core/packages core/cache
# then remove cli and apache
rm -rf backup/core
rm -rf backup/manager/*
# run the php apache script, which basically uses apache user permissions to:
rm -rf backup/core
rm -rf backup/manager/*
# then run both again
rm -rf backup/core
rm -rf backup/manager/*
# run the php apache script, which basically uses apache user permissions to:
rm -rf backup/core
rm -rf backup/manager/*If all error handlers and other protection logic were removed, the actual logic is only about 50 lines of code, but that's very dangerous to run so I'd never do it.
Note that unlike cp or rm, mv does not require the strictest permissions contained in the directory, since the files themselves are not altered. I'd tried using mv backup/core > /dev/null, but the OS wouldn't be tricked that way, heh.
A template folder in manager, which, again, is the same mistake modx makes with allowing user content into core, would be handled the same, assuming the template folders are sane:
mv manager/* backup/manager/ install... mv backup/manager/usertemplate manager/
that's the same you have to do for the packages and components directories anyway, again, user modified content should NOT be in core or manager, that's a real mistake, it should be in a user folder, like:
core/
modx-user/
- manager-templates
- components
- packages
site/connectors/
- manager-templates
- other web stuff from plugins
site/managerdepends on if the contents need to be web accessible or just server accessible. And modx already has the user content directory /connectors so they were obviously aware that you need that approach but they didn't complete it in manager and core.
However, once you know which directories have to be moved back from the backup directories, it's trivial to then move them back before deleting and accessing for example manager. That would be a single line, well actually 3, since it would be a conditional:
if not null manager-template-directory;then
mv backup/manager/template-directory site/manager/
fiNow, if the templates are allowed to scatter js and css and templates all through manager, that's a massive design error which should be flagged and banned. But if all the template stuff goes into the template directory, then it's no different from moving components and packages back into core.
I think sometimes projects get a bit too wedded to the gui, sigh, when the gui is not always the proper solution to all problems, in fact, when it comes to server admin, while the low learning curves guis can provide are useful, there are far too many cases where gui tools are used that should never be used, phpmyadmin for example comes to mind. And where the cli is actually a better solution, maybe not newbie easy, but technically better. I know of few cases in the areas I work with where guis are better, though I make them for the staff etc, but better? no, almost never.
By admin of course, I mean the actual admin stuff, not users updating pages, etc, which is why we use modx, I mean the real admin stuff, like updating the install, running databases, etc.
What's kind of funny is that I find that scripting gui solutions usually takes about 10x more time than it takes to script a far more robust cli solution, you don't need to make the forms, the js, the css, the html, and adding features is trivial, usually just a line or two of code. If you look at for example the cli installer for modx, it's got args, easy to parse, easy to add more to, really an extra option often may be just a few more lines of code in a cli tool, but it's a major endeavor in a gui application. And it's got predictable xml files, which are likewise really easy to edit, and once you understand the xml options, far less opaque than the gui stuff. Plus once you do one config, and one cli thing, you know how to do all subsequent ones, makes it really easy, copy paste, etc.
This may or may not be related to issue #32.
I decided to test upgradeModx since if it works for us it's a huge time saver, so testing on local dev system first.
Current modx: 2.4.1-pl installed from the 2.4.1-pl-advanced modx installer. PHP 5.6.26-1 Apache 2.4
Does not use standard connections/ manager/ or core/ directory names, but checking inside the 'manager' directory, I can see files were updated there. Same in [manager]/controllers, files are updated. Note that 'core' is below document root.
It does not look to me like any of the 'core' files were updated.
I probably should have checked if upgrading from a start install using 2.4.1-pl-advanced to 2.5.7-pl would actually work, but I figured I'd give it a try just in case.
Result: upgradeModx installs fine, the button is there, it alerts me that 2.5.7-pl is available, I click install it, it uses curl, appears to download the new modx, the installer starts, asks for the actual location of core, I tell it, the install completes, no errors reported, I login, and it's still 2.4.1.
Note that upon not finding the default location for 'core' it asked me during install process where it was, I told it, and it went on, seemingly content.
The version in Settings is:
Settings Version | settings_version | 2.4.1-plSo now I appear to be running the /manager from 2.5.7-pl and the core/ from 2.4.1, so I'm glad I tested this locally, heh.
Nothing appeared to go wrong with the install/upgrade process, it finished happily stating it was done.
I've noticed a few times in the past that when I'd changed permissions without realizing it, those permissions would block certain upgrade events, since the installer didn't actually check all the permissions.
I'd like to move our site to using this upgrade tool since the main thing I always, without exception, put off, is upgrades, due to what a headache the manual ones are to do.
Let me know if you need any debugging data and I'll generate it for you.
I cleared browser cache etc, manually cleared core/cache just to be sure, but it's clear that the core and db were not upgraded, since both still say they are running 2.4.1-pl
I ran the upgradeModx installer twice just to see if something would change, but it's the same each time, modx version is 2.4.1-pl, it upgrades, I tell it the true location of 'core', it checks permissions, it passes checks, and installs, and says it is done, logs me out, and claims to be successful, but it isn't.