search

contao / contao-manager

Contao Manager
GNU Lesser General Public License v3.0
83 stars 33 forks source link

Install a bundle not possible - Manager can not delete a folder #674

Closed planepix closed 2 years ago

planepix commented 2 years ago

Hi there,

i tried to install a bundle ( for e.g. EasyThemes ) under a Contao 4.9.26 installation. Hosting by Hostpoint. PHP 7.4.28

After starting the manager an error occurs by deleting a folder:

In Filesystem.php line 332:

  Could not delete /home/www/domain.ch/var/cache/prod/Containe  
  rOnUKWe4:

Tried to install another bundle runs into the same error.

It's also mentioned in the german forum: https://community.contao.org/de/showthread.php?82604-Konsolenfehler-beim-Update-von-4-9-22-auf-4-9-26-(In-Filesystem-php-Could-not-delete)&p=555935#post555935

Maren got the same error with same settings and also Hostpoint as Hoster. Deleting the vendor folder does not work in my installation.

aschempp commented 2 years ago

The issue might happen if someone is accessing your website while the cache is being deleted. This has been improved in Contao 4.13, but have you tried to delete the cache multiple times?

mlwebworker commented 2 years ago

Ich habe diesen Fehler heute und gestern mit leicht unterschiedlichen Meldungen (was den nicht zu löschenden Container betrifft) bei fast allen Updates bei Hostpoint-Installationen gehabt, aber bei keinem anderen Provider. Die Installationen lassen sich nach dem Fehler weder durch Reparaturmodus noch im abgesicherten Modus (composer update) wieder zum Leben erwecken. Bei mir hat jeweils nur das Löschen des kompletten vendor-Ordners zum Erfolg geführt. Nicht aufgetreten ist es bei einer Hostpoint-Installation, die auf 4.13 war und bei der nur Erweiterungen upgedatet wurden.

Ich habe mir deshalb eine Testinstallation bei einem Hostpointkunden angelegt

Kann ich noch etwas testen um dem Fehler auf den Grund zu gehen?

aschempp commented 2 years ago

Vielen Dank für die ausführliche Analyse! Die gerade veröffentlichte Version 1.5.1 erlaubt automatisch alle Plugins. Kannst du es vielleicht damit nochmals testen?

padgrob commented 2 years ago

Ich habe soeben mit der Version 1.5.1 getestet, erhalte aber leider immer noch denselben Fehler in der Konsole.

Script Contao\ManagerBundle\Composer\ScriptHandler::initializeApplication handling the post-install-cmd event terminated with an exception

In Filesystem.php line 332:

Could not delete xxxxx/xxxxx/xxxx/var/cache/prod/ContainerX4
dKgav:

planepix commented 2 years ago

Ich habe den Manager auf 1.5.1 aktualisiert. Klappt. Update auf 4.9.27 klappt nicht - wieder ein Ordner, der nicht gelöscht werden kann.

Atherel commented 2 years ago

Mir ist aufgefallen, dass nach dem Fehler

In Filesystem.php line 332:

  Could not delete /home/username/www/contao/var/cache/prod/ContainerEgFC4Pu:

immer die selben vier Dateien im entsprechenden Ordner übrig bleiben. Egal ob man ein Paket installieren oder die Contao Installation updaten will:

AuthenticatedTokenHandler_e3677d3.php
ConditionAuthenticationHandler_a1ee12f.php
EntityManager_9a5be93.php
IpWhitelistHandler_06fe2b0.php
aschempp commented 2 years ago

After investigating this for 5 hours (!!) yesterday with the help of @fritzmg and @Toflar, we finally understand what is going on.

  1. The changes introduced in https://github.com/symfony/symfony/pull/44070 will lead to changes in the default inherited variables when launching subprocesses.
  2. before running post-install scripts, Composer is changing the PATH variable to its bin directory (https://github.com/composer/composer/blob/main/src/Composer/EventDispatcher/EventDispatcher.php#L568)
  3. Contao (4.9) uses a post-install script to purge the cache directory (https://github.com/contao/contao/blob/4.9/manager-bundle/src/Composer/ScriptHandler.php#L45) using the Composer\Util\Filesystem class
  4. Composer Filesystem tries to remove the directory using rm -rf command (https://github.com/composer/composer/blob/main/src/Composer/Util/Filesystem.php#L117-L123), which fails because the command is not found (not in PATH variable)
  5. Composer then falls back to deleting the files using PHP (https://github.com/composer/composer/blob/main/src/Composer/Util/Filesystem.php#L219). This somehow failes most likely because of the notoriously slow I/O performance on Hostpoint. I would guess deleting the directory fails because it is not yet empty, even though PHP already deleted all the files in it.

There are multiple solutions to this problem, and/or multiple issues that should be fixed

  1. Maybe https://github.com/symfony/symfony/pull/44070 needs to be improved
  2. Composer should probably not override the PATH in https://github.com/composer/composer/blob/main/src/Composer/EventDispatcher/EventDispatcher.php#L568 but maybe just append its own path?
  3. Composer already handles slow IO but only on Windows (https://github.com/composer/composer/blob/main/src/Composer/Util/Filesystem.php#L317-L321), enabling this everywhere would probably have solved the final error message

Here's what the $env argument passed to the process looks like before the change in https://github.com/symfony/symfony/pull/44070

Array
(
    [0] => SERVER_PORT=443
    [1] => SERVER_ADDR=127.0.0.2
    [2] => HTTP_HOST=test.example.com
    [3] => SSL_VERSION_INTERFACE=mod_ssl/2.4.52
    [4] => CONTEXT_DOCUMENT_ROOT=/%kernel.project_dir%/web
    [5] => QUERY_STRING=
    [6] => SSL_SECURE_RENEG=false
    [7] => SSL_CIPHER=TLS_AES_256_GCM_SHA384
    [8] => SSL_CIPHER_EXPORT=false
    [9] => HTTP_ACCEPT_LANGUAGE=en
    [10] => SSL_SESSION_RESUMED=Initial
    [11] => SSL_SERVER_M_VERSION=3
    [12] => DOCUMENT_ROOT=/%kernel.project_dir%/web
    [13] => LD_LIBRARY_PATH=/usr/local/lib:/usr/local/lib
    [14] => HTTP_REFERER=https://test.example.com/contao-manager.phar.php/
    [15] => SCRIPT_NAME=/contao-manager.phar.php
    [16] => SSL_VERSION_LIBRARY=OpenSSL/1.1.1m
    [17] => H2_STREAM_ID=39
    [18] => SSL_SERVER_I_DN_CN=R3
    [19] => SSL_SERVER_A_SIG=sha256WithRSAEncryption
    [20] => H2_PUSHED_ON=
    [21] => FCGI_ROLE=RESPONDER
    [22] => HTTP_USER_AGENT=Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/15.3 Safari/605.1.15
    [23] => H2_PUSH=on
    [24] => H2_STREAM_TAG=580-39
    [25] => PWD=/%kernel.project_dir%/
    [26] => SSL_SESSION_ID=...
    [27] => REQUEST_METHOD=GET
    [28] => UNIQUE_ID=Yi9o8u4iGiLDSXsmNgraaAACRAo
    [29] => SERVER_SOFTWARE=Apache
    [30] => SERVER_ADMIN=webmaster@test.example.com
    [31] => SERVER_NAME=test.example.com
    [32] => SERVER_PROTOCOL=HTTP/2.0
    [33] => PHP_PATH=/usr/bin/php
    [34] => HTTP_ACCEPT_ENCODING=gzip, deflate, br
    [35] => REQUEST_URI=/contao-manager.phar.php/api/task
    [36] => SERVER_SIGNATURE=
    [37] => HTTPS=on
    [38] => SSL_SERVER_M_SERIAL=0328739695295B4D2F5E67771428ED7AD7A8
    [39] => SSL_CLIENT_VERIFY=NONE
    [40] => SSL_PROTOCOL=TLSv1.3
    [41] => SSL_SERVER_SAN_DNS_0=test.example.com
    [42] => SSL_SERVER_I_DN_C=US
    [43] => GATEWAY_INTERFACE=CGI/1.1
    [44] => CONTEXT_PREFIX=
    [45] => PHPRC=/usr/local/php74/etc
    [46] => REQUEST_SCHEME=https
    [47] => HTTP_X_REQUESTED_WITH=XMLHttpRequest
    [48] => SSL_SERVER_V_END=Jun  6 13:39:03 2022 GMT
    [49] => SSL_CIPHER_USEKEYSIZE=256
    [50] => SSL_TLS_SNI=test.example.com
    [51] => SSL_SERVER_S_DN_CN=test.example.com
    [52] => SSL_SERVER_I_DN=CN=R3,O=Let's Encrypt,C=US
    [53] => SSL_CIPHER_ALGKEYSIZE=256
    [54] => HTTP2=on
    [55] => SSL_SERVER_A_KEY=rsaEncryption
    [56] => SSL_COMPRESS_METHOD=NULL
    [57] => SSL_SERVER_S_DN=CN=test.example.com
    [58] => H2_PUSHED=
    [59] => H2PUSH=on
    [60] => PATH_TRANSLATED=redirect:/index.php/task
    [61] => HTTP_COOKIE=contao_manager_auth=...; csrf_https-contao_csrf_token=...
    [62] => REMOTE_PORT=38150
    [63] => SSL_SERVER_V_START=Mar  8 13:39:04 2022 GMT
    [64] => REMOTE_ADDR=2a02:21b0:644c:8ba3:700e:5a52:c0c9:39f4
    [65] => PATH_INFO=/api/task
    [66] => SCRIPT_FILENAME=/%kernel.project_dir%/web/contao-manager.phar.php
    [67] => HTTP_ACCEPT=application/json
    [68] => SSL_SERVER_I_DN_O=Let's Encrypt
    [69] => GIT_ASKPASS=echo
    [70] => LANGUAGE=C
    [71] => COMPOSER_DEV_MODE=0
)

and after it

Array
(
    [0] => SHELL_VERBOSITY=3
    [1] => GIT_ASKPASS=echo
    [2] => LANGUAGE=C
    [3] => COMPOSER_DEV_MODE=0
    [4] => PATH=/%kernel.project_dir%/vendor/bin:
    [5] => PWD=/%kernel.project_dir%
    [6] => PHP_PATH=/usr/bin/php
    [7] => PHPRC=/usr/local/php74/etc
)

As one can see, the getDefaultEnv() method no longer returns environment variables from the webserver process (including $_SERVER) but the ones set by the composer runtime. Which actually seems rather correct to me, even though this also seems like a major change?

Maybe @seldaek and/or @nicolas-grekas have an opinion whether we should open issues/PRs at Composer or Symfony?

Seldaek commented 2 years ago

What Composer version is being run?

  1. Composer should probably not override the PATH in https://github.com/composer/composer/blob/main/src/Composer/EventDispatcher/EventDispatcher.php#L568 but maybe just append its own path?

It is not overriding but prepending to the existing PATH, which should not cause issues. Platform::putEnv($pathEnv, $binDir.PATH_SEPARATOR.$pathValue);

I don't know why you have an empty PATH there.. that could be an env issue on hostpoint? Because even your first env array with 72 items has no PATH in it at all?!

  1. Composer already handles slow IO but only on Windows (https://github.com/composer/composer/blob/main/src/Composer/Util/Filesystem.php#L317-L321), enabling this everywhere would probably have solved the final error message

If you can prove it helps I'm happy to do that change, but AFAIK on unix this should not have any impact.

aschempp commented 2 years ago

What Composer version is being run?

Latest (2.2.7) (see https://github.com/contao/contao-manager/blob/1.5.1/composer.lock#L86-L92)

  1. Composer should probably not override the PATH in https://github.com/composer/composer/blob/main/src/Composer/EventDispatcher/EventDispatcher.php#L568 but maybe just append its own path?

It is not overriding but prepending to the existing PATH, which should not cause issues. Platform::putEnv($pathEnv, $binDir.PATH_SEPARATOR.$pathValue);

You're absolutely right, my mistake.

I don't know why you have an empty PATH there.. that could be an env issue on hostpoint? Because even your first env array with 72 items has no PATH in it at all?!

Sometimes I just need that soft hug with a hard stick…! 😂 I noticed in my logs that the path variable is gone when the Contao Manager starts a background process. Which lead me to find https://github.com/contao/contao-manager/blob/main/api/System/ServerInfo.php#L120, which exactly is our issue. We're unsetting all existing variables to work around issues when running PHP on the command line using a CGI binary (some hosters do not have a real CLI binary). If we don't unset the env variables, the CGI binary on command line thinks its running a web process. But maybe this has also been fixed by the changes in Symfony 😎

aschempp commented 2 years ago

The problem should be fixed in version 1.5.2 🎉 thank you @Seldaek @Toflar @fritzmg

planepix commented 2 years ago

Wow, what a ride! Thank you all. Can report the installation now runs the update of Contao and also able to install thirdparty bundles.

padgrob commented 2 years ago

THANK YOU ALL!