Mutagen enabled: Drupal Package Manager hits exception when installing a module via the Project Browser

[rpkoller]

Expected Behavior

ddev macOS, Colima 0.5.4 + mutagen (Also Docker Desktop + mutagen, also Linux + mutagen)

i would expect that the install of a module runs through without any problem in the Project Browser UI

Actual Behavior

A few weeks or months back I was able to test the install process of new modules via project browser's ui in drupal. everything worked as expected. when i started to test again a few weeks ago i ran all of a sudden into consistent errors. described here: https://www.drupal.org/project/automatic_updates/issues/3343793 and here https://www.drupal.org/project/automatic_updates/issues/3344542

for quite a while i had the suspicion the error was caused by the changes in package manager and the hardening that was done there. phenaproxima suspected in a comment in one of the issues it might be an issue on the ddev end. but then a few days ago @chrisfromredfin provided a drupalpod instance (gitpod + ddev) with a test environment for drupalcon that folks are able to test project browser there.

https://gitpod.io/#DP_PROJECT_NAME=project_browser,DP_ISSUE_FORK=,DP_ISSUE_BRANCH=,DP_PROJECT_TYPE=project_module,DP_MODULE_VERSION=1.0.x,DP_CORE_VERSION=10.0.x,DP_OLIVERO=1,DP_PATCH_FILE=,DP_REINSTALL=1,DP_INSTALL_PROFILE=standard/https://github.com/chrisfromredfin/DrupalPod

i was afraid that people might run into the same error i'Ve consistently expierenced. but surprisingly i was able to run an installation of a module without any exception and error. but since gitpod runs on linux mutagen was disabled.

i've then tried to set up a new site locally. this time i had mutagen disabled as well. there the install ran through without any problem as well. so i suspected the problem might be cause by mutagen. but i then activated mutagen for the gitpod instance. also with mutagen enabled a module install ran through flawless.

@rfay noted that sshfs (the mounttype i use) might cause surprising issues. i've now went ahead and set up a colima profile with 9p as the mounttype. but the same result as with sshfs. unable to install a module via the project browser ui with mutagen enabled

Steps To Reproduce

1) copy composer,json found at https://gist.github.com/rpkoller/3617dbb09995c24da02d6949e7147724 into an empty directory 2) ddev config --project-type=drupal10 --docroot=web --create-docroot --mutagen-enabled=true --php-version="8.2" --database=mariadb:"10.3" 3) ddev start 4) ddev composer install 5) ddev drush si -y standard --account-pass=admin 6) ddev drush en package_manager project_browser admin_toolbar admin_toolbar_tools admin_toolbar_links_access_filter 7) /admin/config/development/project_browser check the allow installing via the ui checkbox and save 8) go to /admin/modules/browse and click the add and install button on the metatag module card

Anything else?

ddev head version colima 0.5.4 qemu 8.0.0 limactl version 0.15.1

[rfay]

@rpkoller you said that you enabled mutagen on gitpod and had no trouble with that configuration, true?

[rpkoller]

yes the install of a module in project browser with and without mutagen enabled worked flawlessly. i mentioned that in the issue summary as well:

but i then activated mutagen for the gitpod instance. also with mutagen enabled a module install ran through flawless.

[rfay]

At first I was able to recreate these problems; I also recreated on Docker Desktop 4.19.0 + mutagen.

But now I can't make it happen anywhere. It's working fine.

Some things I did along the way include

• ddev mutagen reset
• ddev mutagen logs
• ddev mutagen status -l
• ddev mutagen monitor

[rfay]

After the failure, if you clean up with

git reset --hard && git clean -fd && ddev mutagen reset && ddev start

It fixes it every time (assuming the code was committed before anything was done with project browser)

[rpkoller]

an updated list how to reproduce the issue:

moved the list of steps to reproduce the problem to the issue summary

[rfay]

I tested this on Ubuntu 22.04, with current DDEV HEAD. It had the exact same behavior, and the exact same fix fixed it, git reset --hard && git clean -fd && ddev mutagen reset && ddev start

[rfay]

One of the things we can do is ask the developers of Project Browser for a summary of what happens when you install a module. What directories are used? Where is the code staged?

[wimleers]

Composer has specifically had to work around problems on virtualized filesystems — for example: https://github.com/composer/composer/issues/9627

Can you try setting COMPOSER_RUNTIME_ENV=virtualbox?

[rpkoller]

i'll also post my answer in here for folks not following along https://www.drupal.org/project/automatic_updates/issues/3343793

I've created a new test setup and added

web_environment:
 - COMPOSER_RUNTIME_ENV=virtualbox

in the project's config.yaml and then went through the rest of the steps outlined in the issue summary in https://github.com/ddev/ddev/issues/4938. unfortunately that didn't fix the issue. i still ran into the identical installexception.

[rfay]

Thanks for joining us in this conversation @wimleers !

I didn't expect the virtualbox-like approach to solve this, because DDEV with mutagen is actually installing on a normal linux filesystem (docker volume inside docker container), no funny mounting things going on, which is why it's normally rock-solid with composer.

In fact, with colima people normally run with the sshfs filesystem mount type, and without mutagen that actually does have the virtualbox-type behavior, where composer doing many, many changes all at once will fail. there are several issues about it. And it's why we always tell people to use mutagen with colima.

Here's what /var/www/html (project root) looks like in the container; it's just ext4 with nothing else going on:

rfay@d9-web:/var/www/html$ df -T .
Filesystem                     Type 1K-blocks      Used Available Use% Mounted on
/dev/disk/by-label/data-volume ext4 205313000 139382392  55428516  72% /var/www

However... with Project Browser... is there a staging setup before stuff gets done on the real project filesystem? That might possibly be outside /var/www/html? Where would it be?

(COMPOSER_RUNTIME_ENV=virtualbox may very well solve the colima-sshfs-without-mutagen composer problem, which is pretty awesome, I'm looking forward to trying that out.)

[rfay]

Suspecting the use of /tmp (which is a different filesystem and filesystem type than /var/www/html (ext4 when using mutagen) I tried changing the tmpdir in settings.php:

$settings['file_temp_path'] = '/var/www/tmp';

which puts the Drupal's temp dir on the same filesystem and type. But I don't see PB's operations going on there...

It failed just the same.

Does PB respect file_temp_path ?

[chrisfromredfin]

It should respect it, as far as I know, because it's using a SharedTempStore(Factory) from core...

[rfay]

And... Doing a ddev mutagen reset && ddev start right before step 8 (/admin/modules/browse and install) ALSO fixes it. I have no idea how that could be.

[rpkoller]

To narrow the root cause further down I've remembered that there was the option to use rsync as the file syncer when ted bowman mentioned it in the thread on the drupal slack. i've followed the steps setting up a new test instance except i've entered those two lines between step 6 and 7 :

ddev drush config:set package_manager.settings file_syncer rsync
ddev drush config:set package_manager.settings executables.rsync /usr/bin/rsync

with rsync active i was able to install pathauto , webform and metatag in a row and no error happened. so the root cause for the error is somewhere in the php file syncer?

[rpkoller]

and i've tested an observation by @rfay and am able to confirm it. if you keep the PHP file sync and go through step 1-7 and then run a ddev mutagen reset && ddev start then you are also able to install a module via the project browser ui without the error.

[chrisfromredfin]

There are reports of this happening without ddev/mutagen, but on native also (i.e. not-even-docker-is-involved).

Related issue in php-tuf/composer-stager: https://github.com/php-tuf/composer-stager/issues/162

[rfay]

It looks like we're making progress in understanding this. The permissions are going to be the key.

It turns out that mutagen does not actually manage permissions, see

• https://github.com/mutagen-io/mutagen/issues/282

So what is happening here that is fixed by ddev mutagen reset (which deletes the project_mutagen docker volume where all the code is) is

1. We have the funky perms described in https://github.com/php-tuf/composer-stager/issues/162 - Those are created on the project_mutagen docker volume as described
2. ddev mutagen reset actually destroys that volume (but it was all synced to the host)
3. ddev start syncs the code back to a new volume... but per https://github.com/mutagen-io/mutagen/issues/282 it uses default perms.
4. Now everything works

[rfay]

It turns out this is about the combined behavior of composer-stage and mutagen, but it's not a bug related to DDEV. It's something that's will need to be worked on in Project Browser and such though. Closing.