Target `upload_dirs` do not have any files in them inside container, when using mutagen after a `ddev pull pantheon` or `ddev import-files`

tesliker commented 2 weeks ago

Preliminary checklist

[X] I am using the latest stable version of DDEV (see upgrade guide)
[X] I have searched existing issues
[X] I have checked the troubleshooting guide
[X] I have run ddev debug test to include output below

Output of `ddev debug test`

Expand `ddev debug test` diagnostic information

``` ======== OS Information ======== Darwin MacBook-Pro-5 22.4.0 Darwin Kernel Version 22.4.0: Mon Mar 6 21:00:17 PST 2023; root:xnu-8796.101.5~3/RELEASE_X86_64 x86_64 ProductName: macOS ProductVersion: 13.3.1 ProductVersionExtra: (a) BuildVersion: 22E772610a ======== User information ======== uid=501(tesliker) gid=20(staff) groups=20(staff),12(everyone),61(localaccounts),79(_appserverusr),80(admin),81(_appserveradm),98(_lpadmin),33(_appstore),100(_lpoperator),204(_developer),250(_analyticsusers),395(com.apple.access_ftp),398(com.apple.access_screensharing),399(com.apple.access_ssh),400(com.apple.access_remote_ae),701(com.apple.sharepoint.group.1) ======== ddev version ======== ITEM VALUE DDEV version v1.23.0 architecture amd64 cgo_enabled 0 db ddev/ddev-dbserver-mariadb-10.11:v1.23.0 ddev-ssh-agent ddev/ddev-ssh-agent:v1.23.0 docker 25.0.5 docker-api 1.44 docker-compose v2.26.1 docker-platform orbstack mutagen 0.17.2 os darwin router ddev/ddev-traefik-router:v1.23.0 web ddev/ddev-webserver:v1.23.0-1 ======== curl -I of http://tryddevproject-8996.ddev.site from outside ======== % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 HTTP/1.1 200 OK Content-Type: text/html; charset=UTF-8 Date: Wed, 15 May 2024 16:49:35 GMT Server: nginx Vary: Accept-Encoding ======== Full curl of http://tryddevproject-8996.ddev.site from outside ======== % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 286 0 286 0 0 31930 0 --:--:-- --:--:-- --:--:-- 71500 Success accessing database... db via TCP/IP
ddev is working.
The output file for Discord or issue queue is in /var/folders/k0/y_1rz5rd1kj8d0h8pyqk04_h0000gn/T/ddev-debug-test.txt
file:///var/folders/k0/y_1rz5rd1kj8d0h8pyqk04_h0000gn/T/ddev-debug-test.txt
======== Full curl of https://tryddevproject-8996.ddev.site from outside ======== % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 curl: (35) LibreSSL/3.3.6: error:1404B458:SSL routines:ST_CONNECT:tlsv1 unrecognized name ======== Project ownership on host ======== drwxr-xr-x@ 4 tesliker staff 128 May 15 12:47 ../tryddevproject-8996 ======== Project ownership in container ======== drwxr-xr-x 1 tesliker dialout 16 May 15 16:49 //var/www/html ======== In-container filesystem ======== Filesystem Type 1K-blocks Used Available Use% Mounted on /dev/vdb1 btrfs 778501156 22373792 756127364 3% /var/www ======== Thanks for running the diagnostic! ======== Running ddev launch in 3 seconds Running ddev launch HTTP may redirect to HTTPS in your browser See https://ddev.readthedocs.io/en/stable/users/usage/commands/#launch Waiting 10 seconds to run ddev stop --unlist ```

Expected Behavior

Run:

ddev config
ddev start
(configure provider yml)
ddev pull

The website should be available at *.ddev.site with no errors

Actual Behavior

When I initialize a new project, I have this error:

User warning: mkdir(): Permission Denied in Drupal\Component\PhpStorage\FileStorage->createDirectory() (line 123 of core/lib/Drupal/Component/PhpStorage/FileStorage.php).
Drupal\Component\PhpStorage\FileStorage->createDirectory('sites/default/files/php', 511) (Line: 104)
Drupal\Component\PhpStorage\FileStorage->createDirectory('sites/default/files/php/twig', 511) (Line: 73)
Drupal\Component\PhpStorage\FileStorage->ensureDirectory('sites/default/files/php/twig') (Line: 66)
Drupal\Component\PhpStorage\MTimeProtectedFastFileStorage->save('657330a214eb2_file-upload-help.html.twi_JxsofamZIKhnF5cOCQHHpqEtI', '&lt;?php

My sites/default/files folder is empty.

Once I run sudo chmod -R 777 . and restart the containers, the error goes away.

If I run sudo chmod -R 777 . before creating the project, I still get the errors and have to run it again and restart.

It is just weird that I have to do this for every project the first time. Not sure What I am doing wrong.

Steps To Reproduce

No response

Anything else?

No response

rfay commented 2 weeks ago

You shouldn't need sudo anywhere here, because the directories should already be owned by you.

if you can show ls -ld sites/default/files && ls -ld sites/default/files/* at various points during your trouble, it will help to understand this.

I assume you're using a custom hosting provider, can you share it? If not, can you say which one you're using?

With Drupal the default settings.ddev.php sets up Drupal so it doesn't try to constrain permissions. You left out a huge part of the ddev debug test (which goes to a file, you can just attach the file if you want). As a result there's a lot I can't see.

I imagine the problem is that your files tarball contains files which have limited permissions. You can easily check this by untarring the tarball in another directory and taking a look at it. If you want to share the tarball with me, I can take a look as well.

cd ~/tmp && tar -zxf <pathToTarball.tar.gz> && ls -l <directory-where-it-is>

tesliker commented 2 weeks ago

It seems this is actually a problem with the files_pull_command: in the pantheon.yaml.

So everytime I pull, the permissions are getting messed up. Not sure the right approach to fix. Adjusted the script like so, but it still does not help.

    mkdir -p files
    tar --strip-components=1 -C files -zxf files.tgz
    # Ensure the directory has the correct permissions
    sudo chmod -R 755 files
    # Ensure the directory is owned by the web server user, typically www-data in ddev containers
    sudo chown -R www-data:www-data files

rfay commented 2 weeks ago

Please use chmod ugo+rw (which makes your files writable by user and group) instead of chmod 755 (which makes all your files executable).

www-data is not who files are owned by in a DDEV setup. Where did that come from? Files in a DDEV project should be owned by your user, and mounted into the container with your user also.

It looks like you're not using https://github.com/ddev/ddev/blob/master/pkg/ddevapp/dotddev_assets/providers/pantheon.yaml.example at all?

tesliker commented 2 weeks ago

Here is the full ddev debug ddev-debug-test.txt

tesliker commented 2 weeks ago

Apologies, that was my attempt to further debug and fix the permissions issue. The code was:

files_pull_command:
  command: |
    set -x   # You can enable bash debugging output by uncommenting
    set -eu -o pipefail
    ls /var/www/html/.ddev >/dev/null # This just refreshes stale NFS if possible
    pushd /var/www/html/.ddev/.downloads >/dev/null;
    terminus backup:get ${project} --element=files --to=files.tgz
    mkdir -p files && tar --strip-components=1 -C files -zxf files.tgz

rfay commented 2 weeks ago

Thanks for providing the full ddev debug test output. Everything seems fine there.

I happen to note that you don't seem to have run mkcert -install, so please do that. It has nothing to do with this issue, but you will then be able to use https URLs successfully.

rfay commented 2 weeks ago

Go ahead and use terminus inside the container to grab the files tarball (or maybe it's already there waiting for you in .ddev/.downloads) and extract it and look at the permissions it's using.

Mostly you don't want to use sudo for anything anywhere.

rfay commented 2 weeks ago

If it turns out that you can share privileges on the project so I can test against it, I'm happy to do that. Normally we only have our one simple test project. Alternately, if you want to PM me the files tarball that you can download, I'll take a look at that. I certainly haven't had this problem before.

rfay commented 2 weeks ago

Thanks for sending the tarball. It has almost nothing in it, so I assume the upstream project is almost empty?

When I untar it everything comes out appropriately and it looks fine, and none of the directories like sites/default/files/php are populated, so nothing should be the fault of the tarball.

You're not running ddev with a different user than the sites/default/files directory is owned by are you?

Do you have any trouble if you just ddev pull pantheon --skip-files and then start up?

(I don't think I've gotten you to show the output of ls -ld sites/default/files or ls -la sites/default/files yet.)

tesliker commented 2 weeks ago

Yes upstream project is almost empty. This error happens with all of the websites big and small.

ls -ld web/sites/default/files
drwxr-xr-x@ 7 tesliker  staff  224 May 15 13:32 web/sites/default/files

My user that I run ddev with is tesliker

I will try --skip-files next

tesliker commented 2 weeks ago

If I skip the files, Drupal does not throw the mkdir errors

rfay commented 2 weeks ago

And if you just --skip-db (to make things faster) what does ls -ld web/sites/default/files look like right after the pull?

(Is there any chance you have a .ddev/docker-compose.*.yaml, or have altered .ddev/mutagen/mutagen.yml?)

Does this problem go away if you ddev config --performance-mode=none ?

rfay commented 2 weeks ago

What does id -a && ddev exec id -a say?

rfay commented 2 weeks ago

Maybe you should just make an appointment and we'll look at it together with a screenshare. https://cal.com/randyfay/30min

tesliker commented 2 weeks ago

@rfay It does look like the problem went away once I turned off performance-mode.

I performed ddev pantheon pull with no issues.

I wanted to turn it back on to see if the issue happened again, but when I ran ddev pantheon pull I got this:

You're about to delete the current database and files and replace with the results of a fresh pull.
Would you like to continue? [Y/n] (yes): 
Authenticating...
 [notice] Logged in via machine token.
Obtaining databases...
+ set -eu -o pipefail
+ ls /var/www/html/.ddev
+ pushd /var/www/html/.ddev/.downloads
+ terminus backup:get agf-retiree.dev --element=db --to=db.sql.gz
 [notice] Downloading agf-retiree_dev_2024-05-15T16-30-35_UTC_database.sql.gz to db.sql.gz
Pull failed: failed to find downloaded files in /Users/tesliker/work/agf-retiree/.ddev/.downloads: <nil>

rfay commented 2 weeks ago

I'm sure you knew that you had to ddev restart when changing between mutagen and none.

My bet is if you did this a few times in a row it might be intermittent.

Someone else did have trouble with Orbstack and this situation recently, but has been impossible to sort out and unpredictable. I imagine it would be intermittent for you here as well.

Tomorrow we'll have DDEV v1.23.1, which has slightly better error reporting in situations like what you may have, so if you're able to reproduce the "nothing found in .downloads" with DDEV v1.23.1, details about it will be so very welcome.

That problem (related to what you just saw, but not related to your original problem) is a long thread in Discord: https://discord.com/channels/664580571770388500/1232668554550186024

rfay commented 2 weeks ago

Since your upstream project is so very limited, is there any possibility you could give me access to it so I could try to recreate?

Also, you say ddev pantheon pull a number of times, but I'm sure you're actually doing ddev pull pantheon, right?

rfay commented 2 weeks ago

@tesliker - I was just debugging an unrelated item and wanted to ask if you're running PhpStorm with the DDEV Plugin going. It could affect the intermittent behavior you just had.

rfay commented 2 weeks ago

Tommy and I studied this pretty carefully (thanks!) and what we found was this:

He's using Orbstack 1.5.1 on amd64 mac
The problem only happens when mutagen is enabled
His ddev pull pantheon pulls a pretty trivial set of files
After ddev pull pantheon the files directory is properly populated on the host, but it's empty inside the container, and it's not possible to mkdir inside the files dir
ddev restart makes the files dir mount correctly and show the files on the host inside the container.

I'm able to reproduce this with our test pantheon project on Orbstack 1.5.1 also (arm64 mac). In my case, I just didn't have any twig action happening, so had never seen the problem.

Our tests on macOS + OrbStack really should be showing this behavior (TestPantheonPull should see this problem. Apparently it doesn't)

We have OrbStack 1.5.0 on tb-macos-arm64-5 and tb-macos-arm64-6, but 1.5.1 on tb-macos-arm64-7.

However, we don't do a ddev pull test on the macOS tests, only on Linux.

rfay commented 2 weeks ago

This is a bug in DDEV, at least with Drupal, possibly with other CMSs. It does not have to do with Orbstack (easy to recreate with Docker Desktop).

Here's the location of the failure:

https://github.com/ddev/ddev/blob/e694c20e870ab98c6996e1c0b2af0b4ea5e52326/pkg/ddevapp/drupal.go#L575-L580

What's happening is the drupalImportFilesAction() is removing the entire mounted path (sites/default/files) on the host side. But when mutagen is enabled, that is the source for the in-container files directory, and when we delete and create a new one, the old one is no longer mounted.

The correct behavior here will be to purge the target files directory, rather than removing it.

rfay commented 2 weeks ago

@tesliker could you please take a look at the artifacts in https://github.com/ddev/ddev/pull/6199#issuecomment-2115870068 and see if they solve your problem?

You'll want the macOS-amd64 one, and you need to put it in your PATH in a place before /usr/local/bin in your case, or run it explicitly. In your situation I often put it in ~/bin and make sure ~/bin is first in PATH.

tesliker commented 2 weeks ago

@rfay Confirmed, this works. Great job, thanks!

rfay commented 2 weeks ago

Thanks for taking the time to work through this! This will be in the next release of DDEV, but of course you can use the HEAD version when this gets pulled or you also have other workarounds.

ddev / ddev