Prebuild fails to run for some users

tyler36 commented 1 year ago

Describe the bug

When using DrupalPod, it states it can't find a project, installs the project files then says can't find project and install the project then says ....

Looking through terminal output, the below catches my eye:

mc: <ERROR> Unable to stat `gcs/drupalpod/main/ready-made-envs.tar.gz`. Insufficient permissions to access this file `https://storage.googleapis.com/drupalpod/main/ready-made-envs.tar.gz`.
mc: <ERROR> Unable to stat `gcs/drupalpod/main/ready-made-envs.tar.gz`. Insufficient permissions to access this file `https://storage.googleapis.com/drupalpod/main/ready-made-envs.tar.gz`.

To Reproduce

Visit https://www.drupal.org/project/drupal/issues/3178202
Click DrupalPod extension
Enter the following:
- Drupal code: 10.0.x
- Choose patch: https://www.drupal.org/files/issues/2023-04-03/3178202-61.patch (first)
- Install profile: demo_umami
- Click "Open Dev Env"

render-block

Accept Gitpod default.
Watch terminal ....

...
 [success] Successfully enabled: devel
 [success] Successfully installed theme: claro
You have specified a project type of drupal9 but no project of that type is found in /workspace/ready-made-envs/10.0.0/web
Warning: MOST EXISTING CONTENT in the composer root (/workspace/ready-made-envs/10.0.0) will be deleted by the composer create-project operation. Only .ddev, .git and .tarballs will be preserved.
Removing any existing files in composer root
Creating a "drupal/recommended-project:~10.0.0" project at "/tmp/oyTKvA"
Installing drupal/recommended-project (10.0.10)
...

Expected behavior

GipodPod installs Drupal with patch.

Version and configuration information (please complete the following information)

Issue page https://www.drupal.org/project/drupal/issues/3178202#comment-15231630
Issue branch 11.x-dev
Drupal core version 10.0.x
Drupal core

Additional context

tyler36 commented 1 year ago

Did screenshare with @shaal. Was able to reproduce problem and all my settings appeared correct. After trying several thing, we were unable to fix issue.

@shaal opened an issue on Gitpod: Other users can't load my prebuild

rfay commented 1 year ago

Thanks for continuing to follow this up. Now that you understand it better, could you give it a better title? Maybe something like Insufficient permissions to access this filehttps://storage.googleapis.com/drupalpod/main/ready-made-envs.tar.gz``

rfay commented 1 year ago

I tried using the github user ddltest and got the same failure @tyler36 had.

+ mc find gcs/drupalpod/main/ready-made-envs.tar.gz
mc: <ERROR> Unable to stat `gcs/drupalpod/main/ready-made-envs.tar.gz`. Insufficient permissions to access this path `https://storage.googleapis.com/drupalpod/main/ready-made-envs.tar.gz`.
+ echo '*** Upload environments file to Google Cloud'
+ echo drupalpod/main/ready-made-envs.tar.gz
+ mc cp /workspace/ready-made-envs.tar.gz gcs/drupalpod/main/ready-made-envs.tar.gz
mc: <ERROR> Failed to copy `/workspace/ready-made-envs.tar.gz`. Insufficient permissions to access this path `https://storage.googleapis.com/drupalpod/main/ready-made-envs.tar.gz`

real    14m57.850s
user    0m50.128s
sys     0m46.976s
cp: cannot stat '../ready-made-envs/9.4.0/.': No such file or directory
Failed to restore snapshot demo_umami for project drupalpod: no snapshot found for name demo_umami: open /workspace/DrupalPod/.ddev/db_snapshots: no such file or directory

So I imagine this is a complete blocker for all new gitpod users.

I'm happy to give you full access to the ddltest account @shaal

tyler36 commented 1 year ago

Gitpod lists me on the "Open Source" billing plan, if it helps.

rfay commented 1 year ago

I tried the same with the github drud-test-machine-account. On this one gitpod walked me through the whole onboarding routine, so it had apparently never been used with gitpod before.

The same failure happened.

gist: https://gist.github.com/rfay/1a6aff3a52f01999d4da905ceef12fe9

I suspect that DrupalPod is unusable for all new gitpod users.

shaal commented 1 year ago

I created a separated new Github account + Gitpod. That user was able to open DrupalPod workspace WITH prebuild

rfay commented 1 year ago

Here's a workspace snapshot: https://gitpod.io#snapshot/3d7b1c7d-75b1-4cb2-a4cc-c913092b60f9

I don't even know how to recover from this failure. I think we might need to go without prebuilds?

What does one do to recover when the prebuild fails?

tyler36 commented 1 year ago

When I try https://gitpod.io/#snapshot/3d7b1c7d-75b1-4cb2-a4cc-c913092b60f9 it appears to run the pre-builds.

States "loading prebuilds" on loading screen
Starts very quickly
I don't see permission errors
I don't see it downloading DDEV images

See logs here

rfay commented 1 year ago

Right, but does it work? Doesn't work for me, because the prebuild wasn't loaded. Even composer.json isn't there.

tyler36 commented 1 year ago

Right, but does it work?

@rfay I thought your workspace was a fork of DrupalPod which doesn't contain Drupal. My fork of DrupalPod produces a similar result to your workspace.

https://github.com/shaal/ddev-gitpod is the repo has the Drupal composer.json. My fork of that works (workign Drupal site) but the @shaal does not.

rfay commented 1 year ago

No, I just clicked the drupalpod icon on the repo. Is that wrong?

If somebody tells me how to successfully use Drupalpod anywhere I'll be happy to give it some time. Failed at everything I've tried (even with my own user).

shaal commented 1 year ago

@rfay that's how I usually test it, for loading Drupal with umami profile on DrupalPod. But currently this is broken (but unrelated to the prebuild not loading for some users issue)

rfay commented 1 year ago

@rfay that's how I usually test it, for loading Drupal with umami profile on DrupalPod. But currently this is broken (but unrelated to the prebuild not loading for some users issue)

I think you had suggested this testing technique to me.

Can you please tell me a correct way to test? Give me some issue I can go to that will work? Tell me what choices I should make in the form?

shaal commented 1 year ago

The solution should be that if prebuild failed - the current workspace will install Drupal simply without using the ready-made-envs (which is what helps environments to be prepared faster when prebuild do work)

tyler36 commented 1 year ago

It appears to be working now following the the steps in the OP.

I no longer see DDEV images download.
It "starts" with DDEV starting the ssh-agent, db and web containers.

Starting DrupalPod... 
Downloading https://github.com/docker/compose/releases/download/v2.21.0/docker-compose-linux-x86_64 ... 
docker-compose 56.78 MiB / 56.78 MiB [==============================] 100.00% 0s
Download complete. 
 Container ddev-ssh-agent  Recreate 
 Container ddev-ssh-agent  Recreated 
...

I see composer download the project

Creating a "drupal/recommended-project:10.0.x-dev" project at "./temp-composer-files"
Info from https://repo.packagist.org: #StandWithUkraine
Installing drupal/recommended-project (10.0.x-dev be77349ecf5d9fdc5f9aea15e4aab15f4092dad3)
  - Downloading drupal/recommended-project (10.0.x-dev be77349)
 ...
drupalpod-setup.sh script ran for 85 seconds

Simple browser opens the "umami" homepage.
- Previously, I was unable to reach this step.
- I now appear to have a functioning Drupal site.

tyler36 commented 1 year ago

I tried testing on some other project issue pages and am hitting an error:

Tamper: https://www.drupal.org/project/tamper/issues/3049621

Lock file operations: 1 install, 0 updates, 0 removals
  - Locking mglaman/composer-drupal-lenient (1.0.3)
Writing lock file
> repos/add-project-as-symlink.sh
Replace project with a symlink
repos/add-project-as-symlink.sh: line 5: cd: web/modules/contrib: No such file or directory
Script repos/add-project-as-symlink.sh handling the post-update-cmd event returned with error code 1

Installation failed, reverting ./composer.json and ./composer.lock to their original content.
Failed to execute command composer require mglaman/composer-drupal-lenient --no-install: exit status 1

real    0m1.594s
user    0m0.128s
sys     0m0.167s

Not sure if this is related to the changes to fix this issue or not.

tyler36 commented 1 year ago

Here I a couple I was able to replicate the mglaman/composer-drupal-lenient issue on:

Failing patch

Visit https://www.drupal.org/project/tamper/issues/3049621
Use DrupalPod extension with the following options:
- Drupal core: 10.0.x
- Choose patch: https://www.drupal.org/files/issues/2023-09-14/tamper-copy-values-to-keys-3049621-9.patch
- Install profile: Standard

Failing branch

Visit https://www.drupal.org/project/tamper/issues/3362615
Use DrupalPod extension with the following options:
- Branch: 3362615-json-array-with
- Drupal core: 10.0.x
- Install profile: Standard

tyler36 commented 1 year ago

With the merge of #134 , the issue seems to be resolved.

Prebuilds work (it skips the downloading of DDEV images)
composer-drupal-lenient error does not appear.
Drupal welcome can be displayed.

rfay commented 1 year ago

Thanks @tyler36 @shaal !

shaal / DrupalPod