Issue while pulling an Ibexa Project from PSH

[flovntp]

Hi,

I've got an issue while pulling my Ibexa project (Symfony) from Platform.sh.

This is steps to reproduce (maybe i did something wrong)....

1. platform get lg4gpxi5g22he
2. cd vntp
3. brew unlink ddev && brew install --HEAD --fetch-head drud/ddev/ddev (because i had already ddev installed and discover that i needed to install the new version)
4. ddev config
5. ddev get platformsh/ddev-platformsh

And this is the error i get

could not process pre-install action '#ddev-nodisplay
#todo: This simplistic approach doesn't work for postgres; volume name is different
#and the database type file is different too
if docker volume ls | grep ${DDEV_PROJECT}-mariadb && docker run --rm -w /db -v ${DDEV_PROJECT}-mariadb:/db busybox cat db_mariadb_version.txt; then
  export current_db_version="$(docker run --rm -w /db -v ${DDEV_PROJECT}-mariadb:/db busybox cat db_mariadb_version.txt)"
  if [ -z "${current_db_version}" ]; then
    echo "something went wrong, current_db_version is empty" && false
  fi
  {{ $dbheader := index (split ":" .platformapp.relationships.database) "_0" }} 
  {{ $dbtype := replace "postgresql" "postgres" (get (get .services $dbheader) "type") }}
  # echo "dbheader={{$dbheader}} dbtype={{$dbtype}} "
  export upstream_db="{{ regexReplaceAll "oracle-" (regexReplaceAll ":.*$" $dbtype "") "" }}:{{ regexReplaceAll "^.*:" $dbtype "" }}"
  echo "Current db is ${current_db_version/_/:} (${current_db_version})"
  echo "New db is ${upstream_db}"
  if [ "${current_db_version/_/:}" != "${upstream_db}" ]; then
    printf "There is an existing database in this project that doesn\'t match the upstream database type.\nPlease back up this database and then \"ddev delete\" so the new database can be created.\n"
    false
  else
    echo "database versions match: ${current_db_version/_/:}=${upstream_db}"
  fi
fi
': Unable to run action set -eu -o pipefail
#ddev-nodisplay
#todo: This simplistic approach doesn't work for postgres; volume name is different
#and the database type file is different too
if docker volume ls | grep ${DDEV_PROJECT}-mariadb && docker run --rm -w /db -v ${DDEV_PROJECT}-mariadb:/db busybox cat db_mariadb_version.txt; then
  export current_db_version="$(docker run --rm -w /db -v ${DDEV_PROJECT}-mariadb:/db busybox cat db_mariadb_version.txt)"
  if [ -z "${current_db_version}" ]; then
    echo "something went wrong, current_db_version is empty" && false
  fi

  # echo "dbheader=mysqldb dbtype=mariadb:10.4 "
  export upstream_db="mariadb:10.4"
  echo "Current db is ${current_db_version/_/:} (${current_db_version})"
  echo "New db is ${upstream_db}"
  if [ "${current_db_version/_/:}" != "${upstream_db}" ]; then
    printf "There is an existing database in this project that doesn\'t match the upstream database type.\nPlease back up this database and then \"ddev delete\" so the new database can be created.\n"
    false
  else
    echo "database versions match: ${current_db_version/_/:}=${upstream_db}"
  fi
fi
: exit status 1, output=local     vntp-mariadb
Unable to find image 'busybox:latest' locally
latest: Pulling from library/busybox
50783e0dfb64: Pulling fs layer
50783e0dfb64: Verifying Checksum
50783e0dfb64: Download complete
50783e0dfb64: Pull complete
Digest: sha256:ef320ff10026a50cf5f0213d35537ce0041ac1d96e9b7800bafd8bc9eff6c693
Status: Downloaded newer image for busybox:latest
mariadb_10.3
Current db is mariadb:10.3 (mariadb_10.3)
New db is mariadb:10.4
There is an existing database in this project that doesn't match the upstream database type.
Please back up this database and then "ddev delete" so the new database can be created.

So i executed ddev delete && ddev get platformsh/ddev-platformsh but same error occurs.

Am i doing something wrong ?

@rfay I give you access to my PSH project (VNTP, dev branch), let me know if the invite expired, i'm in vacation but i'm able to connect and invite you again.

[rfay]

Thanks so much for this!

The problem is that in a late PR that went into v1.20.0, ddev get does a start before proceeding, which means there's no easy way for you to to this as an all-in-one.

I've been exploring this issue elsewhere and ddev may need a pre-start section where this kind of check takes place. It's also quite likely that the start-on-get behavior can be rolled back.

In the meantime, I'd love to have you proceed -

1. Just edit your .ddev/config.yaml (or create one with ddev config)
2. Set the mariadb version in the project to the mariadb version of your platform project (Mariadb 10.4, right)? This will prevent the check that goes wrong.
3. ddev delete -Oy
4. `ddev get platformsh/ddev-platformsh
5. See how it goes.

Thanks so much for testing it! And thanks for making this issue, which was discovered in discord the other day as well.

[rfay]

@flovntp thanks for helping with this issue, and thanks for giving me access to your project on Platform.sh.

I need you to give me access to your gitlab project as well, gitlab.com/flovntp/pilatus, or I can't check out the code.

Thanks!

[flovntp]

@rfay do you have a gitlab.com account ? i send an invite with your PSH address (randy.fay@platform.sh), let me know if you need to use another email address :wink:

[flovntp]

@rfay I still have an error :

Network ddev-vntp_default  Created 
Container ddev-vntp-dba  Started 
Container ddev-vntp-db  Started 
Container ddev-vntp-db  Healthy 
Container ddev-vntp-web  Started 
Container ddev-router  Started 
Instrumentation is opted in, but SegmentKey is not available. This usually means you have a locally-built ddev binary or one from a PR build. It's not an error. Please report it if you're using an official release build.

Using existing PLATFORMSH_CLI_TOKEN.

Using existing PLATFORM_PROJECT from project config.yaml.

Using existing PLATFORM_ENVIRONMENT from project config.yaml.

local     vntp-mariadb
mariadb_10.4
Current db is mariadb:10.4 (mariadb_10.4)
New db is mariadb:10.4
database versions match: mariadb:10.4=mariadb:10.4

could not process pre-install action '#ddev-nodisplay
# set -x
platform_routes=$(cat <<-ENDROUTES
{
"${DDEV_PRIMARY_URL}": {
  "primary": true,
  "id": null,
  "production_url": "${DDEV_PRIMARY_URL}",
  "attributes": {},
  "upstream": "drupal",
  "type": "upstream",
  "original_url": "https://{default}/"  }
}
ENDROUTES
)

cat <<-EOF >.ddev/config.platformsh.yaml
# #ddev-generated
# Generated configuration based on platform.sh project configuration
{{ $dbheader := index (split ":" .platformapp.relationships.database) "_0" }} 
{{ $dbtype := replace "postgresql" "postgres" (get (get .services $dbheader) "type") }}
php_version: {{ trimPrefix "php:" .platformapp.type }}
database:
  type: {{ regexReplaceAll "oracle-" (regexReplaceAll ":.*$" $dbtype "") "" }}
  version: {{ regexReplaceAll "^.*:" $dbtype "" }}
docroot: {{ dig "web" "locations" "/" "root" "notfound" .platformapp }}
web_environment:
# TODO: Review which of these matters and which can be dummied up
- "PLATFORM_MOUNTS={{ range $key, $value := .platformapp.mounts }}{{ $key }} {{ end }}"
- "PLATFORM_APP_DIR=/var/www/html"
- PLATFORM_PROJECT_ENTROPY=$(echo $RANDOM | shasum -a 256 | awk '{print $1}')
# Consider commit hash for PLATFORM_TREE_ID
- "PLATFORM_TREE_ID=2dc356f2fea13ef683f9adc5fc5bd28e05ad992a"
- "PLATFORM_DIR=/var/www/html"
- "PLATFORM_ROUTES=$( if base64 --version >/dev/null 2>&1; then echo -n ${platform_routes} | base64 -w0; else echo -n ${platform_routes} | base64; fi)"
- PLATFORM_VARIABLES=e30=
- PATH=$PATH:/var/www/html/.global/bin

hooks:
  post-start:
{{ if eq .platformapp.build.flavor "composer" }}
  - composer: install
{{ end }}

# TODO: Separate the hooks into separate exec lines
{{ if .platformapp.hooks.build }}
  # platformsh build hooks
  - exec: '{{ trimAll "\n" .platformapp.hooks.build | replace "\n\n" "\n" | splitList "\n"  | join ` && ` }}'
{{ end }}

# TODO: Separate the hooks into separate exec lines
{{ if .platformapp.hooks.deploy }}
  # platformsh deploy hooks
  - exec: '{{ trimAll "\n" .platformapp.hooks.deploy | replace "\n\n" "\n" | splitList "\n"  | join ` && ` }}'
{{ end }}

{{ if .platformapp.hooks.post_deploy }}
  # platformsh post_deploy hooks
  - exec: '{{ trimAll "\n" .platformapp.hooks.post_deploy | replace "\n\n" "\n" | splitList "\n"  | join ` && ` }}'
{{ end }}

EOF
': Unable to run action set -eu -o pipefail
#ddev-nodisplay
# set -x
platform_routes=$(cat <<-ENDROUTES
{
"${DDEV_PRIMARY_URL}": {
  "primary": true,
  "id": null,
  "production_url": "${DDEV_PRIMARY_URL}",
  "attributes": {},
  "upstream": "drupal",
  "type": "upstream",
  "original_url": "https://{default}/"  }
}
ENDROUTES
)

cat <<-EOF >.ddev/config.platformsh.yaml
# #ddev-generated
# Generated configuration based on platform.sh project configuration

php_version: 7.4
database:
  type: mariadb
  version: 10.4
docroot: public
web_environment:
# TODO: Review which of these matters and which can be dummied up
- "PLATFORM_MOUNTS=config/graphql/types/ezplatform public/var var/cache var/encore var/log var/sessions "
- "PLATFORM_APP_DIR=/var/www/html"
- PLATFORM_PROJECT_ENTROPY=$(echo $RANDOM | shasum -a 256 | awk '{print $1}')
# Consider commit hash for PLATFORM_TREE_ID
- "PLATFORM_TREE_ID=2dc356f2fea13ef683f9adc5fc5bd28e05ad992a"
- "PLATFORM_DIR=/var/www/html"
- "PLATFORM_ROUTES=$( if base64 --version >/dev/null 2>&1; then echo -n ${platform_routes} | base64 -w0; else echo -n ${platform_routes} | base64; fi)"
- PLATFORM_VARIABLES=e30=
- PATH=$PATH:/var/www/html/.global/bin

hooks:
  post-start:

# TODO: Separate the hooks into separate exec lines

  # platformsh build hooks
  - exec: 'set -e && # Install newer version of Node.js, as current default (v6LTS) does not play well with WebpackEncore && unset NPM_CONFIG_PREFIX && curl -o- https://raw.githubusercontent.com/creationix/nvm/v0.33.8/install.sh | dash && export NVM_DIR="$HOME/.nvm" && [ -s "$NVM_DIR/nvm.sh" ] && \. "$NVM_DIR/nvm.sh" && nvm current && nvm install 12 && if [ -z "$COMPOSER_AUTH" ]; then &&     echo "TIP: If you need to authenticate against Github/Gitlab/updates.ez.no, use COMPOSER_AUTH env variable" &&     echo "See: https://docs.platform.sh/guides/general/composer-auth.html#set-the-envcomposer_auth-project-variable" && fi && composer install --no-dev --prefer-dist --no-progress --no-interaction --optimize-autoloader'

# TODO: Separate the hooks into separate exec lines

  # platformsh deploy hooks
  - exec: 'set -e && unset NPM_CONFIG_PREFIX && command -v nvm && nvm use 12 && # Mainly relevant for eZ Platform demo usage, for own setup adapt this or remove and rely on migrations. && if [ ! -f public/var/.platform.installed ]; then &&     # To workaround issues with p.sh Varnish we clear container cache & temporary set Symfony Proxy &&     rm -Rf var/cache/$APP_ENV/*.* &&     SKIP_HTTPCACHE_PURGE="1" php -d memory_limit=-1 `which composer` &&     touch public/var/.platform.installed && fi && # Now that mounts are available, clear cache on mount. && # Note: Skip on PE Cluster setup using e.g. "if [$PLATFORM_BRANCH" != 'production']; then" & get p.sh to enable this on internal per node "pre_start" hook && sh bin/platformsh_prestart_cacheclear.sh && # If you also need to clear Redis cache on every deploy, you can either use this command or redis-cli && # Normally this should only be needed if cached data structures changes (upgrades), or you change data via sql (e.g. restore backup) && ##php bin/console cache:pool:clear cache.redis && # Example of additional deploy hooks if you use doctrine and/or kaliop migration bundle && ##php bin/console doctrine:migrations:migrate --no-interaction --allow-no-migration && php bin/console kaliop:migration:migrate --no-interaction --no-debug && # When using Solr, there are two cases where you'll need to rebuild indexes: && # - When Solr / search configuration changes && # - On database import/restore && # So in development it might be convenient to rebuild indexes, slowing down deploy time && ##php bin/console ezplatform:reindex --processes=auto && # When using Varnish/Fastly, HttpCache is not cleared when clearing Symfony Cache folder above, you need to && # clear cache yourself when for instance templates or config affecting responses change, for instance with: && ##bin/console fos:httpcache:invalidate:tag ez-all && # Depending on your VCL, buy default this would trigger a soft purge (expiry) and allow grace period, however && # even so if your change only affected certain subset of content, ideally you should only clear specific tags: && ##bin/console fos:httpcache:invalidate:tag l44 c33 ct2 ...'

EOF
: exit status 1, output=bash: line 17: NVM_DIR: unbound variable

[rfay]

Thanks so much @flovntp - I was able to use my gitlab account to get that, just had to add my platform.sh account to it. Now I'll be able to try out your situation.

The work I've done for your situation is in a PR, https://github.com/platformsh/ddev-platformsh/pull/25, but it does require the current HEAD of ddev. But now that I can try out your project we'll be able to make some progress.

Thanks!

[rfay]

Well, there's progress, but a number of problems, and this is a great example to work on because it has much more exotic things like trying to install nvm itself, etc.

I first tried to install the default branch, "main", but main doesn't have platform stuff. Then I tried the "master" branch, and made more progress with that.

rfay@rfay-mbp-2021:~/workspace/vntp$ ddev get ../ddev-platformsh/
Using existing PLATFORMSH_CLI_TOKEN.

Using existing PLATFORM_PROJECT from project config.yaml.

Using existing PLATFORM_ENVIRONMENT from project config.yaml.

# #ddev-generated

Installed file /Users/rfay/workspace/vntp/.ddev/web-build/Dockerfile.platformsh
Installed file /Users/rfay/workspace/vntp/.ddev/homeadditions/.bashrc.d/platformsh-environment.sh
Installed file /Users/rfay/workspace/vntp/.ddev/providers/platform.yaml
Installed file /Users/rfay/.ddev/commands/web/platform
Downloaded add-on ../ddev-platformsh/, use `ddev restart` to enable.
rfay@rfay-mbp-2021:~/workspace/vntp$ ddev restart
Restarting project vntp...
Container ddev-vntp-dba  Removed
Container ddev-vntp-web  Removed
Container ddev-vntp-db  Removed
Network ddev-vntp_default  Removed
Network ddev-vntp_default  Created
Container ddev-vntp-dba  Started
Container ddev-vntp-web  Started
Container ddev-vntp-db  Started
Container ddev-router  Running
nvm is not compatible with the npm config "prefix" option: currently set to "/mnt/ddev-global-cache/nvm_dir/vntp-web/versions/node/v10.15.3"
Run `npm config delete prefix` or `nvm use --delete-prefix v10.15.3 --silent` to unset it.
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 12540  100 12540    0     0  62388      0 --:--:-- --:--:-- --:--:-- 62388
=> nvm is already installed in /home/rfay/.nvm, trying to update using git
=> => Compressing and cleaning up git repository

=> Appending nvm source string to /home/rfay/.profile
=> bash_completion source string already in /home/rfay/.profile
nvm is not compatible with the npm config "prefix" option: currently set to "/mnt/ddev-global-cache/nvm_dir/vntp-web/versions/node/v10.15.3"
Run `npm config delete prefix` or `nvm use --delete-prefix v10.15.3 --silent` to unset it.
npm WARN config global `--global`, `--local` are deprecated. Use `--location=global` instead.
=> You currently have modules installed globally with `npm`. These will no
=> longer be linked to the active version of Node when you install a new node
=> with `nvm`; and they may (depending on how you construct your `$PATH`)
=> override the binaries of modules installed with `nvm`:

/usr/lib
├── corepack@0.10.0
├── gulp-cli@2.3.0
└── yarn@1.22.19
=> If you wish to uninstall them at a later point (or re-install them under your
=> `nvm` Nodes), you can remove them from the system Node as follows:

     $ nvm use system
     $ npm uninstall -g a_module

=> Close and reopen your terminal to start using nvm or run the following to use it now:

export NVM_DIR="$HOME/.nvm"
[ -s "$NVM_DIR/nvm.sh" ] && \. "$NVM_DIR/nvm.sh"  # This loads nvm
nvm is not compatible with the npm config "prefix" option: currently set to "/mnt/ddev-global-cache/nvm_dir/vntp-web/versions/node/v10.15.3"
Run `npm config delete prefix` or `nvm use --delete-prefix v10.15.3 --silent` to unset it.
Task failed: Exec command 'set -e
# Install newer version of Node.js, as current default (v6LTS) does not play well with WebpackEncore
unset NPM_CONFIG_PREFIX
curl -o- https://raw.githubusercontent.com/creationix/nvm/v0.33.8/install.sh | dash
export NVM_DIR="$HOME/.nvm"
[ -s "$NVM_DIR/nvm.sh" ] && \. "$NVM_DIR/nvm.sh"
nvm current
nvm install 10.15.3
if [ -z "$COMPOSER_AUTH" ]; then
    echo "TIP: If you need to authenticate against Github/Gitlab/updates.ez.no, use COMPOSER_AUTH env variable"
    echo "See: https://docs.platform.sh/guides/general/composer-auth.html#set-the-envcomposer_auth-project-variable"
fi
composer install --no-dev --prefer-dist --no-progress --no-interaction --optimize-autoloader
' in container/service 'web': exit status 11
A task failure does not mean that ddev failed, but your hook configuration has a command that failed.
nvm is not compatible with the npm config "prefix" option: currently set to "/mnt/ddev-global-cache/nvm_dir/vntp-web/versions/node/v10.15.3"
Run `npm config delete prefix` or `nvm use --delete-prefix v10.15.3 --silent` to unset it.
nvm
nvm is not compatible with the npm config "prefix" option: currently set to "/mnt/ddev-global-cache/nvm_dir/vntp-web/versions/node/v10.15.3"
Run `npm config delete prefix` or `nvm use --delete-prefix v10.15.3` to unset it.
Task failed: Exec command 'set -e
unset NPM_CONFIG_PREFIX
command -v nvm && nvm use 10.15.3
# Mainly relevant for eZ Platform demo usage, for own setup adapt this or remove and rely on migrations.
if [ ! -f public/var/.platform.installed ]; then
    # To workaround issues with p.sh Varnish we clear container cache & temporary set Symfony Proxy
    rm -Rf var/cache/$APP_ENV/*.*
    SKIP_HTTPCACHE_PURGE="1" php -d memory_limit=-1 `which composer` ezplatform-install
    touch public/var/.platform.installed
fi
# Now that mounts are available, clear cache on mount.
# Note: Skip on PE Cluster setup using e.g. "if [$PLATFORM_BRANCH" != 'production']; then" & get p.sh to enable this on internal per node "pre_start" hook
sh bin/platformsh_prestart_cacheclear.sh
# If you also need to clear Redis cache on every deploy, you can either use this command or redis-cli
# Normally this should only be needed if cached data structures changes (upgrades), or you change data via sql (e.g. restore backup)
##php bin/console cache:pool:clear cache.redis
# Example of additional deploy hooks if you use doctrine and/or kaliop migration bundle
##php bin/console doctrine:migrations:migrate --no-interaction --allow-no-migration
##php bin/console kaliop:migration:migrate --no-interaction --no-debug
# When using Solr, there are two cases where you'll need to rebuild indexes:
# - When Solr / search configuration changes
# - On database import/restore
# So in development it might be convenient to rebuild indexes, slowing down deploy time
##php bin/console ezplatform:reindex --processes=auto
# When using Varnish/Fastly, HttpCache is not cleared when clearing Symfony Cache folder above, you need to
# clear cache yourself when for instance templates or config affecting responses change, for instance with:
##bin/console fos:httpcache:invalidate:tag ez-all
# Depending on your VCL, buy default this would trigger a soft purge (expiry) and allow grace period, however
# even so if your change only affected certain subset of content, ideally you should only clear specific tags:
##bin/console fos:httpcache:invalidate:tag l44 c33 ct2 ...
' in container/service 'web': exit status 11
A task failure does not mean that ddev failed, but your hook configuration has a command that failed.
Restarted vntp
Your project can be reached at https://vntp.ddev.site https://127.0.0.1:61937

The biggest problems we have are in both sets of hooks:

nvm is not compatible with the npm config "prefix" option: currently set to "/mnt/ddev-global-cache/nvm_dir/vntp-web/versions/node/v10.15.3" Run npm config delete prefix or nvm use --delete-prefix v10.15.3 --silent to unset it.

This seems to be caused by https://stackoverflow.com/a/58559982/215713 - and it may only be with super-old node versions, because it hasn't been reported elsewhere.

DDEV's in-container .nvm is in fact a link, as in the Stack Overflow link.

ls -l ~/.nvm
lrwxrwxrwx 1 rfay dialout 39 Aug 20 21:24 /home/rfay/.nvm -> /mnt/ddev-global-cache/nvm_dir/vntp-web

Since node 10.15.3 is several years out of support, it's not clear to me whether this has to do with node 10 or what.

Your build hooks are:

        # Install newer version of Node.js, as current default (v6LTS) does not play well with WebpackEncore
        unset NPM_CONFIG_PREFIX
        curl -o- https://raw.githubusercontent.com/creationix/nvm/v0.33.8/install.sh | dash
        export NVM_DIR="$HOME/.nvm"
        [ -s "$NVM_DIR/nvm.sh" ] && \. "$NVM_DIR/nvm.sh"
        nvm current
        nvm install 10.15.3

        if [ -z "$COMPOSER_AUTH" ]; then
            echo "TIP: If you need to authenticate against Github/Gitlab/updates.ez.no, use COMPOSER_AUTH env variable"
            echo "See: https://docs.platform.sh/guides/general/composer-auth.html#set-the-envcomposer_auth-project-variable"
        fi

        composer install --no-dev --prefer-dist --no-progress --no-interaction --optimize-autoloader

So ... DDEV already has nvm installed, and this tries to install over the top of it. We can obviously improve it to check for nvm's existence before overwriting everything, but we'd hope that we wouldn't have to do that.

And of course, then it uses node 10 for a "newer" nodejs, but node 10 has been end-of-life since 2021-04-30.

And I think if we ever get to if [ -z "$COMPOSER_AUTH" ]; I imagine it will fail as needing to be if [ -z "${COMPOSER_AUTH:-}" ];, not sure.

So this is a great, great example of how hard it is to take something complex written for Platform and just place it into another context.

It's easy enough to solve for this exact case with modest changes, checking for nvm existing for example. But IMO it's not going to be possible to solve every problem that we will encounter.

If you want to spend a few minutes with this I'd love to do it together.

Also, ddev pull platform with environment main gets

[ClientException] Client error response [url] https://api.platform.sh/api/projects/lg4gpxi5g22he/environments/mai n/deployments/current [status code] 400 [reason phrase] Bad Request [message] The server could not comply with the request since it is either malformed or otherwise incorrect. [detail] {"app lications":"No .platform.app.yaml file found anywhere in the repository.","routes":"Error loa ding file: No .platform\/routes.yaml file found in the repository."} [title] Bad Request

[rfay]

I spent some more time with this, and unfortunately it can't be made to work because it depends on python 2, which we won't be supporting.

While chasing it, I added to config.yaml

fail_on_hook_fail: true
hooks:
  post-start:
    - exec: npm config delete prefix && npm config set prefix ~/.nvm/versions/node/v12.22.12

to sort out nvm problems.

And in your build hooks in .platform.app.yaml I changed to:

hooks:
    # Build hook, done before connected to services, disk is still writable here
    build: |
        set -e +u

because DDEV by default executes with set -e -u, not allowing unset variables, and COMPOSER_AUTH is unset in there.

Closing, this one won't go with python 2, but thanks so much for trying it out, and it helped me learn a lot about Platform.sh