Old Multidevs not being deleted

[maskedjellybean]

During the build process there is a step called "Delete old Multidevs to make space for a new one". This does not work, which eventually causes builds to fail until I manually delete old Multidevs.

I can see the reason why but I'm not sure how to fix it. The build step output looks like this:

#!/bin/bash -eo pipefail
terminus -n build:env:delete:ci "$TERMINUS_SITE" --keep=2 --yes
 [notice] No environments matched the provided pattern "ci-".
CircleCI received exit code 0

Notice that it is looking for Multidevs that start with "ci-". None of the Multidevs created by this orb start with "ci-". They all start with "pr-". This makes some sense because our workflow is based on pull requests and looks like this:

• Branch off of master to work on a feature.
• Create a PR on Github to merge the feature branch back into master.
• CircleCI automatically creates a Multidev env representing the merge of the PR. The Multidev is named starting with "pr-".
• After QA on the Multidev, the PR is merged on Github. CircleCI automatically deploys to Dev env. (at the point the Multidev is no longer needed and could be deleted)

Is there any way for the "Delete old Multidevs to make space for a new one" step to delete Multidevs whose PR has been merged? If not, what is the purpose of this step?

[maskedjellybean]

After updating to version 0.5.1 I think this is resolved. At least the output appears to be doing more:

#!/bin/bash -eo pipefail
if [[ $TERMINUS_ENV = ci-* || $TERMINUS_ENV = pr-*   ]]
then
    echo "May need to delete old ci- or pr- environments to make room for this one"
    echo "Getting list of all environments"
    export ENV_LIST=$(terminus env:list $TERMINUS_SITE --field=id)
    echo "Checking if current environment is in list of existing environments"
    if [[ $(echo "${ENV_LIST}" | grep -x ${TERMINUS_ENV})  ]]
    then
        echo "${TERMINUS_ENV} found in the list of environments"
        exit 0
    else
        echo "${TERMINUS_ENV} not found in the list of environments."
        echo "Running clean-up script to delete old pr- environments"
        terminus -n build:env:delete:pr "$TERMINUS_SITE" --yes
        if [[ $TERMINUS_ENV = ci-*  ]]
        then
            echo "Running clean-up script to delete old ci- environments"
            terminus -n build:env:delete:ci "$TERMINUS_SITE" --keep=2 --yes
        else
            echo "Skipping deletion of ci- envs"
        fi
    fi
fi

May need to delete old ci- or pr- environments to make room for this one
Getting list of all environments
Checking if current environment is in list of existing environments
pr-43 not found in the list of environments.
Running clean-up script to delete old pr- environments
 [notice] Rsync pr-24.41bb5bb9-3ef0-4449-b837-8f30bd6b5e17@appserver.pr-24.41bb5bb9-3ef0-4449-b837-8f30bd6b5e17.drush.in:code/build-metadata.json => /tmp/build-metadata.json
 [notice] Rsync pr-36.41bb5bb9-3ef0-4449-b837-8f30bd6b5e17@appserver.pr-36.41bb5bb9-3ef0-4449-b837-8f30bd6b5e17.drush.in:code/build-metadata.json => /tmp/build-metadata.json
 [notice] Rsync pr-34.41bb5bb9-3ef0-4449-b837-8f30bd6b5e17@appserver.pr-34.41bb5bb9-3ef0-4449-b837-8f30bd6b5e17.drush.in:code/build-metadata.json => /tmp/build-metadata.json
 [notice] Rsync pr-37.41bb5bb9-3ef0-4449-b837-8f30bd6b5e17@appserver.pr-37.41bb5bb9-3ef0-4449-b837-8f30bd6b5e17.drush.in:code/build-metadata.json => /tmp/build-metadata.json
 [notice] Rsync pr-38.41bb5bb9-3ef0-4449-b837-8f30bd6b5e17@appserver.pr-38.41bb5bb9-3ef0-4449-b837-8f30bd6b5e17.drush.in:code/build-metadata.json => /tmp/build-metadata.json
 [notice] Rsync pr-39.41bb5bb9-3ef0-4449-b837-8f30bd6b5e17@appserver.pr-39.41bb5bb9-3ef0-4449-b837-8f30bd6b5e17.drush.in:code/build-metadata.json => /tmp/build-metadata.json
 [notice] Rsync pr-40.41bb5bb9-3ef0-4449-b837-8f30bd6b5e17@appserver.pr-40.41bb5bb9-3ef0-4449-b837-8f30bd6b5e17.drush.in:code/build-metadata.json => /tmp/build-metadata.json
 [notice] Rsync pr-41.41bb5bb9-3ef0-4449-b837-8f30bd6b5e17@appserver.pr-41.41bb5bb9-3ef0-4449-b837-8f30bd6b5e17.drush.in:code/build-metadata.json => /tmp/build-metadata.json
 [notice] No environments have build metadata.
Skipping deletion of ci- envs
CircleCI received exit code 0

This doesn't seem to be reflected in the deletion of Multidevs in the Pantheon dashboard however. There's still more Multidevs than there are open PRs. We'll see if we run out of space for Multidevs again. It's still unclear to me what the orb is using the determine whether a Multidev is "old" and can be deleted.

[digitalsutton]

(TL;DR - if you didn't use Terminus Build Tools to create your project, this probably isn't working for you)

I've experienced this same issue and it is not resolved with version 0.5.1. Here's what I've been able to figure out so far:

• As seen in @maskedjellybean output above, the script is looking for the file build-metadata.json inside the code directory on Pantheon.
• This file appears to be generated on the Pantheon platform side, but I've not been able to figure out how (updated: I sorta figured out how below). When you use Terminus Build Tools to create a new project, you get a default cirlcleci config that - interestingly - does NOT use this orb. I was hoping to find something there about creating that file, but no dice.
• Without the build-metatdata.json file, the deletion of old mulltidevs gets skipped (as seen in the output example above).

After some digging, I found that file gets generated in the push_back_to_github function that comes with with pantheon-systems/quicksilver-pushback package. The function is called via pantheon.yml like this (from a vanilla Build Tools site):

workflows:
  sync_code:
    after:
      - type: webphp
        description: Push changes back to GitHub if needed
        script: private/scripts/quicksilver/quicksilver-pushback/push-back.php

I haven't tried this yet, but in theory if you add pantheon-systems/quicksilver-pushback as a project dependency and setup your pantheon.yml to call that script, it should work.

I hope this helps somebody.

[maskedjellybean]

Thanks for the info! Did you end up trying the pushback script? I'm hesitant to try it without understanding everything it does... I'll take a closer look at it. Definitely seems like an oversight if this orb relies on a file created by an optional/unrelated script.

[enotick]

I tried this one - still no luck same problem - no environments have build metadata.

[stevector]

I think I figured it out: https://github.com/pantheon-systems/circleci-orb/pull/52 Testers wanted!

[enotick]

@stevector tested linking to the new version of the orb as shown in the PR - no luck same problem. Does it have to have push_back extension? I removed mine since it wasn't doing anything.

Also perhaps it would be easier / good idea to allow for multidev spin off option and that it can be set to skip?

[stevector]

@enotick can you link me to the repo/build output where you tested it?

I'm getting deletions working appropriately here: https://app.circleci.com/pipelines/github/stevector/d9umami/25/workflows/f231c86f-c0c0-44f9-b506-c3b72a9fddd1/jobs/25

Running clean-up script to delete old pr- environments
 [notice] Rsync pr-1.eb744546-238b-41f2-9953-ce3ca96edea1@appserver.pr-1.eb744546-238b-41f2-9953-ce3ca96edea1.drush.in:code/build-metadata.json => /tmp/build-metadata.json
 [notice] Call GitHub API: GET repos/stevector/*******/pulls?state=all
 [notice] Deleted the multidev environment pr-3.
 [notice] Deleted the multidev environment pr-1.

I don't think the push_back extension is needed for this to work. To my understanding, it also needs this url property to be set. But Build Tools is the thing that sets the property.

[enotick]

@stevector thank you for all the help. The solution indeed works. For other people who might be struggling 2 gotchas:

1. It will not be applicable to the old multidevs - prior to the applied orb fix. So you will have to delete those manually
2. You will need to add GITHUB_TOKEN to the Circle CI environment vars for your project. Give the toke repo scope. It will only delete closed / merged PRs and will not try to clean up if it doesn't need to spin up a new multidev.

[maskedjellybean]

Thanks! Sorry I'm late to testing this. I've updated to 0.5.2 and I'll let you know what happens when I hit the max number of multidevs.