Verify that sidecars exist before declaring che devfile registry release successful

nickboldt commented 3 years ago

Is your task related to a problem? Please describe.

As seen in https://github.com/eclipse/che/issues/19486 we had a situation w/ multiarch sidecar & happy path builds where...

the build failed to create the containers because not all arches requested to be built exist for the base images from which we're building...
despite failing to create sidecars, the devfile registry GH action thought all was well, and merrily reported success.

Describe the solution you'd like

We did a couple quickfixes today:

Quickfix to remove multiarch with buildx: https://github.com/eclipse-che/che-devfile-registry/commit/65a109f8793ad98a564f5a972c16556b953d36c4
Quickfix to throw error if make-release.sh fails: https://github.com/eclipse-che/che-devfile-registry/commit/ee4dbcb0f74b7fef21ddd00c4d1f2b4af3d45778

(needs validation -- we should have seen build failures for nightlies for the last week?)

But what we really need is a full validation of 'do the containers I need in the registry exist, and do the tags I expect also exist', like we do for other containers, eg., like this:

IMAGES_LIST=(
    quay.io/eclipse/che-endpoint-watcher
    quay.io/eclipse/che-keycloak
    quay.io/eclipse/che-postgres
    quay.io/eclipse/che-dev
    quay.io/eclipse/che-server
    quay.io/eclipse/che-dashboard-dev
    quay.io/eclipse/che-e2e
)

if [[ ${PHASES} == *"3"* ]] || [[ ${PHASES} == *"4"* ]]; then
    # verify images all created from IMAGES_LIST
    for image in "${IMAGES_LIST[@]}"; do
        verifyContainerExistsWithTimeout ${image}:${CHE_VERSION} 60
    done
fi

-- https://github.com/eclipse/che-release/blob/master/make-release.sh#L271-L286

Additional context

To compute a list of sidecar base images for the Che devfile registry, and then vet them for which arches they support:

curl -sSLo- https://raw.githubusercontent.com/eclipse-che/che-devfile-registry/master/arbitrary-users-patch/base_images | sed -r -e "s#che-.+ +##g" > /tmp/base-images.txt
for d in $(cat /tmp/base-images.txt); do if [[ $d != *"/"* ]]; then d=docker.io/$d; fi; echo; echo $d; skopeo inspect docker://${d} --raw | jq -r .manifests[].platform.architecture 2>&1 | tr "\n" " "; rm -f /tmp/out.txt; echo; done
rm -f  /tmp/base-images.txt

Output (where null = single arch, likely just amd64):

docker.io/antora/antora:2.3.3
null

registry.access.redhat.com/devtools/llvm-toolset-rhel7
amd64 ppc64le s390x 

mcr.microsoft.com/dotnet/core/sdk:2.2-stretch
null

mcr.microsoft.com/dotnet/core/sdk:3.1.301-buster
null

docker.io/golang:1.12-stretch
amd64 arm arm64 386 ppc64le s390x 

docker.io/golang:1.14-stretch
amd64 arm arm64 386 

docker.io/gradle:6.6.1-jdk11
amd64 arm arm64 ppc64le s390x 

docker.io/maven:3.6.3-jdk-11
amd64 arm64 

docker.io/maven:3.6.1-jdk-8
amd64 

docker.io/node:10.16
amd64 arm arm64 ppc64le s390x 

docker.io/node:12.18
amd64 arm arm64 

registry.access.redhat.com/ubi8/nodejs-10
amd64 arm64 ppc64le s390x 

registry.centos.org/che-stacks/centos-nodejs
null

quay.io/eclipse/che-php-base:7.4
null

centos/python-36-centos7:1
null

docker.io/python:3.8.6-slim
amd64 arm arm arm64 386 mips64le ppc64le s390x 

quay.io/quarkus/centos-quarkus-maven:21.0-java11
null

docker.io/rust:1.39.0-slim
amd64 arm arm64 386

ericwill commented 3 years ago

It would be cool if we had a job that ran every night and checked that the digests in the base_images file are valid. That way we don't need to wait until release time to find out an image was deleted/moved etc.

nickboldt commented 3 years ago

IMAGES_LIST array in the above sample code can be created from content in https://github.com/eclipse-che/che-devfile-registry/blob/master/arbitrary-users-patch/base_images ... every item in the left column maps to quay.io/eclipse/[image here]:[Che-release-tag]

checked that the digests in the base_images file are valid

What if we just regen the file and if a change is found, a PR is submitted?

Discussed this on 4/21, and @mkuznyetsov agreed to open a new issue for this work as a followup task to this issue, which should be resolved as of the 7.29.1 release.

nickboldt commented 3 years ago

additional PR https://github.com/eclipse-che/che-devfile-registry/pull/375 needed to fix missing script variable, wrong path to base images file, and avoid 20 sec waits before each check.

Also made the checks run in parallel instead of sequentially because hey, might as well go fast fast fast, right ?

nickboldt commented 3 years ago

running to verify in https://github.com/eclipse-che/che-devfile-registry/runs/2410671243?check_suite_focus=true

nickboldt commented 3 years ago

2021-04-22T14:27:11.1831289Z        [1/3] Verify quay.io/eclipse/che-java11-maven:7.29.1 exists...
2021-04-22T14:27:11.1832126Z        [1/3] Verify quay.io/eclipse/che-java8-maven:7.29.1 exists...
2021-04-22T14:27:11.1833005Z        [1/3] Verify quay.io/eclipse/che-nodejs10-community:7.29.1 exists...
2021-04-22T14:27:11.1833932Z        [1/3] Verify quay.io/eclipse/che-nodejs12-community:7.29.1 exists...
2021-04-22T14:27:11.1834803Z        [1/3] Verify quay.io/eclipse/che-nodejs10-ubi:7.29.1 exists...
2021-04-22T14:27:11.1835641Z        [1/3] Verify quay.io/eclipse/che-nodejs8-centos:7.29.1 exists...
2021-04-22T14:27:11.1836456Z        [1/3] Verify quay.io/eclipse/che-php-7:7.29.1 exists...
2021-04-22T14:27:11.1837238Z        [1/3] Verify quay.io/eclipse/che-python-3.8:7.29.1 exists...
2021-04-22T14:27:11.1838028Z        [1/3] Verify quay.io/eclipse/che-quarkus:7.29.1 exists...
2021-04-22T14:27:11.1838797Z        [1/3] Verify quay.io/eclipse/che-rust-1.39:7.29.1 exists...

...

2021-04-22T14:27:11.9169554Z + echo '[INFO] Found quay.io/eclipse/che-cpp-rhel7:7.29.1 (tag = 7.29.1)'
...

nickboldt commented 3 years ago

Followup in https://github.com/eclipse/che/issues/19660

eclipse-che / che