search

cypress-io / cypress-docker-images

Docker images with Cypress dependencies and browsers
https://on.cypress.io/continuous-integration
MIT License
1.02k stars 380 forks source link

"Push Factory Image" failure checking `cypress/factory:4.0.1` #1093

Closed MikeMcC399 closed 4 months ago

MikeMcC399 commented 4 months ago

Issue

The CircleCI workflow job "Push Factory Image" is failing to get the status of cypress/factory:4.0.1 from Docker Hub cypress/factory.

The error is

message: 'unsupported schema version 2 in cypress/factory:4.0.1 manifest',

Examining the manifest of cypress/factory:4.0.1 with the following command shows that it differs in format from the manifest for cypress/factory:4.0.0 (and from cypress/base, cypress/browsers and cypress/included)

docker manifest inspect cypress/factory:4.0.1

shows

   "schemaVersion": 2,
   "mediaType": "application/vnd.oci.image.index.v1+json",

instead of

   "schemaVersion": 2,
   "mediaType": "application/vnd.docker.distribution.manifest.list.v2+json",

Logs

https://app.circleci.com/pipelines/github/cypress-io/cypress-docker-images/1959/workflows/3c12d584-8378-4f1d-8722-bcc504b21e2d/jobs/59620 from PR

shows

npm WARN exec The following package was not found and will be installed: docker-image-not-found@1.1.1
npm WARN deprecated inflight@1.0.6: This module is not supported, and leaks memory. Do not use it. Check out lru-cache if you want a good and tested way to coalesce async requests by a key value, which is much more comprehensive and powerful.
npm WARN deprecated rimraf@2.4.5: Rimraf versions prior to v4 are no longer supported
npm WARN deprecated uuid@3.4.0: Please upgrade  to version 7 or higher.  Older versions may use Math.random() in certain circumstances, which is known to be problematic.  See https://v8.dev/blog/math-random for details.
npm WARN deprecated glob@6.0.4: Glob versions prior to v9 are no longer supported
npm WARN deprecated debug@4.1.1: Debug versions >=3.2.0 <3.2.7 || >=4 <4.3.1 have a low-severity ReDos regression when used in a Node.js environment. It is recommended you upgrade to 3.2.7 or 4.3.1. (https://github.com/visionmedia/debug/issues/797)
npm WARN deprecated tough-cookie@2.0.0: ReDoS vulnerability parsing Set-Cookie https://nodesecurity.io/advisories/130
got an error fetching info about Docker image docker.io/cypress/factory:4.0.1
got an error other than image not found
RestError [InvalidContentError]: unsupported schema version 2 in cypress/factory:4.0.1 manifest
    at _afterCall (/home/circleci/.npm/_npx/299ed24e1d89a4d2/node_modules/docker-registry-client/lib/registry-client-v2.js:1469:24)
    at IncomingMessage.finish (/home/circleci/.npm/_npx/299ed24e1d89a4d2/node_modules/docker-registry-client/lib/docker-json-client.js:171:13)
    at Object.onceWrapper (node:events:632:28)
    at IncomingMessage.emit (node:events:530:35)
    at endReadableNT (node:internal/streams/readable:1696:12)
    at process.processTicksAndRejections (node:internal/process/task_queues:82:21) {
  jse_shortmsg: 'unsupported schema version 2 in cypress/factory:4.0.1 manifest',
  jse_info: {},
  message: 'unsupported schema version 2 in cypress/factory:4.0.1 manifest',
  [stack]: 'InvalidContentError: unsupported schema version 2 in cypress/factory:4.0.1 manifest\n' +
    '    at _afterCall (/home/circleci/.npm/_npx/299ed24e1d89a4d2/node_modules/docker-registry-client/lib/registry-client-v2.js:1469:24)\n' +
    '    at IncomingMessage.finish (/home/circleci/.npm/_npx/299ed24e1d89a4d2/node_modules/docker-registry-client/lib/docker-json-client.js:171:13)\n' +
    '    at Object.onceWrapper (node:events:632:28)\n' +
    '    at IncomingMessage.emit (node:events:530:35)\n' +
    '    at endReadableNT (node:internal/streams/readable:1696:12)\n' +
    '    at process.processTicksAndRejections (node:internal/process/task_queues:82:21)',
  body: {
    code: 'InvalidContent',
    message: 'unsupported schema version 2 in cypress/factory:4.0.1 manifest'
  },
  restCode: 'InvalidContent',
  cause: <ref *1> [Function: we_cause] {
    [length]: 1,
    [name]: 'we_cause',
    [arguments]: null,
    [caller]: null,
    [prototype]: { [constructor]: [Circular *1] }
  },
  name: 'InvalidContentError',
  statusCode: 400
}
exiting with code 1
Docker hub has image cypress/factory:4.0.1 or not responding
We should stop in this case

Analysis

Manifest

cypress/factory:4.0.1

$ docker manifest inspect cypress/factory:4.0.1
{
   "schemaVersion": 2,
   "mediaType": "application/vnd.oci.image.index.v1+json",
   "manifests": [
      {
         "mediaType": "application/vnd.oci.image.manifest.v1+json",
         "size": 865,
         "digest": "sha256:bbf8a2e6fda64adb6c87e9793493240667e9e3a43874e71563f5963076942b85",
         "platform": {
            "architecture": "arm64",
            "os": "linux"
         }
      },
      {
         "mediaType": "application/vnd.oci.image.manifest.v1+json",
         "size": 865,
         "digest": "sha256:7cfe1c11128c9e4a119b7ad3a6bb4ce13a705cc8f00436f4174a964e102fe4fd",
         "platform": {
            "architecture": "amd64",
            "os": "linux"
         }
      },
      {
         "mediaType": "application/vnd.oci.image.manifest.v1+json",
         "size": 566,
         "digest": "sha256:936a49c206ffcdfeb0faed320876c164b671f862cd97e815a8cfa9ba3c61eff6",
         "platform": {
            "architecture": "unknown",
            "os": "unknown"
         }
      },
      {
         "mediaType": "application/vnd.oci.image.manifest.v1+json",
         "size": 566,
         "digest": "sha256:fdb5a0baf2cb8b69d44a4d435b0ff6fdd1b6877de919cf0a74bc4aae0c93a4e1",
         "platform": {
            "architecture": "unknown",
            "os": "unknown"
         }
      }
   ]
}

cypress/factory:4.0.0

$ docker manifest inspect cypress/factory:4.0.0
{
   "schemaVersion": 2,
   "mediaType": "application/vnd.docker.distribution.manifest.list.v2+json",
   "manifests": [
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "size": 899,
         "digest": "sha256:7a9498698dc9ceacb1a3e93cc602436f6e135c734a701681ef0fe7304e2cc601",
         "platform": {
            "architecture": "arm64",
            "os": "linux"
         }
      },
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "size": 899,
         "digest": "sha256:abbafb58e45003f37c1e2a0466c36af8707917a7b682eed9546ed928b2ceeec3",
         "platform": {
            "architecture": "amd64",
            "os": "linux"
         }
      }
   ]
}

Log Push Factory Image for 4.0.1

https://app.circleci.com/pipelines/github/cypress-io/cypress-docker-images/1944/workflows/d32dcb15-f148-4cf7-a4d1-3b1f48ab6141/jobs/59206

Check for existence

npm WARN exec The following package was not found and will be installed: docker-image-not-found@1.1.1
npm WARN deprecated inflight@1.0.6: This module is not supported, and leaks memory. Do not use it. Check out lru-cache if you want a good and tested way to coalesce async requests by a key value, which is much more comprehensive and powerful.
npm WARN deprecated rimraf@2.4.5: Rimraf versions prior to v4 are no longer supported
npm WARN deprecated glob@6.0.4: Glob versions prior to v9 are no longer supported
npm WARN deprecated uuid@3.4.0: Please upgrade  to version 7 or higher.  Older versions may use Math.random() in certain circumstances, which is known to be problematic.  See https://v8.dev/blog/math-random for details.
npm WARN deprecated debug@4.1.1: Debug versions >=3.2.0 <3.2.7 || >=4 <4.3.1 have a low-severity ReDos regression when used in a Node.js environment. It is recommended you upgrade to 3.2.7 or 4.3.1. (https://github.com/visionmedia/debug/issues/797)
npm WARN deprecated tough-cookie@2.0.0: ReDoS vulnerability parsing Set-Cookie https://nodesecurity.io/advisories/130
got an error fetching info about Docker image docker.io/cypress/factory:4.0.1
Got definite NotFoundError
exiting with code 0
Docker hub says image cypress/factory:4.0.1 does not exist

building Docker image for target factory

#14 exporting to image
#14 exporting manifest sha256:265f8134ce492f9219dd8c71ff531f33fa38e2e4254873e981ef94bba7427103 0.0s done
#14 exporting config sha256:a8eb011e2c5ca149e4f573a8b902f10596a378dbcebc94eb67ea1a95a21911fa 0.0s done
#14 exporting manifest sha256:206ac86fb72641f54d01c8068bafa8e216169cd4ce433aa408de646176e885eb 0.0s done
#14 exporting config sha256:de63602eb3ffcd58ce5dbde4fa501245b01748e3e17b91a78406557327465997 0.0s done
#14 exporting manifest list sha256:d287e6a9570956dbd6f70efb871bf86db5aebb94bb331dcf47ddb65d5fba205b 0.0s done
#14 pushing layers
#14 pushing layers 5.4s done
#14 pushing manifest for docker.io/cypress/factory:latest@sha256:d287e6a9570956dbd6f70efb871bf86db5aebb94bb331dcf47ddb65d5fba205b
#14 pushing manifest for docker.io/cypress/factory:latest@sha256:d287e6a9570956dbd6f70efb871bf86db5aebb94bb331dcf47ddb65d5fba205b 0.5s done
#14 pushing layers 0.2s done
#14 pushing manifest for docker.io/cypress/factory:4.0.1@sha256:d287e6a9570956dbd6f70efb871bf86db5aebb94bb331dcf47ddb65d5fba205b
#14 pushing manifest for docker.io/cypress/factory:4.0.1@sha256:d287e6a9570956dbd6f70efb871bf86db5aebb94bb331dcf47ddb65d5fba205b 0.3s done
#14 DONE 25.3s
MikeMcC399 commented 4 months ago

@jennifer-shehane

I don't understand why this issue has occurred and it is in an area where I have no experience! I can't tell if was a CircleCI glitch or something else.

If it is not addressed it will prevent any new cypress/factory images from being published.

Depending on the root cause it may also affect the automatic publication of cypress/base, cypress/browsers and cypress/included images.

MikeMcC399 commented 4 months ago

There is a possibility that the image cypress/factory:4.0.1 is corrupted through 3 parallel-running PRs which each checked if there was an image cypress/factory:4.0.1 present, found there was not, and then each built one and published it:

The 2nd of the PRs https://github.com/cypress-io/cypress-docker-images/pull/1079 also effectively bumped the Docker client version from Docker 20.10.18 to 26.0.2. This might have all been too much in one go!

Perhaps cypress/factory:4.0.1 should be manually deleted so that it can be recreated?

MikeMcC399 commented 4 months ago

It seems there were some breaking changes in https://github.com/docker/buildx

MikeMcC399 commented 4 months ago

After this is merged it will require a manual delete of the incompatible cypress/factory:4.0.1 image

jennifer-shehane commented 4 months ago

K, let me know if that works and if the deletion is required.

MikeMcC399 commented 4 months ago

@jennifer-shehane

Sorry that a revert was necessary, but at least only one image was caught up in this. I only noticed by accident when I was looking into another failure, as the workflow itself does not return an error. It's only visible when you examine the logs manually.

Status

https://app.circleci.com/pipelines/github/cypress-io/cypress-docker-images/1967/workflows/63050ae3-636c-46a2-a570-c5a14187d2e1/jobs/59867/parallel-runs/0/steps/0-103?invite=true#step-103-1209_95

RestError [InvalidContentError]: unsupported schema version 2 in cypress/factory:4.0.1 manifest

So, yes please, deletion of cypress/factory:4.0.1 on Docker Hub is needed. I don't think it checks on AWS, so I'm guessing that it will just overwrite there.

jennifer-shehane commented 4 months ago

I deleted the 4.0.1 tag in cypress/factory. Hopefully that worked.

MikeMcC399 commented 4 months ago

@jennifer-shehane

I deleted the 4.0.1 tag in cypress/factory. Hopefully that worked.

Thank you! That should be good, and the next merge to master will test that out.

In the meantime I checked by hand and docker-image-not-found is happy again! 😃 

$ docker-image-not-found --repo cypress/factory:4.0.1
got an error fetching info about Docker image docker.io/cypress/factory:4.0.1
Got definite NotFoundError
exiting with code 0

It does however still fail when no tag is defined, which defaults to latest, and latest is still linked to the problematic manifest. I expect that will get overwritten at the next release. We'll see!

MikeMcC399 commented 4 months ago

Closing as resolved. I will be monitoring what happens next!

Edit: