Manifest with digest 'sha256:884ac2144c3ee154bd67271e99bc6ef00f430092750167729b0321ab55fde5ec' has media type 'application/vnd.docker.distribution.manifest.list.v2+json', but client accepts 'application/vnd.docker.distribution.manifest.v2+json'."

[cybercent]

Actual behavior A clear and concise description of what the bug is. Tried to deploy on AppEngine Flexible code I already deployed before.

Error:

Starting Step #1
Step #1: Already have image (with digest): gcr.io/kaniko-project/executor@sha256:f87c11770a4d3ed33436508d206c584812cd656e6ed08eda1cff5c1ee44f5870
Step #1: INFO[0000] Downloading base image gcr.io/distroless/base@sha256:884ac2144c3ee154bd67271e99bc6ef00f430092750167729b0321ab55fde5ec 
Step #1: error building image: getting stage builder for stage 0: MANIFEST_UNKNOWN: "Manifest with digest 'sha256:884ac2144c3ee154bd67271e99bc6ef00f430092750167729b0321ab55fde5ec' has media type 'application/vnd.docker.distribution.manifest.list.v2+json', but client accepts 'application/vnd.docker.distribution.manifest.v2+json'."
Finished Step #1
ERROR
ERROR: build step 1 "gcr.io/kaniko-project/executor@sha256:f87c11770a4d3ed33436508d206c584812cd656e6ed08eda1cff5c1ee44f5870" failed: step exited with non-zero status: 1

Expected behavior A clear and concise description of what you expected to happen.

Expected to build.

To Reproduce Steps to reproduce the behavior:

1. ...
2. ...

Additional Information

• Dockerfile Please provide either the Dockerfile you're trying to build or one that can reproduce this error.
• Build Context Please provide or clearly describe any files needed to build the Dockerfile (ADD/COPY commands)

• Kaniko Image (fully qualified with digest)

  Triage Notes for the Maintainers

  Description	Yes/No
  Please check if this a new feature you are proposing	- [ ]
  Please check if the build works in docker but not in kaniko	- [ ]
  Please check if this error is seen when you use --cache flag	- [ ]
  Please check if your dockerfile is a multistage dockerfile	- [ ]

[popinary]

Experiencing the same issue on all App Engine Flexible Environment.

The App Engine standard environment deployments succeed.

[FeCastle]

Same issue. App Engine Flex Environment. I tried go1.13 but it did not help.

[MariaKeskinen]

Same issue here. Go1.13 didn't help. Any other tips to try to solve the issue?

[albeebe]

Same issue. Nothing appears to be working and our dev team has grinded to a halt this week. It's bad

[albeebe]

I can confirm this solution from a Google engineer worked (at least for me).

Downgrade gcloud to the previous version

gcloud components update --version 315.0.0

[sbuss]

Re @albeebe's comment: The fix required us to roll back our GAE Flex Go builder images and for you to downgrade your cloudsdk version. We are investigating the root cause, which may (or may not) lie in Kaniko.

[sbuss]

We are still working on our root cause analysis, but the initial results indicate it could be an incompatibility with a recent change to use manifest lists for the :latest tag in https://github.com/GoogleContainerTools/distroless. It seems Kaniko may be incompatible with this change.

[tejal29]

Thanks @sbuss. Kaniko will try to add support manifest list soon!

[jonjohnsonjr]

@sbuss @tejal29 does this reproduce at HEAD? I'm just speculating, but I'm fairly confident this was fixed almost two years ago.

The executor image is very old:

$ crane config gcr.io/kaniko-project/executor@sha256:f87c11770a4d3ed33436508d206c584812cd656e6ed08eda1cff5c1ee44f5870 | jq .created
"2018-11-07T18:29:09.932346179Z"

I believe the relevant changes to go-containerregistry were merged into kaniko in https://github.com/GoogleContainerTools/kaniko/pull/680

[tejal29]

Thanks @jonjohnsonjr. I looked in the code and on master i think the v1.3.0 should fix it. The error line "error building image: getting stage builder for stage 0" itself is pretty old.

[jonjohnsonjr]

Is this fixed?

[jhssilva]

Currently, I'm stuck on this issue. The build doesn't go further. Do we know a work around?

[imjasonh]

Currently, I'm stuck on this issue. The build doesn't go further. Do we know a work around?

Are you using the latest kaniko release? It sounds like this may have been fixed a while ago.