Manifest fetch fails with 406 error

[masonoise]

I'm getting failed to fetch digest: 406 Not Acceptable that I've narrowed down to the check command. The current theory is that our registry is returning type application/json while check is looking for application/vnd.docker.distribution.manifest.v2+json. See this line: https://github.com/concourse/docker-image-resource/blob/master/cmd/check/main.go#L49

If the header can be configurable, or simply add application/json, that would be great.

Our private registry is an Artifactory Docker-type repository, which may explain why the type is different than what's expected.

[concourse-bot]

Hi there!

We use Pivotal Tracker to provide visibility into what our team is working on. A story for this issue has been automatically created.

The current status is as follows:

• [x] #119292001 Added application/json to manifestRequest Accept header
• [x] #119236061 Manifest fetch fails with 406 error

This comment, as well as the labels on the issue, will be automatically updated as the status in Tracker changes.

[databus23]

Ok, looks like its a regression introduced by #34, which fixed the resource for docker distribution >=2.3.1.

Could you check the Content-Type returned by your registry:

curl -I https://$HOST/v2/$ORG/$REPO/manifests/$TAG

[databus23]

Also, what version of Artifactory are you running? V2 Schema 2 should be supported since 4.4.3 according to https://www.jfrog.com/jira/browse/RTFACT-9056

[masonoise]

Yeah, I checked the registry and it is sending back Content-Type: application/json.

We are running Artifactory 4.4.2 rev 40093. Sounds like maybe 4.4.3 would help so I'll see when an upgrade might be planned.

[masonoise]

I did confirm that changing that line to manifestRequest.Header.Add("Accept", "application/json") fixes the problem, so the accept-header is the issue. I could submit a PR adding that header to the existing one, but if you want to look into making the header configurable that's probably a better long-term solution. Let me know -- thanks.

[databus23]

While I'm not in charge of this resource I'm not convinced that making this configurable is the way to go. Ideally the resource should implement the same behaviour as the docker daemon itself to fetch the correct manifest. Would be good to know what docker is actually sending to get the correct manifest in all cases. Unfortunately the docker code base is quite big and its not obvious to me what the client is actually sending.

[masonoise]

I added application/json to the accept header and testing seems good so I submitted a PR for the change. I'm not entirely sure how to do a complete build of everything in this package to create a new Dockerfile but building and testing the check command was straightforward. Thanks.

[databus23]

Unable to figure out from code what docker is doing itself I used mitmproxy to take a closer look: This is what I get when doing docker pull alpine (docker 1.11.1):

GET /v2/library/alpine/manifests/latest HTTP/1.1
host: registry-1.docker.io
User-Agent: docker/1.11.1 go/go1.5.4 git-commit/5604cbe kernel/4.4.8-boot2docker os/linux arch/amd64 UpstreamClient(Docker-Client/1.11.1 \(darwin\))
Accept: application/vnd.docker.distribution.manifest.list.v2+json
Accept: application/vnd.docker.distribution.manifest.v1+prettyjws
Accept: application/json
Accept: application/vnd.docker.distribution.manifest.v2+json
Authorization: Bearer [...]

docker pull progrium/busybox on the other hand yields:

GET /v2/progrium/busybox/manifests/latest HTTP/1.1
host: registry-1.docker.io
User-Agent: docker/1.11.1 go/go1.5.4 git-commit/5604cbe kernel/4.4.8-boot2docker os/linux arch/amd64 UpstreamClient(Docker-Client/1.11.1 \(darwin\))
Accept: application/vnd.docker.distribution.manifest.v2+json
Accept: application/vnd.docker.distribution.manifest.list.v2+json
Accept: application/vnd.docker.distribution.manifest.v1+prettyjws
Accept: application/json
Authorization: Bearer [...]

I'm a bit baffled that the ordering of Accept headers seems to be different depending on which image is pulled. At least it shows that docker is also sending application/json as an acceptable Content-Type.

[concourse-bot]

Hello again!

All stories related to this issue have been accepted, so I'm going to automatically close this issue.

At the time of writing, the following stories have been accepted:

• #119292001 Added application/json to manifestRequest Accept header
• #119236061 Manifest fetch fails with 406 error

If you feel there is still more to be done, or if you have any questions, leave a comment and we'll reopen if necessary!