klar getting 404 from clair?

[chino]

Not sure if this is something I might be doing wrong.

Below I'm attempting to run a test against public centos container image.

Clair/Klar/Docker/Registry are all running locally in a single linux VM with --net=host to simplify it all.

I'm using docker registry:2 (28525f9a6e46), klar (tried git:1e20efb and 1.5-rc2), and clair:latest (be223c092e09).

----> HTTP REQUEST:
GET /v2/centos/manifests/latest HTTP/1.1
Host: localhost:5000
Accept: application/vnd.docker.distribution.manifest.v2+json

<---- HTTP RESPONSE:
HTTP/1.1 200 OK
Content-Length: 529
Content-Type: application/vnd.docker.distribution.manifest.v2+json
Date: Wed, 11 Oct 2017 01:59:46 GMT
Docker-Content-Digest: sha256:822de5245dc5b659df56dd32795b08ae42db4cc901f3462fc509e91e97132dc0
Docker-Distribution-Api-Version: registry/2.0
Etag: "sha256:822de5245dc5b659df56dd32795b08ae42db4cc901f3462fc509e91e97132dc0"
X-Content-Type-Options: nosniff

{
   "schemaVersion": 2,
   "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
   "config": {
      "mediaType": "application/vnd.docker.container.image.v1+json",
      "size": 1863,
      "digest": "sha256:196e0ce0c9fbb31da595b893dd39bc9fd4aa78a474bbdc21459a3ebe855b7768"
   },
   "layers": [
      {
         "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
         "size": 73386947,
         "digest": "sha256:d9aaf4d82f249dc101a6638ff5177fe926cdebfa6c42d874dfa5029533da0e72"
      }
   ]
}
Analysing 1 layers
----> HTTP REQUEST:
POST /v1/layers HTTP/1.1
Host: localhost:6060
Content-Type: application/json

{"Layer":{"Name":"196e0ce0c9fbb31da595b893dd39bc9fd4aa78a474bbdc21459a3ebe855b7768d9aaf4d82f249dc101a6638ff5177fe926cdebfa6c42d874dfa5029533da0e72","Path":"http://localhost:5000/v2/centos/blobs/sha256:d9aaf4d82f249dc101a6638ff5177fe926cdebfa6c42d874dfa5029533da0e72","ParentName":"","Format":"Docker","Features":null,"Headers":{"Authorization":""}}}
<---- HTTP RESPONSE:
HTTP/1.1 404 Not Found
Content-Length: 10
Content-Type: text/plain; charset=utf-8
Date: Wed, 11 Oct 2017 01:59:46 GMT
X-Content-Type-Options: nosniff

Not Found

Push layer 0 failed: Can't even read an error message: invalid character 'N' looking for beginning of value
----> HTTP REQUEST:
GET /v1/layers/196e0ce0c9fbb31da595b893dd39bc9fd4aa78a474bbdc21459a3ebe855b7768d9aaf4d82f249dc101a6638ff5177fe926cdebfa6c42d874dfa5029533da0e72?vulnerabilities HTTP/1.1
Host: localhost:6060

<---- HTTP RESPONSE:
HTTP/1.1 404 Not Found
Content-Length: 10
Content-Type: text/plain; charset=utf-8
Date: Wed, 11 Oct 2017 01:59:46 GMT
X-Content-Type-Options: nosniff

Not Found

Analyse image http://localhost:5000/v2/centos:latest failed: Analyze error 404: Not Found

Found 0 vulnerabilities

[hashmap]

@chino as you can see in logs POST /v1/layers gets 404 Not Found, this is response from Clair. Can you check what exact version of Clair you run? There was a similar issue https://github.com/optiopay/klar/issues/55 when the reason was:

ok, found the solution for my issue: dont use quay.io/coreos/clair-git as image...
using quay.io/coreos/clair worked fine :-)

[hashmap]

I checked the current state of Clair, it seems that old API support was dropped in master branch, so far use a stable 2.x version of Clair

[chino]

I figured it was something like that :]

Now I seem to hit another error.

This is with running quay.io/coreos/clair:latest (7b87f5ea984c).

I would have to reference the clair api to debug but if you know what's going on off hand.

[root@localhost vagrant]#     KLAR_TRACE=true /vagrant/klar.sh localhost:5000/centos
----> HTTP REQUEST:
GET /v2/centos/manifests/latest HTTP/1.1
Host: localhost:5000
Accept: application/vnd.docker.distribution.manifest.v2+json

<---- HTTP RESPONSE:
HTTP/1.1 200 OK
Content-Length: 529
Content-Type: application/vnd.docker.distribution.manifest.v2+json
Date: Wed, 11 Oct 2017 05:44:26 GMT
Docker-Content-Digest: sha256:822de5245dc5b659df56dd32795b08ae42db4cc901f3462fc509e91e97132dc0
Docker-Distribution-Api-Version: registry/2.0
Etag: "sha256:822de5245dc5b659df56dd32795b08ae42db4cc901f3462fc509e91e97132dc0"
X-Content-Type-Options: nosniff

{
   "schemaVersion": 2,
   "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
   "config": {
      "mediaType": "application/vnd.docker.container.image.v1+json",
      "size": 1863,
      "digest": "sha256:196e0ce0c9fbb31da595b893dd39bc9fd4aa78a474bbdc21459a3ebe855b7768"
   },
   "layers": [
      {
         "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
         "size": 73386947,
         "digest": "sha256:d9aaf4d82f249dc101a6638ff5177fe926cdebfa6c42d874dfa5029533da0e72"
      }
   ]
}
Analysing 1 layers
----> HTTP REQUEST:
POST /v1/layers HTTP/1.1
Host: localhost:6060
Content-Type: application/json

{"Layer":{"Name":"196e0ce0c9fbb31da595b893dd39bc9fd4aa78a474bbdc21459a3ebe855b7768d9aaf4d82f249dc101a6638ff5177fe926cdebfa6c42d874dfa5029533da0e72","Path":"http://localhost:5000/v2/centos/blobs/sha256:d9aaf4d82f249dc101a6638ff5177fe926cdebfa6c42d874dfa5029533da0e72","ParentName":"","Format":"Docker","Features":null,"Headers":{"Authorization":""}}}
<---- HTTP RESPONSE:
HTTP/1.1 400 Bad Request
Content-Type: application/json;charset=utf-8
Date: Wed, 11 Oct 2017 05:44:27 GMT
Server: clair

{"Error":{"Message":"could not find layer"}}

Push layer 0 failed: Push error 400: {"Error":{"Message":"could not find layer"}}

----> HTTP REQUEST:
GET /v1/layers/196e0ce0c9fbb31da595b893dd39bc9fd4aa78a474bbdc21459a3ebe855b7768d9aaf4d82f249dc101a6638ff5177fe926cdebfa6c42d874dfa5029533da0e72?vulnerabilities HTTP/1.1
Host: localhost:6060

<---- HTTP RESPONSE:
HTTP/1.1 404 Not Found
Content-Type: application/json;charset=utf-8
Date: Wed, 11 Oct 2017 05:44:27 GMT
Server: clair

{"Error":{"Message":"the resource cannot be found"}}

Analyse image http://localhost:5000/v2/centos:latest failed: Analyze error 404: {"Error":{"Message":"the resource cannot be found"}}

Found 0 vulnerabilities

[root@localhost vagrant]#     skopeo inspect --tls-verify=false docker://localhost:5000/centos
{
    "Name": "localhost:5000/centos",
    "Digest": "sha256:822de5245dc5b659df56dd32795b08ae42db4cc901f3462fc509e91e97132dc0",
    "RepoTags": [
        "latest"
    ],
    "Created": "2017-09-14T15:13:25.699933088Z",
    "DockerVersion": "17.06.2-ce",
    "Labels": {
        "build-date": "20170911",
        "license": "GPLv2",
        "name": "CentOS Base Image",
        "vendor": "CentOS"
    },
    "Architecture": "amd64",
    "Os": "linux",
    "Layers": [
        "sha256:d9aaf4d82f249dc101a6638ff5177fe926cdebfa6c42d874dfa5029533da0e72"
    ]
}

[hashmap]

Please make sure that your registry is reachable as localhost:5000 from Clair container, I assume it's not localhost of your docker host, but clair container itslef.

[chino]

I had used --net=host for everything to keep it simple but I'll check tomorrow to make sure.

[chino]

[root@localhost vagrant]# docker exec -it clair /bin/sh
/go/src/github.com/coreos/clair # nc -vz localhost 5000
localhost (127.0.0.1:5000) open

I'm running registry:2 as described here: https://docs.docker.com/registry/#tldr

Perhaps it's something now between clair<>registry versions?

[hashmap]

Does curl http://localhost:5000/v2/centos/blobs/sha256:d9aaf4d82f249dc101a6638ff5177fe926cdebfa6c42d874dfa5029533da0e72 work?

[hashmap]

@chino is this issue still valid?

[chino]

Sorry, haven't tried running it again.

[ggulati2]

@hashmap I get below 404 error for all layers while scanning an ECR Repo and I have built both clair and klar using GO (not using docker images)

Push layer failed: Can't even read an error message: invalid character 'N' looking for beginning of value ----> HTTP REQUEST: POST /v1/layers HTTP/1.1 Host: localhost:6060 Content-Type: application/json

[agnihotrigaurav]

Similar issue i am getting if i am running clair locally not in docker container. But if i run clair in docker, it works fine. Is building clair binary from source code not stable?

[hashmap]

@ggulati2 please try the latest version of Klar

[hashmap]

@agnihotrigaurav sorry, did you ask about clair or klar?

[hashmap]

Closing because of inactivity, I hope the latest release fixed it. Let me know if the issue needs to be reopened.