Unable to push image to clair: 400 Error

[adityacs]

Running clair as docker container. From the logs I see that ubuntu, debian update fails.

Is this the reason for clair not analyzing correctly? Kindly help

Thanks

[adityacs]

clair logs

{"Event":"could not download Debian's update","Level":"error","Location":"debian.go:68","Time":"2017-05-29 11:48:17.849983","error":"Get https://security-tracker.debian.org/tracker/data/json: dial tcp 128.31.0.67:443: i/o timeout"}
{"Event":"an error occured when fetching update","Level":"error","Location":"updater.go:220","Time":"2017-05-29 11:48:17.850068","error":"could not download requested resource","updater name":"debian"}
{"Event":"could not branch Ubuntu repository","Level":"error","Location":"ubuntu.go:177","Time":"2017-05-29 11:48:18.618409","error":"exit status 3","output":"bzr: ERROR: Unrecognised container format: '\u003chtml\u003e'\n"}
{"Event":"an error occured when fetching update","Level":"error","Location":"updater.go:220","Time":"2017-05-29 11:48:18.618487","error":"could not download requested resource","updater name":"ubuntu"}
{"Event":"adding metadata to vulnerabilities","Level":"info","Location":"updater.go:253","Time":"2017-05-29 11:48:18.618520"}
{"Event":"could not get NVD data feed hash","Level":"warning","Location":"nvd.go:137","Time":"2017-05-29 11:48:19.549053","data feed name":"2002","error":"invalid .meta file format"}
{"Event":"could not download layer","Level":"warning","Location":"driver.go:129","Time":"2017-05-29 11:48:44.205584","error":"Get https://registry.igloo.in/v2/devops/build/blobs/sha256:cfc728c1c5584d8e0ae69368fc9c34d54d72651355573ba42554c2469a0a6299: x509: certificate signed by unknown authority"}
{"Event":"failed to extract data from path","Level":"error","Location":"worker.go:122","Time":"2017-05-29 11:48:44.205643","error":"could not find layer","layer":"sha256:cfc728c1c5584d8e0ae69368fc9c34d54d72651355573ba42554c2469a0a6299","path":"https://registry.igloo.in/v2/devops/build/blobs/sha256:cfc728c1c5584d8e0ae69368fc9c34d54d72651355573ba42554c2469a0a6299"}

clairctl logs

2017-05-29 12:17:54.060592 D | config: Using config file: clairctl.yml
2017-05-29 12:17:54.060867 D | dockerdist: Downloading manifest for registry.igloo.in/devops/build:v1
2017-05-29 12:17:54.061053 D | dockerdist: Retrieving repository client
2017-05-29 12:17:54.193172 D | dockerdist: endpoint.TLSConfig.InsecureSkipVerify: true
2017-05-29 12:17:54.266045 D | dockerdist: manifest type: *schema2.DeserializedManifest
2017-05-29 12:17:54.266073 D | dockerdist: retrieved schema2 manifest, no verification
2017-05-29 12:17:54.266088 I | config: retrieving interface for local IP
2017-05-29 12:17:54.266094 D | config: no interface provided, looking for docker0
2017-05-29 12:17:54.266337 I | clair: Pushing Layer 1/10 [sha256:cfc72]
2017-05-29 12:17:54.292156 D | clair: Saving sha256:cfc728c1c5584d8e0ae69368fc9c34d54d72651355573ba42554c2469a0a6299[https://registry.igloo.in/v2]
2017-05-29 12:17:54.325614 I | clair: adding layer 1/10 [sha256:cfc72]: receiving http error: 400
client quit unexpectedly
2017-05-29 12:17:54.325640 C | cmd: pushing image "registry.igloo.in/devops/build:v1": receiving http error: 400

[jgsqware]

@jzelinskie this is strange:

{"Event":"could not download layer","Level":"warning","Location":"driver.go:129","Time":"2017-05-29 11:48:44.205584","error":"Get https://registry.igloo.in/v2/devops/build/blobs/sha256:cfc728c1c5584d8e0ae69368fc9c34d54d72651355573ba42554c2469a0a6299: x509: certificate signed by unknown authority"}

how can we add certificates in clair?

[jzelinskie]

@jgsqware it depends on how you're running it.

There's a field in the config called certFile you can use, but also Clair should defer to the OS for certificate validation. Adding the certificates to your OS usually involves adding your certs to /usr/local/share/ca-certificates/ and running update-ca-certificates.

[adityacs]

@jzelinskie I am running clair in container. Should I update the clair docker image with certs and then use it or should I mount certs form host to clair container? How does that work?

[adityacs]

@jgsqware @jzelinskie I am running clair with -insecure-tls. Now I am getting below error

clairctl logs

./clairctl --config clairctl.yml --log-level Debug pull registry.igloo.in/devops/build:v1
2017-05-29 18:44:33.321370 D | config: Using config file: clairctl.yml
2017-05-29 18:44:33.321522 D | dockerdist: Downloading manifest for registry.igloo.in/devops/build:v1
2017-05-29 18:44:33.321605 D | dockerdist: Retrieving repository client
2017-05-29 18:44:33.457176 D | dockerdist: endpoint.TLSConfig.InsecureSkipVerify: true
2017-05-29 18:44:33.530370 D | dockerdist: manifest type: *schema2.DeserializedManifest
2017-05-29 18:44:33.530388 D | dockerdist: retrieved schema2 manifest, no verification

Image: registry.igloo.in/devops/build:v1
 10 layers found
  ➜ sha256:cfc728c1c5584d8e0ae69368fc9c34d54d72651355573ba42554c2469a0a6299
  ➜ sha256:b57e2e3098701a05265ec1d030ccbc3e1a7f36f37869d688ab9c9f072010d217
  ➜ sha256:ccba75e2910d81609bde82b6ae60098aacc2d6810381987b41bfdd69b3ea7864
  ➜ sha256:7a112056ca40e378302cc627964c7270e9b4731a3d64a7b764e1a4b365db3323
  ➜ sha256:d4cf50edca36051c34d1b7ac1b1c54ea39d2eab71cb6d6d741c4329dddf292ac
  ➜ sha256:393675e9a7366ab349e06203422be2b6539ac687146fba2f25a0d72df630d4ec
  ➜ sha256:fa586482b9b86f72c20261e89b43e3230b6c7cff2662ab4d422d39076b59c5b1
  ➜ sha256:6df815a641c768d673ebea726604ccad1d0968550f8f3c9c751abdfaedbd9226
  ➜ sha256:c7a3f3efb3e28fe9cbc51def2db531ad08516fa7113eef06a4b8ad8378f9edca
  ➜ sha256:5f4d3b590ba8cae65a8a3c12d44cd80da96bcc573c71199d543089885edc8cb1

swarmd-10 clair-config # ./clairctl --config clairctl.yml --log-level Debug push registry.igloo.in/devops/build:v1
2017-05-29 18:44:36.983179 D | config: Using config file: clairctl.yml
2017-05-29 18:44:36.983347 D | dockerdist: Downloading manifest for registry.igloo.in/devops/build:v1
2017-05-29 18:44:36.983428 D | dockerdist: Retrieving repository client
2017-05-29 18:44:37.116644 D | dockerdist: endpoint.TLSConfig.InsecureSkipVerify: true
2017-05-29 18:44:37.187963 D | dockerdist: manifest type: *schema2.DeserializedManifest
2017-05-29 18:44:37.187984 D | dockerdist: retrieved schema2 manifest, no verification
2017-05-29 18:44:37.188128 I | config: retrieving interface for local IP
2017-05-29 18:44:37.188208 D | config: no interface provided, looking for docker0
2017-05-29 18:44:37.188579 I | clair: Pushing Layer 1/10 [sha256:cfc72]
2017-05-29 18:44:37.215745 D | clair: Saving sha256:cfc728c1c5584d8e0ae69368fc9c34d54d72651355573ba42554c2469a0a6299[https://registry.igloo.in/v2]
2017-05-29 18:44:37.252764 I | clair: adding layer 1/10 [sha256:cfc72]: receiving http error: 400
client quit unexpectedly
2017-05-29 18:44:37.252787 C | cmd: pushing image "registry.igloo.in/devops/build:v1": receiving http error: 400
swarmd-10 clair-config # docker pull https://registry.igloo.in/v2/devops/build/blobs/sha256:cfc728c1c5584d8e0ae69368fc9c34d54d72651355573ba42554c2469a0a6299
invalid reference format

clair logs

{"Event":"processing layer","Level":"debug","Location":"worker.go:73","Time":"2017-05-29 18:43:51.678695","engine version":3,"format":"Docker","layer":"sha256:cfc728c1c5584d8e0ae69368fc9c34d54d72651355573ba42554c2469a0a6299","parent layer":"","path":"https://registry.igloo.in/v2/devops/build/blobs/sha256:cfc728c1c5584d8e0ae69368fc9c34d54d72651355573ba42554c2469a0a6299"}
{"Event":"could not download layer: expected 2XX","Level":"warning","Location":"driver.go:135","Time":"2017-05-29 18:43:51.689734","status code":401}
{"Event":"failed to extract data from path","Level":"error","Location":"worker.go:122","Time":"2017-05-29 18:43:51.689790","error":"could not find layer","layer":"sha256:cfc728c1c5584d8e0ae69368fc9c34d54d72651355573ba42554c2469a0a6299","path":"https://registry.igloo.in/v2/devops/build/blobs/sha256:cfc728c1c5584d8e0ae69368fc9c34d54d72651355573ba42554c2469a0a6299"}
{"Event":"Handled HTTP request","Level":"info","Location":"router.go:57","Time":"2017-05-29 18:43:51.690669","elapsed time":12060987,"method":"POST","remote addr":"172.18.2.190:53210","request uri":"/v1/layers","status":"400"}

[adityacs]

@jgsqware @jzelinskie Any suggestions on above?

[jgsqware]

1. Can your Clair instance have access to your registry?
2. Have you logged in docker before running clairctl
3. Are you using the last built version of Clairctl?

Le mer. 31 mai 2017 05:54, Aditya C S notifications@github.com a écrit :

@jgsqware https://github.com/jgsqware @jzelinskie https://github.com/jzelinskie Any suggestions on above?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/coreos/clair/issues/398#issuecomment-305075323, or mute the thread https://github.com/notifications/unsubscribe-auth/ADJrqyNhUIR9XFsnLfN439wE8egVuz_oks5r_OSDgaJpZM4Ng9pJ .

[adityacs]

1. Can your Clair instance have access to your registry? Yes, clair container is able to access registry 2. Have you logged in docker before running clairctl YES 3. Are you using the last built version of Clairctl? YES

[jzelinskie]

The Clair logs indicate that your registry is returning a 401 when Clair attempts to access the layer. If this image is private, you might have include more values in the headers object in your POST request. I'm not sure if/how clairctl surfaces this configuration.

[jgsqware]

Last version of Clairctl use the authorization header before sending the request to clair

Le ven. 2 juin 2017 18:20, Jimmy Zelinskie notifications@github.com a écrit :

The Clair logs indicate that your registry is returning a 401 when Clair attempts to access the layer. If this image is private, you might have include more values in the headers object https://github.com/coreos/clair/blob/master/Documentation/api_v1.md#example-request in your POST request. I'm not sure if/how clairctl surfaces this configuration.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/coreos/clair/issues/398#issuecomment-305836762, or mute the thread https://github.com/notifications/unsubscribe-auth/ADJrq6vnjZso18s1lHbd10Jjzb8t66Joks5sADZAgaJpZM4Ng9pJ .

[patty08]

I've got the same problem, i'm blocking.

1. postgres: i should create data base?
2. clair-database/source: how about the good config file and the right syntax.
3. clairctl: how about the config file?

My logs:

{"Event":"pgsql: could not open database: dial tcp 172.25.0.2:5432: getsockopt: connection refused","Level":"fatal","Location":"main.go:96","Time":"2017-07-13 14:30:53.726673"}
clair_clair | {"Event":"pgsql: could not open database: dial tcp 172.25.0.2:5432: getsockopt: connection refused","Level":"fatal","Location":"main.go:96","Time":"2017-07-13 14:30:54.776597"}
clair_clair | {"Event":"running database migrations","Level":"info","Location":"pgsql.go:216","Time":"2017-07-13 14:30:55.909787"}
clair_clair | {"Event":"database migration ran successfully","Level":"info","Location":"pgsql.go:223","Time":"2017-07-13 14:30:56.631225"}
clair_clair | {"Event":"starting main API","Level":"info","Location":"api.go:52","Time":"2017-07-13 14:30:56.631609","port":6060}
clair_clair | {"Event":"starting health API","Level":"info","Location":"api.go:85","Time":"2017-07-13 14:30:56.631658","port":6061}
clair_clair | {"Event":"updater service started","Level":"info","Location":"updater.go:80","Time":"2017-07-13 14:30:56.632119","lock identifier":"4013724c-3ce7-4d98-a354-732534e8143f"}
clair_clair | {"Event":"notifier service is disabled","Level":"info","Location":"notifier.go:77","Time":"2017-07-13 14:30:56.632329"}
clair_clair | {"Event":"updating vulnerabilities","Level":"info","Location":"updater.go:167","Time":"2017-07-13 14:30:56.640089"}
clair_clair | {"Event":"fetching vulnerability updates","Level":"info","Location":"updater.go:213","Time":"2017-07-13 14:30:56.640175"}
clair_clair | {"Event":"Start fetching vulnerabilities","Level":"info","Location":"alpine.go:52","Time":"2017-07-13 14:30:56.640321","package":"Alpine"}
clair_clair | {"Event":"Start fetching vulnerabilities","Level":"info","Location":"oracle.go:119","Time":"2017-07-13 14:30:56.640354","package":"Oracle Linux"}
clair_clair | {"Event":"Start fetching vulnerabilities","Level":"info","Location":"ubuntu.go:88","Time":"2017-07-13 14:30:56.640404","package":"Ubuntu"}
clair_clair | {"Event":"Start fetching vulnerabilities","Level":"info","Location":"debian.go:63","Time":"2017-07-13 14:30:56.640576","package":"Debian"}
clair_clair | {"Event":"Start fetching vulnerabilities","Level":"info","Location":"rhel.go:92","Time":"2017-07-13 14:30:56.641434","package":"RHEL"}
clair_clair | {"Event":"finished fetching","Level":"info","Location":"updater.go:227","Time":"2017-07-13 14:31:00.187675","updater name":"alpine"}
clair_clair | {"Event":"Debian buster is not mapped to any version number (eg. Jessie-\u003e8). Please update me.","Level":"warning","Location":"debian.go:128","Time":"2017-07-13 14:31:15.682543"}
clair_clair | {"Event":"finished fetching","Level":"info","Location":"updater.go:227","Time":"2017-07-13 14:31:15.682592","updater name":"debian"}
clair_clair | {"Event":"finished fetching","Level":"info","Location":"updater.go:227","Time":"2017-07-13 14:34:15.931967","updater name":"ubuntu"}
clair_clair | {"Event":"could not download layer: expected 2XX","Level":"warning","Location":"driver.go:135","Time":"2017-07-13 14:40:01.923938","status code":404}
clair_clair | {"Event":"failed to extract data from path","Level":"error","Location":"worker.go:122","Time":"2017-07-13 14:40:01.925094","error":"could not find layer","layer":"5c99d3991584b5b30941bb2ac10f2bf188e711217be1c4fe58ed38d9d512cf8b","path":"http://172.17.0.1:42731/local/docker.io/cloudunit/jenkins/blobs/5c99d3991584b5b30941bb2ac10f2bf188e711217be1c4fe58ed38d9d512cf8b/layer.tar"}
clair_clair | {"Event":"Handled HTTP request","Level":"info","Location":"router.go:57","Time":"2017-07-13 14:40:01.927947","elapsed time":43480956,"method":"POST","remote addr":"172.25.0.1:32932","request uri":"/v1/layers","status":"400"}
clair_clair | {"Event":"finished fetching","Level":"info","Location":"updater.go:227","Time":"2017-07-13 14:48:28.911007","updater name":"rhel"}
clair_clair | {"Event":"finished fetching","Level":"info","Location":"updater.go:227","Time":"2017-07-13 14:51:41.819086","updater name":"oracle"}
clair_clair | {"Event":"adding metadata to vulnerabilities","Level":"info","Location":"updater.go:253","Time":"2017-07-13 14:51:41.831352"}

clairctl logs: Clairctl version 1.2.8

2017-07-13 16:43:22.404746 D | config: Using config file: config.yml
2017-07-13 16:43:22.404910 I | config: retrieving interface for local IP
2017-07-13 16:43:22.404916 D | config: no interface provided, looking for docker0
2017-07-13 16:43:22.405330 D | server: Update local server port from "0" to "35495"
2017-07-13 16:43:22.405339 I | server: Starting Server on 172.17.0.1:35495
2017-07-13 16:43:22.410566 D | dockercli: docker image to save: cloudunit/jenkins:latest
2017-07-13 16:43:22.410598 D | dockercli: saving in: /tmp/clairctl/cloudunit/jenkins/blobs
2017-07-13 16:44:06.134573 I | config: retrieving interface for local IP
2017-07-13 16:44:06.134595 D | config: no interface provided, looking for docker0
2017-07-13 16:44:06.135263 I | clair: using http://172.17.0.1:35495/local as local url
2017-07-13 16:44:06.135274 I | clair: Pushing Layer 1/32 [5c99d3991584]
2017-07-13 16:44:06.135329 D | clair: Saving 5c99d3991584b5b30941bb2ac10f2bf188e711217be1c4fe58ed38d9d512cf8b[https://registry-1.docker.io/v2]
2017-07-13 16:44:09.466339 I | clair: adding layer 1/32 [5c99d3991584]: pushing layer to clair: Post http://clair_clair:6060/v1/layers: dial tcp: lookup clair_clair: no such host
client quit unexpectedly
2017-07-13 16:44:09.466411 C | cmd: pushing image "cloudunit/jenkins:latest": pushing layer to clair: Post http://clair_clair:6060/v1/layers: dial tcp: lookup clair_clair: no such host

NEED HELP PLEASE. Thanks a lot.

[FrankJLhota]

I have had the same issue working through a clairctl container. I logged into the docker.io/library registry, then started up the clairctl containers, then ran clairctl --log-level debug analyze ubuntu:16.04 -l in the clairctl container; it failed with this output:

2017-07-13 20:09:37.586000 D | config: Using config file: /home/clairctl/clairctl.yml
2017-07-13 20:09:37.586239 D | dockercli: docker image to save: ubuntu:16.04
2017-07-13 20:09:37.586247 D | dockercli: saving in: /tmp/ubuntu/blobs
2017-07-13 20:09:38.385015 I | config: retrieving interface for local IP
2017-07-13 20:09:38.385028 D | config: no interface provided, looking for docker0
2017-07-13 20:09:38.385075 D | config: docker0 not found, looking for first connected broadcast interface
2017-07-13 20:09:38.385363 I | server: Starting Server on 172.21.0.4:44480
2017-07-13 20:09:38.390334 I | config: retrieving interface for local IP
2017-07-13 20:09:38.390342 D | config: no interface provided, looking for docker0
2017-07-13 20:09:38.390389 D | config: docker0 not found, looking for first connected broadcast interface
2017-07-13 20:09:38.390475 I | clair: using http://172.21.0.4:44480/local as local url
2017-07-13 20:09:38.390495 I | clair: Pushing Layer 1/5 [0cfd9cb2ea20]
2017-07-13 20:09:38.390530 D | clair: Saving 0cfd9cb2ea20b891dad7b2c5e46b18686848e692d49f9cad3261f3428bbfbfc9[https://registry-1.docker.io/v2]
2017-07-13 20:09:38.400090 I | clair: adding layer 1/5 [0cfd9cb2ea20]: receiving http error: 400
client quit unexpectedly
2017-07-13 20:09:38.400112 C | cmd: pushing image "ubuntu:16.04": receiving http error: 400

The clair logs are

Attaching to clairctl_clair_1
clair_1     | {"Event":"pgsql: could not open database: dial tcp 172.21.0.3:5432: getsockopt: connection refused","Level":"fatal","Location":"main.go:96","Time":"2017-07-13 20:09:15.895903"}
clair_1     | {"Event":"pgsql: could not open database: dial tcp 172.21.0.3:5432: getsockopt: connection refused","Level":"fatal","Location":"main.go:96","Time":"2017-07-13 20:09:16.530895"}
clair_1     | {"Event":"pgsql: could not open database: dial tcp 172.21.0.3:5432: getsockopt: connection refused","Level":"fatal","Location":"main.go:96","Time":"2017-07-13 20:09:17.351787"}
clair_1     | {"Event":"pgsql: could not open database: dial tcp 172.21.0.3:5432: getsockopt: connection refused","Level":"fatal","Location":"main.go:96","Time":"2017-07-13 20:09:18.267208"}
clair_1     | {"Event":"running database migrations","Level":"info","Location":"pgsql.go:216","Time":"2017-07-13 20:09:19.499089"}
clair_1     | {"Event":"database migration ran successfully","Level":"info","Location":"pgsql.go:223","Time":"2017-07-13 20:09:19.795034"}
clair_1     | {"Event":"starting main API","Level":"info","Location":"api.go:52","Time":"2017-07-13 20:09:19.795254","port":6060}
clair_1     | {"Event":"starting health API","Level":"info","Location":"api.go:85","Time":"2017-07-13 20:09:19.795342","port":6061}
clair_1     | {"Event":"notifier service is disabled","Level":"info","Location":"notifier.go:77","Time":"2017-07-13 20:09:19.795377"}
clair_1     | {"Event":"updater service started","Level":"info","Location":"updater.go:80","Time":"2017-07-13 20:09:19.795761","lock identifier":"af1bd852-e96d-4f2c-b23a-4d8018b4deb2"}
clair_1     | {"Event":"attempting to obtain update lock","Level":"debug","Location":"updater.go:99","Time":"2017-07-13 20:09:19.798409"}
clair_1     | {"Event":"updating vulnerabilities","Level":"info","Location":"updater.go:167","Time":"2017-07-13 20:09:19.803639"}
clair_1     | {"Event":"fetching vulnerability updates","Level":"info","Location":"updater.go:213","Time":"2017-07-13 20:09:19.803820"}
clair_1     | {"Event":"Start fetching vulnerabilities","Level":"info","Location":"oracle.go:119","Time":"2017-07-13 20:09:19.804016","package":"Oracle Linux"}
clair_1     | {"Event":"Start fetching vulnerabilities","Level":"info","Location":"rhel.go:92","Time":"2017-07-13 20:09:19.804005","package":"RHEL"}
clair_1     | {"Event":"Start fetching vulnerabilities","Level":"info","Location":"ubuntu.go:88","Time":"2017-07-13 20:09:19.804253","package":"Ubuntu"}
clair_1     | {"Event":"Start fetching vulnerabilities","Level":"info","Location":"alpine.go:52","Time":"2017-07-13 20:09:19.804480","package":"Alpine"}
clair_1     | {"Event":"Start fetching vulnerabilities","Level":"info","Location":"debian.go:63","Time":"2017-07-13 20:09:19.805288","package":"Debian"}
clair_1     | {"Event":"Debian buster is not mapped to any version number (eg. Jessie-\u003e8). Please update me.","Level":"warning","Location":"debian.go:128","Time":"2017-07-13 20:09:25.280003"}
clair_1     | {"Event":"finished fetching","Level":"info","Location":"updater.go:227","Time":"2017-07-13 20:09:25.280049","updater name":"debian"}
clair_1     | {"Event":"finished fetching","Level":"info","Location":"updater.go:227","Time":"2017-07-13 20:09:30.855547","updater name":"alpine"}
clair_1     | {"Event":"processing layer","Level":"debug","Location":"worker.go:73","Time":"2017-07-13 20:09:38.397822","engine version":3,"format":"Docker","layer":"0cfd9cb2ea20b891dad7b2c5e46b18686848e692d49f9cad3261f3428bbfbfc9","parent layer":"","path":"http://172.21.0.4:44480/local/docker.io/library/ubuntu/blobs/0cfd9cb2ea20b891dad7b2c5e46b18686848e692d49f9cad3261f3428bbfbfc9/layer.tar"}
clair_1     | {"Event":"could not download layer: expected 2XX","Level":"warning","Location":"driver.go:135","Time":"2017-07-13 20:09:38.399628","status code":404}
clair_1     | {"Event":"failed to extract data from path","Level":"error","Location":"worker.go:122","Time":"2017-07-13 20:09:38.399675","error":"could not find layer","layer":"0cfd9cb2ea20b891dad7b2c5e46b18686848e692d49f9cad3261f3428bbfbfc9","path":"http://172.21.0.4:44480/local/docker.io/library/ubuntu/blobs/0cfd9cb2ea20b891dad7b2c5e46b18686848e692d49f9cad3261f3428bbfbfc9/layer.tar"}
clair_1     | {"Event":"Handled HTTP request","Level":"info","Location":"router.go:57","Time":"2017-07-13 20:09:38.399952","elapsed time":2203915,"method":"POST","remote addr":"172.21.0.4:53202","request uri":"/v1/layers","status":"400"}

Please advise.

[shashankkoppar]

@jgsqware solved the issue!! Could you just add proxyFromenviornment in here https://github.com/coreos/clair/blob/master/ext/imagefmt/driver.go#L123 `

    tr := &http.Transport
            Proxy:           http.ProxyFromEnvironment,
        TLSClientConfig: &tls.Config{InsecureSkipVerify: insecureTLS},
    }

`

[the-nw1-group]

I have the same issue - clair/clairctl/postgres all running in containers. compose file:

version: '2.1'

services:
  postgres:
    image: postgres:9.6
    restart: unless-stopped
    volumes:
      - /opt/clair/docker-compose-data/postgres-data/:/var/lib/postgresql/data:rw
    environment:
      - POSTGRES_PASSWORD=ChangeMe
      - POSTGRES_USER=clair
      - POSTGRES_DB=clair

  clair:
    user: root
    image: quay.io/coreos/clair:v2.0.0
    restart: unless-stopped
    volumes:
      - /opt/clair/docker-compose-data/clair-config/:/config/:ro
      - /opt/clair/docker-compose-data/clair-tmp/:/tmp/:rw
    depends_on:
      postgres:
        condition: service_started
    command: [--log-level=debug, --config, /config/config.yml,  --insecure-tls]

  clairctl:
    user: root
    image: jgsqware/clairctl:latest
    restart: unless-stopped
    environment:
      - DOCKER_API_VERSION=1.24
    volumes:
      - /opt/clair/docker-compose-data/clairctl-reports/:/reports/:rw
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - /etc/docker/certs.d/:/etc/docker/certs.d/:ro
    depends_on:
      clair:
        condition: service_started

Running the following:

[] docker exec clair_clairctl_1 docker login -u <uid>-p <pwd> <private-registry-ip>:443
Login Succeeded
[] docker exec clair_clairctl_1 clairctl pull <private-registry-ip>:443/<image>:<tag>
Image: <private-registry-ip>:443/<image>:<tag>
 3 layers found
  ➜ sha256:45a2e645736c4c66ef34acce2407ded21f7a9b231199d3b92d6c9776df264729
  ➜ sha256:798039b8222d87075e2136f497db3595ed8f5049f8ac3768a1bb188d95efecb4
  ➜ sha256:1fe39dff86fdf7c981fe94535e3c541f2c15ad0234e35e4961a82637abbe9537
[] docker exec clair_clairctl_1 clairctl push <private-registry-ip>:443/<image>:<tag>
client quit unexpectedly
2017-08-03 12:23:51.558964 C | cmd: pushing image "<private-registry-ip>:443/<image>:<tag>": receiving http error: 400

clair logs:

{"Event":"could not download layer: expected 2XX","Level":"warning","Location":"driver.go:135","Time":"2017-08-03 12:23:51.555562","status code":401}
{"Event":"failed to extract data from path","Level":"error","Location":"worker.go:122","Time":"2017-08-03 12:23:51.555686","error":"could not find layer","layer":"sha256:45a2e645736c4c66ef34acce2407ded21f7a9b231199d3b92d6c9776df264729","path":"https://<private-registry-ip>:443/v2/<image>/blobs/sha256:45a2e645736c4c66ef34acce2407ded21f7a9b231199d3b92d6c9776df264729"}
{"Event":"Handled HTTP request","Level":"info","Location":"router.go:57","Time":"2017-08-03 12:23:51.558505","elapsed time":131859526,"method":"POST","remote addr":"172.19.0.4:51798","request uri":"/v1/layers","status":"400"}

After installing curl into the clair container, running:

[] docker exec clair_clair_1 curl -k https://<private-registry-ip>:443/<image>/blobs/sha256:45a2e645736c4c66ef34acce2407ded21f7a9b231199d3b92d6c9776df264729
{"errors":[{"code":"UNAUTHORIZED","message":"authentication required","detail":[{"Type":"repository","Class":"","Name":"<image>","Action":"pull"}]}]}

which at least shows my clair container can see my private repo.

[shashankkoppar]

@the-nw1-group i had the same issue too, have you updated your ca certificates for registry in clair contajner?

[the-nw1-group]

I'm happy for the time being enabling the --insecure-tls flag on Clair, once it's all working, I'll go back and add the CA certificates, and configure Clair correctly with that.

[shashankkoppar]

I actually tried that, still that error wouldn't go away, anyway try clairctl -log-level debug analyze image name and also try clairctl -log-level debug analyze image name --local as well

[shashankkoppar]

*--log-level debug

[the-nw1-group]

Pretty much the same:

[] docker exec clair_clairctl_1 clairctl --log-level debug analyze <private-registry-ip>:443/<image>:<tag>
2017-08-03 14:25:11.530695 D | config: Using config file: /home/clairctl/clairctl.yml
2017-08-03 14:25:11.531115 D | dockerdist: Downloading manifest for <private-registry-ip>:443/<image>:<tag>
2017-08-03 14:25:11.531333 D | dockerdist: Retrieving repository client
2017-08-03 14:25:11.825310 D | dockerdist: endpoint.TLSConfig.InsecureSkipVerify: true
2017-08-03 14:25:12.034639 D | dockerdist: manifest type: *schema2.DeserializedManifest
2017-08-03 14:25:12.034690 D | dockerdist: retrieved schema2 manifest, no verification
2017-08-03 14:25:12.034774 I | config: retrieving interface for local IP
2017-08-03 14:25:12.034793 D | config: no interface provided, looking for docker0
2017-08-03 14:25:12.035674 D | config: docker0 not found, looking for first connected broadcast interface
2017-08-03 14:25:12.035905 I | clair: Pushing Layer 1/3 [sha256:45a2e]
2017-08-03 14:25:12.093401 D | clair: Saving sha256:45a2e645736c4c66ef34acce2407ded21f7a9b231199d3b92d6c9776df264729[https://10.102.71.97:443/v2]
2017-08-03 14:25:12.147381 D | clair: auth.insecureSkipVerify: true
2017-08-03 14:25:12.147449 D | clair: request.URL.String(): https://<private-registry-ip>:443/v2/<image>/blobs/sha256:45a2e645736c4c66ef34acce2407ded21f7a9b231199d3b92d6c9776df264729
2017-08-03 14:25:12.212154 I | clair: pull from clair is unauthorized
2017-08-03 14:25:12.290188 I | clair: adding layer 1/3 [sha256:45a2e]: receiving http error: 400
client quit unexpectedly
2017-08-03 14:25:12.290238 C | cmd: pushing image "<private-registry-ip>:443/<image>:<tag>": receiving http error: 400

and from the clair logs:

{"Event":"could not download layer: expected 2XX","Level":"warning","Location":"driver.go:135","Time":"2017-08-03 14:25:12.287681","status code":401}
{"Event":"failed to extract data from path","Level":"error","Location":"worker.go:122","Time":"2017-08-03 14:25:12.287794","error":"could not find layer","layer":"sha256:45a2e645736c4c66ef34acce2407ded21f7a9b231199d3b92d6c9776df264729","path":"https://<private-registry-ip>:443/v2/<image>/blobs/sha256:45a2e645736c4c66ef34acce2407ded21f7a9b231199d3b92d6c9776df264729"}
{"Event":"Handled HTTP request","Level":"info","Location":"router.go:57","Time":"2017-08-03 14:25:12.289647","elapsed time":73881157,"method":"POST","remote addr":"172.19.0.4:51952","request uri":"/v1/layers","status":"400"}

And with --local, Clair's getting a 404 instead of a 401

[] docker exec clair_clairctl_1 clairctl --log-level debug analyze <private-registry-ip>:443/<image>:<tag> --local
2017-08-03 14:29:17.397432 D | config: Using config file: /home/clairctl/clairctl.yml
2017-08-03 14:29:17.398497 D | dockercli: docker image to save: <private-registry-ip>:443/<image>:<tag>
2017-08-03 14:29:17.398540 D | dockercli: saving in: /tmp/<private-registry-ip>/blobs
2017-08-03 14:29:25.006568 I | config: retrieving interface for local IP
2017-08-03 14:29:25.006650 D | config: no interface provided, looking for docker0
2017-08-03 14:29:25.007563 D | config: docker0 not found, looking for first connected broadcast interface
2017-08-03 14:29:25.008335 I | server: Starting Server on 172.19.0.4:44480
2017-08-03 14:29:25.013110 I | config: retrieving interface for local IP
2017-08-03 14:29:25.013147 D | config: no interface provided, looking for docker0
2017-08-03 14:29:25.013266 D | config: docker0 not found, looking for first connected broadcast interface
2017-08-03 14:29:25.013514 I | clair: using http://172.19.0.4:44480/local as local url
2017-08-03 14:29:25.013557 I | clair: Pushing Layer 1/3 [0b1fad843228]
2017-08-03 14:29:25.072451 D | clair: Saving 0b1fad8432280fea43a4749497065ec2198ece532236c298ab72029fc907aab6[https://<private-registry-ip>:443/v2]
2017-08-03 14:29:25.137805 I | clair: adding layer 1/3 [0b1fad843228]: receiving http error: 400
client quit unexpectedly
2017-08-03 14:29:25.137855 C | cmd: pushing image "<private-registry-ip>:443/<image>:<tag>": receiving http error: 400

and Clair logs:

{"Event":"could not download layer: expected 2XX","Level":"warning","Location":"driver.go:135","Time":"2017-08-03 14:29:25.136372","status code":404}
{"Event":"failed to extract data from path","Level":"error","Location":"worker.go:122","Time":"2017-08-03 14:29:25.136489","error":"could not find layer","layer":"0b1fad8432280fea43a4749497065ec2198ece532236c298ab72029fc907aab6","path":"http://172.19.0.4:44480/local/<private-registry-ip>:443/<image>/blobs/0b1fad8432280fea43a4749497065ec2198ece532236c298ab72029fc907aab6/layer.tar"}
{"Event":"Handled HTTP request","Level":"info","Location":"router.go:57","Time":"2017-08-03 14:29:25.137394","elapsed time":6950966,"method":"POST","remote addr":"172.19.0.4:51956","request uri":"/v1/layers","status":"400"}

[shashankkoppar]

Can you curl http://172.19.0.4:44480/local/:443//blobs/0b1fad8432280fea43a4749497065ec2198ece532236c298ab72029fc907aab6/layer.tar in container?

[shashankkoppar]

if u can , then you have to update CA certificate and check or else you might be missing some proxy to reach your blob storage

[the-nw1-group]

Yes I just tried that, and I get connection refused, rather than not found.

[shashankkoppar]

Sorry that looks local one, can u execute without --local and tail clair logs you might see {"Event":"failed to extract data from path","Level":"error","Location":"worker.go:122","Time":"2017-06-04 23:12:25.109273","error":"could not find layer","layer":"sha256:983bfa07a342e316f08afd066894505088de985d46a9af743920aa9cafd17e7a","path":"http://localhost:5000/v2/hello-world/blobs/sha256:983bfa07a342e316f08afd066894505088de985d46a9af743920aa9cafd17e7a"} some error like above

[shashankkoppar]

try downloading the layer manually in container from your blob storage

[the-nw1-group]

I've tried that, and get:

[] docker exec clair_clair_1 curl -k https://<private-registry-ip>:443/<image>/blobs/sha256:45a2e645736c4c66ef34acce2407ded21f7a9b231199d3b92d6c9776df264729
{"errors":[{"code":"UNAUTHORIZED","message":"authentication required","detail":[{"Type":"repository","Class":"","Name":"<image>","Action":"pull"}]}

As the private registry requires a login.

[shashankkoppar]

actually clairctl uses ~/.docker/config.json for credentials have you executed docker login command before clairctl analyze ? or could you just add ~/.docker/config.json ?

[the-nw1-group]

Yes I've run docker login, and clairctl does get the image from my private registry, I think the issue is when clair tries to download the layer.

Although there's another issue when using --local, it might be related to https://github.com/jgsqware/clairctl/issues/74.

Either way I'm a bit stuck, as I can't change the port our registry runs on, and I can't remove logins.

[shashankkoppar]

Have you added it as insecure registry in clairctl.yml file?

[shashankkoppar]

clair: port: 6060 healthPort: 6061 uri: http://localhost report: path: ./reports format: html docker: insecure-registries:

• "localhost:5000"

if not try something like this once?

[the-nw1-group]

Same I'm afraid with the --local, but fails to connect to my private registry otherwise, as it uses https, and clairctl tries to use http. I'm not sure if it uses the scheme when I added https://:

clair:
  port: 6060
  healthPort: 6061
  uri: http://clair
  priority: Low
  report:
    path: /reports
    format: html
clairctl:
  port: 44480
  tempfolder: /tmp
docker:
  insecure-registries:
    - "https://<private-registry-ip>:443"

[jault3]

I am having very similar issues as above and nothing so far has fixed my 404 errors. I cloned clair locally and am running it on my localhost - not in a container. Running this on OSX 10.12.6.

$ go install github.com/coreos/clair
$ clair -log-level debug -config config.yaml
{"Event":"pagination key is empty, generating...","Level":"warning","Location":"config.go:110","Time":"2017-11-02 15:46:01.923524"}
{"Detectors":"lsb-release,os-release,redhat-release,alpine-release,apt-sources","Event":"Clair registered components","Level":"info","Listers":"dpkg,rpm,apk","Location":"main.go:101","Time":"2017-11-02 15:46:01.924189","Updaters":"ubuntu,alpine,debian,oracle,rhel"}
{"Event":"running database migrations","Level":"info","Location":"pgsql.go:270","Time":"2017-11-02 15:46:01.938298"}
{"Event":"database migration ran successfully","Level":"info","Location":"pgsql.go:277","Time":"2017-11-02 15:46:01.942651"}
{"Event":"starting grpc server","Level":"info","Location":"server.go:155","Time":"2017-11-02 15:46:01.942884","addr":"[::]:6060"}
{"Event":"notifier service is disabled","Level":"info","Location":"notifier.go:76","Time":"2017-11-02 15:46:01.943046"}
{"Event":"starting health API","Level":"info","Location":"api.go:62","Time":"2017-11-02 15:46:01.942994","addr":"0.0.0.0:6061"}
{"Event":"updater service started","Level":"info","Location":"updater.go:91","Time":"2017-11-02 15:46:01.943033","lock identifier":"0074e3bb-a5d6-4ff7-a358-1c2e3ff4956a"}
{"Event":"grpc server is configured without client certificate authentication","Level":"warning","Location":"server.go:199","Time":"2017-11-02 15:46:01.943339"}
{"Event":"updater sleeping","Level":"debug","Location":"updater.go:177","Time":"2017-11-02 15:46:01.945850","scheduled time":"2017-11-02T22:30:49Z"}
{"Event":"Handled HTTP request","Level":"info","Location":"server.go:105","Time":"2017-11-02 15:46:16.102597","elapsed time (ms)":0.027573,"method":"POST","remote addr":"[::1]:62343","request uri":"/v1/layers","status":"404"}

Clair config is identical to the config.example.yaml included in the repository.

I am running a postgres docker container with this command docker run -d -e POSTGRES_PASSWORD="" -p 5432:5432 postgres:9.6.

I want to analyze this docker image:

$ docker images
REPOSITORY                                     TAG                       IMAGE ID            CREATED             SIZE
vuln                                           v1                        911a7366f994        About an hour ago   267MB

I have clairctl cloned locally and running on localhost

$ go install github.com/jgsqware/clairctl
$ clairctl --config ~/.clairctl.yml --log-level debug push --local vuln:v1
2017-11-02 15:54:45.821161 D | config: Using config file: ~/.clairctl.yml
2017-11-02 15:54:45.821416 I | config: retrieving interface for local IP
2017-11-02 15:54:45.821420 D | config: no interface provided, looking for docker0
2017-11-02 15:54:45.821495 D | config: docker0 not found, looking for first connected broadcast interface
2017-11-02 15:54:45.822290 D | server: Update local server port from "0" to "62509"
2017-11-02 15:54:45.822298 I | server: Starting Server on 192.168.24.91:62509
2017-11-02 15:54:45.826975 D | dockercli: docker image to save: vuln:v1
2017-11-02 15:54:45.826986 D | dockercli: saving in: /tmp/clairctl/vuln/blobs
2017-11-02 15:54:49.855995 I | config: retrieving interface for local IP
2017-11-02 15:54:49.856024 D | config: no interface provided, looking for docker0
2017-11-02 15:54:49.856103 D | config: docker0 not found, looking for first connected broadcast interface
2017-11-02 15:54:49.856181 I | clair: using http://192.168.24.91:62509/local as local url
2017-11-02 15:54:49.856192 I | clair: Pushing Layer 1/6 [75b4349eaaff]
2017-11-02 15:54:49.856196 D | clair: RegistryURI: docker.io
2017-11-02 15:54:49.856922 D | clair: Saving 75b4349eaaff6a8b9cd775ae284e340770cf9ab9a8a2f5843dd14ad6ffc3cdbf[https://registry-1.docker.io/v2]
2017-11-02 15:54:49.874385 I | clair: adding layer 1/6 [75b4349eaaff]: receiving http error: 404
client quit unexpectedly
2017-11-02 15:54:49.874404 C | cmd: pushing image "vuln:v1": receiving http error: 404

Clairctl config

$ cat ~/.clairctl.yml
docker:
  insecure-registries:
    - "localhost:5000"

No matter what I try I still get these 404 errors from clair. I've also tried running a local docker registry. Tried to tag my local image, push to my local registry, then push localhost:5000/vuln:v1 with clairctl and the exact same results and errors.

I also tried making the ProxyFromEnvironment change suggested above and it did not help. I have also logged into docker hub (even though that shouldn't matter) and my local unauthenticated docker registry.

Anyone have any ideas on this?

EDIT: I should note that I am running the latest master branch of clairctl and the v2.0.1 tag of clair. Docker version is 17.09.0-ce both server and client. EDIT: I also cannot curl any routes directly to clair. I get the same 404:

$ curl -i localhost:6060/v1/namespaces
HTTP/1.1 404 Not Found
Content-Type: text/plain; charset=utf-8
X-Content-Type-Options: nosniff
Date: Thu, 02 Nov 2017 21:58:49 GMT
Content-Length: 10

Not Found

[jault3]

I apologize, I was building clair incorrectly. It is working now after I ran go install github.com/coreos/clair/cmd/clair. I was missing the /cmd/clair before, resulting in an old binary which was based off master.

[jzelinskie]

I'm closing this thread because it's become stale and as it mostly revolved around a clairctl issue. Please post a new issue if you're still experiencing a problem with these symptoms.

[xuqiang76]

docker run --net=host --name clair -d -p 6060-6061:6060-6061 -v $PWD/clair_config:/config quay.io/coreos/clair:v2.1.2 -config=/config/config.yaml -insecure-tls