Clair failed: can't push layer to Clair

[kody-abe]

Failed to analyze using API v1: push image https://{{registryPath}} to Clair failed: can't push layer to Clair: Post https://{{clairService}}:443/v1/layers: EOF

Failed to analyze using API v3: push image https://{{registryPath}} to Clair failed: rpc error: code = Unavailable desc = transport is closing

Not sure what is happening here - Nothing really showing up on the Clair logs either. Any ideas?

[hashmap]

Klar can't connect to Clair, make sure it's reachable

[kody-abe]

@hashmap Thanks for the reply. Clair is reachable and other images are fine. This image seems to be a bit larger than other ones we have been testing.

[kody-abe]

@hashmap Is there a debug mode or something that I can enable to give more insights into the request/response to validate your thought?

[kody-abe]

@hashmap So, I ran clair in debug mode while it was running and this is the output. Does not seem like anything happening on the Clair side. Is there a debug mode for klar?

{"Event":"processing layer","Level":"debug","Location":"worker.go:73","Time":"2018-07-19 20:00:23.991102","engine version":3,"format":"Docker","layer":"f147e18a299753abf7b854feac527d723004b69c336772cf55152f16f4aee7e266bb560c702cb73fd1d1627ac3de0f67da8521f27d9e37de13dff384ff19168d","parent layer":"","path":"https://AWSENDPOINT/v2/APP/blobs/sha256:66bb560c702cb73fd1d1627ac3de0f67da8521f27d9e37de13dff384ff19168d"}
{"Event":"detected namespace","Level":"debug","Location":"driver.go:85","Time":"2018-07-19 20:00:31.436549","name":"os-release","namespace":"debian:9"}
{"Event":"detected namespace","Level":"debug","Location":"worker.go:149","Time":"2018-07-19 20:00:31.436632","detected namespace":"debian:9","layer":"f147e18a299753abf7b854feac527d723004b69c336772cf55152f16f4aee7e266bb560c702cb73fd1d1627ac3de0f67da8521f27d9e37de13dff384ff19168d"}
{"Event":"detected features","Level":"debug","Location":"worker.go:137","Time":"2018-07-19 20:00:31.437108","feature count":56,"layer":"f147e18a299753abf7b854feac527d723004b69c336772cf55152f16f4aee7e266bb560c702cb73fd1d1627ac3de0f67da8521f27d9e37de13dff384ff19168d"}
{"Event":"Handled HTTP request","Level":"info","Location":"router.go:57","Time":"2018-07-19 20:00:31.799002","elapsed time":7808047372,"method":"POST","remote addr":"100.101.254.241:47002","request uri":"/v1/layers","status":"201"}
{"Event":"processing layer","Level":"debug","Location":"worker.go:73","Time":"2018-07-19 20:00:31.845089","engine version":3,"format":"Docker","layer":"f147e18a299753abf7b854feac527d723004b69c336772cf55152f16f4aee7e2a3b36953fbb38b46dd17f1c75f15c716ce94063137a489d28126c1646a5e137c","parent layer":"f147e18a299753abf7b854feac527d723004b69c336772cf55152f16f4aee7e266bb560c702cb73fd1d1627ac3de0f67da8521f27d9e37de13dff384ff19168d","path":"https://AWSENDPOINT/v2/APP/blobs/sha256:a3b36953fbb38b46dd17f1c75f15c716ce94063137a489d28126c1646a5e137c"}
{"Event":"detected namespace (from parent)","Level":"debug","Location":"worker.go:157","Time":"2018-07-19 20:00:33.726817","detected namespace":"debian:9","layer":"f147e18a299753abf7b854feac527d723004b69c336772cf55152f16f4aee7e2a3b36953fbb38b46dd17f1c75f15c716ce94063137a489d28126c1646a5e137c"}
{"Event":"detected features","Level":"debug","Location":"worker.go:137","Time":"2018-07-19 20:00:33.727670","feature count":78,"layer":"f147e18a299753abf7b854feac527d723004b69c336772cf55152f16f4aee7e2a3b36953fbb38b46dd17f1c75f15c716ce94063137a489d28126c1646a5e137c"}
{"Event":"Handled HTTP request","Level":"info","Location":"router.go:57","Time":"2018-07-19 20:00:33.838991","elapsed time":1993997161,"method":"POST","remote addr":"100.101.254.241:47002","request uri":"/v1/layers","status":"201"}
{"Event":"processing layer","Level":"debug","Location":"worker.go:73","Time":"2018-07-19 20:00:33.847590","engine version":3,"format":"Docker","layer":"f147e18a299753abf7b854feac527d723004b69c336772cf55152f16f4aee7e27503ef05f58f6158452a6fe2b6959ff164998b72551aef0dfc1951867d00220f","parent layer":"f147e18a299753abf7b854feac527d723004b69c336772cf55152f16f4aee7e2a3b36953fbb38b46dd17f1c75f15c716ce94063137a489d28126c1646a5e137c","path":"https://AWSENDPOINT/v2/APP/blobs/sha256:7503ef05f58f6158452a6fe2b6959ff164998b72551aef0dfc1951867d00220f"}
{"Event":"detected namespace (from parent)","Level":"debug","Location":"worker.go:157","Time":"2018-07-19 20:00:34.438394","detected namespace":"debian:9","layer":"f147e18a299753abf7b854feac527d723004b69c336772cf55152f16f4aee7e27503ef05f58f6158452a6fe2b6959ff164998b72551aef0dfc1951867d00220f"}
{"Event":"detected features","Level":"debug","Location":"worker.go:137","Time":"2018-07-19 20:00:34.439208","feature count":84,"layer":"f147e18a299753abf7b854feac527d723004b69c336772cf55152f16f4aee7e27503ef05f58f6158452a6fe2b6959ff164998b72551aef0dfc1951867d00220f"}
{"Event":"Handled HTTP request","Level":"info","Location":"router.go:57","Time":"2018-07-19 20:00:34.534400","elapsed time":686883486,"method":"POST","remote addr":"100.101.254.241:47002","request uri":"/v1/layers","status":"201"}
{"Event":"processing layer","Level":"debug","Location":"worker.go:73","Time":"2018-07-19 20:00:34.543572","engine version":3,"format":"Docker","layer":"f147e18a299753abf7b854feac527d723004b69c336772cf55152f16f4aee7e252bbcb900ba900cd4f9409b4bf95a314d92dd43763a785faaab45c38b12d548e","parent layer":"f147e18a299753abf7b854feac527d723004b69c336772cf55152f16f4aee7e27503ef05f58f6158452a6fe2b6959ff164998b72551aef0dfc1951867d00220f","path":"https://AWSENDPOINT/v2/APP/blobs/sha256:52bbcb900ba900cd4f9409b4bf95a314d92dd43763a785faaab45c38b12d548e"}
{"Event":"detected namespace (from parent)","Level":"debug","Location":"worker.go:157","Time":"2018-07-19 20:00:44.224960","detected namespace":"debian:9","layer":"f147e18a299753abf7b854feac527d723004b69c336772cf55152f16f4aee7e252bbcb900ba900cd4f9409b4bf95a314d92dd43763a785faaab45c38b12d548e"}
{"Event":"detected features","Level":"debug","Location":"worker.go:137","Time":"2018-07-19 20:00:44.225858","feature count":105,"layer":"f147e18a299753abf7b854feac527d723004b69c336772cf55152f16f4aee7e252bbcb900ba900cd4f9409b4bf95a314d92dd43763a785faaab45c38b12d548e"}
{"Event":"Handled HTTP request","Level":"info","Location":"router.go:57","Time":"2018-07-19 20:00:44.332670","elapsed time":9789182674,"method":"POST","remote addr":"100.101.254.241:47002","request uri":"/v1/layers","status":"201"}
{"Event":"processing layer","Level":"debug","Location":"worker.go:73","Time":"2018-07-19 20:00:44.339496","engine version":3,"format":"Docker","layer":"f147e18a299753abf7b854feac527d723004b69c336772cf55152f16f4aee7e2c8c1dc87abda235361bf3b044c31ef1dc487c447e4f46f8ef44a431891589a64","parent layer":"f147e18a299753abf7b854feac527d723004b69c336772cf55152f16f4aee7e252bbcb900ba900cd4f9409b4bf95a314d92dd43763a785faaab45c38b12d548e","path":"https://AWSENDPOINT/v2/APP/blobs/sha256:c8c1dc87abda235361bf3b044c31ef1dc487c447e4f46f8ef44a431891589a64"}
{"Event":"detected namespace (from parent)","Level":"debug","Location":"worker.go:157","Time":"2018-07-19 20:01:19.435743","detected namespace":"debian:9","layer":"f147e18a299753abf7b854feac527d723004b69c336772cf55152f16f4aee7e2c8c1dc87abda235361bf3b044c31ef1dc487c447e4f46f8ef44a431891589a64"}
{"Event":"detected features","Level":"debug","Location":"worker.go:137","Time":"2018-07-19 20:01:19.437748","feature count":205,"layer":"f147e18a299753abf7b854feac527d723004b69c336772cf55152f16f4aee7e2c8c1dc87abda235361bf3b044c31ef1dc487c447e4f46f8ef44a431891589a64"}
{"Event":"Handled HTTP request","Level":"info","Location":"router.go:57","Time":"2018-07-19 20:01:19.991724","elapsed time":35652341283,"method":"POST","remote addr":"100.101.254.241:47002","request uri":"/v1/layers","status":"201"}
{"Event":"processing layer","Level":"debug","Location":"worker.go:73","Time":"2018-07-19 20:01:19.996525","engine version":3,"format":"Docker","layer":"f147e18a299753abf7b854feac527d723004b69c336772cf55152f16f4aee7e240df8e1e18284d5e7e20d1cada407589b0eee3ca572f520c902d73389ae79206","parent layer":"f147e18a299753abf7b854feac527d723004b69c336772cf55152f16f4aee7e2c8c1dc87abda235361bf3b044c31ef1dc487c447e4f46f8ef44a431891589a64","path":"https://AWSENDPOINT/v2/APP/blobs/sha256:40df8e1e18284d5e7e20d1cada407589b0eee3ca572f520c902d73389ae79206"}
{"Event":"detected namespace (from parent)","Level":"debug","Location":"worker.go:157","Time":"2018-07-19 20:01:20.830303","detected namespace":"debian:9","layer":"f147e18a299753abf7b854feac527d723004b69c336772cf55152f16f4aee7e240df8e1e18284d5e7e20d1cada407589b0eee3ca572f520c902d73389ae79206"}
{"Event":"detected features","Level":"debug","Location":"worker.go:137","Time":"2018-07-19 20:01:20.832586","feature count":210,"layer":"f147e18a299753abf7b854feac527d723004b69c336772cf55152f16f4aee7e240df8e1e18284d5e7e20d1cada407589b0eee3ca572f520c902d73389ae79206"}
{"Event":"Handled HTTP request","Level":"info","Location":"router.go:57","Time":"2018-07-19 20:01:20.868020","elapsed time":871601184,"method":"POST","remote addr":"100.101.254.241:47002","request uri":"/v1/layers","status":"201"}
{"Event":"processing layer","Level":"debug","Location":"worker.go:73","Time":"2018-07-19 20:01:20.873190","engine version":3,"format":"Docker","layer":"f147e18a299753abf7b854feac527d723004b69c336772cf55152f16f4aee7e2408f1d85a239c24fa049e8635350fe86a2b057cf8bec2931a694e5761cb815c8","parent layer":"f147e18a299753abf7b854feac527d723004b69c336772cf55152f16f4aee7e240df8e1e18284d5e7e20d1cada407589b0eee3ca572f520c902d73389ae79206","path":"https://AWSENDPOINT/v2/APP/blobs/sha256:408f1d85a239c24fa049e8635350fe86a2b057cf8bec2931a694e5761cb815c8"}
{"Event":"detected namespace (from parent)","Level":"debug","Location":"worker.go:157","Time":"2018-07-19 20:01:24.924473","detected namespace":"debian:9","layer":"f147e18a299753abf7b854feac527d723004b69c336772cf55152f16f4aee7e2408f1d85a239c24fa049e8635350fe86a2b057cf8bec2931a694e5761cb815c8"}
{"Event":"detected features","Level":"debug","Location":"worker.go:137","Time":"2018-07-19 20:01:24.926898","feature count":210,"layer":"f147e18a299753abf7b854feac527d723004b69c336772cf55152f16f4aee7e2408f1d85a239c24fa049e8635350fe86a2b057cf8bec2931a694e5761cb815c8"}
{"Event":"Handled HTTP request","Level":"info","Location":"router.go:57","Time":"2018-07-19 20:01:24.935821","elapsed time":4062750522,"method":"POST","remote addr":"100.101.254.241:47002","request uri":"/v1/layers","status":"201"}
{"Event":"processing layer","Level":"debug","Location":"worker.go:73","Time":"2018-07-19 20:01:24.943505","engine version":3,"format":"Docker","layer":"f147e18a299753abf7b854feac527d723004b69c336772cf55152f16f4aee7e2b4b5443b19a746df185c567adce1bff448926c5e0a3e69b0d13be428c9ed71b6","parent layer":"f147e18a299753abf7b854feac527d723004b69c336772cf55152f16f4aee7e2408f1d85a239c24fa049e8635350fe86a2b057cf8bec2931a694e5761cb815c8","path":"https://AWSENDPOINT/v2/APP/blobs/sha256:b4b5443b19a746df185c567adce1bff448926c5e0a3e69b0d13be428c9ed71b6"}
{"Event":"detected namespace (from parent)","Level":"debug","Location":"worker.go:157","Time":"2018-07-19 20:01:25.053333","detected namespace":"debian:9","layer":"f147e18a299753abf7b854feac527d723004b69c336772cf55152f16f4aee7e2b4b5443b19a746df185c567adce1bff448926c5e0a3e69b0d13be428c9ed71b6"}

[hashmap]

@kody-abe sure, add env var KLAR_TRACE=true

[piu28]

I am also facing the same issue. Details are below: klar-2.3.0 clair using helm - repo: quay.io/coreos/clair-git kubernetes version 1.11 docker registry: AWS ECR

Command: KLAR_TRACE=true CLAIR_ADDR="http://clair.domain.com:80" CLAIR_OUTPUT=High CLAIR_THRESHOLD=10 DOCKER_USER=AWS DOCKER_PASSWORD=${PASSWORD} klar ACCOUNT.dkr.ecr.ap-south-1.amazonaws.com/REPONAME

I am running Clair service using NodePort in Kubernetes.

Can someone help me here to scan the docker images for vulnerabilities?

Also, Please give more details on webhook notification endpoint in Clair. Some examples would definitely help.

Thanks.

[kody-abe]

@piu28 At this point we seem to be looking good on our side. We updated Clair to 2.0.3 and Klar to 2.3.0

The clair-git repo seems to be using HEAD and we had issues using that version. I would try:

image:
  repository: quay.io/coreos/clair
  tag: "v2.0.3"