Can't pull image from docker hub

[Karreg]

Hello,

I'm writing this, following the discussion in #1 I'm testing clair and klar since a few days, and I managed to make it work with a private registry (using Artifactory).

However, when I try to scan an image from docker hub, it does not work. But I did not use DOCKER_USER and DOCKER_PASSWORD

What I don't understand is that in your Readme, you require DOCKER_USER and DOCKER_PASSWORD even for docker hub, but pulling from docker hub does not require authentication.

Is there something I should know about how klar and clair work, that requires authentication even for docker hub ?

Here is the command line and the result:

KLAR_TRACE=true CLAIR_ADDR=127.0.0.1:6060 CLAIR_OUTPUT=High CLAIR_THRESHOLD=10 klar debian:9
clair timeout 1m0s
docker timeout: 1m0s
no whitelist file
----> HTTP REQUEST:
GET /v2/library/debian/manifests/9 HTTP/1.1
Host: registry-1.docker.io
Accept: application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.v1+prettyjws

Get error
Can't pull image: Get https://registry-1.docker.io/v2/library/debian/manifests/9: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)

Pulling does work:

docker pull debian:9                                                                                                                                    9: Pulling from library/debian
0bd44ff9c2cf: Pull complete 
Digest: sha256:b978de83fd0b0fc7e08d00108c8ea6b5314af46a8d508b085e82d76d33c8661a
Status: Downloaded newer image for debian:9

Thanks for the help.

[hashmap]

Hi,

What I don't understand is that in your Readme, you require DOCKER_USER and DOCKER_PASSWORD even for docker hub, but pulling from docker hub does not require authentication.

Technically docker hub supports private images, so it may require user/password. It still requires a token even for anonymous access, so if no user/password is specified Klar will get a temporary token for you. You refer to conversation in 2016 when we had an issue with this auth dance it's why I gave an advice to specify credentials as workaround. It's not needed now for sure, readme just show usage of such parameters, admittedly it looks misleading. I ran your command on my machine, see HTTP exhcange below.

You are getting a timeout error:

request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)

try to set DOCKER_TIMEOUT, the default is 1 minute, which usually more than enough, I don't know why it's so slow for you.

KLAR_TRACE=true CLAIR_ADDR=127.0.0.1:6060 CLAIR_OUTPUT=High CLAIR_THRESHOLD=10 ./klar debian:9
clair timeout 1m0s
docker timeout: 1m0s
no whitelist file
----> HTTP REQUEST:
GET /v2/library/debian/manifests/9 HTTP/1.1
Host: registry-1.docker.io
Accept: application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.v1+prettyjws

<---- HTTP RESPONSE:
HTTP/1.1 401 Unauthorized
Content-Length: 157
Content-Type: application/json; charset=utf-8
Date: Wed, 11 Jul 2018 13:30:11 GMT
Docker-Distribution-Api-Version: registry/2.0
Strict-Transport-Security: max-age=31536000
Www-Authenticate: Bearer realm="https://auth.docker.io/token",service="registry.docker.io",scope="repository:library/debian:pull"

{"errors":[{"code":"UNAUTHORIZED","message":"authentication required","detail":[{"Type":"repository","Class":"","Name":"library/debian","Action":"pull"}]}]}

----> HTTP REQUEST:
GET /v2/library/debian/manifests/9 HTTP/1.1
Host: registry-1.docker.io
Accept: application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.v1+prettyjws
Authorization: Bearer eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCIsIng1YyI6WyJNSUlDK2pDQ0FwK2dBd0lCQWdJQkFEQUtCZ2dxaGtqT1BRUURBakJHTVVRd1FnWURWUVFERXpzeVYwNVpPbFZMUzFJNlJFMUVVanBTU1U5Rk9reEhOa0U2UTFWWVZEcE5SbFZNT2tZelNFVTZOVkF5VlRwTFNqTkdPa05CTmxrNlNrbEVVVEFlRncweE9EQXlNVFF5TXpBMk5EZGFGdzB4T1RBeU1UUXlNekEyTkRkYU1FWXhSREJDQmdOVkJBTVRPMVpCUTFZNk5VNWFNenBNTkZSWk9sQlFTbGc2VWsxQlZEcEdWalpQT2xZMU1sTTZRa2szV2pwU1REVk9PbGhXVDBJNlFsTmFSanBHVTFRMk1JSUJJakFOQmdrcWhraUc5dzBCQVFFRkFBT0NBUThBTUlJQkNnS0NBUUVBMGtyTmgyZWxESnVvYjVERWd5Wi9oZ3l1ZlpxNHo0OXdvNStGRnFRK3VPTGNCMDRyc3N4cnVNdm1aSzJZQ0RSRVRERU9xNW5keEVMMHNaTE51UXRMSlNRdFY1YUhlY2dQVFRkeVJHUTl2aURPWGlqNFBocE40R0N0eFV6YTNKWlNDZC9qbm1YbmtUeDViOElUWXBCZzg2TGNUdmMyRFVUV2tHNy91UThrVjVPNFFxNlZKY05TUWRId1B2Mmp4YWRZa3hBMnhaaWNvRFNFQlpjWGRneUFCRWI2YkRnUzV3QjdtYjRRVXBuM3FXRnRqdCttKzBsdDZOR3hvenNOSFJHd3EwakpqNWtZbWFnWHpEQm5NQ3l5eDFBWFpkMHBNaUlPSjhsaDhRQ09GMStsMkVuV1U1K0thaTZKYVNEOFZJc2VrRzB3YXd4T1dER3U0YzYreE1XYUx3SURBUUFCbzRHeU1JR3ZNQTRHQTFVZER3RUIvd1FFQXdJSGdEQVBCZ05WSFNVRUNEQUdCZ1JWSFNVQU1FUUdBMVVkRGdROUJEdFdRVU5XT2pWT1dqTTZURFJVV1RwUVVFcFlPbEpOUVZRNlJsWTJUenBXTlRKVE9rSkpOMW82VWt3MVRqcFlWazlDT2tKVFdrWTZSbE5VTmpCR0JnTlZIU01FUHpBOWdEc3lWMDVaT2xWTFMxSTZSRTFFVWpwU1NVOUZPa3hITmtFNlExVllWRHBOUmxWTU9rWXpTRVU2TlZBeVZUcExTak5HT2tOQk5sazZTa2xFVVRBS0JnZ3Foa2pPUFFRREFnTkpBREJHQWlFQWdZTWF3Si9uMXM0dDlva0VhRjh2aGVkeURzbERObWNyTHNRNldmWTFmRTRDSVFEbzNWazJXcndiSjNmU1dwZEVjT3hNazZ1ZEFwK2c1Nkd6TjlRSGFNeVZ1QT09Il19.eyJhY2Nlc3MiOlt7InR5cGUiOiJyZXBvc2l0b3J5IiwibmFtZSI6ImxpYnJhcnkvZGViaWFuIiwiYWN0aW9ucyI6WyJwdWxsIl19XSwiYXVkIjoicmVnaXN0cnkuZG9ja2VyLmlvIiwiZXhwIjoxNTMxMzE2MTExLCJpYXQiOjE1MzEzMTU4MTEsImlzcyI6ImF1dGguZG9ja2VyLmlvIiwianRpIjoiOTdDN1hMeFM1aE01YUdScmlwS0MiLCJuYmYiOjE1MzEzMTU1MTEsInN1YiI6IiJ9.pjFBaOBYZbwz0CEtgdO3X-5jPrs-9WKvuZF7NoXeSgKPF_2CbrlGnOQzvd1W17dE79HNPkraDyL3MjZNhR_wE0FNBkNJ3ymWDryQNG54WYZDdOxFEtRvGynbJoLmmQgPq0_bhm66J5SIUXajwJqB1d46VvXoKL7kTOBUfdVzkOWO2NUp6-jB8pCQYjhm-2rWULn1j03S0fjFuGa0l9SCGVeOlv_c8_Xqd2KQdxu73Nhc5Vde1JIRlGRLwKFoaPD_z2FkNf0xFCJATXrq0x_kCc4Muw9vjMh3eDjJ__YHYL-oFlCTEhP5a1WctKlJT8HsrSF1WZOBg8l7__myILEkbQ

<---- HTTP RESPONSE:
HTTP/1.1 200 OK
Content-Length: 529
Content-Type: application/vnd.docker.distribution.manifest.v2+json
Date: Wed, 11 Jul 2018 13:30:12 GMT
Docker-Content-Digest: sha256:6ee341d1cf3da8e6ea059f8bc3af9940613c4287205cd71d7c6f9e1718fdcb9b
Docker-Distribution-Api-Version: registry/2.0
Etag: "sha256:6ee341d1cf3da8e6ea059f8bc3af9940613c4287205cd71d7c6f9e1718fdcb9b"
Strict-Transport-Security: max-age=31536000

{
   "schemaVersion": 2,
   "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
   "config": {
      "mediaType": "application/vnd.docker.container.image.v1+json",
      "size": 1503,
      "digest": "sha256:9a5d7185d3a6ede04c29cf89e4fa879f40e33324ced380decaf8919ffc5a4ade"
   },
   "layers": [
      {
         "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
         "size": 45319224,
         "digest": "sha256:0bd44ff9c2cf1129ef8cea689b3e10e6498f64d2f8d5532caae55841b474bf3a"
      }
   ]
}

...
Found 53 vulnerabilities
Unknown: 4
Negligible: 19
Low: 8
Medium: 9
High: 13
....

[Karreg]

Thanks for the reply. I'll try to work with the timeout. If you don't see something that could have an impact, I do... Proxies. I hope this will work, because my only workaround here is to pull image from docker hub, and push it in artifactory so I can scan it with klar/clair...

I'll keep you informed with the results :)

[khba]

Hello, I've also been testing Klar and just tried the same as you above and I also hit the same timeout error even with the CLAIR and DOCKER timeouts set to 5m :

KLAR_TRACE=true CLAIR_ADDR=localhost:6060 CLAIR_OUTPUT=High CLAIR_THRESHOLD=10 DOCKER_TIMEOUT=5 CLAIR_TIMEOUT=5 ./klar

Failed to analyze using API v1: push image https://registry-1.docker.io/v2/library/debian:9 to Clair failed: can't push layer to Clair: Post http://localhost:6060/v1/layers: net/http: request canceled (Client.Timeout exceeded while awaiting headers) Failed to analyze using API v1: push image https://registry-1.docker.io/v2/library/debian:9 to Clair failed: can't push layer to Clair: Post http://localhost:6060/v1/layers: net/http: request canceled (Client.Timeout exceeded while awaiting headers)

Also why is it trying to push the image to Dockerhub instead of simply pulling + analysing it ?

Thanks,

[hashmap]

@khba your case is a bit different, you got timeout on pushing to Clair, not on pulling from docker hub (push image https://registry-1.docker.io/v2/library/debian:9 to Clair failed). Klar is a client of Clair (and docker registry to be precise), it doesn't analyze image, but rather orchestrates the process. Make sure your Clair installation can access docker hub.

[Karreg]

Increasing the timeout won't help. I'm suspecting a proxy issue. But http_proxy and https_proxy are correctly set. Is there something that should be done in klar to handle the proxy defined at system level?

Since the request is:

----> HTTP REQUEST:
GET /v2/library/debian/manifests/9 HTTP/1.1
Host: registry-1.docker.io

I tried a curl and the result is...

$ curl -L https://registry-1.docker.io/v2/library/debian/manifests/9 | jq .                                                       {
{
  "errors": [
    {
      "code": "UNAUTHORIZED",
      "message": "authentication required",
      "detail": [
        {
          "Type": "repository",
          "Class": "",
          "Name": "library/debian",
          "Action": "pull"
        }
      ]
    }
  ]
}

We're back at previous question, it seems authentication is needed?

[hashmap]

@Karreg you mean you set HTTP_PROXY env var? Does your proxy require auth in any form?

No, curl works for you, if you check my trace you will see the same 401, it's how docker hub works, I described it:

It still requires a token even for anonymous access, so if no user/password is specified Klar will get a temporary token for you.

[Karreg]

OK, so that means that the temporary token stuff is not working? How can it be traced to find the root cause? I guess the curl fails because of the absence of this token so the test is worth nothing. I'll try grabing a token too tu passe to curl. I'll create an account in the meantime, to validate the communication.

[Karreg]

That's weird, I get a token with:

curl -L "https://auth.docker.io/token?service=registry-1.docker.io&scope=repository:debian:pull"

And I pass it to curl like this:

curl -L -H "Authorization: Bearer <token>" https://registry-1.docker.io/v2/library/debian/manifests/9 | jq .

And I still get

{
  "errors": [
    {
      "code": "UNAUTHORIZED",
      "message": "authentication required",
      "detail": [
        {
          "Type": "repository",
          "Class": "",
          "Name": "library/debian",
          "Action": "pull"
        }
      ]
    }
  ]
}

It looks like a regression or issue or breaking change from docker hub?

[hashmap]

I doubt so, klar + docker hub work on my machine as usual:)

[khba]

In my case it works when I run the whole thing on another host, I suspect the push was being blocked by one of the multiple security features I have activated on the first host. Will dig on that later.

[Karreg]

OK I got it... I made it work through curl with this sequence:

response=`curl -L "https://auth.docker.io/token?service=registry.docker.io&scope=repository:library/debian:pull"`
token=`echo $response | jq -r .token`
echo "Token is $token"
curl -L -H "Authorization: Bearer $token" https://registry-1.docker.io/v2/library/debian/manifests/9 | jq .

Result is:

{
  "schemaVersion": 1,
  "name": "library/debian",
  "tag": "9",
  "architecture": "amd64",
  "fsLayers": [
    {
      "blobSum": "sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4"
    },
    {
      "blobSum": "sha256:0bd44ff9c2cf1129ef8cea689b3e10e6498f64d2f8d5532caae55841b474bf3a"
    }
  ],
  "history": [
    {
      "v1Compatibility": "{\"architecture\":\"amd64\",\"config\":{\"Hostname\":\"\",\"Domainname\":\"\",\"User\":\"\",\"AttachStdin\":false,\"AttachStdout\":false,\"AttachStderr\":false,\"Tty\":false,\"OpenStdin\":false,\"StdinOnce\":false,\"Env\":[\"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin\"],\"Cmd\":[\"bash\"],\"ArgsEscaped\":true,\"Image\":\"sha256:d8c5e1f55abb310301cea3de258aadefcb2bd103f8bc4854672ac3302b4a5294\",\"Volumes\":null,\"WorkingDir\":\"\",\"Entrypoint\":null,\"OnBuild\":null,\"Labels\":null},\"container\":\"329ff59fa63d9166d26cc0c01dd31778158b93ab9bc6a3401c5a09113516361e\",\"container_config\":{\"Hostname\":\"329ff59fa63d\",\"Domainname\":\"\",\"User\":\"\",\"AttachStdin\":false,\"AttachStdout\":false,\"AttachStderr\":false,\"Tty\":false,\"OpenStdin\":false,\"StdinOnce\":false,\"Env\":[\"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin\"],\"Cmd\":[\"/bin/sh\",\"-c\",\"#(nop) \",\"CMD [\\\"bash\\\"]\"],\"ArgsEscaped\":true,\"Image\":\"sha256:d8c5e1f55abb310301cea3de258aadefcb2bd103f8bc4854672ac3302b4a5294\",\"Volumes\":null,\"WorkingDir\":\"\",\"Entrypoint\":null,\"OnBuild\":null,\"Labels\":{}},\"created\":\"2018-06-26T21:24:58.974831097Z\",\"docker_version\":\"17.06.2-ce\",\"id\":\"9afcd3b9d5e171c7bf9d5e0fd8d82a2756658212355d2e70355a78760c5ef76b\",\"os\":\"linux\",\"parent\":\"b2d4aa5b3efda0873d8a8c531a231724fb65a6ad9da63e9174dce98d0e659038\",\"throwaway\":true}"
    },
    {
      "v1Compatibility": "{\"id\":\"b2d4aa5b3efda0873d8a8c531a231724fb65a6ad9da63e9174dce98d0e659038\",\"created\":\"2018-06-26T21:24:58.552395569Z\",\"container_config\":{\"Cmd\":[\"/bin/sh -c #(nop) ADD file:f21d7c14104d5d9fa99f271177e765a3472f5a69398bb78f34f7401e9b2df837 in / \"]}}"
    }
  ],
  "signatures": [
    {
      "header": {
        "jwk": {
          "crv": "P-256",
          "kid": "VEMC:NZ5F:L2SE:QRRU:BI36:YVJB:INRB:HENP:4DNK:ECLM:R5QE:R443",
          "kty": "EC",
          "x": "QBJbtH4d9PS5sRyvCknNmy2HvJ1PadoyV5gLm94nRhA",
          "y": "54XKjhFMmFVhkOOEdmrn_OQ8LP_A24sgwPC3a1tfSe4"
        },
        "alg": "ES256"
      },
      "signature": "oN6dPjDN6QXTZsZcek1qLAjNYx2TN-NzY0ttQt-0UMHLh7dtmd4P5PUILhu0Hrb9x7UQagB5zz3qE6zkdZYxvA",
      "protected": "eyJmb3JtYXRMZW5ndGgiOjIxMzAsImZvcm1hdFRhaWwiOiJDbjAiLCJ0aW1lIjoiMjAxOC0wNy0xM1QxMjozMTowOVoifQ"
    }
  ]
}

Meaning that I can get the manifest, but klar is still failing... Looks like there's something missing on klar configuration... Is klar correctly handling http_proxy and https_proxy environment variables? Does it need to?

[vippujain]

@Karreg i see you sucessfully tested on private artifacotry repo. Can you share the way to connect klar with private artifactory repo?