jgsqware / clairctl

Tracking container vulnerabilities with Clair Control for CoreOS Clair
Apache License 2.0
231 stars 83 forks source link

HTTP 400 on Push and Analyze #49

Open goffinf opened 7 years ago

goffinf commented 7 years ago

Similar problem as https://github.com/jgsqware/clairctl/issues/18 (Detailed Guidance for setup of CLAIRCTL), however since that issue is now discussing other topics I wanted to return to the basic problem of getting the http error: 400 client quit unexpectedly error for push or analyze to see what it is that I am missing ..

Clairctl health-check ...

` $ clairctl health --log-level debug 2017-06-04 16:08:01.755098 D | config: Using config file: /home/docker/clairctl.yml 2017-06-04 16:08:01.756459 D | clair: requesting health on: http://172.18.0.4:6061/health

Clair: ✔ `

But PUSH ...

` $ clairctl push hello-world:latest --log-level debug

2017-06-04 15:59:48.660635 D | config: Using config file: /home/docker/clairctl.yml 2017-06-04 15:59:48.662555 D | dockerdist: Downloading manifest for hello-world:latest 2017-06-04 15:59:48.664492 D | dockerdist: Retrieving repository client 2017-06-04 15:59:48.666448 D | dockerdist: endpoint.TLSConfig.InsecureSkipVerify: true 2017-06-04 15:59:52.316571 D | dockerdist: manifest type: *schema2.DeserializedManifest 2017-06-04 15:59:52.316594 D | dockerdist: retrieved schema2 manifest, no verification 2017-06-04 15:59:52.317774 I | config: retrieving interface for local IP 2017-06-04 15:59:52.317787 D | config: no interface provided, looking for docker0 2017-06-04 15:59:52.319476 I | clair: Pushing Layer 1/1 [sha256:78445] 2017-06-04 15:59:52.320515 D | clair: Saving sha256:78445dd45222097f5f8d5a16e48dc19c4ca162dcdb80010ab6f1ccfc7e2c0fa3[https://registry-1.docker.io/v2] 2017-06-04 15:59:53.022542 I | clair: adding layer 1/1 [sha256:78445]: receiving http error: 400 client quit unexpectedly 2017-06-04 15:59:53.024542 C | cmd: pushing image "hello-world:latest": receiving http error: 400 ` Setup is as follows ...

Clairctl binary installed (not running inside a container). Clairctl version: 1.2.6

Docker version: 17.05.0-ce

clairctl.yml: ` clair: port: 6060 healthPort: 6061 uri: http://clair report: path: ./reports format: html docker: insecure-registries:

docker-compose.yml:

` version: '2' services: postgres: container_name: clair_postgres image: postgres:latest restart: unless-stopped environment: POSTGRES_PASSWORD: password

clair: container_name: clair_clair

image: quay.io/coreos/clair:latest

image: quay.io/coreos/clair-git:latest
restart: unless-stopped
depends_on:
  - postgres
ports:
  - "6060-6061:6060-6061"
links:
  - postgres
volumes:
  - /tmp:/tmp
  - ./clair_config:/config
command: [-config, /config/config.yaml]

registry: container_name: registry image: registry:2 restart: unless-stopped ports:

Pull appears to work ok ..

` docker@clairctl:~$ clairctl pull hello-world:latest --log-level debug 2017-06-04 16:07:19.617159 D | config: Using config file: /home/docker/clairctl.yml 2017-06-04 16:07:19.625274 D | dockerdist: Downloading manifest for hello-world:latest 2017-06-04 16:07:19.625920 D | dockerdist: Retrieving repository client 2017-06-04 16:07:19.626598 D | dockerdist: endpoint.TLSConfig.InsecureSkipVerify: true 2017-06-04 16:07:25.464485 D | dockerdist: manifest type: *schema2.DeserializedManifest 2017-06-04 16:07:25.464544 D | dockerdist: retrieved schema2 manifest, no verification

Image: docker.io/library/hello-world:latest 1 layers found ➜ sha256:78445dd45222097f5f8d5a16e48dc19c4ca162dcdb80010ab6f1ccfc7e2c0fa3 `

I have tried using localhost and the clair container IP as the uri: value in clairctl.yml but neither work.

nsloookup clair fails

Really not sure what else to try. I assume clair cannot communicate with clairctl. I note mention of the need for a reverse proxy but thought that came OOTB.

Help appreciated.

Regards

Fraser.

jgsqware commented 7 years ago

Hello, thanks for reporting, have you logged into docker with the docker login command before?

To download layer of official images you need to be logged

Le dim. 4 juin 2017 18:14, goffinf notifications@github.com a écrit :

Similar problem as #18 https://github.com/jgsqware/clairctl/issues/18 (Detailed Guidance for setup of CLAIRCTL), however since that issue is now discussing other topics I wanted to return to the basic problem of getting the http error: 400 client quit unexpectedly error for push or analyze to see what it is that I am missing ..

Clairctl health-check ...

` docker@clairctl:$ docker@clairctl:$ clairctl health --log-level debug 2017-06-04 16:08:01.755098 D | config: Using config file: /home/docker/clairctl.yml 2017-06-04 16:08:01.756459 D | clair: requesting health on: http://172.18.0.4:6061/health

Clair: ✔ `

But PUSH ...

` $ clairctl push hello-world:latest --log-level debug

2017-06-04 15:59:48.660635 D | config: Using config file: /home/docker/clairctl.yml 2017-06-04 15:59:48.662555 D | dockerdist: Downloading manifest for hello-world:latest 2017-06-04 15:59:48.664492 D | dockerdist: Retrieving repository client 2017-06-04 15:59:48.666448 D | dockerdist: endpoint.TLSConfig.InsecureSkipVerify: true 2017-06-04 15:59:52.316571 D | dockerdist: manifest type: *schema2.DeserializedManifest 2017-06-04 15:59:52.316594 D | dockerdist: retrieved schema2 manifest, no verification 2017-06-04 15:59:52.317774 I | config: retrieving interface for local IP 2017-06-04 15:59:52.317787 D | config: no interface provided, looking for docker0 2017-06-04 15:59:52.319476 I | clair: Pushing Layer 1/1 [sha256:78445] 2017-06-04 15:59:52.320515 D | clair: Saving sha256:78445dd45222097f5f8d5a16e48dc19c4ca162dcdb80010ab6f1ccfc7e2c0fa3[ https://registry-1.docker.io/v2] 2017-06-04 15:59:53.022542 I | clair: adding layer 1/1 [sha256:78445]: receiving http error: 400 client quit unexpectedly 2017-06-04 15:59:53.024542 C | cmd: pushing image "hello-world:latest": receiving http error: 400 ` Setup is as follows ...

Clairctl binary installed (not running inside a container). Clairctl version: 1.2.6

Docker version: 17.05.0-ce

clairctl.yml:

clair: port: 6060 healthPort: 6061 uri: http://clair report: path: ./reports format: html docker: insecure-registries: - "localhost:5000"

docker-compose.yml:

` version: '2' services: postgres: container_name: clair_postgres image: postgres:latest restart: unless-stopped environment: POSTGRES_PASSWORD: password

clair: container_name: clair_clair

image: quay.io/coreos/clair:latest

image: quay.io/coreos/clair-git:latest restart: unless-stopped depends_on:

  • postgres ports:
  • "6060-6061:6060-6061" links:
  • postgres volumes:
  • /tmp:/tmp
  • ./clair_config:/config command: [-config, /config/config.yaml]

registry: container_name: registry image: registry:2 restart: unless-stopped ports:

  • "5000:5000" `

Pull appears to work ok ..

` docker@clairctl:~$ clairctl pull hello-world:latest --log-level debug 2017-06-04 16:07:19.617159 D | config: Using config file: /home/docker/clairctl.yml 2017-06-04 16:07:19.625274 D | dockerdist: Downloading manifest for hello-world:latest 2017-06-04 16:07:19.625920 D | dockerdist: Retrieving repository client 2017-06-04 16:07:19.626598 D | dockerdist: endpoint.TLSConfig.InsecureSkipVerify: true 2017-06-04 16:07:25.464485 D | dockerdist: manifest type: *schema2.DeserializedManifest 2017-06-04 16:07:25.464544 D | dockerdist: retrieved schema2 manifest, no verification

Image: docker.io/library/hello-world:latest 1 layers found ➜ sha256:78445dd45222097f5f8d5a16e48dc19c4ca162dcdb80010ab6f1ccfc7e2c0fa3 `

I have tried using localhost and the clair container IP as the uri: value in clairctl.yml but neither work.

nsloookup clair fails

Really not sure what else to try. I assume clair cannot communicate with clairctl. I note mention of the need for a reverse proxy but thought that came OOTB.

Help appreciated.

Regards

Fraser.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/jgsqware/clairctl/issues/49, or mute the thread https://github.com/notifications/unsubscribe-auth/ADJrq7v6wuYQVmOUsua8tCGDKRE0QX2Dks5sAtf4gaJpZM4Nvaqx .

goffinf commented 7 years ago

Yes,

docker@clairctl:~$ docker login Username: goffinf Password: Login Succeeded

docker@clairctl:~$ clairctl analyze hello-world:latest --log-level debug 2017-06-04 22:28:34.957602 D | config: Using config file: /home/docker/clairctl.yml 2017-06-04 22:28:34.958019 D | dockerdist: Downloading manifest for hello-world:latest 2017-06-04 22:28:34.958238 D | dockerdist: Retrieving repository client 2017-06-04 22:28:36.358459 D | dockerdist: endpoint.TLSConfig.InsecureSkipVerify: true 2017-06-04 22:28:38.323521 D | dockerdist: manifest type: *schema2.DeserializedManifest 2017-06-04 22:28:38.323543 D | dockerdist: retrieved schema2 manifest, no verification 2017-06-04 22:28:38.323943 I | config: retrieving interface for local IP 2017-06-04 22:28:38.323967 D | config: no interface provided, looking for docker0 2017-06-04 22:28:38.324653 I | clair: Pushing Layer 1/1 [sha256:78445] 2017-06-04 22:28:38.325434 D | clair: Saving sha256:78445dd45222097f5f8d5a16e48dc19c4ca162dcdb80010ab6f1ccfc7e2c0fa3[https://registry-1.docker.io/v2] 2017-06-04 22:28:38.911448 I | clair: adding layer 1/1 [sha256:78445]: receiving http error: 400 client quit unexpectedly 2017-06-04 22:28:38.911486 C | cmd: pushing image "hello-world:latest": receiving http error: 400

I also thought that with clairctl I could use a private registry. As you see from the docker-compose above I started a docker registry on localhost:5000, but if I push to that and then run analyze it fails in exactly the same way ..

docker@clairctl:$ docker push localhost:5000/hello-world The push refers to a repository [localhost:5000/hello-world] 98c944e98de8: Pushed latest: digest: sha256:2075ac87b043415d35bb6351b4a59df19b8ad154e578f7048335feeb02d0f759 size: 524 docker@clairctl:~$ clairctl analyze localhost:5000/hello-world:latest --log-level debug 2017-06-04 22:24:25.528790 D | config: Using config file: /home/docker/clairctl.yml 2017-06-04 22:24:25.529508 D | dockerdist: Downloading manifest for localhost:5000/hello-world:latest 2017-06-04 22:24:25.530102 D | dockerdist: Retrieving repository client 2017-06-04 22:24:25.531270 D | dockerdist: endpoint.TLSConfig.InsecureSkipVerify: true 2017-06-04 22:24:25.553704 D | dockerdist: manifest type: *schema2.DeserializedManifest 2017-06-04 22:24:25.553725 D | dockerdist: retrieved schema2 manifest, no verification 2017-06-04 22:24:25.553971 I | config: retrieving interface for local IP 2017-06-04 22:24:25.553981 D | config: no interface provided, looking for docker0 2017-06-04 22:24:25.555101 I | clair: Pushing Layer 1/1 [sha256:983bf] 2017-06-04 22:24:25.555248 D | clair: Saving sha256:983bfa07a342e316f08afd066894505088de985d46a9af743920aa9cafd17e7a[http://localhost:5000/v2] 2017-06-04 22:24:25.711773 I | clair: adding layer 1/1 [sha256:983bf]: receiving http error: 400 client quit unexpectedly 2017-06-04 22:24:25.711803 C | cmd: pushing image "localhost:5000/hello-world:latest": receiving http error: 400

Note: I changed the clairctl.yml uri value to point to localhost since 'http://clair' did not resolve

` docker@clairctl:~$ cat clairctl.yml

clair: port: 6060 healthPort: 6061 # uri: http://clair uri: http://localhost report: path: ./reports format: html docker: insecure-registries:

Any other thoughts ?

Kind Regards

Fraser

goffinf commented 7 years ago

From the clair container logs ...

When trying .. clairctl analyze hello-world:latest

{"Event":"could not download layer: expected 2XX","Level":"warning","Location":"driver.go:135","Time":"2017-06-04 23:08:38.866291","status code":401} {"Event":"failed to extract data from path","Level":"error","Location":"worker.go:122","Time":"2017-06-04 23:08:38.866450","error":"could not find layer","layer":"sha256:78445dd45222097f5f8d5a16e48dc19c4ca162dcdb80010ab6f1ccfc7e2c0fa3","path":"https://registry-1.docker.io/v2/library/hello-world/blobs/sha256:78445dd45222097f5f8d5a16e48dc19c4ca162dcdb80010ab6f1ccfc7e2c0fa3"} {"Event":"Handled HTTP request","Level":"info","Location":"router.go:57","Time":"2017-06-04 23:08:38.866736","elapsed time":370576353,"method":"POST","remote addr":"172.18.0.1:52322","request uri":"/v1/layers","status":"400"}

... and .... if trying ... clairctl analyze localhost:5000/hello-world:latest

{"Event":"could not download layer","Level":"warning","Location":"driver.go:129","Time":"2017-06-04 23:12:25.108460","error":"Get http://localhost:5000/v2/hello-world/blobs/sha256:983bfa07a342e316f08afd066894505088de985d46a9af743920aa9cafd17e7a: dial tcp 127.0.0.1:5000: getsockopt: connection refused"} {"Event":"failed to extract data from path","Level":"error","Location":"worker.go:122","Time":"2017-06-04 23:12:25.109273","error":"could not find layer","layer":"sha256:983bfa07a342e316f08afd066894505088de985d46a9af743920aa9cafd17e7a","path":"http://localhost:5000/v2/hello-world/blobs/sha256:983bfa07a342e316f08afd066894505088de985d46a9af743920aa9cafd17e7a"} {"Event":"Handled HTTP request","Level":"info","Location":"router.go:57","Time":"2017-06-04 23:12:25.115614","elapsed time":86736193,"method":"POST","remote addr":"172.18.0.1:52338","request uri":"/v1/layers","status":"400"}

gauravjos commented 7 years ago

Facing same issue +1

jdel commented 7 years ago

I cannot reproduce this issue:

$ clairctll analyze hello-world

Image: docker.io/hello-world:latest
 1 layers found

  ➜ Analysis [sha256:78445] found 0 vulnerabilities.

$ clairctll version

Clairctl version b2094f1

edit: clairctll is an alias i use to run clairctl from a docker container

goffinf commented 7 years ago

Thx @jdel . I'm not surprised you can't replicate because I'm sure the mis-configuration is on my (our) side. Could you possibly share how you are running Clair and clairctl and the config you are using for both, where these are located and whether you did any other setup steps which we might have missed.

If we could follow your steps precisely I'm sure we would get to the bottom of it.

Much appreciated

Fraser.

gauravjos commented 7 years ago

@jdel If its possible please share the clairctl config.yml

jdel commented 7 years ago

@goffinf, @gauravjos: you can find my setup in #47

The easiest would be to clone my fork https://github.com/jdel/clairctl.git

Then cd in the directory and run IMAGE_NAME=your-hub-account/clairctl hooks/build in a terminal. This will create a clairctl container.

Mind that you will have to amend the docker-compose.yml and change the image jgsqware/clairctl image with your-hub-account/clairctl.

Then, you can run docker-compose up -d to run everything. Clair will need some time to pull all vulnerabilities list before you can use it.

Finally, you can run docker-compose exec clairctl clairctl health, docker-compose exec clairctl clairctl version, docker-compose exec clairctl clairctl push XXX... and so on.

For reference, the clair.yml config file I use is here https://github.com/jdel/clairctl/blob/master/docker-compose-data/clair-config/config.yml

goffinf commented 7 years ago

Thx Julien,

after a few minor adjustments I was able to get things working using your fork.

Regards

Fraser.

jdel commented 7 years ago

@goffinf: my fork has been merged here. you can get the latest docker image built from master at: jgsqware/clairctl:master

Other tags (latest, 1.6.2, ...) will be coming shortly.

gauravjos commented 7 years ago

I am still getting error when try to scan image

>docker-compose exec clairctl clairctl push gauravsj/demo-php-app:blue client quit unexpectedly 2017-06-13 09:38:06.811711 C | cmd: pushing image "gauravsj/demo-php-app:blue": receiving http error: 400

Psst .. I have already logged in using docker login cmd.

Following is clair container log {"Event":"processing layer","Level":"debug","Location":"worker.go:73","Time":"2017-06-13 09:38:05.758614","engine version":3,"format":"Docker","layer":"sha256:7095154754192bfc2306f3b2b841ef82771b7ad39526537234adb1e74ae81a93","parent layer":"","path":"https://registry-1.docker.io/v2/gauravsj/demo-php-app/blobs/sha256:7095154754192bfc2306f3b2b841ef82771b7ad39526537234adb1e74ae81a93"} {"Event":"could not download layer: expected 2XX","Level":"warning","Location":"driver.go:135","Time":"2017-06-13 09:38:06.811360","status code":401} {"Event":"failed to extract data from path","Level":"error","Location":"worker.go:122","Time":"2017-06-13 09:38:06.811407","error":"could not find layer","layer":"sha256:7095154754192bfc2306f3b2b841ef82771b7ad39526537234adb1e74ae81a93","path":"https://registry-1.docker.io/v2/gauravsj/demo-php-app/blobs/sha256:7095154754192bfc2306f3b2b841ef82771b7ad39526537234adb1e74ae81a93"} {"Event":"Handled HTTP request","Level":"info","Location":"router.go:57","Time":"2017-06-13 09:38:06.811556","elapsed time":1053002203,"method":"POST","remote addr":"172.21.0.3:33576","request uri":"/v1/layers","status":"400"}

pireslaert commented 7 years ago

+1 , having the exact same issue using the clairctl locally outside the container

lcgkm commented 7 years ago

@gauravjos @pireslaert Patch #32 or #52

pireslaert commented 7 years ago

Thanks @lcgkm will hv a go with the latest patch

jgsqware commented 7 years ago

@gauravjos can you run in debug like this:

docker-compose exec clairctl clairctl --log-level debug push gauravsj/demo-php-app:blue

and paste the logs?

can you pull the image too?

docker-compose exec clairctl clairctl --log-level debug pull gauravsj/demo-php-app:blue
jgsqware commented 7 years ago

@gauravjos could you try with the latest 1.2.7 version?

It4lik commented 7 years ago

Hi, I have the same issue. I really don't understand anythoing, from my point of view, this should be really simple to analyze a local image (or an image from a private repo, or a public repo) with CLI tool since you're able to dive into each image layers. I really don't understand what's going on here. What I mean is that, I don't know, such tool is supposed to be very simple to understand/use and I don't even the purpose of the clairctl pull subcommand. Maybe i just didn't find it, but maybe it would be great to explain users how Clairctl is supposed to work ?

Here's my setup :

I used the official docker-compose.yml to setup Clair + Postgres, locally, on my laptop (this is supposed to work isn't it ?) Here is my docker-compose file anyway :

version: '2'
services:
  postgres:
    container_name: clair_postgres
    image: postgres:latest
    restart: unless-stopped
    environment:
      POSTGRES_PASSWORD: password
    networks:
      clair-net:
        aliases:
          - db 
          - db.clair

  clair:
    container_name: clair_clair
    image: quay.io/coreos/clair-git:latest
    restart: unless-stopped
    depends_on:
      - postgres
    ports:
      - "6060-6061:6060-6061"
    links:
      - postgres
    volumes:
      - /tmp:/tmp
      - ./clair_config:/config
    command: [-config, /config/config.yaml]
    networks:
      clair-net:
        aliases:
          - clair
          - clair.clair

networks:
  clair-net:

Here is my ./clair_config:/config.yaml file

# Copyright 2015 clair authors
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

# The values specified here are the default values that Clair uses if no configuration file is specified or if the keys are not defined.
clair:
  database:
    # Database driver
    type: pgsql
    options:
      # PostgreSQL Connection string
      # https://www.postgresql.org/docs/current/static/libpq-connect.html#LIBPQ-CONNSTRING
      source: host=db port=5432 user=postgres sslmode=disable statement_timeout=60000 password=password

      # Number of elements kept in the cache
      # Values unlikely to change (e.g. namespaces) are cached in order to save prevent needless roundtrips to the database.
      cachesize: 16384

  api:
    # API server port
    port: 6060

    # Health server port
    # This is an unencrypted endpoint useful for load balancers to check to healthiness of the clair server.
    healthport: 6061

    # Deadline before an API request will respond with a 503
    timeout: 900s

    # 32-bit URL-safe base64 key used to encrypt pagination tokens
    # If one is not provided, it will be generated.
    # Multiple clair instances in the same cluster need the same value.
    paginationkey:

    # Optional PKI configuration
    # If you want to easily generate client certificates and CAs, try the following projects:
    # https://github.com/coreos/etcd-ca
    # https://github.com/cloudflare/cfssl
    servername:
    cafile:
    keyfile:
    certfile:

  updater:
    # Frequency the database will be updated with vulnerabilities from the default data sources
    # The value 0 disables the updater entirely.
    interval: 2h

  notifier:
    # Number of attempts before the notification is marked as failed to be sent
    attempts: 3

    # Duration before a failed notification is retried
    renotifyinterval: 2h

    http:
      # Optional endpoint that will receive notifications via POST requests
      endpoint:

      # Optional PKI configuration
      # If you want to easily generate client certificates and CAs, try the following projects:
      # https://github.com/cloudflare/cfssl
      # https://github.com/coreos/etcd-ca
      servername:
      cafile:
      keyfile:
      certfile:

      # Optional HTTP Proxy: must be a valid URL (including the scheme).
      proxy:

I tried to build a custom .yml file for clairctl --config flag :

clair:
  port: 6060
  healthPort: 6061
  uri: http://localhost
  report:
    path: ./reports
    format: html

Here is the error, exactly the same as everyone else before :

it4@my-user:~$ clairctl --config ThrashDir/clair_config/config.yaml pull private.reg.url/library/my-company-hello-world:latest

Image: private.reg.url/library/my-company-hello-world:latest
 1 layers found
  ➜ sha256:a5d4636169b4cd5851f3cea8bdc1c0f3e458d939e0572c84d8b27fdb2808ef13

it4@my-user:~$ clairctl --log-level debug --config ThrashDir/clair_config/config.yaml push private.reg.url/library/my-company-hello-world:latest
2017-06-27 14:50:19.890731 D | config: Using config file: ThrashDir/clair_config/config.yaml
2017-06-27 14:50:19.890937 D | dockerdist: Downloading manifest for private.reg.url/library/my-company-hello-world:latest
2017-06-27 14:50:19.891061 D | dockerdist: Retrieving repository client
2017-06-27 14:50:20.231385 D | dockerdist: endpoint.TLSConfig.InsecureSkipVerify: true
2017-06-27 14:50:20.593708 D | dockerdist: manifest type: *schema2.DeserializedManifest
2017-06-27 14:50:20.593725 D | dockerdist: retrieved schema2 manifest, no verification
2017-06-27 14:50:20.593744 I | config: retrieving interface for local IP
2017-06-27 14:50:20.593749 D | config: no interface provided, looking for docker0
2017-06-27 14:50:20.593969 I | clair: Pushing Layer 1/1 [sha256:a5d46]
2017-06-27 14:50:20.598824 D | clair: Saving sha256:a5d4636169b4cd5851f3cea8bdc1c0f3e458d939e0572c84d8b27fdb2808ef13[https://private.reg.url/v2]
2017-06-27 14:50:20.607281 D | clair: auth.insecureSkipVerify: true
2017-06-27 14:50:20.607305 D | clair: request.URL.String(): https://private.reg.url/v2/library/my-company-hello-world/blobs/sha256:a5d4636169b4cd5851f3cea8bdc1c0f3e458d939e0572c84d8b27fdb2808ef13
2017-06-27 14:50:20.683200 I | clair: pull from clair is unauthorized
2017-06-27 14:50:20.831415 I | clair: adding layer 1/1 [sha256:a5d46]: receiving http error: 400
client quit unexpectedly
2017-06-27 14:50:20.831453 C | cmd: pushing image "private.reg.url/library/my-company-hello-world:latest": receiving http error: 400

In the debug log message, why is it trying to use docker0 ?! I never specified it, and I'm not able to change this value with any black-magic-parameter in the .yml file. I saw the corresponding lines in the source code, but am I really supposed to edit the source code to make this work ?

jgsqware commented 7 years ago

Thanks for reporting

Clairctl interact with Clair server. How Clair is working: you send via the API the URL of each layer you want Clair to analyze. Clair download it from your registry.

Clairctl purpose is to send the layers URL for you.

So first the error 400 you received in Clair log have two possibilities:

  1. Clair cannot access your registry.
  2. Clair cannot login to your registry

So, could you ping your private registry from Clair container?

Which version of clairctl are you using? clairctl version

when you use the --local flag, clair should download the layers from your machine. By default, clairctl create a local server on docker0 interface to allow Clair to download the layers.

You can change this interface via the clairctl.yml file

clairctl:
  interface: br-5ae2491b4bb1

So, could you check this and tell us if there is anything else?

Le mar. 27 juin 2017 14:54, It4lik notifications@github.com a écrit :

Hi, I have the same issue. I really don't understand anythoing, from my point of view, this should be really simple to analyze a local image (or an image from a private repo, or a public repo) with CLI tool since you're able to dive into each image layers. I really don't understand what's going on here. What I mean is that, I don't know, such tool is supposed to be very simple to understand/use and I don't even the purpose of the clairctl pull subcommand. Maybe i just didn't find it, but maybe it would be great to explain users how Clairctl is supposed to work ?

Here's my setup :

  • fully-functional private registry with his own cert (it's using TLS), on his own host
  • I can successfully connect to my private registry using docker login command
  • I can successfully push/pull any images to/from my private registry

I used the official docker-compose.yml to setup Clair + Postgres, locally, on my laptop (this is supposed to work isn't it ?) Here is my docker-compose file anyway :

version: '2' services: postgres: container_name: clair_postgres image: postgres:latest restart: unless-stopped environment: POSTGRES_PASSWORD: password networks: clair-net: aliases:

  • db
  • db.clair

    clair: container_name: clair_clair image: quay.io/coreos/clair-git:latest restart: unless-stopped depends_on:

    • postgres ports:
    • "6060-6061:6060-6061" links:
    • postgres volumes:
    • /tmp:/tmp
    • ./clair_config:/config command: [-config, /config/config.yaml] networks: clair-net: aliases:
  • clair
  • clair.clair

networks: clair-net:

Here is my ./clair_config:/config.yaml file

Copyright 2015 clair authors

#

Licensed under the Apache License, Version 2.0 (the "License");

you may not use this file except in compliance with the License.

You may obtain a copy of the License at

#

http://www.apache.org/licenses/LICENSE-2.0

#

Unless required by applicable law or agreed to in writing, software

distributed under the License is distributed on an "AS IS" BASIS,

WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.

See the License for the specific language governing permissions and

limitations under the License.

The values specified here are the default values that Clair uses if no configuration file is specified or if the keys are not defined.

clair: database:

Database driver

type: pgsql
options:
  # PostgreSQL Connection string
  # https://www.postgresql.org/docs/current/static/libpq-connect.html#LIBPQ-CONNSTRING
  source: host=db port=5432 user=postgres sslmode=disable statement_timeout=60000 password=password

  # Number of elements kept in the cache
  # Values unlikely to change (e.g. namespaces) are cached in order to save prevent needless roundtrips to the database.
  cachesize: 16384

api:

API server port

port: 6060

# Health server port
# This is an unencrypted endpoint useful for load balancers to check to healthiness of the clair server.
healthport: 6061

# Deadline before an API request will respond with a 503
timeout: 900s

# 32-bit URL-safe base64 key used to encrypt pagination tokens
# If one is not provided, it will be generated.
# Multiple clair instances in the same cluster need the same value.
paginationkey:

# Optional PKI configuration
# If you want to easily generate client certificates and CAs, try the following projects:
# https://github.com/coreos/etcd-ca
# https://github.com/cloudflare/cfssl
servername:
cafile:
keyfile:
certfile:

updater:

Frequency the database will be updated with vulnerabilities from the default data sources

# The value 0 disables the updater entirely.
interval: 2h

notifier:

Number of attempts before the notification is marked as failed to be sent

attempts: 3

# Duration before a failed notification is retried
renotifyinterval: 2h

http:
  # Optional endpoint that will receive notifications via POST requests
  endpoint:

  # Optional PKI configuration
  # If you want to easily generate client certificates and CAs, try the following projects:
  # https://github.com/cloudflare/cfssl
  # https://github.com/coreos/etcd-ca
  servername:
  cafile:
  keyfile:
  certfile:

  # Optional HTTP Proxy: must be a valid URL (including the scheme).
  proxy:

I tried to build a custom .yml file for clairctl --config flag :

clair: port: 6060 healthPort: 6061 uri: http://localhost

report: path: ./reports format: html

Here is the error, exactly the same as everyone else before :

it4@my-user:~$ clairctl --config ThrashDir/clair_config/config.yaml pull private.reg.url/library/my-company-hello-world:latest

Image: private.reg.url/library/my-company-hello-world:latest 1 layers found ➜ sha256:a5d4636169b4cd5851f3cea8bdc1c0f3e458d939e0572c84d8b27fdb2808ef13

it4@my-user:~$ clairctl --log-level debug --config ThrashDir/clair_config/config.yaml push private.reg.url/library/my-company-hello-world:latest 2017-06-27 14:50:19.890731 D | config: Using config file: ThrashDir/clair_config/config.yaml 2017-06-27 14:50:19.890937 D | dockerdist: Downloading manifest for private.reg.url/library/my-company-hello-world:latest 2017-06-27 14:50:19.891061 D | dockerdist: Retrieving repository client 2017-06-27 14:50:20.231385 D | dockerdist: endpoint.TLSConfig.InsecureSkipVerify: true 2017-06-27 14:50:20.593708 D | dockerdist: manifest type: *schema2.DeserializedManifest 2017-06-27 14:50:20.593725 D | dockerdist: retrieved schema2 manifest, no verification 2017-06-27 14:50:20.593744 I | config: retrieving interface for local IP 2017-06-27 14:50:20.593749 D | config: no interface provided, looking for docker0 2017-06-27 14:50:20.593969 I | clair: Pushing Layer 1/1 [sha256:a5d46] 2017-06-27 14:50:20.598824 D | clair: Saving sha256:a5d4636169b4cd5851f3cea8bdc1c0f3e458d939e0572c84d8b27fdb2808ef13[https://private.reg.url/v2] 2017-06-27 14:50:20.607281 D | clair: auth.insecureSkipVerify: true 2017-06-27 14:50:20.607305 D | clair: request.URL.String(): https://private.reg.url/v2/library/my-company-hello-world/blobs/sha256:a5d4636169b4cd5851f3cea8bdc1c0f3e458d939e0572c84d8b27fdb2808ef13 2017-06-27 https://private.reg.url/v2/library/my-company-hello-world/blobs/sha256:a5d4636169b4cd5851f3cea8bdc1c0f3e458d939e0572c84d8b27fdb2808ef132017-06-27 14:50:20.683200 I | clair: pull from clair is unauthorized 2017-06-27 14:50:20.831415 I | clair: adding layer 1/1 [sha256:a5d46]: receiving http error: 400 client quit unexpectedly 2017-06-27 14:50:20.831453 C | cmd: pushing image "private.reg.url/library/my-company-hello-world:latest": receiving http error: 400

In the debug log message, why is it trying to use docker0 ?! I never specified it, and I'm not able to change this value with any black-magic-parameter in the .yml file. I saw the corresponding lines in the source code, but am I really supposed to edit the source code to make this work ?

— You are receiving this because you commented.

Reply to this email directly, view it on GitHub https://github.com/jgsqware/clairctl/issues/49#issuecomment-311348992, or mute the thread https://github.com/notifications/unsubscribe-auth/ADJrq9wRYd_1d8QzUhEwyo26kr1P_feeks5sIPt3gaJpZM4Nvaqx .

It4lik commented 7 years ago

I'm using the latest Clair version (from your release page) :

it4@my-user:~$ clairctl version

Clairctl version 1.2.8

I can successfully ping my private registry from the Clair container. (no reason I can't, but I understand you asked)

it4@my-user:~$ docker exec clair_clair ping private.reg.url
PING private.reg.url (private-ip): 56 data bytes
64 bytes from private-ip: seq=0 ttl=53 time=17.097 ms
64 bytes from private-ip: seq=1 ttl=53 time=16.940 ms
64 bytes from private-ip: seq=2 ttl=53 time=16.990 ms
^C
## this works

Okay, is there any place in the documentation where you specify the clairctl section in the .yml ? If no I think you should, I tried to add an interface parameter in the "clair" section but no success.

--local flag is not shown in help subcommand, maybe it should ? It just sets up a local Clair server ?

Okay, so there's no "image" concept in clairctl (neither in Clair), it only treats with layers. But we can still push images as a whole group of layers, because this is what users are expected. Am I right ?

I also used docker logs clair_container to see some more logs. I had some x509: certificate signed by unknown authority errors. I used sh in the clair container to add my certificates to the system store. Now I can successfully issue curl commands from within the container, to my https registry, without -k flag (-k is used to disable the cert check).


Back to the problem : still unable to make it work.

I add my interface under clairctl section in the .yml file and it does show up in the log output :)

Without the --local flag, it still fails with 400. If I specify it, it also fails with HTTP 400, but after hanging 3 minutes doing "nothing" See yourself :

jodem commented 7 years ago

Try with sudo, worked for me

shashankkoppar commented 7 years ago

I am too facing the same issue. clairctl can pull layers but while pushing getting 400 error!

./clairctl-linux-amd64 --config /clair-test/clairctl.yml --log-level debug health 2017-07-18 15:10:39.523009 D | config: Using config file: /clair-test/clairctl.yml 2017-07-18 15:10:39.523308 D | clair: requesting health on: http://100.73.100.8:50080/health Clair: ✔

Image: docker.io/library/node:alpine 3 layers found ➜ sha256:88286f41530e93dffd4b964e1db22ce4939fffa4a4c665dab8591fbab03d4926 ➜ sha256:c89e59f636b629afedbd71d08c8e8e177938aa2169cb6c59a0787061e3b9fb88 ➜ sha256:cfb692fa93ee42683ad80015283bbda369fbeb58dc29d6f7fae912efa5c39516

shashankkoppar commented 7 years ago

Please help to resolve the issue! Why clairctl facing issues in pushing when it can pull which means it can access registry right? My clairctl yml file clairctl: port: 44480 tempfolder: /tmp clair: port: 50079 healthPort: 50080 uri: http://100.73.100.8 report: path: ./reports format: html
docker: insecure-registries:

jdel commented 7 years ago

@ShashanKK123, if you want to analyze official images, you need to issue the docker login command. Can you confirm that you have logged in before clairctl analyze ?

shashankkoppar commented 7 years ago

@jdel yes i have executed docker login command and also pulled the docker image as well which i am trying to analyze. By the way , clair is running in kubernetes pod and i am trying to access it from one of my jenkins slave using proxy. Do lemme know if there is anything i have to do?

jdel commented 7 years ago

I have no knowledge of k8s unfortunately. I see clairctl can talk to clair, but can clair reach clairctl ?

shashankkoppar commented 7 years ago

Clair will try to connect to clairctl in localhost right?

jdel commented 7 years ago

clairctl will try to find the best interface and ip address to use:

2017-07-18 15:10:45.316462 I | config: retrieving interface for local IP
2017-07-18 15:10:45.316466 D | config: no interface provided, looking for docker0
2017-07-18 15:10:45.316542 D | config: docker0 not found, looking for first connected broadcast interface

You might need to specify the ip address in your config file.

Also, what's in your clair logs ?

shashankkoppar commented 7 years ago

Sure @jdel lemme try it and get back to you!

shashankkoppar commented 7 years ago
2017-07-20T00:08:32.141972352Z {"Event":"fetching vulnerability updates","Level":"info","Location":"updater.go:228","Time":"2017-07-20 00:08:32.141528"}
2017-07-20T00:08:32.142117596Z {"Event":"Start fetching vulnerabilities","Level":"info","Location":"ubuntu.go:88","Time":"2017-07-20 00:08:32.141738","package":"Ubuntu"}
2017-07-20T00:08:32.142228807Z {"Event":"Start fetching vulnerabilities","Level":"info","Location":"debian.go:63","Time":"2017-07-20 00:08:32.141814","package":"Debian"}
2017-07-20T00:08:32.142273830Z {"Event":"Start fetching vulnerabilities","Level":"info","Location":"oracle.go:119","Time":"2017-07-20 00:08:32.141943","package":"Oracle Linux"}
2017-07-20T00:08:32.142346545Z {"Event":"Start fetching vulnerabilities","Level":"info","Location":"rhel.go:92","Time":"2017-07-20 00:08:32.142031","package":"RHEL"}
2017-07-20T00:08:32.142597806Z {"Event":"Start fetching vulnerabilities","Level":"info","Location":"alpine.go:52","Time":"2017-07-20 00:08:32.142106","package":"Alpine"}
2017-07-20T00:08:33.348392811Z {"Event":"finished fetching","Level":"info","Location":"updater.go:242","Time":"2017-07-20 00:08:33.348029","updater name":"rhel"}
2017-07-20T00:08:34.439981816Z {"Event":"finished fetching","Level":"info","Location":"updater.go:242","Time":"2017-07-20 00:08:34.439545","updater name":"oracle"}
2017-07-20T00:08:34.443682501Z {"Event":"finished fetching","Level":"info","Location":"updater.go:242","Time":"2017-07-20 00:08:34.443351","updater name":"alpine"}
2017-07-20T00:08:41.754402517Z {"Event":"finished fetching","Level":"info","Location":"updater.go:242","Time":"2017-07-20 00:08:41.753894","updater name":"debian"}
2017-07-20T00:15:18.149293937Z {"Event":"could not branch Ubuntu repository","Level":"error","Location":"ubuntu.go:177","Time":"2017-07-20 00:15:18.149031","error":"signal: killed","output":""}
2017-07-20T00:15:18.149403195Z {"Event":"an error occured when fetching update","Level":"error","Location":"updater.go:235","Time":"2017-07-20 00:15:18.149189","error":"could not download requested resource","updater name":"ubuntu"}
2017-07-20T00:15:18.149638886Z {"Event":"adding metadata to vulnerabilities","Level":"info","Location":"updater.go:268","Time":"2017-07-20 00:15:18.149268"}
2017-07-20T00:15:28.223533879Z {"Event":"could not download layer","Level":"warning","Location":"driver.go:129","Time":"2017-07-20 00:15:28.143569","error":"Get https://privateregstorage.blob.core.windows.net/registry//docker/registry/v2/blobs/sha256/a3/a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4/data?se=2017-07-20T00%3A33%3A20Z\u0026sig=GVDWphAdct8j81pDqe7%2B3nBpGWzBNjwftDWg%2FD8LCFc%3D\u0026sp=r\u0026sr=b\u0026sv=2015-02-21: dial tcp 40.115.175.24:443: getsockopt: connection timed out"}
2017-07-20T00:15:28.243769112Z {"Event":"failed to extract data from path","Level":"error","Location":"worker.go:122","Time":"2017-07-20 00:15:28.143776","error":"could not find layer","layer":"sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4","path":"https://dev-zedregistry1101z.dev.jp.local/v2/esd-srg/consul/blobs/sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4"}
2017-07-20T00:15:28.243875170Z {"Event":"Handled HTTP request","Level":"info","Location":"router.go:58","Time":"2017-07-20 00:15:28.144641","elapsed time":127673517959,"method":"POST","remote addr":"10.244.86.1:55982","request uri":"/v1/layers","status":"400"}
2017-07-20T00:15:45.961037790Z {"Event":"could not decode NVD data feed","Level":"error","Location":"nvd.go:93","Time":"2017-07-20 00:15:45.960808","data feed name":"2008","error":"XML syntax error on line 466516: unescaped \u003c inside quoted string"}
2017-07-20T00:15:45.961339295Z {"Event":"an error occured when loading metadata fetcher","Level":"error","Location":"updater.go:289","Time":"2017-07-20 00:15:45.960942","appender name":"NVD","error":"updater/fetchers: could not parse"}
2017-07-20T00:15:45.965216251Z {"Event":"update finished","Level":"info","Location":"updater.go:213","Time":"2017-07-20 00:15:45.964827"}
shashankkoppar commented 7 years ago

clair logs

shashankkoppar commented 7 years ago

And I also wanted to ask is it possible to run clairctl without docker and by just sharing .docker/config.json?

jgsqware commented 7 years ago

You can use clairctl without docker but not for analyzing local images. It has to save the local image to a tar to expose it Clair

Your error 400 is probably that Clair cannot connect to clairctl to download layer

Le jeu. 20 juil. 2017 02:27, H Shashank Koppar notifications@github.com a écrit :

And I also wanted to ask is it possible to run clairctl without docker and by just sharing .docker/config.json?

— You are receiving this because you commented.

Reply to this email directly, view it on GitHub https://github.com/jgsqware/clairctl/issues/49#issuecomment-316561078, or mute the thread https://github.com/notifications/unsubscribe-auth/ADJrqzuuVbxyluR_jhW_lBbGq-qnXOWIks5sPp7VgaJpZM4Nvaqx .

shashankkoppar commented 7 years ago

@jgsqware I got the error now. The problem is clairctl is running in docker container too, so Clair can not be able to connect to it. So I will change my approach to put clairctl in the pod which has Clair!

jgsqware commented 7 years ago

Or you can make a service on k8s to expose a specific port and configure clairctl to use that port as local server

Le jeu. 20 juil. 2017 07:50, H Shashank Koppar notifications@github.com a écrit :

@jgsqware https://github.com/jgsqware I got the error now. Problem is clairctl is running in docker container too, so Clair can not it So I will change my approach to put clairctl in the pod which has Clair!

— You are receiving this because you were mentioned.

Reply to this email directly, view it on GitHub https://github.com/jgsqware/clairctl/issues/49#issuecomment-316603732, or mute the thread https://github.com/notifications/unsubscribe-auth/ADJrq_9-G1gf58-AjrVU65XQXjsDtGB5ks5sPuqlgaJpZM4Nvaqx .

shashankkoppar commented 7 years ago

@jgsqware i am doing it at present port: 50079 healthPort: 50080 uri: http://100.73.100.8 port and healthport are nodeport and 100.73.100.8 is node ip

shashankkoppar commented 7 years ago

but the problem is clair can not connect to clairctl right? clairctl can easilly access the http://100.73.100.8 which is my clair ip

jgsqware commented 7 years ago

You are showing me the Clair nodeport a supposed.

I was talking about clairctl. To avoid to have it the same pod as Clair you can expose a specific port for Clairctl local server

Le jeu. 20 juil. 2017 07:53, H Shashank Koppar notifications@github.com a écrit :

@jgsqware https://github.com/jgsqware i am doing it at present port: 50079 healthPort: 50080 uri: http://100.73.100.8 port and healthport are nodeport and 100.73.100.8 is node ip

— You are receiving this because you were mentioned.

Reply to this email directly, view it on GitHub https://github.com/jgsqware/clairctl/issues/49#issuecomment-316604163, or mute the thread https://github.com/notifications/unsubscribe-auth/ADJrq6W3B6mziy5rzXf89O82cALjo0uGks5sPutwgaJpZM4Nvaqx .

shashankkoppar commented 7 years ago

Ah, I see, sure thanks! I just wanted to check it, so hence putting in the same pod. Later I will separate it and create service as well. But I tried using clairctl in the same pod as Clair It just gets stuck here

/go/src/github.com/coreos/clair # clairctl --config clairctl.yml --log-level debug analyze node 2017-07-20 05:59:41.114647 D | config: Using config file: clairctl.yml 2017-07-20 05:59:41.114827 D | dockerdist: Downloading manifest for node 2017-07-20 05:59:41.114919 D | dockerdist: Retrieving repository client 2017-07-20 05:59:41.115107 D | dockerdist: endpoint.TLSConfig.InsecureSkipVerify: true 2017-07-20 05:59:44.743802 D | dockerdist: manifest type: *schema2.DeserializedManifest 2017-07-20 05:59:44.743846 D | dockerdist: retrieved schema2 manifest, no verification 2017-07-20 05:59:44.743867 I | config: retrieving interface for local IP 2017-07-20 05:59:44.743874 D | config: no interface provided, looking for docker0 2017-07-20 05:59:44.744028 D | config: docker0 not found, looking for first connected broadcast interface 2017-07-20 05:59:44.744241 I | clair: Pushing Layer 1/8 [sha256:9f070] 2017-07-20 05:59:44.744348 D | clair: Saving sha256:9f0706ba7422412cd468804fee456786f88bed94bf9aea6dde2a47f770d19d27[https://registry-1.docker.io/v2] 2017-07-20 05:59:44.744439 D | clair: auth.insecureSkipVerify: true 2017-07-20 05:59:44.744492 D | clair: request.URL.String(): https://registry-1.docker.io/v2/library/node/blobs/sha256:9f0706ba7422412cd468804fee456786f88bed94bf9aea6dde2a47f770d19d27

shashankkoppar commented 7 years ago

@jgsqware Solved this issue, the problem was it was not able to access proxy from environment variables.

https://github.com/coreos/clair/blob/master/ext/imagefmt/driver.go#L123 `

tr := &http.Transport
        Proxy:           http.ProxyFromEnvironment,
    TLSClientConfig: &tls.Config{InsecureSkipVerify: insecureTLS},
}

`

the-nw1-group commented 7 years ago

I also get this - I've added my comments to: https://github.com/coreos/clair/issues/398

jdel commented 7 years ago

Well spotted !

SyCode7 commented 6 years ago

Hi, I am experiencing this same issue right now ! Are there updates or changes. I installed clairctl from the master via the installation link provided i.e curl -L https://raw.githubusercontent.com/jgsqware/clairctl/master/install.sh | sh. Here is my clair config file

clair: database: # Database driver type: pgsql options: ' # PostgreSQL Connection string# https://www.postgresql.org/docs/current/static/libpq-connect.html#LIBPQ-CONNSTRING source: host=localhost port=5432 user=postgres sslmode=disable statement_timeout=60000 # Number of elements kept in the cache # Values unlikely to change (e.g. namespaces) are cached in order to save prevent needless roundtrips to the database. cachesize: 16384 # 32-bit URL-safe base64 key used to encrypt pagination tokens # If one is not provided, it will be generated. # Multiple clair instances in the same cluster need the same value. paginationkey: api: # v3 grpc/RESTful API server address addr: "0.0.0.0:6060" # Health server address # This is an unencrypted endpoint useful for load balancers to check to healthiness of the clair server. healthaddr: "0.0.0.0:6061" # Deadline before an API request will respond with a 503 timeout: 900s`