clair-local-scan container fails to scan & crashes: fatal: unable to access 'https://git.launchpad.net/ubuntu-cve-tracker/': The requested URL returned error: 503

[phil2k]

Hi,

Since today (using it since 1 year or more), I've noticed that the clar-scanner (clair-scanner_linux_amd64) is not working anymore, getting the following error:

 $ docker run -d --name clair-db arminc/clair-db:latest
 $ docker run -p 6060:6060 --link clair-db:postgres -d --name clair arminc/clair-local-scan:latest
 $ ./clair-scanner_linux_amd64 --ip 172.17.0.1 registry.access.redhat.com/ubi8/ubi
...
Could not analyze layer: POST to Clair failed Post http://127.0.0.1:6060/v1/layers: dial tcp 127.0.0.1:6060: getsockopt: connection refused

After some diggings, I noticed the container was stopped immediately after the scan with a fatal erorr (which I guess that's why the previous erorr with "connection refused" was shown, maybe after a retry after the container crash):

$ docker logs clair
...
panic: runtime error: slice bounds out of range [25:24

But If look before this error in the container log (even before scanning when the container is up & running), I noticed this error, which might be the root cause of that container crash "panic : runtime error: ...":

$ docker logs clair
...
2021-04-21T09:12:40.4501447Z {"Event":"could not pull ubuntu-cve-tracker repository","Level":"error","Location":"ubuntu.go:174","Time":"2021-04-21 09:12:38.463262","error":"exit status 128","output":"Cloning into '.'...\nfatal: unable to access 'https://git.launchpad.net/ubuntu-cve-tracker/': The requested URL returned error: 503\n"}

When I try to locally do a "git clone on https://git.launchpad.net/ubuntu-cve-tracker/" I get the same error:

git clone https://git.launchpad.net/ubuntu-cve-tracker/
Cloning into 'ubuntu-cve-tracker'...
fatal: unable to access 'https://git.launchpad.net/ubuntu-cve-tracker/': The requested URL returned error: 503

But If I do a web-browse on this one https://git.launchpad.net/ubuntu-cve-tracker/ it works, where also I noticed that there are other way of mirroring this git repository for ubuntu-cve-tracker: git://git.launchpad.net/ubuntu-cve-tracker , which works:

git clone git://git.launchpad.net/ubuntu-cve-tracker
Cloning into 'ubuntu-cve-tracker'...
remote: Enumerating objects: 682122, done.
remote: Counting objects: 100% (682122/682122), done.
remote: Compressing objects: 100% (69081/69081), done.
Receiving objects: 100% (682122/682122), 105.11 MiB | 1.37 MiB/s, done.
remote: Total 682122 (delta 620026), reused 673859 (delta 612686)
Resolving deltas: 100% (620026/620026), done.
Updating files: 100% (39942/39942), done.

Because of this it seems that the scanner is not working (even when I tried with different versions/tags of clair-local-scan). Can this remote repository for ubuntu-cve-tracker be changed to the above one which works, until Ubuntu will fix their issue with the https one? If there's another issue, can you please have a look into it ?

Thank you in advance.

Kind regards, Bogdan Velcea

[phil2k]

This root-cause issue was reported here: https://bugs.launchpad.net/ubuntu-cve-tracker/+bug/1925337

[scottames]

Running into this as well.

A possible workaround could be to add a custom gitconfig replacing https with git

e.g.

[url "git://git.launchpad.net/"]
    insteadOf = https://git.launchpad.net/

And adding it to the container at build:

COPY gitconfig /etc/gitconfig

or manually using the mount flag when running docker

[Muzamri]

Hi,

Before this issue -- have anyone tried deploying this on an air-gapped system? (i.e no internet connection) -- it should work.. right?

PS: I'm also currently having issues with scanning (with a machine that has internet connectivity) -- but I do have it deployed on an air-gapped system in another environment, and when I left it, it has scanned successfully.

I'm starting to doubt myself, haha.

Kind regards.

[liamfit]

Also having this issue. Struggling with the workaround suggested by @scottames. It successfully pulls the repo when mounting a custom gitconfig but then I get a load of could not determine a valid package from criterions messages before it eventually exits with the same error as before, i.e. panic: runtime error: slice bounds out of range

{"Event":"running database migrations","Level":"info","Location":"pgsql.go:216","Time":"2021-04-23 12:00:05.176746"}
{"Event":"database migration ran successfully","Level":"info","Location":"pgsql.go:223","Time":"2021-04-23 12:00:05.179025"}
{"Event":"starting main API","Level":"info","Location":"api.go:52","Time":"2021-04-23 12:00:05.179181","port":6060}
{"Event":"starting health API","Level":"info","Location":"api.go:85","Time":"2021-04-23 12:00:05.179186","port":6061}
{"Event":"notifier service is disabled","Level":"info","Location":"notifier.go:77","Time":"2021-04-23 12:00:05.179217"}
{"Event":"updater service started","Level":"info","Location":"updater.go:83","Time":"2021-04-23 12:00:05.179256","lock identifier":"b08e3199-3ce3-4c24-b11f-b96511369db9"}
{"Event":"updating vulnerabilities","Level":"info","Location":"updater.go:192","Time":"2021-04-23 12:00:05.182020"}
{"Event":"fetching vulnerability updates","Level":"info","Location":"updater.go:239","Time":"2021-04-23 12:00:05.182133"}
{"Event":"Start fetching vulnerabilities","Level":"info","Location":"debian.go:63","Time":"2021-04-23 12:00:05.182300","package":"Debian"}
{"Event":"Start fetching vulnerabilities","Level":"info","Location":"alpine.go:52","Time":"2021-04-23 12:00:05.182328","package":"Alpine"}
{"Event":"Start fetching vulnerabilities","Level":"info","Location":"oracle.go:119","Time":"2021-04-23 12:00:05.182339","package":"Oracle Linux"}
{"Event":"Start fetching vulnerabilities","Level":"info","Location":"ubuntu.go:85","Time":"2021-04-23 12:00:05.182766","package":"Ubuntu"}
{"Event":"Start fetching vulnerabilities","Level":"info","Location":"rhel.go:92","Time":"2021-04-23 12:00:05.182843","package":"RHEL"}
{"Event":"Debian bullseye is not mapped to any version number (eg. Jessie-\u003e8). Please update me.","Level":"warning","Location":"debian.go:134","Time":"2021-04-23 12:00:06.291119"}
{"Event":"finished fetching","Level":"info","Location":"updater.go:253","Time":"2021-04-23 12:00:06.291189","updater name":"debian"}
{"Event":"finished fetching","Level":"info","Location":"updater.go:253","Time":"2021-04-23 12:00:06.694850","updater name":"alpine"}
{"Event":"could not determine a valid package from criterions","Level":"warning","Location":"rhel.go:320","Time":"2021-04-23 12:00:12.069447","criterions":"[{kernel version 0:4.18.0-193.el8 is currently running} {kpatch-patch not installed for 0:4.18.0-193.el8} {kernel version equals 0:4.18.0-193.el8} {Red Hat Enterprise Linux 8 is installed}]"}
{"Event":"could not determine a valid package from criterions","Level":"warning","Location":"rhel.go:320","Time":"2021-04-23 12:00:12.069593","criterions":"[{kernel version 0:4.18.0-193.el8 is set to boot up on next boot} {kpatch-patch not installed for 0:4.18.0-193.el8} {kernel version equals 0:4.18.0-193.el8} {Red Hat Enterprise Linux 8 is installed}]"}
{"Event":"could not determine a valid package from criterions","Level":"warning","Location":"rhel.go:320","Time":"2021-04-23 12:00:13.064990","criterions":"[{kernel version 0:4.18.0-193.el8 is currently running} {kpatch-patch not installed for 0:4.18.0-193.el8} {kernel version equals 0:4.18.0-193.el8} {Red Hat Enterprise Linux 8 is installed}]"}
.
.
.
panic: runtime error: slice bounds out of range

goroutine 40 [running]:
github.com/coreos/clair/ext/vulnsrc/rhel.toFeatureVersions(0xc422173d18, 0x2, 0xc42032c858, 0x1, 0x1, 0xc4203169b0, 0x1, 0x1, 0x160, 0x83dc80, ...)
        /go/src/github.com/coreos/clair/ext/vulnsrc/rhel/rhel.go:292 +0xaf4
github.com/coreos/clair/ext/vulnsrc/rhel.parseRHSA(0x7f14692f0240, 0xc425309530, 0xc425309530, 0x7f14692f0240, 0xc425309530, 0x88ae6e, 0x4)
        /go/src/github.com/coreos/clair/ext/vulnsrc/rhel/rhel.go:182 +0x1c7
github.com/coreos/clair/ext/vulnsrc/rhel.(*updater).Update(0xafdf80, 0x8f02c0, 0xc42015a900, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
        /go/src/github.com/coreos/clair/ext/vulnsrc/rhel/rhel.go:145 +0x6f6
github.com/coreos/clair.fetch.func1(0x8f02c0, 0xc42015a900, 0xc42027e0a0, 0xc4201fe0c0, 0x88b11e, 0x4, 0x8e86e0, 0xafdf80)
        /go/src/github.com/coreos/clair/updater.go:243 +0xa5
created by github.com/coreos/clair.fetch
        /go/src/github.com/coreos/clair/updater.go:242 +0x19d

[ocrawford555]

@fitzplb I think what you pasted above (and perhaps this issue as well) is linked to an open issue on Clair v2.x: https://github.com/quay/clair/issues/1249

[scottames]

@fitzplb, @ocrawford555 is correct. The git issue is merely covering up the real underlying issue with Clair v2 in quay/clair#1249.

[ernest-kr]

We have also started seeing this issue. Any other workarounds ?

[chadlwilson]

Please correct me if I am wrong, however my understanding is that the idea of this project is to be able to run with a DB pre-filled with CVEs, and thus if you are OK with CVEs as stale as the success rate of the regular supposed-to-be-daily DB image build, the updater that is failing here should be unnecessary?

In which case, disabling it should be possible, via custom Dockerfile or equivalent on-start hacks

FROM arminc/clair-local-scan:latest
RUN sed -ri "s/interval: 2h/interval: 0/g" /config/config.yaml

This should theoretically also allow it to work fine on an air-gapped system (although I have not tested).

If the prebuilt DB still has issues with the RHEL updates as mentioned in quay/clair#1249, this may only be an acceptable workaround for those not scanning RHEL-based images.