Docker exception during scan

[ybznek]

I have created ticket from pull request https://github.com/OpenSCAP/openscap-daemon/pull/57 (some comments are there) " Deamon should not die on an exception in docker client.

This exception was observed during atomic scan --images on Fedora23.

Addressing:

Exception in thread ed695b131d039f3608c0da4335572f2e1a0ef490b9b67664d046cd6c835655de:
Traceback (most recent call last):
File "/usr/lib/python3.4/site-packages/docker/client.py", line 138, in _raise_for_status
response.raise_for_status()
File "/usr/lib/python3.4/site-packages/requests/models.py", line 840, in raise_for_status
raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: http+docker://localunixsocket/v1.20/containers/012d2e837331e17768b5f2df5
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib64/python3.4/threading.py", line 920, in _bootstrap_inner
self.run()
File "/usr/lib64/python3.4/threading.py", line 868, in run
self._target(*self._args, **self._kwargs)
File "/usr/lib/python3.4/site-packages/openscap_daemon/cve_scanner/cve_scanner.py", line 280, in search_containers
f = Scan(image, cids, output, self.ac)
File "/usr/lib/python3.4/site-packages/openscap_daemon/cve_scanner/scan.py", line 53, in __init__
self.dm_results = self.DM.mount(image_uuid)
File "/usr/lib/python3.4/site-packages/Atomic/mount.py", line 302, in mount
cid, dev_name = self._get_cid_from_mountpoint(self.mountpoint)
File "/usr/lib/python3.4/site-packages/Atomic/mount.py", line 494, in _get_cid_from_mountpoint
graph = self.client.inspect_container(c)["GraphDriver"]
File "/usr/lib/python3.4/site-packages/docker/utils/decorators.py", line 21, in wrapped
return f(self, resource_id, *args, **kwargs)
File "/usr/lib/python3.4/site-packages/docker/api/container.py", line 172, in inspect_container
self._get(self._url("/containers/{0}/json", container)), True
File "/usr/lib/python3.4/site-packages/docker/client.py", line 146, in _result
self._raise_for_status(response)
File "/usr/lib/python3.4/site-packages/docker/client.py", line 141, in _raise_for_status
raise errors.NotFound(e, response, explanation=explanation)
docker.errors.NotFound: 404 Client Error: Not Found ("b'no such id: 012d2e837331e17768b5f2df593a2a4471f8d5a87e7de35391eba2d35e6f4b24'")

"

[ybznek]

I have tried to add some synchronization lock to search_containers method which allows only 1 run at time and problem doesn't occur during my testing.

But I have created simple python script to ensure that there is some race-condition problem with atomic mount. The script run simultaneously several mount/unmount but problem doesn't occur.

self.DM = DockerMount("/tmp", mnt_mkdir=True)
self.path = self.DM.mount(self.id)

I don't know what can cause the problem. It is hard to test some problem which sometimes occurs sometime not.

[isimluk]

There is a race condition.

This is only happening because of _get_cid_from_mountpoint. Basically when mounting the image, we ask for all the containers, and then ask whether some of these containers haven't been already mounted to the location. I think we can avoid that race condition, simply by not asking this question.

This was introduced by @rhatdan, when working on docker-1.10 atomic unmount. See https://github.com/projectatomic/atomic/commit/68ae3bcc6f2e622283db9fbdbe1a896e477f2113.

We didn't need this function call during the atomic mount before. It is perhaps needed only for the atomic unmount. I think @jan-cerny has been testing atomic with this reverted hunk with docker-1.10, but it fails on som many other things.

@rhatdan @baude, do you think we can revert that single hunk starting at https://github.com/projectatomic/atomic/commit/68ae3bcc6f2e622283db9fbdbe1a896e477f2113#diff-ce5473cd96a720a53d0834698c9b2eaaL302 ?

Thanks!

[ybznek]

We have merged workaround with synchronization lock for mount/unmount, until we don't have better solution. https://github.com/OpenSCAP/openscap-daemon/pull/60

[jan-cerny]

Should we close this issue because has been fixed in https://github.com/OpenSCAP/openscap-daemon/pull/60 ? Or do we want to keep this open as a reminder of the workaround? I suggest closing this issue and opening a new issue with title eg. "Remove workaround locks".

[ybznek]

+1 for new opened issue

[mpreisler]

Agreed, feel free to do that. You should have necessary rights, ping me otherwise.

[jan-cerny]

OK. opened new issue https://github.com/OpenSCAP/openscap-daemon/issues/78. Closing this.