Do not fail when cleanup after squashing is not possible

[spolti]

Traceback (most recent call last):
  File "/usr/lib/python3.7/site-packages/docker/api/client.py", line 229, in _raise_for_status
    response.raise_for_status()
  File "/usr/lib/python3.7/site-packages/requests/models.py", line 940, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 409 Client Error: Conflict for url: http+docker://localhost/v1.22/images/sha256:f210100c96aaa86d6ee7b74ca7b760e6dae1804b79839aa390cdb501a0538969?force=False&noprune=False

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/bin/cekit", line 11, in <module>
    load_entry_point('cekit', 'console_scripts', 'cekit')()
  File "/usr/local/lib64/python3.7/site-packages/click/core.py", line 764, in __call__
    return self.main(*args, **kwargs)
...
docker.errors.APIError: 409 Client Error: Conflict ("b'conflict: unable to delete f210100c96aa (must be forced) - image is referenced in multiple repositories'")
make: *** [Makefile:9: build] Error 1

This issue can be reproduced with this project: https://github.com/spolti/quarkus-images/tree/quarkus-cekit-poc But, remove the manual docker-squash command and the no-squash flag on the Makefile.

[goldmann]

Full stacktrace:

Traceback (most recent call last):
  File "/usr/lib/python3.7/site-packages/docker/api/client.py", line 229, in _raise_for_status
    response.raise_for_status()
  File "/usr/lib/python3.7/site-packages/requests/models.py", line 940, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 409 Client Error: Conflict for url: http+docker://localhost/v1.22/images/sha256:f210100c96aaa86d6ee7b74ca7b760e6dae1804b79839aa390cdb501a0538969?force=False&noprune=False

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/bin/cekit", line 11, in <module>
    load_entry_point('cekit', 'console_scripts', 'cekit')()
  File "/usr/local/lib64/python3.7/site-packages/click/core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib64/python3.7/site-packages/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib64/python3.7/site-packages/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib64/python3.7/site-packages/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib64/python3.7/site-packages/click/core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib64/python3.7/site-packages/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib64/python3.7/site-packages/click/decorators.py", line 17, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/data/dev/sources/cekit/cekit/cli.py", line 106, in build_docker
    run_build(ctx, 'docker')
  File "/data/dev/sources/cekit/cekit/cli.py", line 282, in run_build
    run_command(ctx, builder_impl)
  File "/data/dev/sources/cekit/cekit/cli.py", line 251, in run_command
    Cekit(common_params).run(clazz, params)
  File "/data/dev/sources/cekit/cekit/cli.py", line 341, in run
    command.execute()
  File "/data/dev/sources/cekit/cekit/builder.py", line 60, in execute
    self.run()
  File "/data/dev/sources/cekit/cekit/builders/docker_builder.py", line 186, in run
    image_id = self._squash(docker_client, image_id)
  File "/data/dev/sources/cekit/cekit/builders/docker_builder.py", line 160, in _squash
    return squash.run()
  File "/usr/lib/python3.7/site-packages/docker_squash/squash.py", line 59, in run
    return self.squash(image)
  File "/usr/lib/python3.7/site-packages/docker_squash/squash.py", line 102, in squash
    self._cleanup()
  File "/usr/lib/python3.7/site-packages/docker_squash/squash.py", line 77, in _cleanup
    self.docker.remove_image(image_id, force=False, noprune=False)
  File "/usr/lib/python3.7/site-packages/docker/utils/decorators.py", line 19, in wrapped
    return f(self, resource_id, *args, **kwargs)
  File "/usr/lib/python3.7/site-packages/docker/api/image.py", line 480, in remove_image
    return self._result(res, True)
  File "/usr/lib/python3.7/site-packages/docker/api/client.py", line 235, in _result
    self._raise_for_status(response)
  File "/usr/lib/python3.7/site-packages/docker/api/client.py", line 231, in _raise_for_status
    raise create_api_error_from_http_exception(e)
  File "/usr/lib/python3.7/site-packages/docker/errors.py", line 31, in create_api_error_from_http_exception
    raise cls(e, response=response, explanation=explanation)
docker.errors.APIError: 409 Client Error: Conflict ("b'conflict: unable to delete f210100c96aa (must be forced) - image is referenced in multiple repositories'")
make: *** [Makefile:9: build] Error 1

[goldmann]

I could not reproduce the issue with the provided code.

I suspect that my intial investigation as mentioned in https://github.com/goldmann/docker-squash/issues/181#issuecomment-491197302 is true. Here is it re-pasted:

I'll review that one later, although I think this is, kind of, expected in cases where the image to squash is tagged multiple times. This can happen if you build the image, for example using the same source, multiple times and some tags were already assigned to it. Then you run it again, cached image is used and we are going to squash it. In a clean run, this issue should not exist, because we operate on image ID's and tag only when the process is successful.

The issue we're seeing here is related to an error received from the Docker library. Since this is a cleanup process, we should not fail squashing here and instead a warning message should be logged and the process should be continued (in fact ended, because cleanup is the last stage of squashing). I propose to rename this issue and implement handling errors properly in such cases.

[spolti]

Alright, I'll test that again with the updated docker-squash and if the issue happens again, I'll let you know.

thanks.