goldmann / docker-squash

Docker image squashing tool
MIT License
848 stars 109 forks source link

Parallel execution of docker-squash fails randomly #166

Closed LuckyTyagi closed 6 years ago

LuckyTyagi commented 6 years ago

Hi, I've incorporated docker-squash (version 1.0.5) in my code where it runs after completion of docker build. Here; i have 3 Dockerfiles running in parallel and after that 3 docker-squash runs almost in parallel. But this sometimes fails. Mostly it runs successfully, but randomly few times it fails. I've been dealing with the issue since quite sometime but unable to solve as to why this happens. It's very hard to reproduce. If it helps, i have squashed my base image as well and it contains single layer. I'm attaching log snippet here. Let me know if there is anything else that is needed and please help me resolve the issue.

3 squash are running in parallel so logs are jumbled up. Also; i'm trying to extract debug level logs. I'll attach the same as soon as i have them and if that is needed. Squash-error.txt

goldmann commented 6 years ago

The issue is that the Docker daemon is not able to deliver (save) the container image. This is a problem when the daemon is overloaded. It is visible especially on slower machines where you try to save bigger images or in you case, when you try to save multiple images at the same time. The randomness you see is closely related to timing when the save commands occur.

Your specific issue may be related to the storage driver you use, if this is an option, you can play with different storage driver. What's the output of docker info?

Other option I can suggest is to not saving it at the same time or switch to more powerful machine.

LuckyTyagi commented 6 years ago

Thanks a lot for the response, attaching output of 'docker info'. Please revert if that suggests anything. docker-info-output.txt

goldmann commented 6 years ago

You are using a configuration that is not meant to do any serious work:

WARNING: Usage of loopback devices is strongly discouraged for production use. Use `--storage-opt dm.thinpooldev` to specify a custom block storage device.

You should switch to thin pool provisioning, but this requires dedicated pool created for the storage.

I'm closing this issue now since this is a problem with the Docker configuration, not with the tool itself.