search

adobe / aquarium-bait

Aquarium part to build stable and sturdy nested images
Other
3 stars 2 forks source link

Parallel builds download fails trying to write the same files in local cache #34

Closed sparshev closed 1 year ago

sparshev commented 1 year ago

When there is multiple images are building at once - it's really easy to get into classic parallel write lock situation, for example: 2 different ci images are starting to download jre, completes the operation at the same time and tries to move it to cache directory and fails.

Steps to Reproduce

Run as much parallel builds as possible to increase your chances to get write lock

sparshev commented 1 year ago

There is multiple ways to solve that, but the most easy - is pre-caching. So we need just to fill the files cache before starting the build. It also speeds-up the build (no need to wait for download). I think this is the most promising one.

Another option would be to prepare some centralized script which will manage all the downloads from multiple playbooks - but I already can see how this complicates the system, although actually solving the root cause. Maybe some day we will use something like that.

sparshev commented 1 year ago

Looks like fixed with the change, so closing