[Bug]: Race condition mounting static build files to web/worker containers

KevinMind commented 1 month ago

What happened?

Sometimes when starting web/worker containers we experience a race condition from the static files volume being simultaneously mounted in both containers.

The error occurs in two instances

1) during the docker build while parallel building stages depend on the same build cache mount (pip_dev + pip_prod) 2) during docker compose up, while parallel starting containers depend on the same bind mount (web + worker both depend on data_olympia, data_static_assets, etc)

 Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error mounting "/var/lib/docker/volumes/4ea2bf3bbeca2c0fc0556d30eef0f6a0658efe9640c65066176ca80b3881b3e6/_data" to rootfs at "/data/olympia/static-build": create mount destination for /data/olympia/static-build mount: mkdirat /var/lib/docker/overlay2/17a38c9d8c0bde0b144f811d4425cd748b258d2e39ce6d43722b066f93b8f685/merged/data/olympia/static-build: file exists: unknown

We can avoid this by not mounting to the worker container as it is not needing these files.

What did you expect to happen?

It should not happen

Is there an existing issue for this?

[X] I have searched the existing issues

┆Issue is synchronized with this Jira Task

diox commented 3 weeks ago

This is happening too often to ignore. This is already annoying on PRs, but it also affects deploys sometimes, which creates friction and wastes our time.

diox commented 2 weeks ago

@KevinMind is this a dupe of https://github.com/mozilla/addons/issues/15036 ?

mozilla / addons