criteo / cluster-pack

A library on top of either pex or conda-pack to make your Python code easily available on a cluster
Apache License 2.0
45 stars 21 forks source link

Fix race condition #94

Closed j-rf closed 1 year ago

j-rf commented 2 years ago

Issue - two running process on the same user could copy their pex to the same place Fix - put operation is slow, but remove and rename/mv are fast: Since the expected behavior is to overwrite the destination with a newer pex, and move cannot overwrite:

Why we do not overwrite after the put at rename/mv time if the destination exists (race condition is met):