search

harlowja / fasteners

A python package that provides useful locks.
Apache License 2.0
243 stars 45 forks source link

Does InterProcessLock work with S3/Remote filesystem #113

Open BrandonLiang opened 1 month ago

BrandonLiang commented 1 month ago

e.g. (from https://pypi.org/project/fasteners/)

lock = fasteners.InterProcessLock('s3://my-bucket/path/to/lock.file')

Use case: I have an integration test that modifies my S3 bucket (I have teardown setup). I want this integration test to run both locally and in CI/CD, so I want to use a lock to prevent race conditions but the filesystem is different between local and CI/CD.

BrandonLiang commented 1 month ago

I could not find a good existing solution, so created this PR: https://github.com/harlowja/fasteners/pull/114

psarka commented 1 month ago

Thanks Brandon!

It is much more difficult to implement the S3 support. I would like it to be a separate class, use generic methods to authenticate and to be covered by an extensive test suite.

Also, I'm surprised that this approach works, did you try this from two different machines? Does it really lock?

BrandonLiang commented 1 month ago

yes that approach makes sense. Feel free to refactor or close #114 for now

I haven't tested yet but will do shortly. I don't see any reason why it couldn't but will keep you posted