search

johannesboyne / gofakes3

A simple fake AWS S3 object storage (used for local test-runs against AWS S3 APIs)
MIT License
361 stars 84 forks source link

Race condition listing objects while deleting #89

Closed samc1213 closed 5 months ago

samc1213 commented 8 months ago

I have some code in production which, in parallel, lists objects in a prefix, and then deletes them. Gofakes3 appears to not work correctly for this use case. See the test here. If run repeatedly, eventually it will fail with gofakes3_test.go:230: All objects should be deleted, got 1 left. Can someone spot where the issue is? s3mem/backend appears to be thread-safe, but maybe the issue is occurring at the bucket page listing level?

johannesboyne commented 8 months ago

Hi @samc1213 interesting case! Will have to dig deeper to fully check where such a race condition could appear. Thanks a lot for adding the test case, that helps a lot for debugging. If you in the meantime will find a fix, feel invited to open a PR!

johannesboyne commented 6 months ago

@samc1213 could you please test whether @rodaine 's patch https://github.com/johannesboyne/gofakes3/pull/90 also fixes the race condition you brought up?

I just tested it on my machine (apple silicon) and your test case runs without any issue now. Even with 20,000 files! If so, happy to integrate your Test-Case into the main branch!

johannesboyne commented 5 months ago

Please re-open if it still exists.