aiidateam / disk-objectstore

An implementation of an efficient "object store" (actually, a key-value store) writing files on disk and not requiring a running server
https://disk-objectstore.readthedocs.io
MIT License
15 stars 8 forks source link

Fix concurrency problems on Windows #77

Closed giovannipizzi closed 4 years ago

giovannipizzi commented 4 years ago

Essentially, one of the main issues is that os.rename is atomic on Windows, but while renaming, the file is locked. So there is the possibility that any open statement will fail.

We now create a new concept of a duplicates folder, where we store e.g. a new loose object (one copy per failed attempt, with name <hashkey>.<random-uuid>) when we fail to create the loose object (typically during concurrent access) and we cannot check if the current existing object is correct.

Tests with mocking are added to have full coverage (if run on Windows as well).

After this is merged, #72 still needs to be addressed to clean up these copies. Also this folder should be considered in #13 when validating containers.

Fixes #37

codecov[bot] commented 4 years ago

Codecov Report

Merging #77 into develop will not change coverage. The diff coverage is 100.00%.

Impacted file tree graph

@@            Coverage Diff            @@
##           develop       #77   +/-   ##
=========================================
  Coverage   100.00%   100.00%           
=========================================
  Files            7         7           
  Lines          987      1010   +23     
=========================================
+ Hits           987      1010   +23     
Impacted Files Coverage Δ
disk_objectstore/exceptions.py 100.00% <ø> (ø)
disk_objectstore/container.py 100.00% <100.00%> (ø)
disk_objectstore/utils.py 100.00% <100.00%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more Δ = absolute <relative> (impact), ø = not affected, ? = missing data Powered by Codecov. Last update 2815887...3ee29ea. Read the comment docs.