This PR resolves https://github.com/entropy-lab/entropy/issues/223 by making it safe for multiple different processes on multiple different machines to use InProcessParamStore on a single Entropy ParamStore DB file (params.json). "Safe" here means that no writes are lost due to concurrent access to the DB file and that reads are always consistent with the most recent writes across processes.
The solution implemented here is based on a "lock file" that must be exclusively acquired by a process before the process can read from, or write, to the ParamStore DB file. This is done using the filelock library.
The PR also includes the following changes:
Disables TinyDB's query caching feature which is not safe in a multi-process scenario.
InProcessParamStore ctor now checks out latest commit (if there is one) automatically. As a result users can no longer commit() an empty store.
The checkout() method, when called without arguments, checks out the latest commit (if there is one).
Fixes a bug where __base_commit_id was not set when checkout() was called with commit_num or move_by arguments.
Fixes a bug where list_values() would return an extra DataFrame record for the current value in the store even when the store was not dirty.
This PR resolves https://github.com/entropy-lab/entropy/issues/223 by making it safe for multiple different processes on multiple different machines to use
InProcessParamStore
on a single Entropy ParamStore DB file (params.json
). "Safe" here means that no writes are lost due to concurrent access to the DB file and that reads are always consistent with the most recent writes across processes.The solution implemented here is based on a "lock file" that must be exclusively acquired by a process before the process can read from, or write, to the ParamStore DB file. This is done using the
filelock
library.The PR also includes the following changes:
InProcessParamStore
ctor now checks out latest commit (if there is one) automatically. As a result users can no longercommit()
an empty store.checkout()
method, when called without arguments, checks out the latest commit (if there is one).__base_commit_id
was not set whencheckout()
was called withcommit_num
ormove_by
arguments.list_values()
would return an extraDataFrame
record for the current value in the store even when the store was not dirty.