While one process invokes the store() method and another process tries to read the ExperimentData object (from_file()), there is the possibility that when the _Data object invokes the to_csv method, the object is read from disk.
Because the to_csv method first empties the files and then reads to it, the file can be read empty. This creates an pd.EmptyDataError. Especially when a large number of experiments need to be written.
Possible solution
Work with temporary file creation:
store()
to_csv to .tmp file
Rename the file to .csv
Remove the `.tmp file
from_file()
Check for .tmp files in path. If exists, retry later
The Problem
While one process invokes the
store()
method and another process tries to read theExperimentData
object (from_file()
), there is the possibility that when the_Data
object invokes theto_csv
method, the object is read from disk. Because theto_csv
method first empties the files and then reads to it, the file can be read empty. This creates an pd.EmptyDataError. Especially when a large number of experiments need to be written.Possible solution
Work with temporary file creation:
store()
to_csv
to.tmp
file.csv
from_file()
.tmp
files in path. If exists, retry later