While one process invokes the store() method and another process tries to read the ExperimentData object (from_file()), there is the possibility that when the _Data object invokes the to_csv method, the object is read from disk.
Because the to_csv method first empties the files and then reads to it, the file can be read empty. This creates an pd.EmptyDataError. Especially when a large number of experiments need to be written.
Possible solution
Work with temporary file creation:
store()
to_csv to .tmp file
Rename the file to .csv
Remove the `.tmp file
from_file()
Check for .tmp files in path. If exists, retry later
The Problem
While one process invokes the
store()method and another process tries to read theExperimentDataobject (from_file()), there is the possibility that when the_Dataobject invokes theto_csvmethod, the object is read from disk. Because theto_csvmethod first empties the files and then reads to it, the file can be read empty. This creates an pd.EmptyDataError. Especially when a large number of experiments need to be written.Possible solution
Work with temporary file creation:
store()to_csvto.tmpfile.csvfrom_file().tmpfiles in path. If exists, retry later