Open jinserk opened 4 years ago
One more minor error I found was, when I export the DataConfig
to json, itemsize
info of a DataAttribute
was not exported. Of course I can add it manually.
@jinserk
No, a lot of questions on this project don't bother me. Rather, I am happy to think that this project can be improved.
First question: If there is no information related to the metadata, it means that the save was accidentally broken in the middle.
Therefore, it seems necessary to create a metadata recover function for this case. Or maybe you have forgot datasaver.disconnect
.
Second question : Yes, itemsize
is missed. I'll add this part as soon as possible.
To fixed
metadata.recover
functionitemsize
option also save json file.So, for the first question, please double check that the code was written correctly before modifying this part.
@graykode You're right! I forgot datasaver.disconnect
. Thank you so much! By the way, is this disconnect
not able to be called from DataSaver.__del__()
automatically?
@jinserk Thanks for the great suggestion.
As you suggested, adding it to DataSaver.__del__()
doesn't seem to have any problem in terms of concurrency(multiprocessing). I will reflect on this. Thanks!
@jinserk
The python destructor is not a function that is triggered when the class ends. Therefore, it seems more efficient to manage with python's Context Manager (__enter()__
, __exit__
) :
with DataSaver(...) as datasaver:
datasave(...)
Looks great! I thought that __del__
is called when the instance destructed but it doesn't.. sorry for making you confused!
Hi again,
Sorry for bothering you with several question and bug report, but this looks critical. I made a compressed data bucket and it looks storing well, but when I retrieve the dataset, it has 0 len as follows:
I've checked briefly, and found that the bucket has no
metadata
to read out the meta info of the dataset. Can you fix this error? I have installed the latest master branch code.