search

graykode / matorage

Matorage is tensor(multidimensional matrix) object storage manager for deep learning framework(Pytorch, Tensorflow V2, Keras)
https://matorage.readthedocs.io
Other
73 stars 8 forks source link

When there are many cores during multi-processing, the metadata and the actually uploaded file are not synced. #36

Open graykode opened 3 years ago

graykode commented 3 years ago

I will write test code related to this and conduct experiments.

To implement it correctly, you should check https://github.com/graykode/matorage/blob/master/matorage/uploader.py#L41 to see if the upload is a success(200) code and put it in metadata.json only when it succeeds.