qiuqiao
commented
6 months ago 谢谢，看起来这个解决方法是可行的，如果你有时间开发的话可以提交PR，或者我也可以过几天再修一下这个问题。
Thank you for the suggestion, it seems to be a viable solution. If you have the time to work on it, feel free to submit a PR; otherwise, I might get around to fixing this issue myself in a few days.
When setting
dataloader_workersto a value other than 0, the program raisesThe reason is because Python uses pickle to transfer objects between processes, and the h5py object in dataset instances cannot be pickled. The direct cause of this problem is that the d5py object is created before the worker processes are created.
To be specific, the h5py object is created when calling
get_label_types()here on the main process:https://github.com/qiuqiao/SOFA/blob/51a869d19cfadeb7128bdf6e8689d13e73344116/train.py#L71-L78
One possible solution to this is to close and release the h5py objects after accessing metadata like this, so that the main process open the HDF5 file and get the metadata, then closes it before spawning into worker processes to avoid pickling and transferring it; the worker process opens it, gets data items from it and keeps it open.
If you think the solution above is okay, I can submit a PR to solve this problem.