search

chainer / chainermn

ChainerMN: Scalable distributed deep learning with Chainer
https://chainer.org
MIT License
207 stars 57 forks source link

pickle and mpi error.. using chainercv with chainermn #88

Closed MannyKayy closed 7 years ago

MannyKayy commented 7 years ago

Hi,

I am trying to use chainercv with chainermn.

I used chainercv with some of my new projects and when i attempt to distribute training using chainermn, I receive the following error from the scatter_dataset method. All i am doing is applying a random_flip transform to the training data. I get the error for all my projects that use chainercv and have replicated it using the chainermn mnist example file .

image

I'm not sure as to where to raise this issue so I have raised it in both the chainercv and chainermn repos.

iwiwi commented 7 years ago

What is the type of your dataset? (i.e., train). Method scatter_dataset requires it to be picklable.

MannyKayy commented 7 years ago

@iwiwi : so according to @yuyu2172 in the related issue on chainercv.

The error occurred because function objects can not be pickled. We will look into a way to fix this issue. Thank you very much for reporting.

As can be seen on line 91 in this gist the chainercv TransformDataset and transform functions wrap the dataset and carry out online data-augmentation. It seems they can't be pickled.

iwiwi commented 7 years ago

I see. Let me close this issue here, and let us continue the discussion at ChainerCV repository. Thanks!