Make the batch generator thread-safe

uhh-lt / path2vec

Learning to represent shortest paths and other graph-based measures of node similarities with graph embeddings

Apache License 2.0

33 stars 14 forks source link

Make the batch generator thread-safe #21

Closed akutuzov closed 5 years ago

akutuzov commented 5 years ago

https://github.com/keras-team/keras/issues/1638

https://stackoverflow.com/questions/41194726/python-generator-thread-safety-using-keras

alexanderpanchenko commented 5 years ago

should we close this one or some help is needed?

akutuzov commented 5 years ago

It would be nice if someone took this over, I most probably won't have time for that in the nearest weeks, But it should be done.

alexanderpanchenko commented 5 years ago

ok, could you describe the issue and its solution please in a more detailed way please?

akutuzov commented 5 years ago

It seems that the overhead of using thread-safe batch generators will actually destroy the benefits of using multiple workers in this case. Easier to simply increase the batch size. Closing the issue.