Any body has the problem of Memory Explosion?

TimDettmers / ConvE

Convolutional 2D Knowledge Graph Embeddings resources

MIT License

675 stars 163 forks source link

Any body has the problem of Memory Explosion? #44

Closed zhangzhaocs closed 5 years ago

zhangzhaocs commented 5 years ago

Hi Tim, When I run this code, the memory usage is getting bigger and finally got an explosion. I was wondering if you have the same problem and what's the solution?

Looking forward to your kind feedback. Thanks, tsingker88

TimDettmers commented 5 years ago

I worked on this problem in the past and could resolve some issues, but it seems that there are still issues with the background loader NumPy pointers that are not released.

Two things that might help in this case is: (1) Use fewer loader_threads for the Streambatcher, (2) play around reducing the samples_per_file variable in the StreamToHDF5 class (I think increasing the samples per file should improve the memory profile, but I am not sure anymore).

Let me know if that helps!

zhangzhaocs commented 5 years ago

Your suggestion really helps. Thank, Tim.

1780041410 commented 5 years ago

How did you solve this problem? Can you email to me?

1780041410 commented 5 years ago

1780041410@qq.com