search

DeepGraphLearning / graphvite

GraphVite: A General and High-performance Graph Embedding System
https://graphvite.io
Apache License 2.0
1.22k stars 151 forks source link

Memory OOM saving model. #35

Open llamazing opened 4 years ago

llamazing commented 4 years ago

It costs about 130G memory training model but when to save model it occurs oom error(total memory 195G). Is there extra memory allocation when saving the model?

KiddoZhu commented 4 years ago

There shouldn't be large memory allocation. Maybe it is caused by pickle?

We will check the memory usage later.

llamazing commented 4 years ago

There shouldn't be large memory allocation. Maybe it is caused by pickle?

We will check the memory usage later.

It is caused by pickle. I replace joblib with pickle. The joblib uses less memory when dumps model but it costs more disk capacity.