search

DeepMotionEditing / deep-motion-editing

An end-to-end library for editing and rendering motion of 3D characters with deep learning [SIGGRAPH 2020]
BSD 2-Clause "Simplified" License
1.58k stars 256 forks source link

memory leak #149

Closed buaaswf closed 3 years ago

buaaswf commented 3 years ago

Hi @PeizhuoLi training style transfer as follows, cd style_transfer/data_proc sh gen_dataset.sh python style_transfer/train.py However, it appears the memory leak.

Have you ever met this?

HalfSummer11 commented 3 years ago

Hi, could you provide us with more details, e.g. the error messages? Based on the current information, my suggestion is to check whether for_try has been set to False in config.py. The default option is True in order to quickly check whether the code can function well. But it will output images at a very high frequency which may flood your disk space (but not the memory, though).

buaaswf commented 3 years ago

I have checked that the code here is the memory leak point. https://github.com/DeepMotionEditing/deep-motion-editing/blob/0efe8831709d15f8a7c58d504b0b4f71d9ee11eb/style_transfer/train.py#L131

HalfSummer11 commented 3 years ago

This is where we write latent space visualizations to the TensorBoard, which should work fine if for_try = False (with for_try = True it will probably write too many images). However, if the problem remains, it's also fine to comment out this line.

miaoYuanyuan commented 3 years ago

@buaaswf @HalfSummer11 Hi, when I comment all the code about get_all_plot. the memory also growing as the train continue. after some epochs, it will break and output error: out of memory