search

zju3dv / NeuralRecon

Code for "NeuralRecon: Real-Time Coherent 3D Reconstruction from Monocular Video", CVPR 2021 oral
https://zju3dv.github.io/neuralrecon/
Apache License 2.0
2.03k stars 294 forks source link

generate_gt.py memory issue #129

Open ywm5 opened 1 year ago

ywm5 commented 1 year ago

ray.exceptions.RayTaskError(RayOutOfMemoryError): ray::process_with_single_worker() (pid=17640, ip=127.0.0.1) File "python\ray_raylet.pyx", line 623, in ray._raylet.execute_task File "E:\Anaconda\envs\NeuralReconpython39\lib\site-packages\ray_private\memory_monitor.py", line 162, in raise_if_low_memory raise RayOutOfMemoryError( ray._private.memory_monitor.RayOutOfMemoryError: More than 95% of the memory on node yanwm5 is used (15.08 / 15.73 GB). The top 10 memory consumers are:

"I want to know how much memory is generate_gt.py required to run, but I don't have enough 16GB of memory."

royrx93 commented 1 year ago

I set the n_proc, num_workers, and n_gpu to 1 and it seems to solve the OOM problem on my device.

bloodhunt3r commented 1 year ago

Looks like ray has memory leak issue and up till now there's no fix yet:

https://github.com/ray-project/ray/issues/25779

you need to wait for ray-2.5.0 release