search

autonomousvision / sdfstudio

A Unified Framework for Surface Reconstruction
Apache License 2.0
1.96k stars 183 forks source link

Memory leakage problem in dataloader #233

Open nickle-fang opened 11 months ago

nickle-fang commented 11 months ago

Describe the bug I use my own dataset which has a big image set like about 1k-2k images. The format is mip-nerf-360. The image set is too big so that I have to use the num_images_to_sample_from and num_times_to_repeat_images params to reduce the memory usage. But I found the dataloader may have a memory leakage problem because everytime the iter time reach the num_times_to_repeat_images, the memory usage increases about 4-8GB. Even I use a server with 128GB memory, it will easily get memory overflow problem and the program is killed before the training process is done.

Could you give some suggestions about this issues? Thanks!

soul1996 commented 8 months ago

I'm having the same problem, is there a solution please? Thanks

smil1546ekimbang2 commented 8 months ago

same problem +1