search

nerfstudio-project / nerfacc

A General NeRF Acceleration Toolbox in PyTorch.
https://www.nerfacc.com/
Other
1.38k stars 113 forks source link

program freezes at nerfacc/grid.py/ray_aabb_intersect #260

Closed IamMohitM closed 11 months ago

IamMohitM commented 11 months ago

I'm trying to implement a nerfstudio compatible Model. I'm using the nerfacc Occupancy grid.

While calling the occupancy_grid.sampling method, the program freezes. Specifically, at function ray_aabb_intersect of in nerfacc/grid.py. From my understanding, it gets stuck in an infinite loop while loading a cpp extention.

I'm using nerfacc 0.5.2. Happy to provide more information if required. Also mentioned in issue #2508 on nerfstudio

hesom commented 11 months ago

You have probably solved it on your own by now, but what helped me with the same problem was just deleting ~/.cache/torch_extensions/

IamMohitM commented 11 months ago

@hesom Thanks a lot! I actually kinda moved on from this issue for a while but deleting that folder solved the issue.

IamMohitM commented 9 months ago

The issue persisted because I had multiple nerfstudio projects (and by extension nerfacc). They were in conflict with the extensions folder and causing the freezing again. For each project, I updated the environment variable TORCH_EXTENSIONS_DIR to a different folder which solved the freezing issue.

juuso-oskari commented 8 months ago

Hi @IamMohitM, would you mind telling in more detail what you did? reinstalling nerfacc, or removing the folder ~/.cache/torch_extensions/, or running "export TORCH_EXTENSIONS_DIR="/home/juuso/miniconda3/envs/nerfstudio/torch_extension" does not remove the issue for me and the run still gets killed at the same point.

IamMohitM commented 8 months ago

By any chance, are you running nerfacc from multiple projects?

I had two projects using nerfacc (different venvs), but they shared the same TORCH_EXTENSIONS_DIR, so I changed the location of the extensions dir to not conflict with each other and that seemed to work.

Particularly, when running each project I set the the export TORCH_EXTENSIONS_DIR=temp/torch_extensions, and it would create this directory relative to the project you are working on.

I ran into this issue, when there was a conflict with two different nerfacc installations accessing the same extension dir. I hope this helps.

juuso-oskari commented 8 months ago

I actually found out that the issue was for me running out of GPU memory, due to instant-ngp-bounded method using a dataparser that does not have downscaling for the images. So it ended up loading the full resolution images from mip-nerf-360 dataset to the GPU which are pretty large. That instant_ngp_dataparser.py sure would make use of downscaling functionality.