How much GPU memory is needed for training?

[kxhit]

Hi, thanks for the great work!

I'm curious about how much GPU memory is needed for training the models. I didn't find the relevant explanation in the paper. Thanks!

[y22ma]

Curious about this as well. Struggling to train a scene on a 8GB graphics card without running into memory allocation failure. Just wondering what are the knobs we can play with to reduce memory requirements?

[saunair]

Was wondering if there if we could work on a multi-gpu mode / split the voxel training into parts. Most of the time the training fails at the upsampling step

[y22ma]

Curious about this as well. Struggling to train a scene on a 8GB graphics card without running into memory allocation failure. Just wondering what are the knobs we can play with to reduce memory requirements?

Finally got my model to train! Turns out the background layers take up alot of memory and shaving that down allowed me to train a model on limited memory.

[entmike]

Turns out the background layers take up alot of memory and shaving that down allowed me to train a model on limited memory.

Can you share where you made the changes? I get about 5 epochs into training on a 3080 Ti with the sample Truck dataset and then it bombs out with this error:

RuntimeError: CUDA out of memory. Tried to allocate 1.55 GiB (GPU 0; 11.75 GiB total capacity; 6.67 GiB already allocated; 1.33 GiB free; 7.56 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF

EDIT: I think I may have found (one of) my problems... I was using llff.json instead of I think the proper one for Trucks dataset (tnt.json) and then I got a different OOM error but I found the "background_nlayers": 64, entry based on your comment and I changed it from 64 to 32 and it is at least running now...

EDIT 2: The training still fails after an upsample to [640, 640, 640] regardless of what I tweak with the other background parameters. Looking for any other advice for ways to conserve memory. I'd hate to go to something with even more RAM like a 3090...

[neurall]

I was getting out of memory when training lego_real_night_radial and started upsampling to 512 on 3080 with 11g mem. but thx to hint above i noticed there is on top of syn also syn_nv in config dir with bunch of different constants seems like low mem config . i provided --config configs/syn_nv.json and didnt crashed.training done used memory was in that dataset around 7.7g .That being said output was then unusable compared to output rendered from pretrained downloadable checkpoints. seems like currently one definitely needs 3090 or train in google colab ? UPDATE: it didnt crash but output was garbled and unusable due to me probably selecting wrong config file for that dataset UPDATE2: as I am looking at authors previous work Plenoctrees that this code uses the simmilar code and datasets?(lego ficus etc) I noticed on their page actually mentioning https://github.com/sxyu/plenoctree " in this setting, a 24GB GPU is needed for each scene "

[hailangWu-Aval]

Is that anyone in this project can give some suggestions for the question ?

[pjmsullivan]

I am also having out of memory errors, although, I am using a mobile GPU with only 4 GB of capacity for CUDA. By reducing the settings in the config file (as mentioned by @neurall, @entmike, and @y22ma above), I was able to run Plenoxels, although the settings were so low, the output is unusable.

[neurall]

Hi guys. At least for Cool Tank dataset minimum gpu mem for default 10 epochs as in config seems to be 27g ram and 16g gpu which are typically found only in cloud. So my 3080 is sitting idle. But I wanted to play with this so baad.... Plus given how hard it is to get hands on any gpu nowadays. I created colab

BEWARE for now. it will not work on Free colab but seems to work fine on Colab Pro for me. Reason being

As mentioned. svox2 currently seems to need 27g ram ? to train M60 Tank dataset and 16G gpu where it even then just barely fits free colab has just 12g ram and 15g gpu.

https://colab.research.google.com/drive/1SODy_HiP_kkjL5E4IiBM1XZsD1HofElZ

I set colab as editable for anyone to make it better. so please be carefull to keep it working if you do changes.

For large changes perhaps fork it on github https://github.com/neurall/PlenoxelsColab

If hopefully someone more experienced finds a way to tweak it to fit free colab and still have acceptably good output quality that would be great service to community.

Enjoy. Ladislav

[neurall]

meantime better alternative popped up. nvidia just released alternative that doesn't require titan class gpu for good results, trains in seconds or realtime (you can actually see mesh changing as it trains while you can in realtime tweak training parameters, supercool and they have pretty good mesh extractor too ) instead of minutes and runs on pretty much any gpu https://github.com/NVlabs/instant-ngp. that being said. still kudos to original authors.