taichi-dev / taichi-nerfs

Implementations of NeRF variants based on Taichi + PyTorch
Apache License 2.0
726 stars 49 forks source link

error: Assertion failure: prealloc_size <= total_mem when I try to train the Lego #38

Open Zdmai opened 1 year ago

Zdmai commented 1 year ago

RTX3050 mobile, Ubuntu 22.04.2 LTS x86_64

I don't know how to solve this problem, I try to change the patch_size into 2048 even 512 in train_nsvf_lego.sh and train.py but it can't work for me.

What can I do next.

Screenshot from 2023-04-20 00-36-17

./scripts/train_nsvf_lego.sh [Taichi] version 1.7.0, llvm 15.0.4, commit 40ba02a3, linux, python 3.10.6 [Taichi] Starting on arch=cuda [E 04/20/23 00:28:48.932 11717] [llvm_runtime_executor.cpp:materialize_runtime@570] Assertion failure: prealloc_size <= total_mem

Traceback (most recent call last): File "/home/zzz/lec/taichi-nerfs/train.py", line 279, in taichi_init(hparams) File "/home/zzz/lec/taichi-nerfs/train.py", line 273, in taichi_init ti.init(**taichi_init_args) File "/home/zzz/.local/lib/python3.10/site-packages/taichi/lang/misc.py", line 455, in init impl.get_runtime().prog.materialize_runtime() RuntimeError: [llvm_runtime_executor.cpp:materialize_runtime@570] Assertion failure: prealloc_size <= total_mem

bobcao3 commented 1 year ago

You would need to find the ti.init statement and then lower the memory allocation number.

Zdmai commented 1 year ago

yes! I solve this (last night, but my roommate have to sleep so I go to sleep too) by modify the

device_memory_GB to 1.0 that is work

image

but it have another error for me!

image