Segmentation fault when I try to extract geometry in text-to-3D scene-level generation setting.

[EchoTHChen]

I'm trying to extract the geometry of the generated result when i run the script:

model_name="a_bedroom_with_a_king-size_bed_and_large_wardrobes@20230623-162020"
ckpt_model_path='/group/30042/ozhengchen/aigc/threestudio/outputs/prolificdreamer/'$model_name/ckpts/epoch=0-step=25000.ckpt

python launch.py --config configs/prolificdreamer-geometry.yaml --train --gpu 0 system.prompt_processor.prompt="a bedroom with a king-size bed and large wardrobes" system.geometry_convert_from=$ckpt_model_path

The log shown in terminal is:

Global seed set to 0
[INFO] Initializing geometry from a given checkpoint ...
threestudio/scripts/run_prolificdreamer_bedroom2_geometry.sh: line 5: 27604 Segmentation fault      python launch.py --config configs/prolificdreamer-geometry.yaml --train --gpu 0 system.prompt_processor.prompt="a bedroom with a king-size bed and large wardrobes" system.geometry_convert_from=$ckpt_model_path

I guess that the room-level mesh extraction may take a lot of memory (RAM or VRAM). But I don't know how to solve it. Do you have any idea?

[thuliu-yt16]

Do you run the first stage with prolificdreamer-scene.yaml? I think for this setting, the geometry could be very bad because the camera itself has a fixed position during optimization, even if the visualization seems good.

[bennyguo]

You may need to set system.model.radius in prolificdreamer-geometry.yaml to be the same as prolificdreamer-scene.yaml.

[EchoTHChen]

You may need to set system.model.radius in prolificdreamer-geometry.yaml to be the same as prolificdreamer-scene.yaml.

Segmentation fault happened again when i set the system.geometry.radius=5 to be the same as prolificdreamer-scene.yaml.

[EchoTHChen]

Do you run the first stage with prolificdreamer-scene.yaml? I think for this setting, the geometry could be very bad because the camera itself has a fixed position during optimization, even if the visualization seems good.

Yes. I run the first stage with prolificdreamer-scene.yaml.

[bennyguo]

Maybe you need to debug line by line to see which line caused the segmentation fault in the create_from function in models/geometry/tetrahedra_sdf_grad.py. I've successfully extracted the mesh from a scene using system.geometry.radius=5 although it's pretty bad:

NeRF:

Extracted mesh:

[EchoTHChen]

Hi! After I debug line by line. I found that it occurred in the function initialize_context of threestudio.utils.rasterize.py. If I initialize the context with the type of 'gl'. Line 16: dr.RasterizeGLContext(device=device) will throw the error:Segmentation fault. If I use the context type of 'cuda', the segmentation fault disappeared. But the geometry extracted is bad like this: Almost nothing is extracted. I wonder how to avoid this problem.

The rendered result in first stage trained by NeRF looks like: (Now, I'm using the text prompt:"Inside of a smart home, realistic detailed photo, 4k")

[bennyguo]

Are you able to run the fantasia3d pipeline correctly?

[EchoTHChen]

Are you able to run the fantasia3d pipeline correctly?

It seems that Fantasia3D works correctly when extracting the geometry. ("gl"context type in my environment still caused Segmentation Fault. So I used "cuda" context type instead):

[bennyguo]

Did you try a lower system.geometry_convert_override.isosurface_threshold (default to 25) when extracting the scene geometry?

[EchoTHChen]

The result turns out much better. But a black hole still exists. (Now I set system.geometry_convert_override.isosurface_threshold=10)

[EchoTHChen]

I set system.geometry_convert_override.isosurface_threshold lower (such as 5). The hole disappeared completely.