deepseek-ai / DreamCraft3D

[ICLR 2024] Official implementation of DreamCraft3D: Hierarchical 3D Generation with Bootstrapped Diffusion Prior
https://mrtornado24.github.io/DreamCraft3D/
MIT License
1.88k stars 81 forks source link

Process killed when trying to load Stable Zero123 #42

Closed Inkapa closed 5 months ago

Inkapa commented 5 months ago

Hello, I've just set up dreamcraft3d with WSL, but when running the first command of the first Stage (NeRF & NeuS) the process gets killed after trying to load Stable Zero123 with no traceback or errors. I've tried this with a custom image and the example mushroom_log_rgba.png

I'm running it from WSL (Ubuntu 22.04.3 LTS) with torch 2.2.0 and CUDA 12.3, GPU is an RTX 4090

(venv) bebop@DESKTOP:~/DreamCraft3D$ python launch.py --config configs/dreamcraft3d-coarse-nerf.yaml --train system.prompt_processor.prompt="$prompt" data.image_path="$image_path"

/home/bebop/DreamCraft3D/venv/lib/python3.10/site-packages/controlnet_aux/segment_anything/modeling/tiny_vit_sam.py:654: UserWarning: Overwriting tiny_vit_5m_224 in registry with controlnet_aux.segment_anything.modeling.tiny_vit_sam.tiny_vit_5m_224. This is because the name being registered conflicts with an existing name. Please check if this is not expected.
  return register_model(fn_wrapper)
/home/bebop/DreamCraft3D/venv/lib/python3.10/site-packages/controlnet_aux/segment_anything/modeling/tiny_vit_sam.py:654: UserWarning: Overwriting tiny_vit_11m_224 in registry with controlnet_aux.segment_anything.modeling.tiny_vit_sam.tiny_vit_11m_224. This is because the name being registered conflicts with an existing name. Please check if this is not expected.
  return register_model(fn_wrapper)
/home/bebop/DreamCraft3D/venv/lib/python3.10/site-packages/controlnet_aux/segment_anything/modeling/tiny_vit_sam.py:654: UserWarning: Overwriting tiny_vit_21m_224 in registry with controlnet_aux.segment_anything.modeling.tiny_vit_sam.tiny_vit_21m_224. This is because the name being registered conflicts with an existing name. Please check if this is not expected.
  return register_model(fn_wrapper)
/home/bebop/DreamCraft3D/venv/lib/python3.10/site-packages/controlnet_aux/segment_anything/modeling/tiny_vit_sam.py:654: UserWarning: Overwriting tiny_vit_21m_384 in registry with controlnet_aux.segment_anything.modeling.tiny_vit_sam.tiny_vit_21m_384. This is because the name being registered conflicts with an existing name. Please check if this is not expected.
  return register_model(fn_wrapper)
/home/bebop/DreamCraft3D/venv/lib/python3.10/site-packages/controlnet_aux/segment_anything/modeling/tiny_vit_sam.py:654: UserWarning: Overwriting tiny_vit_21m_512 in registry with controlnet_aux.segment_anything.modeling.tiny_vit_sam.tiny_vit_21m_512. This is because the name being registered conflicts with an existing name. Please check if this is not expected.
  return register_model(fn_wrapper)
Seed set to 0
[INFO] Loading Deep Floyd ...

A mixture of fp16 and non-fp16 filenames will be loaded.
Loaded fp16 filenames:
[safety_checker/model.fp16.safetensors, text_encoder/model.fp16-00002-of-00002.safetensors, unet/diffusion_pytorch_model.fp16.safetensors, text_encoder/model.fp16-00001-of-00002.safetensors]
Loaded non-fp16 filenames:
[watermarker/diffusion_pytorch_model.safetensors
If this behavior is not expected, please check your folder structure.
Loading pipeline components...: 100%|█████████████████████████████████████| 3/3 [00:00<00:00, 13.94it/s]
[INFO] Loaded Deep Floyd!
[INFO] Loading Stable Zero123 ...
LatentDiffusion: Running in eps-prediction mode
DiffusionWrapper has 859.53 M params.
Keeping EMAs of 688.
making attention of type 'vanilla' with 512 in_channels
Working with z of shape (1, 4, 32, 32) = 4096 dimensions.
making attention of type 'vanilla' with 512 in_channels
Killed

The same issue arises whether I use stable_zero123.ckpt that I got from https://huggingface.co/stabilityai/stable-zero123 or zero123-xl.ckpt that is downloaded by the load/zero123/download.sh script

Inkapa commented 5 months ago

Managed to get Stable Zero123 to load after increasing WSL2's swap amount through .wslconfig (cf: https://learn.microsoft.com/en-us/windows/wsl/wsl-config#wslconfig) and using the data.height=128 data.width=128 data.random_camera.height=128 data.random_camera.width=128 arguments

fantasy-fish commented 1 month ago

May I know what the memory size is on your machine?