search

deepseek-ai / DreamCraft3D

[ICLR 2024] Official implementation of DreamCraft3D: Hierarchical 3D Generation with Bootstrapped Diffusion Prior
https://mrtornado24.github.io/DreamCraft3D/
MIT License
1.88k stars 81 forks source link

Unclear installation steps + run error #16

Open Iliceth opened 6 months ago

Iliceth commented 6 months ago

Hi, I have followed the steps provided and installed threestudio, which went fine.

But then, it is unclear if I have to git clone this repo and run from inside that directory, which results in missing yaml-files, or should overwrite the files in the threestudio directory with this repo's files, because when I only get clone and then run, the load-directory is missing, so yeah. :-/

My command is:

python launch.py --config configs/dreamcraft3d-coarse-nerf.yaml --train system.prompt_processor.prompt="young woman, scarf, white marble, bust, statue, victorian" data.image_path="load/images/Z1.png"

I tried with your repo and copied the load-directory from threestudio. This resulted in this error:

Seed set to 0
[INFO] Loading Deep Floyd ...
unet\diffusion_pytorch_model.fp16.safetensors not found

A mixture of fp16 and non-fp16 filenames will be loaded.
Loaded fp16 filenames:
[text_encoder/pytorch_model.fp16-00002-of-00002.bin, safety_checker/pytorch_model.fp16.bin, text_encoder/pytorch_model.fp16-00001-of-00002.bin, unet/diffusion_pytorch_model.fp16.bin]
Loaded non-fp16 filenames:
[watermarker/diffusion_pytorch_model.bin
If this behavior is not expected, please check your folder structure.
Loading pipeline components...: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 3/3 [00:07<00:00,  2.53s/it]
[INFO] Loaded Deep Floyd!
[INFO] Loading Stable Zero123 ...
LatentDiffusion: Running in eps-prediction mode
DiffusionWrapper has 859.53 M params.
Keeping EMAs of 688.
making attention of type 'vanilla' with 512 in_channels
Working with z of shape (1, 4, 32, 32) = 4096 dimensions.
making attention of type 'vanilla' with 512 in_channels
Traceback (most recent call last):
  File "C:\Users\Andy\Documents\DreamCraft3D\launch.py", line 252, in <module>
    main(args, extras)
  File "C:\Users\Andy\Documents\DreamCraft3D\launch.py", line 120, in main
    system: BaseSystem = threestudio.find(cfg.system_type)(
  File "C:\Users\Andy\Documents\DreamCraft3D\threestudio\systems\base.py", line 45, in __init__
    self.configure()
  File "C:\Users\Andy\Documents\DreamCraft3D\threestudio\systems\dreamcraft3d.py", line 45, in configure
    self.guidance_3d = threestudio.find(self.cfg.guidance_3d_type)(
  File "C:\Users\Andy\Documents\DreamCraft3D\threestudio\utils\base.py", line 83, in __init__
    self.configure(*args, **kwargs)
  File "C:\Users\Andy\Documents\DreamCraft3D\threestudio\models\guidance\stable_zero123_guidance.py", line 137, in configure
    self.prepare_embeddings(self.cfg.cond_image_path)
  File "C:\Users\Andy\Anaconda3\envs\dreamcraft3d\lib\site-packages\torch\amp\autocast_mode.py", line 16, in decorate_autocast
    return func(*args, **kwargs)
  File "C:\Users\Andy\Documents\DreamCraft3D\threestudio\models\guidance\stable_zero123_guidance.py", line 149, in prepare_embeddings
    assert os.path.exists(image_path)
AssertionError

If I go for a version in which I overwrite the original threestudio with your repo's files, I get this error instead:

Seed set to 0
[INFO] Loading Deep Floyd ...
unet\diffusion_pytorch_model.fp16.safetensors not found

A mixture of fp16 and non-fp16 filenames will be loaded.
Loaded fp16 filenames:
[text_encoder/pytorch_model.fp16-00001-of-00002.bin, safety_checker/pytorch_model.fp16.bin, text_encoder/pytorch_model.fp16-00002-of-00002.bin, unet/diffusion_pytorch_model.fp16.bin]
Loaded non-fp16 filenames:
[watermarker/diffusion_pytorch_model.bin
If this behavior is not expected, please check your folder structure.
Loading pipeline components...: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 3/3 [00:07<00:00,  2.53s/it]
[INFO] Loaded Deep Floyd!
[INFO] Loading Stable Zero123 ...
LatentDiffusion: Running in eps-prediction mode
DiffusionWrapper has 859.53 M params.
Keeping EMAs of 688.
making attention of type 'vanilla' with 512 in_channels
Working with z of shape (1, 4, 32, 32) = 4096 dimensions.
making attention of type 'vanilla' with 512 in_channels
[INFO] Loaded Stable Zero123!
[INFO] Using prompt [young woman, scarf, white marble, bust, statue, victorian] and negative prompt []
[INFO] Using view-dependent prompts [side]:[young woman, scarf, white marble, bust, statue, victorian, side view] [front]:[young woman, scarf, white marble, bust, statue, victorian, front view] [back]:[young woman, scarf, white marble, bust, statue, victorian, back view] [overhead]:[young woman, scarf, white marble, bust, statue, victorian, overhead view]
C:\Users\Andy\Anaconda3\envs\dreamcraft3d\lib\site-packages\controlnet_aux\segment_anything\modeling\tiny_vit_sam.py:654: UserWarning: Overwriting tiny_vit_5m_224 in registry with controlnet_aux.segment_anything.modeling.tiny_vit_sam.tiny_vit_5m_224. This is because the name being registered conflicts with an existing name. Please check if this is not expected.
  return register_model(fn_wrapper)
C:\Users\Andy\Anaconda3\envs\dreamcraft3d\lib\site-packages\controlnet_aux\segment_anything\modeling\tiny_vit_sam.py:654: UserWarning: Overwriting tiny_vit_11m_224 in registry with controlnet_aux.segment_anything.modeling.tiny_vit_sam.tiny_vit_11m_224. This is because the name being registered conflicts with an existing name. Please check if this is not expected.
  return register_model(fn_wrapper)
C:\Users\Andy\Anaconda3\envs\dreamcraft3d\lib\site-packages\controlnet_aux\segment_anything\modeling\tiny_vit_sam.py:654: UserWarning: Overwriting tiny_vit_21m_224 in registry with controlnet_aux.segment_anything.modeling.tiny_vit_sam.tiny_vit_21m_224. This is because the name being registered conflicts with an existing name. Please check if this is not expected.
  return register_model(fn_wrapper)
C:\Users\Andy\Anaconda3\envs\dreamcraft3d\lib\site-packages\controlnet_aux\segment_anything\modeling\tiny_vit_sam.py:654: UserWarning: Overwriting tiny_vit_21m_384 in registry with controlnet_aux.segment_anything.modeling.tiny_vit_sam.tiny_vit_21m_384. This is because the name being registered conflicts with an existing name. Please check if this is not expected.
  return register_model(fn_wrapper)
C:\Users\Andy\Anaconda3\envs\dreamcraft3d\lib\site-packages\controlnet_aux\segment_anything\modeling\tiny_vit_sam.py:654: UserWarning: Overwriting tiny_vit_21m_512 in registry with controlnet_aux.segment_anything.modeling.tiny_vit_sam.tiny_vit_21m_512. This is because the name being registered conflicts with an existing name. Please check if this is not expected.
  return register_model(fn_wrapper)
A matching Triton is not available, some optimizations will not be enabled.
Error caught was: No module named 'triton'
Process SpawnProcess-1:
Traceback (most recent call last):
  File "C:\Users\Andy\Anaconda3\envs\dreamcraft3d\lib\multiprocessing\process.py", line 314, in _bootstrap
    self.run()
  File "C:\Users\Andy\Anaconda3\envs\dreamcraft3d\lib\multiprocessing\process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "C:\Users\Andy\Documents\threestudio\threestudio\models\prompt_processors\deepfloyd_prompt_processor.py", line 63, in spawn_func
    text_encoder = T5EncoderModel.from_pretrained(
  File "C:\Users\Andy\Anaconda3\envs\dreamcraft3d\lib\site-packages\transformers\modeling_utils.py", line 2505, in from_pretrained
    raise EnvironmentError(
OSError: DeepFloyd/IF-I-XL-v1.0 does not appear to have a file named pytorch_model.8bit.bin but there is a file without the variant 8bit. Use `variant=None` to load this model from those weights.
Traceback (most recent call last):
  File "C:\Users\Andy\Documents\threestudio\launch.py", line 252, in <module>
    main(args, extras)
  File "C:\Users\Andy\Documents\threestudio\launch.py", line 120, in main
    system: BaseSystem = threestudio.find(cfg.system_type)(
  File "C:\Users\Andy\Documents\threestudio\threestudio\systems\base.py", line 45, in __init__
    self.configure()
  File "C:\Users\Andy\Documents\threestudio\threestudio\systems\dreamcraft3d.py", line 50, in configure
    self.prompt_processor = threestudio.find(self.cfg.prompt_processor_type)(
  File "C:\Users\Andy\Documents\threestudio\threestudio\utils\base.py", line 83, in __init__
    self.configure(*args, **kwargs)
  File "C:\Users\Andy\Documents\threestudio\threestudio\models\prompt_processors\base.py", line 337, in configure
    self.load_text_embeddings()
  File "C:\Users\Andy\Documents\threestudio\threestudio\models\prompt_processors\base.py", line 395, in load_text_embeddings
    self.text_embeddings = self.load_from_cache(self.prompt)[None, ...]
  File "C:\Users\Andy\Documents\threestudio\threestudio\models\prompt_processors\base.py", line 413, in load_from_cache
    raise FileNotFoundError(
FileNotFoundError: Text embedding file .threestudio_cache/text_embeddings\aca2a23effdf0d18401b69d88fd03720.pt for model DeepFloyd/IF-I-XL-v1.0 and prompt [young woman, scarf, white marble, bust, statue, victorian] not found.

What is the right way? And could you add that to the installation guide please?

MrTornado24 commented 6 months ago

Hi did you install triton by pip install triton?

Iliceth commented 6 months ago

@MrTornado24 I did not install triton, as this is a Windows installation and if I'm right it's not available for Windows and optional, but that last thing might not be true? Although it's only in one of the cases above that it points at triton missing, both cases have different errors.

nullx1337 commented 6 months ago

Hey, i have the same issue as @Iliceth. My system is a Ubuntu 22.04 with a RTX 4090.

(venv) root@7f0a3c7d2231:~/DreamCraft3D# python launch.py --config configs/dreamcraft3d-coarse-nerf.yaml --train system.prompt_processor.prompt="$prompt" data.image_path="$image_path"
/root/DreamCraft3D/venv/lib/python3.10/site-packages/requests/__init__.py:102: RequestsDependencyWarning: urllib3 (1.26.13) or chardet (5.2.0)/charset_normalizer (2.0.12) doesn't match a supported version!
  warnings.warn("urllib3 ({}) or chardet ({})/charset_normalizer ({}) doesn't match a supported "
/root/DreamCraft3D/venv/lib/python3.10/site-packages/controlnet_aux/segment_anything/modeling/tiny_vit_sam.py:654: UserWarning: Overwriting tiny_vit_5m_224 in registry with controlnet_aux.segment_anything.modeling.tiny_vit_sam.tiny_vit_5m_224. This is because the name being registered conflicts with an existing name. Please check if this is not expected.
  return register_model(fn_wrapper)
/root/DreamCraft3D/venv/lib/python3.10/site-packages/controlnet_aux/segment_anything/modeling/tiny_vit_sam.py:654: UserWarning: Overwriting tiny_vit_11m_224 in registry with controlnet_aux.segment_anything.modeling.tiny_vit_sam.tiny_vit_11m_224. This is because the name being registered conflicts with an existing name. Please check if this is not expected.
  return register_model(fn_wrapper)
/root/DreamCraft3D/venv/lib/python3.10/site-packages/controlnet_aux/segment_anything/modeling/tiny_vit_sam.py:654: UserWarning: Overwriting tiny_vit_21m_224 in registry with controlnet_aux.segment_anything.modeling.tiny_vit_sam.tiny_vit_21m_224. This is because the name being registered conflicts with an existing name. Please check if this is not expected.
  return register_model(fn_wrapper)
/root/DreamCraft3D/venv/lib/python3.10/site-packages/controlnet_aux/segment_anything/modeling/tiny_vit_sam.py:654: UserWarning: Overwriting tiny_vit_21m_384 in registry with controlnet_aux.segment_anything.modeling.tiny_vit_sam.tiny_vit_21m_384. This is because the name being registered conflicts with an existing name. Please check if this is not expected.
  return register_model(fn_wrapper)
/root/DreamCraft3D/venv/lib/python3.10/site-packages/controlnet_aux/segment_anything/modeling/tiny_vit_sam.py:654: UserWarning: Overwriting tiny_vit_21m_512 in registry with controlnet_aux.segment_anything.modeling.tiny_vit_sam.tiny_vit_21m_512. This is because the name being registered conflicts with an existing name. Please check if this is not expected.
  return register_model(fn_wrapper)
WARNING[XFORMERS]: xFormers can't load C++/CUDA extensions. xFormers was built for:
    PyTorch 2.1.2+cu121 with CUDA 1201 (you have 2.1.2+cu118)
    Python  3.10.13 (you have 3.10.12)
  Please reinstall xformers (see https://github.com/facebookresearch/xformers#installing-xformers)
  Memory-efficient attention, SwiGLU, sparse and more won't be available.
  Set XFORMERS_MORE_DETAILS=1 for more details
Seed set to 0
[INFO] Loading Deep Floyd ...

A mixture of fp16 and non-fp16 filenames will be loaded.
Loaded fp16 filenames:
[unet/diffusion_pytorch_model.fp16.safetensors, text_encoder/model.fp16-00002-of-00002.safetensors, safety_checker/model.fp16.safetensors, text_encoder/model.fp16-00001-of-00002.safetensors]
Loaded non-fp16 filenames:
[watermarker/diffusion_pytorch_model.safetensors
If this behavior is not expected, please check your folder structure.
Loading pipeline components...:  33%|█████████████████████████▋                                                   | 1/3 [00:00<00:01,  1.74it/s]You are using the default legacy behaviour of the <class 'transformers.models.t5.tokenization_t5.T5Tokenizer'>. This is expected, and simply means that the `legacy` (previous) behavior will be used so nothing changes for you. If you want to use the new behaviour, set `legacy=False`. This should only be set if you understand what it means, and thouroughly read the reason why this was added as explained in https://github.com/huggingface/transformers/pull/24565
Loading pipeline components...: 100%|█████████████████████████████████████████████████████████████████████████████| 3/3 [00:00<00:00,  4.03it/s]
[INFO] Loaded Deep Floyd!
[INFO] Loading Stable Zero123 ...
LatentDiffusion: Running in eps-prediction mode
DiffusionWrapper has 859.53 M params.
Keeping EMAs of 688.
making attention of type 'vanilla' with 512 in_channels
Working with z of shape (1, 4, 32, 32) = 4096 dimensions.
making attention of type 'vanilla' with 512 in_channels
[INFO] Loaded Stable Zero123!
[INFO] Using prompt [minimalsitic stone statue ow an owl] and negative prompt []
[INFO] Using view-dependent prompts [side]:[minimalsitic stone statue ow an owl, side view] [front]:[minimalsitic stone statue ow an owl, front view] [back]:[minimalsitic stone statue ow an owl, back view] [overhead]:[minimalsitic stone statue ow an owl, overhead view]
Processing Propmp: minimalsitic stone statue ow an owl
Processing Propmp:
Processing Propmp: minimalsitic stone statue ow an owl, side view
Processing Propmp: minimalsitic stone statue ow an owl, front view
Processing Propmp: minimalsitic stone statue ow an owl, back view
Processing Propmp: minimalsitic stone statue ow an owl, overhead view
Processing Propmp:
Processing Propmp:
Processing Propmp:
Processing Propmp:
/root/DreamCraft3D/venv/lib/python3.10/site-packages/requests/__init__.py:102: RequestsDependencyWarning: urllib3 (1.26.13) or chardet (5.2.0)/charset_normalizer (2.0.12) doesn't match a supported version!
  warnings.warn("urllib3 ({}) or chardet ({})/charset_normalizer ({}) doesn't match a supported "
/root/DreamCraft3D/venv/lib/python3.10/site-packages/controlnet_aux/segment_anything/modeling/tiny_vit_sam.py:654: UserWarning: Overwriting tiny_vit_5m_224 in registry with controlnet_aux.segment_anything.modeling.tiny_vit_sam.tiny_vit_5m_224. This is because the name being registered conflicts with an existing name. Please check if this is not expected.
  return register_model(fn_wrapper)
/root/DreamCraft3D/venv/lib/python3.10/site-packages/controlnet_aux/segment_anything/modeling/tiny_vit_sam.py:654: UserWarning: Overwriting tiny_vit_11m_224 in registry with controlnet_aux.segment_anything.modeling.tiny_vit_sam.tiny_vit_11m_224. This is because the name being registered conflicts with an existing name. Please check if this is not expected.
  return register_model(fn_wrapper)
/root/DreamCraft3D/venv/lib/python3.10/site-packages/controlnet_aux/segment_anything/modeling/tiny_vit_sam.py:654: UserWarning: Overwriting tiny_vit_21m_224 in registry with controlnet_aux.segment_anything.modeling.tiny_vit_sam.tiny_vit_21m_224. This is because the name being registered conflicts with an existing name. Please check if this is not expected.
  return register_model(fn_wrapper)
/root/DreamCraft3D/venv/lib/python3.10/site-packages/controlnet_aux/segment_anything/modeling/tiny_vit_sam.py:654: UserWarning: Overwriting tiny_vit_21m_384 in registry with controlnet_aux.segment_anything.modeling.tiny_vit_sam.tiny_vit_21m_384. This is because the name being registered conflicts with an existing name. Please check if this is not expected.
  return register_model(fn_wrapper)
/root/DreamCraft3D/venv/lib/python3.10/site-packages/controlnet_aux/segment_anything/modeling/tiny_vit_sam.py:654: UserWarning: Overwriting tiny_vit_21m_512 in registry with controlnet_aux.segment_anything.modeling.tiny_vit_sam.tiny_vit_21m_512. This is because the name being registered conflicts with an existing name. Please check if this is not expected.
  return register_model(fn_wrapper)
WARNING[XFORMERS]: xFormers can't load C++/CUDA extensions. xFormers was built for:
    PyTorch 2.1.2+cu121 with CUDA 1201 (you have 2.1.2+cu118)
    Python  3.10.13 (you have 3.10.12)
  Please reinstall xformers (see https://github.com/facebookresearch/xformers#installing-xformers)
  Memory-efficient attention, SwiGLU, sparse and more won't be available.
  Set XFORMERS_MORE_DETAILS=1 for more details
You are using the default legacy behaviour of the <class 'transformers.models.t5.tokenization_t5.T5Tokenizer'>. This is expected, and simply means that the `legacy` (previous) behavior will be used so nothing changes for you. If you want to use the new behaviour, set `legacy=False`. This should only be set if you understand what it means, and thouroughly read the reason why this was added as explained in https://github.com/huggingface/transformers/pull/24565
Process SpawnProcess-1:
Traceback (most recent call last):
  File "/usr/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap
    self.run()
  File "/usr/lib/python3.10/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/root/DreamCraft3D/threestudio/models/prompt_processors/deepfloyd_prompt_processor.py", line 63, in spawn_func
    text_encoder = T5EncoderModel.from_pretrained(
  File "/root/DreamCraft3D/venv/lib/python3.10/site-packages/transformers/modeling_utils.py", line 3634, in from_pretrained
    raise ValueError(
ValueError:
                        Some modules are dispatched on the CPU or the disk. Make sure you have enough GPU RAM to fit
                        the quantized model. If you want to dispatch the model on the CPU or the disk while keeping
                        these modules in 32-bit, you need to set `load_in_8bit_fp32_cpu_offload=True` and pass a custom
                        `device_map` to `from_pretrained`. Check
                        https://huggingface.co/docs/transformers/main/en/main_classes/quantization#offload-between-cpu-and-gpu
                        for more details.

Traceback (most recent call last):
  File "/root/DreamCraft3D/launch.py", line 252, in <module>
    main(args, extras)
  File "/root/DreamCraft3D/launch.py", line 120, in main
    system: BaseSystem = threestudio.find(cfg.system_type)(
  File "/root/DreamCraft3D/threestudio/systems/base.py", line 45, in __init__
    self.configure()
  File "/root/DreamCraft3D/threestudio/systems/dreamcraft3d.py", line 50, in configure
    self.prompt_processor = threestudio.find(self.cfg.prompt_processor_type)(
  File "/root/DreamCraft3D/threestudio/utils/base.py", line 83, in __init__
    self.configure(*args, **kwargs)
  File "/root/DreamCraft3D/threestudio/models/prompt_processors/base.py", line 337, in configure
    self.load_text_embeddings()
  File "/root/DreamCraft3D/threestudio/models/prompt_processors/base.py", line 396, in load_text_embeddings
    self.text_embeddings = self.load_from_cache(self.prompt)[None, ...]
  File "/root/DreamCraft3D/threestudio/models/prompt_processors/base.py", line 414, in load_from_cache
    raise FileNotFoundError(
FileNotFoundError: Text embedding file .threestudio_cache/text_embeddings/cd17b07ca6b0c460977ba5516694f2d0.pt for model DeepFloyd/IF-I-XL-v1.0 and prompt [minimalsitic stone statue ow an owl] not found.
FrozenSilent commented 6 months ago

I also encountered this FileNotFoundError. Does anyone have some clues on how to solve this problem?

kamata1729 commented 6 months ago

You can fix the FileNotFoundError by changing below 2 lines into local_files_only=False https://github.com/deepseek-ai/DreamCraft3D/blob/624aa2b8ce16beafc2c72f1356e6cc5710c577b3/threestudio/models/prompt_processors/deepfloyd_prompt_processor.py#L61

https://github.com/deepseek-ai/DreamCraft3D/blob/624aa2b8ce16beafc2c72f1356e6cc5710c577b3/threestudio/models/prompt_processors/deepfloyd_prompt_processor.py#L70