JSON files not found

[ngc-shj]

Thank you for publishing a great job. I am getting an error that the JSON files are not found. Where can I get these JSON files?

1) model_index.json

Traceback (most recent call last):
  File "/path/to/venv/vistadream/VistaDream/demo.py", line 7, in <module>
    vistadream()
  File "/path/to/venv/vistadream/VistaDream/pipe/c2f_recons.py", line 182, in __call__
    self._MCS_Refinement()
  File "/path/to/venv/vistadream/VistaDream/pipe/c2f_recons.py", line 149, in _MCS_Refinement
    refiner = HackSD_MCS(device='cuda',use_lcm=True,denoise_steps=self.mcs_iterations)
  File "/path/to/venv/vistadream/VistaDream/ops/mcs.py", line 33, in __init__
    self._load_model()
  File "/path/to/venv/vistadream/VistaDream/ops/mcs.py", line 39, in _load_model
    self.model = Hack_SDPipe_Stepwise.from_pretrained(self.sd_ckpt)
  File "/path/to/venv/vistadream/lib/python3.10/site-packages/huggingface_hub/utils/_validators.py", line 114, in _inner_fn
    return fn(*args, **kwargs)
  File "/path/to/venv/vistadream/lib/python3.10/site-packages/diffusers/pipelines/pipeline_utils.py", line 1117, in from_pretrained
    config_dict = cls.load_config(cached_folder)
  File "/path/to/venv/vistadream/lib/python3.10/site-packages/huggingface_hub/utils/_validators.py", line 114, in _inner_fn
    return fn(*args, **kwargs)
  File "/path/to/venv/vistadream/lib/python3.10/site-packages/diffusers/configuration_utils.py", line 370, in load_config
    raise EnvironmentError(
OSError: Error no file named model_index.json found in directory tools/StableDiffusion/ckpt.

2) preprocessor_config.json

Traceback (most recent call last):
  File "/path/to/venv/vistadream/VistaDream/demo.py", line 7, in <module>
    vistadream()
  File "/path/to/venv/vistadream/VistaDream/pipe/c2f_recons.py", line 182, in __call__
    self._MCS_Refinement()
  File "/path/to/venv/vistadream/VistaDream/pipe/c2f_recons.py", line 149, in _MCS_Refinement
    refiner = HackSD_MCS(device='cuda',use_lcm=True,denoise_steps=self.mcs_iterations)
  File "/path/to/venv/vistadream/VistaDream/ops/mcs.py", line 33, in __init__
    self._load_model()
  File "/path/to/venv/vistadream/VistaDream/ops/mcs.py", line 39, in _load_model
    self.model = Hack_SDPipe_Stepwise.from_pretrained(self.sd_ckpt)
  File "/path/to/venv/vistadream/lib/python3.10/site-packages/huggingface_hub/utils/_validators.py", line 114, in _inner_fn
    return fn(*args, **kwargs)
  File "/path/to/venv/vistadream/lib/python3.10/site-packages/diffusers/pipelines/pipeline_utils.py", line 1271, in from_pretrained
    loaded_sub_model = load_sub_model(
  File "/path/to/venv/vistadream/lib/python3.10/site-packages/diffusers/pipelines/pipeline_utils.py", line 528, in load_sub_model
    loaded_sub_model = load_method(cached_folder, **loading_kwargs)
  File "/path/to/venv/vistadream/lib/python3.10/site-packages/transformers/image_processing_base.py", line 206, in from_pretrained
    image_processor_dict, kwargs = cls.get_image_processor_dict(pretrained_model_name_or_path, **kwargs)
  File "/path/to/venv/vistadream/lib/python3.10/site-packages/transformers/image_processing_base.py", line 335, in get_image_processor_dict
    resolved_image_processor_file = cached_file(
  File "/path/to/venv/vistadream/lib/python3.10/site-packages/transformers/utils/hub.py", line 373, in cached_file
    raise EnvironmentError(
OSError: tools/StableDiffusion/ckpt does not appear to have a file named preprocessor_config.json. Checkout 'https://huggingface.co/tools/StableDiffusion/ckpt/tree/main' for available files.

[HpWang-whu]

Hi @ngc-shj, I have just resolved this issue at this commit. You can pull the latest version of the project to address it. Another modification may be to rename the current tools/StableDiffusion/ckpt folder to tools/StableDiffusion/lcm_ckpt.

Yours,

[ngc-shj]

Thank you for your quick response. I'll give it a try!

[ngc-shj]

I did a git pull, placed the download file properly, and ran demo.py, which finished successfully and generated the following files!

$ find . -type f -ls | grep 'Oct 25'
 30160965     412 -rw-r--r--   1 users users     419478 Oct 25 00:13 ./data/sd_readingroom/video_dpt.mp4
 30160924  215928 -rw-r--r--   1 users users  221107092 Oct 25 00:12 ./data/sd_readingroom/scene.pth
 30160964     944 -rw-r--r--   1 users users     963279 Oct 25 00:13 ./data/sd_readingroom/video_rgb.mp4
$

[HpWang-whu]

Cool!

[ngc-shj]

In order for demo.py to succeed, I had to execute the following command:

cd tools/DepthPro
pip install -e .
cd ../..

pushd tools/OneFormer/oneformer/modeling/pixel_decoder/ops/
sh make.sh
popd