load_3DFuse error: Unexpected key(s) in state_dict: "cond_stage_model.transformer.text_model.embeddings.position_ids"

[YHQiu]

After generating the point cloud using the image, I encountered an error when trying to load the state_dict of the 3DFuse model. Here is the stack trace of the error. I have checked the source code but couldn't find the cause.

ERROR LOGS:

No module 'xformers'. Proceeding without it.
ControlLDM: Running in eps-prediction mode
DiffusionWrapper has 859.52 M params.
making attention of type 'vanilla' with 512 in_channels
Working with z of shape (1, 4, 32, 32) = 4096 dimensions.
making attention of type 'vanilla' with 512 in_channels
Loaded model config from [cldm_v15.yaml]
Loaded state_dict from [weights/3DFuse_sparse_depth_injector.ckpt]
Traceback (most recent call last):
  File "/root/anaconda3/envs/3DFuse/lib/python3.8/site-packages/gradio/routes.py", line 439, in run_predict
    output = await app.get_blocks().process_api(
  File "/root/anaconda3/envs/3DFuse/lib/python3.8/site-packages/gradio/blocks.py", line 1384, in process_api
    result = await self.call_function(
  File "/root/anaconda3/envs/3DFuse/lib/python3.8/site-packages/gradio/blocks.py", line 1103, in call_function
    prediction = await utils.async_iteration(iterator)
  File "/root/anaconda3/envs/3DFuse/lib/python3.8/site-packages/gradio/utils.py", line 343, in async_iteration
    return await iterator.__anext__()
  File "/root/anaconda3/envs/3DFuse/lib/python3.8/site-packages/gradio/utils.py", line 336, in __anext__
    return await anyio.to_thread.run_sync(
  File "/root/anaconda3/envs/3DFuse/lib/python3.8/site-packages/anyio/to_thread.py", line 33, in run_sync
    return await get_asynclib().run_sync_in_worker_thread(
  File "/root/anaconda3/envs/3DFuse/lib/python3.8/site-packages/anyio/_backends/_asyncio.py", line 877, in run_sync_in_worker_thread
    return await future
  File "/root/anaconda3/envs/3DFuse/lib/python3.8/site-packages/anyio/_backends/_asyncio.py", line 807, in run
    result = context.run(func, *args)
  File "/root/anaconda3/envs/3DFuse/lib/python3.8/site-packages/gradio/utils.py", line 319, in run_sync_iterator_async
    return next(iterator)
  File "/root/anaconda3/envs/3DFuse/lib/python3.8/site-packages/gradio/utils.py", line 688, in gen_wrapper
    yield from f(*args, **kwargs)
  File "gradio_app.py", line 47, in gen_3d
    yield from model.run_gradio(points,exp_instance_dir)
  File "/usr/local/apps/3DFuse/run_3DFuse.py", line 138, in run_gradio
    model = getattr(self, family).make()
  File "/usr/local/apps/3DFuse/run_img_sampling.py", line 18, in make
    model = StableDiffusion(**args)
  File "/usr/local/apps/3DFuse/adapt_sd.py", line 135, in __init__
    model=load_3DFuse(self.checkpoint_root(),dir,alpha)
  File "/usr/local/apps/3DFuse/adapt_sd.py", line 121, in load_3DFuse
    model.load_state_dict(load_state_dict(control['control_weight'], location='cuda'))
  File "/root/anaconda3/envs/3DFuse/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1604, in load_state_dict
    raise RuntimeError('Error(s) in loading state_dict for {}:\n\t{}'.format(
RuntimeError: Error(s) in loading state_dict for ControlLDM:
    Unexpected key(s) in state_dict: "cond_stage_model.transformer.text_model.embeddings.position_ids". 

ERROR HANPPEND SOURCE CODE: Module: adapt_sd.py

def load_3DFuse(control,dir,alpha):
    ######################LOADCONTROL###########################
    model = create_model(control['control_yaml']).cpu()
    model.load_state_dict(load_state_dict(control['control_weight'], location='cuda'))
    state_dict, l = merge("runwayml/stable-diffusion-v1-5",dir,alpha)

    #######################OVERRIDE#############################
    model.load_state_dict(state_dict,strict=False)

    #######################ADDEMBBEDDING########################
    load_embedding(model,l)
    ###############################################################
    return model

OTHER INFO:

git+https://github.com/openai/point-e
git+https://github.com/huggingface/diffusers
git+https://github.com/cloneofsimo/lora.git
git+https://github.com/Ir1d/image-background-remove-tool@2b68f5276d0f2e90f607c7845c60d2bddb79d5ba
I have loosened the version constraints for the opencv-contrib-python and safetensors libraries in the requirements.txt file in order to accommodate the upgraded dependencies of the third-party Git repository's code.

[muyuWang]

Hi , have you solved this problem? I meet the same problem and I have no idea too.

[j0seo]

Hello @YHQiu @muyuWang . I also recently encountered that error while setting up the environment again. It seems like an issue that has arisen over time. Actually, there will be an update supporting various 3D pipelines soon, and I plan to fix it then. For now, you can resolve it by simply adding the strict=False option in torch.load().

[muyuWang]

Thank you for your advice. In fact, I have tried this way yesterday and then another bug existed. It seems still caused by the environment settings.

[hasan1292]

I also recently encountered that error while setting up the environment agai

Hi,

I am having the exact same issue. And I applied strict=False as well but it is not working. Can you please tell how to fix it?

[Ines-Hyeonsu-Kim]

Hello,

It appears that the "transformers" version doesn't match. Maybe "pip install -U transformers==4.27.0" can resolve the issue.

[YHQiu]

Hello @YHQiu @muyuWang . I also recently encountered that error while setting up the environment again. It seems like an issue that has arisen over time. Actually, there will be an update supporting various 3D pipelines soon, and I plan to fix it then. For now, you can resolve it by simply adding the strict=False option in torch.load().

Thank you, this issue has been resolved.