Which checkpoint is used in Unet module of UNetMV2DConditionModel

[zjcs]

When I test the edit/add function, there are some checkpoints url downloading error when downloading checkpoints. As I see, threestudio/utils/wonder3D/configs/mvdiffusion-joint-ortho-6views.yaml define the checkpoints used in test_mvdiffusion_seq.py, such as the definition of pretrained_model_name_or_path and pretrained_unet_path. But only unet is download by the definition of pretrained_unet_path, which cause error when there is no threestudio/utils/wonder3D/ckpts direction exists. Actually, this direction realy do no exists in this repo.

Can you help me to access the unet pretrain model? Use cfg.pretrained_model_name_or_path not cfg.pretrained_unet_path during UNetMV2DConditionModel.from_pretrained_2d, or others?

The definition of mvdiffusion-joint-ortho-6views.yaml:

pretrained_model_name_or_path: 'lambdalabs/sd-image-variations-diffusers'
pretrained_unet_path: './ckpts/'

The pretrain ckpt download in

    image_encoder = CLIPVisionModelWithProjection.from_pretrained(cfg.pretrained_model_name_or_path, subfolder="image_encoder", revision=cfg.revision)
    feature_extractor = CLIPImageProcessor.from_pretrained(cfg.pretrained_model_name_or_path, subfolder="feature_extractor", revision=cfg.revision)
    vae = AutoencoderKL.from_pretrained(cfg.pretrained_model_name_or_path, subfolder="vae", revision=cfg.revision)
    unet = UNetMV2DConditionModel.from_pretrained_2d(cfg.pretrained_unet_path, subfolder="unet", revision=cfg.revision, **cfg.unet_from_pretrained_kwargs)

[zjcs]

You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference. Some weights of UNetMV2DConditionModel were not initialized from the model checkpoint at lambdalabs/sd-image-variations-diffusers and are newly initialized because the shapes did not match:

• conv_in.weight: found shape torch.Size([320, 4, 3, 3]) in the checkpoint and torch.Size([320, 8, 3, 3]) in the model instantiated You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference. use xformers.

When I use cfg.pretrained_model_name_or_path in UNetMV2DConditionModel.from_pretrained_2d, the above log occurs.

The checkpoint is wrong?

[buaacyw]

We use wonder3D to implement our add function. It looks like you have problem downloading the Wonder3D ckpt. I have never encountered this problem before. Could you please tell me the way you prepare your environment? Or you can try download the ckpt from wonder3D manually.

[Dumeowmeow]

Did you run the sh download wonder3d.sh command?

[john123zerg]

Oh I forgot to update the discussion about that. I reinstalled everything with cuda 11.8, ubuntu22,04, rtx4090 and now it works fine.

I had issues because I was using docker which had some path problems. The code functions fine.

Can I ask if I can change the Wonder3D models to something else?

Thank you for the email. John Kim

-----Original Message----- From: @.> To: @.>; Cc: @.>; @.>; Sent: 2024-02-19 (월) 13:17:43 (GMT+09:00) Subject: Re: [buaacyw/GaussianEditor] Which checkpoint is used in Unet module of UNetMV2DConditionModel (Issue #30)

Did you run the sh download wonder3d.sh command? — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you commented.Message ID: @.***>

[john123zerg]

Oh I forgot to update the discussion about that. I reinstalled everything with cuda 11.8, ubuntu22,04, rtx4090 and now it works fine.

I had issues because I was using docker which had some path problems. The code functions fine.

Can I ask if I can change the Wonder3D models to something else?

Thank you for the email. John Kim

-----Original Message----- From: "Yiwen @.> To: @.>; Cc: @.>; @.>; Sent: 2024-02-12 (월) 13:51:55 (GMT+09:00) Subject: Re: [buaacyw/GaussianEditor] Which checkpoint is used in Unet module of UNetMV2DConditionModel (Issue #30)

We use wonder3D to implement our add function. It looks like you have problem downloading the Wonder3D ckpt. I have never encountered this problem before. Could you please tell me the way you prepare your environment? Or you can try download the ckpt from wonder3D manually. — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you commented.Message ID: @.***>

[zjcs]

Did you run the sh download wonder3d.sh command?

Sorry to reply late.

I just directly run the code and the checkpoint download directly via the huggingface. At previouse time, I did two modifies following the origin wonder3D, and the final result is right:

1. threestudio/utils/wonder3D/configs/mvdiffusion-joint-ortho-6views.yaml

   pretrained_model_name_or_path: 'lambdalabs/sd-image-variations-diffusers'

   pretrained_unet_path: './ckpts/'

   pretrained_model_name_or_path: 'flamehaze1115/wonder3d-v1.0'

2. threestudio/utils/wonder3D/test_mvdiffusion_seq.py

   unet = UNetMV2DConditionModel.from_pretrained_2d(cfg.pretrained_unet_path, subfolder="unet", revision=cfg.revision, **cfg.unet_from_pretrained_kwargs)

   unet = UNetMV2DConditionModel.from_pretrained_2d(cfg.pretrained_model_name_or_path, subfolder="unet", revision=cfg.revision, **cfg.unet_from_pretrained_kwargs)

I will try bash wonder3d.sh later.

BTW, why these two places is different to the origin Wonder3D code? A better fine-tune checkpoint?

[HaoZhiG]

@zjcs Hello, I have a similar issue. Did you manage to resolve it?

[zjcs]

@zjcs Hello, I have a similar issue. Did you manage to resolve it?

Please refer the above discussion.