MCG-NJU / BIVDiff

[CVPR 2024] BIVDiff: A Training-free Framework for General-Purpose Video Synthesis via Bridging Image and Video Diffusion Models
43 stars 0 forks source link

Lack of 'clip' directory in stable-diffusion-2-1-base link #2

Closed Garyson1204 closed 1 month ago

Garyson1204 commented 1 month ago

As seen in the SDVideoModel in models/ReuseAndDiffuse/model/model.py:

class SDVideoModel(pl.LightningModule):
    def __init__(self, pretrained_model_path, **kwargs):
        super().__init__()
        self.save_hyperparameters(ignore=["pretrained_model_path"], logger=False)
        # main training module
        self.unet: Union[str, UNet3DConditionModel] = Path(
            pretrained_model_path, "unet"
        ).as_posix()
        # components for training
        self.noise_scheduler_dir = Path(pretrained_model_path, "scheduler").as_posix()
        self.vae = Path(pretrained_model_path, "vae").as_posix()
        self.text_encoder = Path(pretrained_model_path, "text_encoder").as_posix()
        self.tokenizer: Union[str, CLIPTokenizer] = Path(
            pretrained_model_path, "tokenizer"
        ).as_posix()
        # clip model for metric
        self.clip = Path(pretrained_model_path, "clip").as_posix()
        self.clip_processor = Path(pretrained_model_path, "clip").as_posix()
        # define pipeline for inference
        self.val_pipeline = None
        # video frame resolution
        self.resolution = kwargs.get("resolution", 512)
        # use temporal_vae
        self.temporal_vae_path = kwargs.get("temporal_vae_path", None)

There are two lines in this code about the 'clip model for metric', but when I checked the Huggingface link provided by the author, the stable-diffusion-2-1-base repo doesn't have a clip directory. Any idea how to fix this?

ShiFengyuan1999 commented 1 month ago

Hi @Garyson1204, you can download clip from https://huggingface.co/openai/clip-vit-large-patch14 and put it under checkpoints/stable-diffusion-2-1-base/. I have updated README.md. Thanks for your reminder.