As seen in the SDVideoModel in models/ReuseAndDiffuse/model/model.py:
class SDVideoModel(pl.LightningModule):
def __init__(self, pretrained_model_path, **kwargs):
super().__init__()
self.save_hyperparameters(ignore=["pretrained_model_path"], logger=False)
# main training module
self.unet: Union[str, UNet3DConditionModel] = Path(
pretrained_model_path, "unet"
).as_posix()
# components for training
self.noise_scheduler_dir = Path(pretrained_model_path, "scheduler").as_posix()
self.vae = Path(pretrained_model_path, "vae").as_posix()
self.text_encoder = Path(pretrained_model_path, "text_encoder").as_posix()
self.tokenizer: Union[str, CLIPTokenizer] = Path(
pretrained_model_path, "tokenizer"
).as_posix()
# clip model for metric
self.clip = Path(pretrained_model_path, "clip").as_posix()
self.clip_processor = Path(pretrained_model_path, "clip").as_posix()
# define pipeline for inference
self.val_pipeline = None
# video frame resolution
self.resolution = kwargs.get("resolution", 512)
# use temporal_vae
self.temporal_vae_path = kwargs.get("temporal_vae_path", None)
There are two lines in this code about the 'clip model for metric', but when I checked the Huggingface link provided by the author, the stable-diffusion-2-1-base repo doesn't have a clip directory. Any idea how to fix this?
As seen in the
SDVideoModel
inmodels/ReuseAndDiffuse/model/model.py
:There are two lines in this code about the 'clip model for metric', but when I checked the Huggingface link provided by the author, the stable-diffusion-2-1-base repo doesn't have a clip directory. Any idea how to fix this?