Runtime environment crushes after RAM usage goes over 12GB

[volotat]

I tired to generate anything using this colab but each time it crushes after system RAM goes over 12GB. Is there any way to mitigate this problem via controlling number of frames generated or the size of the frame?

[camenduru]

Hi @volotat 👋 please try again maybe works and if modelscope developers add new features I will add we are using this pipeline and thanks ❤ for the https://github.com/volotat/Reference-based-SD-CN-Animation look super cool 🔥 I will try

[camenduru]

and maybe we can edit max_frames in /content/models/configuration.json

{   "framework": "pytorch",
    "task": "text-to-video-synthesis",
    "model": {
        "type": "latent-text-to-video-synthesis",
        "model_args": {
            "ckpt_clip": "open_clip_pytorch_model.bin",
            "ckpt_unet": "text2video_pytorch_model.pth",
            "ckpt_autoencoder": "VQGAN_autoencoder.pth",
            "max_frames": 16,
            "tiny_gpu": 1
        },
        "model_cfg": {
            "unet_in_dim": 4,
            "unet_dim": 320,
            "unet_y_dim": 768,
            "unet_context_dim": 1024,
            "unet_out_dim": 4,
            "unet_dim_mult": [1, 2, 4, 4],
            "unet_num_heads": 8,
            "unet_head_dim": 64,
            "unet_res_blocks": 2,
            "unet_attn_scales": [1, 0.5, 0.25],
            "unet_dropout": 0.1,
            "temporal_attention": "True",
            "num_timesteps": 1000,
            "mean_type": "eps",
            "var_type": "fixed_small",
            "loss_type": "mse"
        }
    },
    "pipeline": {
        "type": "latent-text-to-video-synthesis"
    }
}

[camenduru]

please watch this https://www.youtube.com/watch?v=b8D4am73e6I

[volotat]

Thank you a lot for a quick response and the video. It does work now.