[Test] Clip_skip test!

[toyxyz]

I tested the newly added Clip_skip this time. For anime checkpoints, definitely 2 will generate cleaner and more natural images than 1. Except for Clip_skip, Seed and Prompt are identical.

https://github.com/neggles/animatediff-cli/assets/8006000/51a720fe-a19f-416c-9001-2821f7a704e0

https://github.com/neggles/animatediff-cli/assets/8006000/0ad25913-7a18-4b5e-bba6-922618a454f6

https://github.com/neggles/animatediff-cli/assets/8006000/13a867ac-f6c8-4045-8d49-677805db2a1b

https://github.com/neggles/animatediff-cli/assets/8006000/e6ec06f2-416a-4ea3-8fc0-cc07f73f68f4

[Skquark]

Very cool parameter, makes a difference, but I'm not entirely sure how I'd define it. What does it actually do? Any value in setting clip_skip to 0 or 3?

[Alphyn-gunner]

In simple terms, clip translates the prompt into numeric tokens, that can be understood by SD. It happens in several stages, each stage making the final numeric embedding more precise. Clip skip 2 skips the last stage resulting in a slightly more vague and generalized embedding, as far as I understand. The important practical matter here is that NovelAI was trained with clip skip 2 for some unknown reason, and now all models based on NovelAI, that is to say all models with waifus. work best at clip skip 2. Maybe the devs thought that for anime waifus a more generalized pretty picture was more important than precise interpretation of the prompt. For non-anime models you should basically never use anything but clip skip 1.

[15704080]

I want to know the specific model and JSON. I can't reproduce the same effect

[15704080]

{ "name": "A1", "base": "/root/autodl-tmp/cli/animatediff-cli", "path": "/root/autodl-tmp/cli/animatediff-cli/sdmodel/3Guofeng3_v33.safetensors", "motion_module": "/root/autodl-tmp/cli/animatediff-cli/data/models/motion-module/animatediff/mm_sd_v14.ckpt", "compile": false, "seed": [ 10788741199826055000 ], "scheduler": "euler", "steps": 30, "guidance_scale": 8.5, "clip_skip": 2, "prompt": [ "1girl, solo, best quality, masterpiece, looking at viewer, purple hair, orange hair, gradient hair, blurry background, upper body, dress, flower print, spaghetti strap, bare shoulders" ], "n_prompt": [ "worst quality, low quality, cropped, lowres, text, jpeg artifacts, multiple view" ] }

[15704080]

animatediff generate -c 'config/prompts/A1.json' -W 512 -H 512 -L 72 -C 24

[toyxyz]

I want to know the specific model and JSON. I can't reproduce the same effect

{"name": "animelike", "base": ".", "path": "models\sd\Illustration_Counterfeit-V3.0_fp32.safetensors", "motion_module": "models\motion-module\mm_sd_v14.ckpt", "compile": false, "seed": [12345556123], "scheduler": "k_dpmpp_2m_sde", "steps": 25, "guidance_scale": 7.5, "clip_skip": 2, "prompt": ["masterpiece, best quality, 1girl, solo, white bikini, fov, large breasts, wide hips, narrow waist, shiny skin, cowboy shot, blue ribbon, standing, looking_at_viewer, blonde hair, blue eyes, braid hair, smile, solo, pool, white cloud, blue sky, sunlight"], "n_prompt": ["(worst quality, low quality:1.6)"]}

[15704080]

Can Vae model selection support be added to JSON

[15704080]

https://github.com/neggles/animatediff-cli/assets/128477008/260175a0-9784-4f9b-a0b4-ca21a8f1a6e2

[15704080]

have not rife

[15704080]

https://github.com/neggles/animatediff-cli/assets/128477008/674fbddf-d1e1-4bcb-b3a6-c28b5e274f4b

[toyxyz]

Can Vae model selection support be added to JSON

I merged VAE into the model.

[15704080]

where is the embeddings set

[15704080]

oh i find it, autodl-tmp/cli/animatediff-cli/data/embeddings

[neggles]

I really did not expect this many many people to use this, so it's awesome to see some of these generations :) glad to hear it's working well!

Very cool parameter, makes a difference, but I'm not entirely sure how I'd define it. What does it actually do? Any value in setting clip_skip to 0 or 3?

In general, you'll want clip_skip 2 for models based off the NAI model (which is 99% of anime models) since that's how NAI was trained. clip_skip 0 is technically invalid and might actually break things (I should probably sanity check that number, it needs to be between 1 and 12 inclusive). Sometimes fun stuff happens at clip skip 6 or 7, though, so feel free to mess around with it.

Can Vae model selection support be added to JSON

I merged VAE into the model.

Yeah, that's the way to do it for the time being; converting VAEs to Diffusers format is tricky without just converting an entire checkpoint. I've got a lot going on at the moment but I'll try to add VAE selection as/when/if I get time.

FWIW if you have a diffusers-converted VAE you can drop it into the converted pipeline under data/models/huggingface/<checkpoint_name> (the CLI creates this the first time you feed it a checkpoint) and it'll just use that VAE in future runs.