Closed SoftologyPro closed 1 year ago
s9roll7
commented
1 year ago It is basically the same as using stable diffusion to output the desired picture. Of course, if you want to output a picture like Giger's, you need to choose a model that understands Giger's picture.
SoftologyPro
commented
1 year ago So those 2 example movies are from using Video Stylizaation in AnimateDiff Prompt Travel to process a source video to look like the giger aliens? If so, please share the settings.
I am intrested in styling videos to look like the work of different artists. So for giger I want to have that biomechanical look, not necessarily have just aliens. Or to give a video the look of Van Gough's Starry Night.
If you (or anyone) can share some settings/models/prompts to get started that would be a big help. Thanks.
It just seems like the prompts are ignore when styyling a video. It only seems to use the selected LORA to apply the style. It works very well for that, but I want to be able to tweak and change the styling with prompts (if possible)
s9roll7
commented
1 year ago Can you post the "source video" and "prompt.json" here? I'm not sure what's failing and what you're struggling with.
SoftologyPro
commented
1 year ago Source video https://github.com/s9roll7/animatediff-cli-prompt-travel/assets/47686889/0bf06e54-e2ee-47a7-952f-b87664a149b7
prompt.json
{
"name": "sample",
"path": "models\\sd\\majicmixRealistic_v5Preview.safetensors",
"vae_path": "",
"motion_module": "models\\motion-module\\mm_sd_v15_v2.ckpt",
"compile": false,
"tensor_interpolation_slerp": true,
"seed": [
1523491514
],
"scheduler": "k_dpmpp_sde",
"steps": 25,
"guidance_scale": 7.5,
"clip_skip": 2,
"prompt_fixed_ratio": 0.5,
"head_prompt": "",
"prompt_map": {
"0": "(1girl:0.99),(road:0.55),(teeth:0.37),(outdoors:0.48),(closed_eyes:0.82),(solo:0.94),(earphones:0.47),(vest:0.40),(smile:0.44),(ground_vehicle:0.44),(long_hair:0.81),(black_hair:0.89)",
"1": "(1girl:0.99),(road:0.35),(outdoors:0.37),(closed_eyes:0.80),(solo:0.95),(earphones:0.45),(vest:0.44),(smile:0.50),(long_hair:0.79),(black_hair:0.88)",
"2": "(1girl:0.99),(road:0.39),(outdoors:0.41),(closed_eyes:0.63),(solo:0.94),(earphones:0.62),(vest:0.44),(smile:0.50),(long_hair:0.78),(black_hair:0.85)",
"3": "(1girl:0.98),(road:0.38),(outdoors:0.48),(earphones:0.53),(solo:0.95),(black_eyes:0.44),(vest:0.40),(smile:0.49),(long_hair:0.74),(black_hair:0.84)",
"4": "(1girl:0.98),(outdoors:0.45),(earphones:0.38),(solo:0.93),(looking_at_viewer:0.39),(black_eyes:0.48),(vest:0.48),(smile:0.57),(long_hair:0.75),(black_hair:0.85)",
"5": "(1girl:0.98),(outdoors:0.48),(earphones:0.54),(solo:0.94),(black_eyes:0.48),(vest:0.51),(smile:0.54),(long_hair:0.70),(black_hair:0.83)",
"6": "(ring:0.58),(1girl:0.99),(grin:0.38),(outdoors:0.42),(earphones:0.58),(solo:0.95),(black_eyes:0.37),(vest:0.46),(smile:0.61),(long_hair:0.64),(black_hair:0.84),(jewelry:0.46)",
"7": "(ring:0.66),(1girl:0.99),(road:0.43),(outdoors:0.55),(earphones:0.44),(solo:0.94),(looking_at_viewer:0.37),(black_eyes:0.41),(vest:0.50),(smile:0.50),(long_hair:0.74),(black_hair:0.85),(jewelry:0.41)",
"8": "(ring:0.49),(1girl:0.99),(grin:0.45),(road:0.36),(outdoors:0.55),(earphones:0.64),(solo:0.95),(looking_at_viewer:0.49),(black_eyes:0.46),(vest:0.52),(smile:0.70),(long_hair:0.72),(black_hair:0.86)",
"9": "(ring:0.46),(1girl:0.99),(grin:0.50),(outdoors:0.52),(earphones:0.49),(solo:0.95),(looking_at_viewer:0.40),(black_eyes:0.38),(headphones:0.40),(vest:0.49),(smile:0.73),(long_hair:0.76),(black_hair:0.86),(jewelry:0.35)",
"10": "(ring:0.63),(1girl:0.99),(grin:0.43),(outdoors:0.51),(earphones:0.69),(solo:0.94),(looking_at_viewer:0.50),(black_eyes:0.43),(vest:0.40),(smile:0.73),(long_hair:0.81),(black_hair:0.88),(jewelry:0.44)",
"11": "(ring:0.68),(1girl:0.99),(grin:0.49),(outdoors:0.57),(earphones:0.60),(solo:0.93),(looking_at_viewer:0.54),(black_eyes:0.42),(vest:0.55),(smile:0.79),(long_hair:0.76),(black_hair:0.87),(jewelry:0.47)",
"12": "(ring:0.74),(1girl:0.98),(outdoors:0.42),(earphones:0.54),(solo:0.95),(vest:0.47),(smile:0.35),(long_hair:0.76),(black_hair:0.85),(jewelry:0.52)",
"13": "(1girl:0.99),(grin:0.42),(outdoors:0.48),(solo:0.93),(looking_at_viewer:0.42),(black_eyes:0.44),(vest:0.58),(smile:0.63),(long_hair:0.78),(black_hair:0.85)",
"14": "(1girl:0.99),(grin:0.37),(road:0.40),(outdoors:0.47),(earphones:0.70),(solo:0.94),(black_eyes:0.41),(vest:0.48),(smile:0.56),(long_hair:0.71),(black_hair:0.82)",
"15": "(1girl:0.98),(grin:0.46),(road:0.40),(outdoors:0.58),(earphones:0.67),(solo:0.92),(looking_at_viewer:0.38),(vest:0.55),(smile:0.66),(long_hair:0.75),(black_hair:0.83)",
"16": "(1girl:0.99),(grin:0.52),(teeth:0.36),(outdoors:0.55),(earphones:0.70),(solo:0.94),(looking_at_viewer:0.46),(vest:0.59),(smile:0.70),(long_hair:0.75),(black_hair:0.85)",
"17": "(1girl:0.99),(grin:0.60),(road:0.37),(outdoors:0.63),(earphones:0.68),(solo:0.94),(looking_at_viewer:0.53),(vest:0.47),(smile:0.83),(long_hair:0.74),(black_hair:0.87)",
"18": "(1girl:0.99),(grin:0.40),(outdoors:0.48),(earphones:0.54),(solo:0.95),(smile:0.61),(long_hair:0.76),(black_hair:0.88)",
"19": "(1girl:0.98),(grin:0.54),(road:0.37),(outdoors:0.61),(earphones:0.45),(solo:0.95),(looking_at_viewer:0.47),(vest:0.37),(smile:0.77),(long_hair:0.78),(black_hair:0.88)",
"20": "(1girl:0.98),(grin:0.51),(road:0.38),(outdoors:0.55),(solo:0.94),(looking_at_viewer:0.40),(vest:0.50),(smile:0.72),(long_hair:0.75),(black_hair:0.85)",
"21": "(1girl:0.99),(grin:0.54),(outdoors:0.41),(earphones:0.54),(solo:0.95),(vest:0.44),(smile:0.80),(long_hair:0.75),(black_hair:0.87)",
"22": "(1girl:0.96),(grin:0.47),(outdoors:0.36),(earphones:0.68),(solo:0.94),(black_eyes:0.39),(vest:0.49),(smile:0.73),(long_hair:0.77),(black_hair:0.87)",
"23": "(1girl:0.99),(grin:0.50),(outdoors:0.43),(earphones:0.43),(solo:0.94),(vest:0.53),(smile:0.77),(long_hair:0.71),(black_hair:0.84),(hands_on_hips:0.40)",
"24": "(ring:0.67),(1girl:0.99),(grin:0.50),(road:0.37),(outdoors:0.53),(earphones:0.56),(solo:0.94),(looking_at_viewer:0.39),(black_eyes:0.36),(vest:0.53),(smile:0.73),(long_hair:0.70),(black_hair:0.85),(jewelry:0.54)",
"25": "(ring:0.52),(1girl:0.99),(grin:0.50),(outdoors:0.44),(closed_eyes:0.49),(solo:0.94),(earphones:0.44),(vest:0.47),(smile:0.79),(long_hair:0.65),(black_hair:0.83),(jewelry:0.42)",
"26": "(zipper:0.37),(solo:0.94),(looking_at_viewer:0.51),(long_sleeves:0.36),(long_hair:0.80),(black_hair:0.86),(ring:0.50),(1girl:0.99),(grin:0.52),(outdoors:0.65),(earphones:0.44),(vest:0.37),(smile:0.83),(road:0.40)",
"27": "(1girl:0.98),(grin:0.47),(outdoors:0.55),(solo:0.95),(looking_at_viewer:0.44),(vest:0.37),(smile:0.76),(long_hair:0.76),(black_hair:0.86)",
"28": "(ring:0.56),(1girl:0.99),(grin:0.51),(outdoors:0.51),(closed_eyes:0.42),(solo:0.94),(earphones:0.50),(vest:0.46),(smile:0.80),(long_hair:0.69),(black_hair:0.85),(jewelry:0.48)",
"29": "(1girl:0.99),(grin:0.49),(outdoors:0.49),(earphones:0.68),(solo:0.95),(looking_at_viewer:0.37),(vest:0.38),(smile:0.77),(long_hair:0.69),(black_hair:0.86)",
"30": "(ring:0.69),(1girl:0.99),(grin:0.58),(road:0.46),(outdoors:0.53),(solo:0.93),(vest:0.49),(smile:0.80),(long_hair:0.76),(black_hair:0.87),(jewelry:0.69)",
"31": "(1girl:0.98),(grin:0.46),(road:0.51),(outdoors:0.49),(closed_eyes:0.50),(solo:0.94),(vest:0.53),(smile:0.71),(long_hair:0.67),(black_hair:0.84)",
"32": "(1girl:0.98),(grin:0.54),(road:0.57),(outdoors:0.55),(solo:0.91),(vest:0.52),(jacket:0.36),(smile:0.74),(long_hair:0.71),(black_hair:0.79)",
"33": "(1girl:0.98),(grin:0.55),(road:0.37),(outdoors:0.59),(earphones:0.90),(solo:0.92),(looking_at_viewer:0.42),(earbuds:0.51),(vest:0.53),(smile:0.79),(long_hair:0.69),(black_hair:0.84)",
"34": "(1girl:0.97),(grin:0.56),(road:0.47),(outdoors:0.62),(earphones:0.78),(solo:0.91),(looking_at_viewer:0.48),(black_eyes:0.37),(vest:0.62),(smile:0.75),(long_hair:0.67),(black_hair:0.84)",
"35": "(1girl:0.99),(grin:0.57),(road:0.42),(outdoors:0.53),(earphones:0.64),(solo:0.94),(looking_at_viewer:0.41),(vest:0.58),(smile:0.75),(long_hair:0.65),(black_hair:0.83)",
"36": "(1girl:0.99),(grin:0.56),(road:0.42),(outdoors:0.55),(earphones:0.57),(solo:0.94),(looking_at_viewer:0.43),(black_eyes:0.40),(vest:0.38),(smile:0.79),(long_hair:0.73),(black_hair:0.86)",
"37": "(1girl:0.97),(grin:0.54),(road:0.36),(outdoors:0.57),(earphones:0.47),(solo:0.95),(looking_at_viewer:0.36),(black_eyes:0.36),(vest:0.46),(smile:0.75),(long_hair:0.75),(black_hair:0.88)",
"38": "(1girl:0.97),(grin:0.47),(road:0.40),(outdoors:0.52),(earphones:0.75),(solo:0.94),(looking_at_viewer:0.37),(vest:0.45),(smile:0.69),(long_hair:0.75),(black_hair:0.86)",
"39": "(1girl:0.96),(grin:0.46),(road:0.44),(outdoors:0.56),(earphones:0.72),(solo:0.94),(looking_at_viewer:0.40),(vest:0.48),(smile:0.72),(long_hair:0.78),(black_hair:0.85)",
"40": "(1girl:0.96),(grin:0.39),(road:0.46),(teeth:0.37),(outdoors:0.55),(earphones:0.54),(solo:0.93),(looking_at_viewer:0.37),(vest:0.55),(smile:0.60),(long_hair:0.77),(black_hair:0.86)",
"41": "(running:0.35),(1girl:0.99),(grin:0.43),(road:0.48),(teeth:0.36),(outdoors:0.66),(earphones:0.59),(solo:0.93),(looking_at_viewer:0.47),(black_eyes:0.39),(vest:0.55),(smile:0.72),(long_hair:0.76),(black_hair:0.87)",
"42": "(ring:0.40),(1girl:0.99),(grin:0.55),(road:0.45),(outdoors:0.60),(earphones:0.69),(solo:0.94),(looking_at_viewer:0.42),(vest:0.57),(smile:0.75),(long_hair:0.78),(black_hair:0.86),(jewelry:0.43)",
"43": "(ring:0.78),(1girl:0.98),(grin:0.56),(road:0.50),(outdoors:0.58),(earphones:0.60),(solo:0.92),(looking_at_viewer:0.38),(vest:0.53),(smile:0.72),(long_hair:0.80),(black_hair:0.85),(jewelry:0.64)",
"44": "(ring:0.55),(1girl:0.98),(grin:0.64),(road:0.44),(teeth:0.36),(outdoors:0.60),(solo:0.93),(looking_at_viewer:0.49),(black_eyes:0.36),(vest:0.57),(smile:0.83),(long_hair:0.82),(black_hair:0.88)",
"45": "(ring:0.78),(1girl:0.98),(grin:0.54),(road:0.37),(outdoors:0.62),(earphones:0.44),(solo:0.91),(looking_at_viewer:0.44),(vest:0.54),(smile:0.75),(long_hair:0.81),(black_hair:0.86),(jewelry:0.72)",
"46": "(ring:0.42),(1girl:0.98),(grin:0.56),(road:0.39),(teeth:0.39),(outdoors:0.64),(earphones:0.50),(solo:0.92),(looking_at_viewer:0.49),(vest:0.54),(smile:0.82),(long_hair:0.83),(black_hair:0.88)",
"47": "(1girl:0.98),(grin:0.51),(teeth:0.41),(outdoors:0.57),(earphones:0.79),(solo:0.92),(looking_at_viewer:0.45),(vest:0.54),(smile:0.72),(long_hair:0.80),(black_hair:0.87)",
"48": "(1girl:0.97),(grin:0.51),(outdoors:0.42),(earphones:0.38),(solo:0.93),(vest:0.49),(smile:0.75),(long_hair:0.64),(black_hair:0.85)",
"49": "(ring:0.63),(1girl:0.97),(grin:0.50),(road:0.41),(outdoors:0.35),(earphones:0.44),(solo:0.93),(vest:0.45),(smile:0.69),(ground_vehicle:0.47),(long_hair:0.58),(black_hair:0.83),(jewelry:0.49)",
"50": "(1girl:0.99),(grin:0.49),(road:0.44),(outdoors:0.53),(earphones:0.61),(solo:0.93),(looking_at_viewer:0.39),(vest:0.53),(smile:0.81),(long_hair:0.64),(black_hair:0.86)",
"51": "(1girl:0.99),(grin:0.49),(road:0.36),(outdoors:0.49),(earphones:0.52),(solo:0.94),(smile:0.74),(long_hair:0.60),(black_hair:0.79)",
"52": "(ring:0.74),(1girl:0.98),(grin:0.46),(road:0.53),(outdoors:0.57),(earphones:0.61),(solo:0.91),(vest:0.57),(smile:0.64),(long_hair:0.68),(black_hair:0.83),(jewelry:0.50)",
"53": "(1girl:0.99),(grin:0.53),(outdoors:0.47),(earphones:0.65),(solo:0.94),(vest:0.56),(smile:0.73),(long_hair:0.65),(black_hair:0.85)",
"54": "(1girl:0.99),(grin:0.57),(road:0.44),(outdoors:0.52),(closed_eyes:0.57),(solo:0.94),(vest:0.42),(smile:0.79),(long_hair:0.82),(black_hair:0.88)",
"55": "(1girl:0.99),(grin:0.52),(road:0.46),(outdoors:0.55),(closed_eyes:0.41),(solo:0.94),(vest:0.41),(smile:0.74),(long_hair:0.76),(black_hair:0.87)",
"56": "(1girl:0.98),(grin:0.47),(road:0.40),(outdoors:0.55),(solo:0.95),(vest:0.38),(smile:0.70),(long_hair:0.71),(black_hair:0.87)",
"57": "(teeth:0.40),(solo:0.93),(looking_at_viewer:0.46),(long_hair:0.77),(black_hair:0.85),(1girl:0.98),(grin:0.43),(outdoors:0.66),(earphones:0.51),(black_eyes:0.41),(vest:0.55),(smile:0.69),(ground_vehicle:0.46),(road:0.47)",
"58": "(1girl:0.99),(grin:0.48),(road:0.35),(outdoors:0.48),(earphones:0.56),(solo:0.94),(vest:0.54),(smile:0.77),(long_hair:0.75),(black_hair:0.85)",
"59": "(1girl:0.98),(grin:0.46),(outdoors:0.42),(earphones:0.41),(solo:0.94),(vest:0.52),(smile:0.70),(long_hair:0.70),(black_hair:0.80)"
},
"tail_prompt": "",
"n_prompt": [
"(worst quality, low quality:1.4),nudity,simple background,border,mouth closed,text, patreon,bed,bedroom,white background,((monochrome)),sketch,(pink body:1.4),7 arms,8 arms,4 arms"
],
"is_single_prompt_mode": false,
"lora_map": {
"share/Lora/muffet_v2.safetensors": 1.0,
"share/Lora/add_detail.safetensors": 1.0
},
"motion_lora_map": {
"models/motion_lora/v2_lora_PanLeft.ckpt": 1.0
},
"ip_adapter_map": {
"enable": true,
"input_image_dir": "ip_adapter_image/test",
"save_input_image": true,
"resized_to_square": false,
"scale": 0.5,
"is_plus_face": true,
"is_plus": true
},
"controlnet_map": {
"input_image_dir": "..\\stylize\\2023-10-03T11-34-13-sample-majicmixrealistic_v5preview\\00_controlnet_image",
"max_samples_on_vram": 200,
"max_models_on_vram": 3,
"save_detectmap": true,
"preprocess_on_gpu": true,
"is_loop": false,
"controlnet_tile": {
"enable": true,
"use_preprocessor": true,
"guess_mode": false,
"controlnet_conditioning_scale": 1.0,
"control_guidance_start": 0.0,
"control_guidance_end": 1.0,
"control_scale_list": []
},
"controlnet_ip2p": {
"enable": true,
"use_preprocessor": true,
"guess_mode": false,
"controlnet_conditioning_scale": 0.5,
"control_guidance_start": 0.0,
"control_guidance_end": 1.0,
"control_scale_list": []
},
"controlnet_lineart_anime": {
"enable": true,
"use_preprocessor": true,
"guess_mode": false,
"controlnet_conditioning_scale": 1.0,
"control_guidance_start": 0.0,
"control_guidance_end": 1.0,
"control_scale_list": [
0.5,
0.4,
0.3,
0.2,
0.1
]
},
"controlnet_openpose": {
"enable": true,
"use_preprocessor": true,
"guess_mode": false,
"controlnet_conditioning_scale": 1.0,
"control_guidance_start": 0.0,
"control_guidance_end": 1.0,
"control_scale_list": [
0.5,
0.4,
0.3,
0.2,
0.1
]
},
"controlnet_softedge": {
"enable": true,
"use_preprocessor": true,
"preprocessor": {
"type": "softedge_pidsafe",
"param": {}
},
"guess_mode": false,
"controlnet_conditioning_scale": 1.0,
"control_guidance_start": 0.0,
"control_guidance_end": 1.0,
"control_scale_list": [
0.5,
0.4,
0.3,
0.2,
0.1
]
},
"controlnet_shuffle": {
"enable": true,
"use_preprocessor": true,
"guess_mode": false,
"controlnet_conditioning_scale": 1.0,
"control_guidance_start": 0.0,
"control_guidance_end": 1.0,
"control_scale_list": [
0.5,
0.4,
0.3,
0.2,
0.1
]
},
"controlnet_depth": {
"enable": true,
"use_preprocessor": true,
"guess_mode": false,
"controlnet_conditioning_scale": 1.0,
"control_guidance_start": 0.0,
"control_guidance_end": 1.0,
"control_scale_list": [
0.5,
0.4,
0.3,
0.2,
0.1
]
},
"controlnet_canny": {
"enable": true,
"use_preprocessor": true,
"guess_mode": false,
"controlnet_conditioning_scale": 1.0,
"control_guidance_start": 0.0,
"control_guidance_end": 1.0,
"control_scale_list": [
0.5,
0.4,
0.3,
0.2,
0.1
]
},
"controlnet_inpaint": {
"enable": true,
"use_preprocessor": true,
"guess_mode": false,
"controlnet_conditioning_scale": 1.0,
"control_guidance_start": 0.0,
"control_guidance_end": 1.0,
"control_scale_list": [
0.5,
0.4,
0.3,
0.2,
0.1
]
},
"controlnet_lineart": {
"enable": true,
"use_preprocessor": true,
"guess_mode": false,
"controlnet_conditioning_scale": 1.0,
"control_guidance_start": 0.0,
"control_guidance_end": 1.0,
"control_scale_list": [
0.5,
0.4,
0.3,
0.2,
0.1
]
},
"controlnet_mlsd": {
"enable": true,
"use_preprocessor": true,
"guess_mode": false,
"controlnet_conditioning_scale": 1.0,
"control_guidance_start": 0.0,
"control_guidance_end": 1.0,
"control_scale_list": [
0.5,
0.4,
0.3,
0.2,
0.1
]
},
"controlnet_normalbae": {
"enable": true,
"use_preprocessor": true,
"guess_mode": false,
"controlnet_conditioning_scale": 1.0,
"control_guidance_start": 0.0,
"control_guidance_end": 1.0,
"control_scale_list": [
0.5,
0.4,
0.3,
0.2,
0.1
]
},
"controlnet_scribble": {
"enable": true,
"use_preprocessor": true,
"guess_mode": false,
"controlnet_conditioning_scale": 1.0,
"control_guidance_start": 0.0,
"control_guidance_end": 1.0,
"control_scale_list": [
0.5,
0.4,
0.3,
0.2,
0.1
]
},
"controlnet_seg": {
"enable": true,
"use_preprocessor": true,
"guess_mode": false,
"controlnet_conditioning_scale": 1.0,
"control_guidance_start": 0.0,
"control_guidance_end": 1.0,
"control_scale_list": [
0.5,
0.4,
0.3,
0.2,
0.1
]
},
"controlnet_ref": {
"enable": false,
"ref_image": "ref_image/ref_sample.png",
"attention_auto_machine_weight": 0.3,
"gn_auto_machine_weight": 0.3,
"style_fidelity": 0.5,
"reference_attn": true,
"reference_adain": false,
"scale_pattern": [
1.0
]
}
},
"upscale_config": {
"scheduler": "k_dpmpp_sde",
"steps": 20,
"strength": 0.5,
"guidance_scale": 10,
"controlnet_tile": {
"enable": true,
"controlnet_conditioning_scale": 1.0,
"guess_mode": false,
"control_guidance_start": 0.0,
"control_guidance_end": 1.0
},
"controlnet_line_anime": {
"enable": false,
"controlnet_conditioning_scale": 1.0,
"guess_mode": false,
"control_guidance_start": 0.0,
"control_guidance_end": 1.0
},
"controlnet_ip2p": {
"enable": true,
"controlnet_conditioning_scale": 0.5,
"guess_mode": false,
"control_guidance_start": 0.0,
"control_guidance_end": 1.0
},
"controlnet_ref": {
"enable": false,
"use_frame_as_ref_image": false,
"use_1st_frame_as_ref_image": false,
"ref_image": "ref_image/path_to_your_ref_img.jpg",
"attention_auto_machine_weight": 1.0,
"gn_auto_machine_weight": 1.0,
"style_fidelity": 0.25,
"reference_attn": true,
"reference_adain": false
}
},
"stylize_config": {
"0": {
"width": 512,
"height": 512,
"length": 60,
"context": 16,
"overlap": 4,
"stride": 0
},
"1": {
"steps": 25,
"guidance_scale": 7.5,
"width": 768,
"height": 768,
"length": 60,
"context": 8,
"overlap": 2,
"stride": 0,
"controlnet_tile": {
"enable": true,
"use_preprocessor": true,
"guess_mode": false,
"controlnet_conditioning_scale": 1.0,
"control_guidance_start": 0.0,
"control_guidance_end": 1.0,
"control_scale_list": []
},
"controlnet_ip2p": {
"enable": false,
"use_preprocessor": true,
"guess_mode": false,
"controlnet_conditioning_scale": 0.5,
"control_guidance_start": 0.0,
"control_guidance_end": 1.0,
"control_scale_list": []
},
"ip_adapter": false,
"reference": false,
"interpolation_multiplier": 1
}
},
"output": {
"format": "mp4",
"fps": 10,
"encode_param": {
"crf": 10
}
},
"result": {}
}I think it is a fair output.
SoftologyPro
commented
1 year ago I think I see my problem now. I am not editing the prompts into the correct json file. I put the prompts into the usual prompt_travel.json and not prompt.json for this specifc styling. So I edit prompt.json with the giger prompt as the head and tail prompt as follows
"head_prompt": "(painted by giger:0.99)",
"tail_prompt": "(painted by giger:0.99)",How exactly do head and tail work? Are they added at the start and end of every other prompt? Or do I need to edit the giger prompts manually before every other frame prompt?
And that gives this result. No change and is not in ther style of giger? https://github.com/s9roll7/animatediff-cli-prompt-travel/assets/47686889/cfb40d9b-5f89-4581-950f-68fc686231cb
How do I get a giger look like your 2 example movies?
s9roll7
commented
1 year ago Are they added at the start and end of every other prompt?
yes
"path": "models\\sd\\majicmixRealistic_v5Preview.safetensors", "lora_map": { "share/Lora/muffet_v2.safetensors": 1.0, "share/Lora/add_detail.safetensors": 1.0 },
I don't think this model and lora know the word "giger".
SoftologyPro
commented
1 year ago OK, that would make sense. I cannot file the file muffet_v2.safetensors anywhere on my hard drive. What would a good general model and lora be to use for more general prompts? Is there a list of compatible LORAs that can be used?
Which models/checkpoints/loras did you use for your giger movies above?
dragoriano
commented
1 year ago i think you could use this model for giger https://civitai.com/models/20846/biomechanicals-hr-giger
and/or use this lora
https://civitai.com/models/131840/gigercraftxs https://civitai.com/models/20846/biomechanicals-hr-giger
I have been trying some video stylizations and find it hard to get the prompts to really influence the outcome. The LORA selected seems much more influencial on the end result.
For example, I want to give a video a look like it was painted by giger. If I set the head and tail prompts to
"in the style of a painting by giger"and then set the prompts to(500 so it is past the length of the frames to make sure they all get covered) the end result is nothing like giger.
If I try and change the prompts using brackets and weights like
(a highly detailed airbrush painting in the style of giger:1.0),((painted by giger:2.0))that does not seem to help much if at all. What is the purpose of the brackets and is doubling them up supposed to enforce the prompt?I have tried different models and LORAs without much success. The animation LORAs just give cartoonish results, the "realism" LORAs give big eyed asian girls. No real giger influence.
Any tips on prompts to get the Video Stylization to be able to stylize a video based on an artist prompt? The styling for movies using the cartoon LORAs are so temporally smooth and look great. If only I could control other styles with the prompting?