Question: Is it feasible to implement AnimateDiff in CoreML?

[cylemguang42]

I am a noob admittedly when it comes to programming, so I may be above my head / punching above my weight (pun intended)

I am wondering / attempting if it’s possible to modify the following node:

` class AnimateDiffLoaderWithContext: @classmethod def INPUT_TYPES(s): return { "required": { "model": ("MODEL",), "model_name": (get_available_motion_models(),), "beta_schedule": (BetaSchedules.get_alias_list_with_first_element(BetaSchedules.SQRT_LINEAR),),

"apply_mm_groupnorm_hack": ("BOOLEAN", {"default": True}),

        },
        "optional": {
            "context_options": ("CONTEXT_OPTIONS",),
            "motion_lora": ("MOTION_LORA",),
            "motion_model_settings": ("MOTION_MODEL_SETTINGS",),
            "motion_scale": ("FLOAT", {"default": 1.0, "min": 0.0, "step": 0.001}),
            "apply_v2_models_properly": ("BOOLEAN", {"default": False}),
        }
    }

RETURN_TYPES = ("MODEL",)
CATEGORY = "Animate Diff 🎭🅐🅓"
FUNCTION = "load_mm_and_inject_params"

def load_mm_and_inject_params(self,
    model: ModelPatcher,
    model_name: str, beta_schedule: str,# apply_mm_groupnorm_hack: bool,
    context_options: ContextOptions=None, motion_lora: MotionLoRAList=None, motion_model_settings: MotionModelSettings=None,
    motion_scale: float=1.0, apply_v2_models_properly: bool=False,
):
    # load motion module
    mm = load_motion_module(model_name, motion_lora, model=model, motion_model_settings=motion_model_settings)
    # set injection params
    injection_params = InjectionParams(
            video_length=None,
            unlimited_area_hack=False,
            apply_mm_groupnorm_hack=True,
            beta_schedule=beta_schedule,
            injector=mm.injector_version,
            model_name=model_name,
            apply_v2_models_properly=apply_v2_models_properly,
    )
    if context_options:
        # set context settings TODO: make this dynamic for future purposes
        if type(context_options) == UniformContextOptions:
            injection_params.set_context(
                    context_length=context_options.context_length,
                    context_stride=context_options.context_stride,
                    context_overlap=context_options.context_overlap,
                    context_schedule=context_options.context_schedule,
                    closed_loop=context_options.closed_loop,
                    sync_context_to_pe=context_options.sync_context_to_pe,
            )
            injection_params.noise_type = context_options.noise_type
    if motion_lora:
        injection_params.set_loras(motion_lora)
    # set motion_scale and motion_model_settings
    if not motion_model_settings:
        motion_model_settings = MotionModelSettings()
    motion_model_settings.attn_scale = motion_scale
    injection_params.set_motion_model_settings(motion_model_settings)
    # inject for use in sampling code
    model = inject_params_into_model(model, injection_params)

    return (model,)`

— https://github.com/Kosinkadink/ComfyUI-AnimateDiff-Evolved/blob/main/animatediff/nodes.py

I was analyzing how you did the CoreMLSampler from KSampler, and I first attempted to convert a safetensor motion module file to .mlmodelc format, but to no avail. (https://huggingface.co/CiaraRowles/TemporalDiff/blob/main/temporaldiff-v1-animatediff.safetensors)

Thank you for your time and attention

[aszc-dev]

Hello. First of all, it's great to hear you're conducting your own experiments. The topic of Core ML is very broad, too broad for one person, especially considering how much is happening in Stable Diffusion world lately.

I remember going through Kosinkadink's code to see if there was a simple and easy way of adding AnimateDiff to existing Core ML workflow. I recall coming up to a conclusion that there's no easy way, but can't remember why, how it worked, etc.

In principle, any torch model can be converted to Core ML, but there are limitations - converted model is "closed", weights and structure of the model cannot be modified (that's why LoRAs need to be applied beforehand). As I understand, applying the motion model modifies the model (can't remember how exactly), so any injections must happen before conversion.

Perhaps a new torch model has to be implemented for this to work and to be worth the effort (it has to be ANE compatibile). Currently I don't have enough knowledge to do this, but if I find some time, and no one else provides any solution by then, I'll try to dive deeper into this subject and see what I can come up with.

Cheers