This fixes a shape mismatch with controlnet that at least exists with the reference_only processor. Possibly others too, but I haven't tested and this shouldn't wrongly affect anything except similar shape mismatches.
Basically using controlnet reference_only concats the last_to_v we get along the sequence length dimension. This therefore takes only the first seq_len slices along that axis, resolving the issue.
There are still issues with controlnet, it does not work with batch sizes above 1 and I am still investigating (maybe this is a mercy, it's already slow enough combining this with controlnet at batch 1...). But it can at least be used.
v0xie
commented
5 months ago
This fixes a shape mismatch with controlnet that at least exists with the reference_only processor. Possibly others too, but I haven't tested and this shouldn't wrongly affect anything except similar shape mismatches.
Basically using controlnet reference_only concats the last_to_v we get along the sequence length dimension. This therefore takes only the first seq_len slices along that axis, resolving the issue.
There are still issues with controlnet, it does not work with batch sizes above 1 and I am still investigating (maybe this is a mercy, it's already slow enough combining this with controlnet at batch 1...). But it can at least be used.