Infinitely tileable videos

[isaac-art]

Couldn't see anything when searching so apologies if already done, but I took the circular convolution patch from: https://github.com/TomMoore515/material_stable_diffusion

and applied it before running walk and it worked to produce tileable videos, fork here with notebook for anyone interested: https://github.com/isaac-art/tile-stable-diffusion-videos/

Could be added as enhancement to walk parameters, adding a 'tileable=True/False' option and running the patch?

Demo gif here: https://isaacclarke.com/tiletest/

[nateraw]

Woah this is pretty sweet. What exactly did you add? Would be easier to see if you open PR against this repo - can you do that? Even if its not something we want to merge/we would rather solve a different way, it will be easier for me to determine next steps if I see the diffs. (Sifting through commit diffs is annoying to me haha).

I (maybe naively) assume this is just a model swap, eh? if that's the case, we have open issues related to that. If its something else, I'd love to take a look.

The example videos are 🔥🔥🔥!!!!!! I see huge use for this being background to websites. TYSM for sharing.

[nateraw]

Took a look at the notebook. Looks like its just the patch_conv situation that's been added. It would have to be called pre-init of the pipeline though, eh?

I think this is yet another reason to update to refactor for #28 or #25

[isaac-art]

Hey, yes its just the convolution patch in the notebook that makes it tiled (the rest of the repo changes are me playing around with things).

i wish i knew more but my approach is pretty hacky changing till things work. On reflection I guess this patch is probably fine staying in a notebook really as its small enough to copy-paste in when its needed.

but the pipeline idea sounds good being able to switch to other versions of SD 👍

[nateraw]

hmm looks like one could actually turn this on and off if you have record of the original patch params. Would be really awesome to add enable_tiled_conv and disable_tiled_conv (or reset_tiled_conv, etc) to the pipeline class.

Would love some help with this 😄 If you find time to play with it, just add it to the pipeline, and don't bother adding it to walk. huge refactor coming soon with that

[nateraw]

opting for throwing this in a wrapper around from_pretrained instead, and not worrying about making it disable-able.

After my wip refactor is done, you'll be able to do something like this (names might change):

from stable_diffusion_videos import StableDiffusionWalkPipeline

pipeline = StableDiffusionWalkPipeline.from_pretrained(
    "<model-id>",
    use_auth_token=True,
    torch_dtype=torch.float16,
    revision="fp16",
    tiled=tiled,  # Enables tiling
)

[nateraw]

This is now possible on main/in the 0.5.0 release. Cheers