Patchmatch for outpainting

gadicc commented 1 year ago

Follow on from https://github.com/kiri-art/docker-diffusers-api/pull/1 (merged to patchmatch branch).

Hey @msuess, thanks again for your awesome work here. I've finally had a chance to look at this a bit more, sorry it took me a while.

I'm less familiar with PatchMatch, but from my understanding - and please correct me if I'm wrong - there's nothing diffusers or even GPU specific to the code here, which makes me think it would be better to put in it's own container, and even host somewhere with CPU serverless for faster times and cheaper bills. Thoughts?

Also, 2 days ago, a Stable Diffusion model finetuned for inpainting was released, and apparently works really well for outpainting too. Would love your feedback if you have a chance, to let us know how it compares to patchmatch, and we should proceed with both models (See https://github.com/kiri-art/docker-diffusers-api/blob/main/CHANGELOG.md for more info).

gadicc commented 1 year ago

@msuess, my apologies, looked at this too quickly, and obviously makes sense to keep inside the same container. I need to spend a bit more time with the new pipeline, but I think it only works with the specific SD inpainting model anyway, and I'd like to leave inpainting available for any model, and patchmatch seems the best way to do this. I might be a bit focused on getting all the new SD stuff working at the start of the week, but I hope to have this properly merged by end of the week, and thank you for your patience! (and again for the contribution!).

gadicc commented 1 year ago

@msuess, thanks for your patience. When you have a chance, please take look at the feat/patchmatch branch, I believe this is ready to be merged now (edit: merged to dev!).

My changes (following on from your awesome work):

git clone rather than copy / include PyPatchMatch source
modify patch_match.py to skip recompile on non-nt OSes
Dockerfile build stuff
make it optional... note there are TWO USE_PATCHMATCH lines that need to be set (only if you can't use docker build args... which should be landing at banana soon).
add a basic test... hope I got this right, feedback welcome.

Think that's it! It works for me locally, and here's an example input and output:

girl_with_pearl_earing_outpainting_in.png

        {
            "modelInputs": {
                "prompt": "girl with a pearl earing standing in a big room",
                "init_image": b64encode_file(
                    "girl_with_pearl_earing_outpainting_in.png"
                ),
            },
            "callInputs": {
                "MODEL_ID": "CompVis/stable-diffusion-v1-4",
                "PIPELINE": "StableDiffusionInpaintPipelineLegacy",
                "SCHEDULER": "DDIM",  # Note, as of diffusers 0.3.0, no LMS yet
                "FILL_MODE": "patchmatch",
            },
        },

output outpaint

kiri-art / docker-diffusers-api

Patchmatch for outpainting #4