search

kiri-art / docker-diffusers-api

Diffusers / Stable Diffusion in docker with a REST API, supporting various models, pipelines & schedulers.
https://kiri.art/
MIT License
202 stars 94 forks source link

feat: add outpainting support #1

Closed msuess closed 1 year ago

msuess commented 1 year ago

Hey Gadi!

This PR adds patchmatch outpainting support a la stablediffusion-infinity to this image 🪄 ✨

The actual PyPatchMatch code is not mine, I have lifted it from the amazing Jiayuan Mao, who in turn lifted the C code from Younesse Andam. See more info in PyPatchMatch/README.md. I only removed some code that was related to compiling on travis, as well as the examples directory.

Implementation Notes

I don't really do python or ML development, so take all of this with boatloads of grains of salt.

This feature adds a dependency to OpenCV, which is rather heavy. You need to install libopencv-dev for your platform. I have added it to the docker image. Maybe this could be added to requirements.txt instead?

It also adds an additional build step, you have to cd PyPatchMatch && make before executing the code (this is documented in the updated README.md). It's really quick though and only needs to be done once.

The patchmatch step requires the init_image to be a PNG with alpha channel (the default when calling htmlCanvasElement.toDataURL). A mask_image is not needed, as patchmatch generates it for us. I have left the output format of the inference function as JPEG to not break your existing UI, but it might make sense to change it to PNG down the road.

To enable patchmatch and outpainting, pass INIT_MODE: "patchmatch" and PIPELINE: "StableDiffusionInpaintPipeline" in callInputs.

I have verified that this code works in my private fork of banana-sd-base (that includes my API keys) and copied the changes over, so there's a chance that something is missing/broken. Since I don't have a machine capable of running SD it's quite cumbersome for me to test and I need my banana model to work since I'm actively developing against it. So please test this locally and let me know if anything breaks. Sorry for the inconvenience.

Oh and BTW thanks for all your work on banana-sd-base and kiri.art!

Cheers ✌️

gadicc commented 1 year ago

Hey @msuess!

This is awesome... thanks so, so much for this contribution. This is such an exciting space right now and it's great to be collaborating with others in open source to bring more and more magic which just would not be possible by one person alone.

This all looks great. I'll need a bit of time to play around with it a bit before I can merge to main, but I'm going to make a new patchmatch branch on here in the meantime and will keep you posted. Regrettably I won't have any time over the weekend, but I expect I'll be in touch early next week.

In the meantime, have a lovely weekend, and thanks again for this greatly appreciated contribution! :raised_hands:

gadicc commented 1 year ago

Not possible to leave a PR open after merge, but I'll still post back here once I've had a time to work on it :pray: