search

exx8 / differential-diffusion

399 stars 23 forks source link

How to do outpainting correctly? #26

Open darshats opened 6 months ago

darshats commented 6 months ago

Hi, Thanks for the great work! My question is if this approach be used to do outpainting? Say I have an object without any other surrounding context, I'm having trouble getting it to fill up the context based on prompt. Below on left is the object. The diffdiff map is the second image and outputs are the right two images. The trouble is it always adds another object of the same type in the spatial location that is masked. I used a binary diffdiff map as its ok to outpaint anything as per prompt. Using a gradient mask based on distance also doesnt help as areas close to the object also need complete infilling.

image

neph1 commented 2 months ago

I haven't yet tried out differential-diffusion, but I think this is a case of 'too much freedom, too little context'. I think you would be better off generating a background you think would be suitable, separately, then merge the two images (crudely) and make a mask to let SD work on the area directly surrounding the bottle (inpainting rather than outpainting).

Clement-Lelievre commented 1 month ago

if you can afford that, maybe one thing you could try is leverage the change map by setting the pixels just surrounding the bottle (or your object more generally) to non-zero, to try and force the pipe to respect what was just around the bottle