search

mcmonkeyprojects / SwarmUI

SwarmUI, A Modular Stable Diffusion Web-User-Interface, with an emphasis on making powertools easily accessible, high performance, and extensibility.
MIT License
255 stars 22 forks source link

Inpainting (with mask shrink grow) & segmenting should use the model native resolution (or current resolution settings) IMHO #19

Open CholmesFr opened 1 week ago

CholmesFr commented 1 week ago

Hi! Awesome work with swarmUI. It's almost perfect for my needs apart for an issue with inpainting and segmenting.

Sometime when using inpainting with mask shrink grow, the inpaint is deformed / low quality. Looking at the generated workflow, it seems that swarm use the current image resolution for the inpaint: image

Step to reproduce: 1°) import a big image (or scale an image to a big resolution) 2°) inpaint a small part using the mask shrink grow option 3°) import the workflow (great option for debuging btw) 4°) check the size of SwarmImageScaleForMP => it's set to the current image resolution

When working in native resolution (eg 1024x1024 for SDXL) it's not an issue. However when working with a high resolution image, it's problematic because the inpaint resolution can exceed the range of the model and the memory can explode.

Here an example when trying to inpaint a small part of a big image with a sd1.5 model. After importing it, if I force the native resolution in the workflow we can see that the inpaint lose quality as the resolution grows. image (Same seed but Gandy is not recognizable at >1024, the skin is smoother and the glasses frame is deformed)

One solution would be to always use the native resolution in this case: WorkflowGenerator.cs line 327 :

string scaledImage = CreateNode("SwarmImageScaleForMP", new JObject() { ["image"] = new JArray() { croppedImage, 0 }, ["width"] = UserInput.Get(T2IParamTypes.Model).StandardWidth, // use model native resolution ["height"] = UserInput.Get(T2IParamTypes.Model).StandardHeight, ["can_shrink"] = true // Shrink inpaint if mask is too big });

It works great but the inpaint is always square (not sure if it's better or not but it should be possible to keep the mask ratio if necessary)

The other solution would be to give the user more control and use the selected resolution in the UI. I believe some changes in JS part would be necessary as I don't think that these parameters are accessible in the C# function.

Segmenting use the same portion of code but that's great because it suffer the same issue, so both would be fixed.

Let me know what you think and if I should work on a PR.

jetsetcorvette commented 1 week ago

I see similar bug

If you:

  1. Bring or generate image
  2. Choose "Edit Image"
  3. Mask your area you want.
  4. use "Mask grow/shrink" option
  5. Generate your new content (inpaint)

Now problem is if you want to switch images:

  1. You drag /drop new image and say "reuse parameters"
  2. if you immediately choose "Edit image" the image will be imported into the image editor at very low resolution and very pixelated. And it will not work as expected, the output image will now be low resolution

Workaround when switching images you are working on:

  1. Import image by dragging it to the main Generate tab
  2. Choose "reuse parameters"
  3. Generate again to regenerate the image. This appears to reset the inpainting parameters correctly
  4. Now if you do "edit image" it will appear correctly in the image editor once again