No removing effect

[rzamarefat]

Thanks for the great repo. I was trying to remove an object from an image. I try to use the canny method and also set the prompt to be nothing and decrease the controlnet_conditioning_scale to 0. This works on the default image in the colab but not with any other image. In fact, it produces sth else in the masked area. Could you please explain what else should be done to have removing effect ?

[mikonvergence]

Hi @rzamarefat! Can you give more detail on how you used the pipeline (all set parameter values etc) and share what kind of result you obtained?

[rzamarefat]

I use the following snippet: new_image = pipe( "", num_inference_steps=20, generator=generator, image=image, control_image=canny_image, controlnet_conditioning_scale = 0, mask_image=mask_image ).images[0] The text prompt is empty. The controlnet_conditioning_scale is set to zero. And the image, the mask, and the results I use:

[mikonvergence]

and what would you expect in the output?

[rzamarefat]

Thank U ur quick response. I expect complete removal of the masked area in a cohesive way. Like there is no vase there at all.

[mikonvergence]

It feels like regular inpainting pipeline should be more appropriate in that case.

I am not sure I understand why controlnet would be necessary.

Also, consider that if you mask out only the vase, but leave the reflection, it is likely that the model will still reconstruct the object based on that reflection.

[mikonvergence]

I tested your image in my HF app that is based on this pipeline (https://huggingface.co/spaces/mikonvergence/mask-and-sketch) and got this result:

As I said, it is important to mask out all areas related to the object (shadow, reflection). Also, I increased the steps to 80 for higher quality inpainting.

[rzamarefat]

So, the only difference that you have made was to mask the shadow along with the object itself? Because I've tested it but got some other objects again. I'll check it out again. If anything was different please let me know.

[mikonvergence]

No, in this case I'm using the sketch model, since the canny edge (which you seem to be using) is probably too detailed as a guide. Further, in the app, the user has the opportunity to redraw the guide, so the region of the guide affected by the middle object is removed.