Accessing the 'Modify Content' inpaint method via replicate api

[jareddr]

Hi there,

Once again, thanks for putting this together, it's been an amazing tool for me in a project I'm working on.

Today I built a little interface for inpainting and have been using it to add detail to images. Sometime it works great and does the exact inpaint I was looking for. Sometimes however, it seems to ignore the inpaint prompt and simply redraws the original image a bit.

To investigate I installed Fooocus desktop and messed around with the gradio client. I noticed that it has multiple methods for inpainting. If you choose the 'Modify Content' inpaint method it seems to very strongly follow my requests for a new inpaint object. The objects often look a bit out of place, but it almost always draw them in.

To find out how this was working I dug into the Fooocus code and found this block of code from the gradio webUI

        def inpaint_mode_change(mode):
            assert mode in modules.flags.inpaint_options

            # inpaint_additional_prompt, outpaint_selections, example_inpaint_prompts,
            # inpaint_disable_initial_latent, inpaint_engine,
            # inpaint_strength, inpaint_respective_field

            if mode == modules.flags.inpaint_option_detail:
                return [
                    gr.update(visible=True), gr.update(visible=False, value=[]),
                    gr.Dataset.update(visible=True, samples=modules.config.example_inpaint_prompts),
                    False, 'None', 0.5, 0.0
                ]

            if mode == modules.flags.inpaint_option_modify:
                return [
                    gr.update(visible=True), gr.update(visible=False, value=[]),
                    gr.Dataset.update(visible=False, samples=modules.config.example_inpaint_prompts),
                    True, modules.config.default_inpaint_engine_version, 1.0, 0.0
                ]

            return [
                gr.update(visible=False, value=''), gr.update(visible=True),
                gr.Dataset.update(visible=False, samples=modules.config.example_inpaint_prompts),
                False, modules.config.default_inpaint_engine_version, 1.0, 0.618
            ]

It seems to be setting some advanced_parameters when you pick one

inpaint_disable_initial_latent, inpaint_engine, inpaint_strength, inpaint_respective_field

it looks like the 'modify object' inpaint version wants the following values

inpaint_disable_initial_latent = True
='v2.6'
inpaint_strength=1.0
inpaint_respective_field=0.0

I was able to test this out by overriding the default advanced_parameters with these values, and indeed, the inpaint tool is much more likely to add a new object to a scene!

From what I've seen, the replicate version doesn't currently support changing advanced_params, so I attempted to add a method that would allow access to these other inpaint methods

I wrote some code to add a new cog parameter inpaint_method in the predict.py file.

     inpaint_input_image: Path = Input(default=None, 
                                description="Input image for inpaint or outpaint, keep None for not inpaint or outpaint. Please noticed, `uov_input_image` has bigger priority is not None."),
        inpaint_input_mask: Path = Input(default=None, 
                                description="Input mask for inpaint"),
        inpaint_method: str = Input(default='Balanced', choices=['Modify Object (add objects, change background, etc.)', 'Improve Detail (face, hand, eyes, etc.)', 'Balanced'], description="Changes the technique used to inpaint"),

then in parameters.py

I use this value to correctly set the advanced_params to invoke each different inpaint technique.

At line 169 in parameters.py I added

            #set some advanced_params that impact inpaint_medhod
            if inpaint_method == 'Improve Detail (face, hand, eyes, etc.)':
                inpaint_disable_initial_latent = False
                inpaint_strength = 0.5
                inpaint_respective_field = 0.0
                inpaint_engine = 'None'
            elif inpaint_method == 'Modify Object (add objects, change background, etc.)':
                inpaint_disable_initial_latent = True
                inpaint_strength = 1.0
                inpaint_respective_field = 0.0

This short patch unlocks more of the inpaint techniques available in fooocus.

Let me know if you'd like me to do a pull request and also let me know if you think I've missed something here as well.

Cheers,

J

[mrhan1993]

Thank you for your work. PR please

[rohanath123]

@jareddr Amazing work! Exactly what I was looking for. PR please!

[jareddr]

Added to the realistic preset branch with this pull request: https://github.com/mrhan1993/Fooocus-API/pull/228