[Bug]: Latest Automatic WebUI breaks openOutpaint

[vladmandic]

What happened?

Immediately after opening WebUI, it logs errors if openOutpaint extension is enabled:

Launching Web UI with arguments: --api --xformers --disable-console-progressbars
WARNING:root:Pytorch pre-release version 2.0.0.dev20230118+cu118 - assuming intent to test it
WARNING:root:Pytorch pre-release version 2.0.0.dev20230118+cu118 - assuming intent to test it
LatentDiffusion: Running in eps-prediction mode
DiffusionWrapper has 859.52 M params.
Loading weights [cc6cb27103] from /home/vlado/dev/automatic/models/Stable-diffusion/sd-v15-runwayml.ckpt
Loading VAE weights specified in settings: /home/vlado/dev/automatic/models/VAE/vae-ft-mse-840000-ema-pruned.ckpt
Applying xformers cross attention optimization.
Textual inversion embeddings loaded(1): buffy
Model loaded in 8.0s (0.2s create model, 7.2s load weights).
Running on local URL:  http://127.0.0.1:7860

To create a public link, set `share=True` in `launch()`.
ERROR:    Exception in ASGI application
Traceback (most recent call last):
  File "/home/vlado/.local/lib/python3.10/site-packages/anyio/streams/memory.py", line 94, in receive
    return self.receive_nowait()
  File "/home/vlado/.local/lib/python3.10/site-packages/anyio/streams/memory.py", line 89, in receive_nowait
    raise WouldBlock
anyio.WouldBlock
...
  File "/home/vlado/dev/automatic/modules/api/api.py", line 255, in extras_single_image_api
    reqDict['image'] = decode_base64_to_image(reqDict['image'])
  File "/home/vlado/dev/automatic/modules/api/api.py", line 56, in decode_base64_to_image
    return Image.open(BytesIO(base64.b64decode(encoding)))
  File "/home/vlado/.local/lib/python3.10/site-packages/PIL/Image.py", line 3283, in open
    raise UnidentifiedImageError(msg)
PIL.UnidentifiedImageError: cannot identify image file <_io.BytesIO object at 0x7f5de5d06200>

Issue is that openOutpaint makes an invalid request to /sdapi/v1/extra-single-image

And yes, inside openOutpaint that request is even marked as ignorable/invalid

                const response = await fetch(extras_url, {
            method: "POST",
            headers: {
                Accept: "application/json",
                "Content-Type": "application/json",
            },
            body: JSON.stringify(purposefully_incorrect_data),
        });
        const data = await response.json();

        console.log(
            "[index] purposefully_incorrect_data response, ignore above error"
        );

But the issue is that it now actually breaks WebUI ?!

There has to be a better way than to post invalid request and try to ignore it?

Why not use /sdapi/v1/upscalers endpoint?

And yes, invalid request should not break WebUI - I'll do a PR to do graceful handling of invalid API requests on WebUI server side.

But it still needs to be address in the extension itself.

Steps to reproduce the problem

N/A

What should have happened?

N/A

Commit where the problem happens

fc8e6fb557832320a770af80ec661d59f99878a4

What platforms do you use to access openOutpaint?

Windows

What browsers do you use to access the UI ?

Microsoft Edge

Browser Extensions/Addons

N/A

AUTOMATIC1111 webUI Commandline Arguments

N/A

Additional information

No response

[seijihariki]

The issue was mostly that we didn't have any way to reliably fetch valid upscaler names. The upscaler getting endpoint returned inconsistently, some using the name, others using model_name for valid upscaler parameter value, and it was impossible to determine automatically which used what values for a parameter.

Will see what we can do... still, a 500 error on the API should not crash the webui... I am having the same issue, it returns a 500 but the only thing that happens is that my upscalers list is empty:

[vladmandic]

The problem that on server side, it was not a 500 error, it blindly tried to decode a null image and PIL crashed. I've just added exception handling in PR https://github.com/AUTOMATIC1111/stable-diffusion-webui/pull/7140, now it will be actual 500 error

Now, if you have issues using /sdapi/v1/upscalers endpoint, let me know and i can take a look at it....

[seijihariki]

Can you give it a look now? Upscalers still not working, but at least should always send a valid image. (Now sure why it would not send a valid image before, but well...)

[vladmandic]

sorry, i cant now, just stepped out for dinner (typing on cellphone while waiting for uber)

[zero01101]

Now, if you have issues using /sdapi/v1/upscalers endpoint, let me know and i can take a look at it....

sigh

as mentioned, the "official" endpoints are completely unreliable and have returned trash data since we introduced upscaler support on this app, literally months ago, and even now it doesn't provide complete output - for example, i've got remacri installed but it doesn't show up anywhere in upscalers or realesrgan API endpoints, plus it lists realesrgan models i don't have installed (which is likely by design)

curl -X 'GET' \
  'http://127.0.0.1:7860/sdapi/v1/upscalers' \
  -H 'accept: application/json'

  [
  {
    "name": "None",
    "model_name": null,
    "model_path": "E:\\storage\\stable-diffusion-webui\\models\\None",
    "model_url": null
  },
  {
    "name": "Lanczos",
    "model_name": null,
    "model_path": null,
    "model_url": null
  },
  {
    "name": "Nearest",
    "model_name": null,
    "model_path": null,
    "model_url": null
  },
  {
    "name": "ESRGAN",
    "model_name": "ESRGAN_4x",
    "model_path": "E:\\storage\\stable-diffusion-webui\\models\\ESRGAN",
    "model_url": "https://github.com/cszn/KAIR/releases/download/v1.0/ESRGAN.pth"
  },
  {
    "name": "ESRGAN",
    "model_name": "ESRGAN_4x",
    "model_path": "E:\\storage\\stable-diffusion-webui\\models\\ESRGAN",
    "model_url": "https://github.com/cszn/KAIR/releases/download/v1.0/ESRGAN.pth"
  },
  {
    "name": "RealESRGAN",
    "model_name": null,
    "model_path": "E:\\storage\\stable-diffusion-webui\\models\\RealESRGAN",
    "model_url": null
  },
  {
    "name": "RealESRGAN",
    "model_name": null,
    "model_path": "E:\\storage\\stable-diffusion-webui\\models\\RealESRGAN",
    "model_url": null
  },
  {
    "name": "RealESRGAN",
    "model_name": null,
    "model_path": "E:\\storage\\stable-diffusion-webui\\models\\RealESRGAN",
    "model_url": null
  },
  {
    "name": "RealESRGAN",
    "model_name": null,
    "model_path": "E:\\storage\\stable-diffusion-webui\\models\\RealESRGAN",
    "model_url": null
  },
  {
    "name": "LDSR",
    "model_name": null,
    "model_path": "E:\\storage\\stable-diffusion-webui\\models\\LDSR",
    "model_url": "https://heibox.uni-heidelberg.de/f/578df07c8fc04ffbadf3/?dl=1"
  },
  {
    "name": "SwinIR",
    "model_name": "SwinIR 4x",
    "model_path": "E:\\storage\\stable-diffusion-webui\\models\\SwinIR",
    "model_url": "https://github.com/JingyunLiang/SwinIR/releases/download/v0.0/003_realSR_BSRGAN_DFOWMFC_s64w8_SwinIR-L_x4_GAN.pth "
  }
]

curl -X 'GET' \
  'http://127.0.0.1:7860/sdapi/v1/realesrgan-models' \
  -H 'accept: application/json'

  [
  {
    "name": "R-ESRGAN General 4xV3",
    "path": "https://github.com/xinntao/Real-ESRGAN/releases/download/v0.2.5.0/realesr-general-x4v3.pth",
    "scale": 4
  },
  {
    "name": "R-ESRGAN General WDN 4xV3",
    "path": "https://github.com/xinntao/Real-ESRGAN/releases/download/v0.2.5.0/realesr-general-wdn-x4v3.pth",
    "scale": 4
  },
  {
    "name": "R-ESRGAN AnimeVideo",
    "path": "https://github.com/xinntao/Real-ESRGAN/releases/download/v0.2.5.0/realesr-animevideov3.pth",
    "scale": 4
  },
  {
    "name": "R-ESRGAN 4x+",
    "path": "https://github.com/xinntao/Real-ESRGAN/releases/download/v0.1.0/RealESRGAN_x4plus.pth",
    "scale": 4
  },
  {
    "name": "R-ESRGAN 4x+ Anime6B",
    "path": "https://github.com/xinntao/Real-ESRGAN/releases/download/v0.2.2.4/RealESRGAN_x4plus_anime_6B.pth",
    "scale": 4
  },
  {
    "name": "R-ESRGAN 2x+",
    "path": "https://github.com/xinntao/Real-ESRGAN/releases/download/v0.2.1/RealESRGAN_x2plus.pth",
    "scale": 2
  }
]

previously submitting a null image and intentionally invalid upscaler name to /sdapi/v1/extra-single-image would return a perfectly usable list of installed, available upscalers - earlier before this was submitted, i tried replacing it with a hardcoded 1x1 black pixel so it was actual image data, (seems https://github.com/zero01101/openOutpaint/commit/cb36c2b59984fb986f33544ed4554b160a0f7358 is still sending an empty string?) but now the invalid upscaler also results in a complete error instead of something hackily useful.

Traceback (most recent call last):
  File "D:\storage\stable-diffusion-webui\venv\lib\site-packages\uvicorn\protocols\http\h11_impl.py", line 407, in run_asgi
    result = await app(  # type: ignore[func-returns-value]
  File "D:\storage\stable-diffusion-webui\venv\lib\site-packages\uvicorn\middleware\proxy_headers.py", line 78, in __call__
    return await self.app(scope, receive, send)
  File "D:\storage\stable-diffusion-webui\venv\lib\site-packages\fastapi\applications.py", line 270, in __call__
    await super().__call__(scope, receive, send)
  File "D:\storage\stable-diffusion-webui\venv\lib\site-packages\starlette\applications.py", line 124, in __call__
    await self.middleware_stack(scope, receive, send)
  File "D:\storage\stable-diffusion-webui\venv\lib\site-packages\starlette\middleware\errors.py", line 184, in __call__
    raise exc
  File "D:\storage\stable-diffusion-webui\venv\lib\site-packages\starlette\middleware\errors.py", line 162, in __call__
    await self.app(scope, receive, _send)
  File "D:\storage\stable-diffusion-webui\venv\lib\site-packages\starlette\middleware\base.py", line 106, in __call__
    response = await self.dispatch_func(request, call_next)
  File "E:\storage\stable-diffusion-webui\modules\api\api.py", line 91, in log_and_time
    res: Response = await call_next(req)
  File "D:\storage\stable-diffusion-webui\venv\lib\site-packages\starlette\middleware\base.py", line 80, in call_next
    raise app_exc
  File "D:\storage\stable-diffusion-webui\venv\lib\site-packages\starlette\middleware\base.py", line 69, in coro
    await self.app(scope, receive_or_disconnect, send_no_error)
  File "D:\storage\stable-diffusion-webui\venv\lib\site-packages\starlette\middleware\gzip.py", line 24, in __call__
    await responder(scope, receive, send)
  File "D:\storage\stable-diffusion-webui\venv\lib\site-packages\starlette\middleware\gzip.py", line 44, in __call__
    await self.app(scope, receive, self.send_with_gzip)
  File "D:\storage\stable-diffusion-webui\venv\lib\site-packages\starlette\middleware\cors.py", line 92, in __call__
    await self.simple_response(scope, receive, send, request_headers=headers)
  File "D:\storage\stable-diffusion-webui\venv\lib\site-packages\starlette\middleware\cors.py", line 147, in simple_response
    await self.app(scope, receive, send)
  File "D:\storage\stable-diffusion-webui\venv\lib\site-packages\starlette\middleware\exceptions.py", line 79, in __call__
    raise exc
  File "D:\storage\stable-diffusion-webui\venv\lib\site-packages\starlette\middleware\exceptions.py", line 68, in __call__
    await self.app(scope, receive, sender)
  File "D:\storage\stable-diffusion-webui\venv\lib\site-packages\fastapi\middleware\asyncexitstack.py", line 21, in __call__
    raise e
  File "D:\storage\stable-diffusion-webui\venv\lib\site-packages\fastapi\middleware\asyncexitstack.py", line 18, in __call__
    await self.app(scope, receive, send)
  File "D:\storage\stable-diffusion-webui\venv\lib\site-packages\starlette\routing.py", line 706, in __call__
    await route.handle(scope, receive, send)
  File "D:\storage\stable-diffusion-webui\venv\lib\site-packages\starlette\routing.py", line 276, in handle
    await self.app(scope, receive, send)
  File "D:\storage\stable-diffusion-webui\venv\lib\site-packages\starlette\routing.py", line 66, in app
    response = await func(request)
  File "D:\storage\stable-diffusion-webui\venv\lib\site-packages\fastapi\routing.py", line 237, in app
    raw_response = await run_endpoint_function(
  File "D:\storage\stable-diffusion-webui\venv\lib\site-packages\fastapi\routing.py", line 165, in run_endpoint_function
    return await run_in_threadpool(dependant.call, **values)
  File "D:\storage\stable-diffusion-webui\venv\lib\site-packages\starlette\concurrency.py", line 41, in run_in_threadpool
    return await anyio.to_thread.run_sync(func, *args)
  File "D:\storage\stable-diffusion-webui\venv\lib\site-packages\anyio\to_thread.py", line 31, in run_sync
    return await get_asynclib().run_sync_in_worker_thread(
  File "D:\storage\stable-diffusion-webui\venv\lib\site-packages\anyio\_backends\_asyncio.py", line 937, in run_sync_in_worker_thread
    return await future
  File "D:\storage\stable-diffusion-webui\venv\lib\site-packages\anyio\_backends\_asyncio.py", line 867, in run
    result = context.run(func, *args)
  File "E:\storage\stable-diffusion-webui\modules\api\api.py", line 258, in extras_single_image_api
    result = postprocessing.run_extras(extras_mode=0, image_folder="", input_dir="", output_dir="", save_output=False, **reqDict)
  File "E:\storage\stable-diffusion-webui\modules\postprocessing.py", line 103, in run_extras
    return run_postprocessing(extras_mode, image, image_folder, input_dir, output_dir, show_extras_results, *args, save_output=save_output)
  File "E:\storage\stable-diffusion-webui\modules\postprocessing.py", line 56, in run_postprocessing
    scripts.scripts_postproc.run(pp, args)
  File "E:\storage\stable-diffusion-webui\modules\scripts_postprocessing.py", line 126, in run
    script.process(pp, **process_args)
  File "E:\storage\stable-diffusion-webui\scripts\postprocessing_upscale.py", line 83, in process
    assert upscaler1 or (upscaler_1_name is None), f'could not find upscaler named {upscaler_1_name}'
AssertionError: could not find upscaler named fake_upscaler

currently my thought is to just simply provide a hardcoded list of fallback upscalers that are guaranteed to be "available" to a fresh install until this gets actually solved on the API side temporarily implemented hardcoded list as of https://github.com/zero01101/openOutpaint/commit/85776d86371cd8d07cf92661499da71dfa4ea53a

[AUTOMATIC1111]

/sdapi/v1/upscalers should be working properly in latest version :^)

[relocator]

I'm also getting a crash at startup on a fresh install. I updated to newest Automatic a few mins ago, and the openoutpaint tab is gone, and I'm getting errors in startup. Seems like an issue with adding the tab to the webUI?

[seijihariki]

Seems some problem with setting up the arguments for openOutpaint. Added error handling to avoid fully breaking even if arguments were not set up properly.

[zero01101]

/sdapi/v1/upscalers should be working properly in latest version :^)

the man himself! confirmed and muchly obliged :D fixed in https://github.com/zero01101/openOutpaint/commit/bd5581fdbf3a7dbe1637ece4684a0fb18129aacd

Seems like an issue with adding the tab to the webUI?

yeah, closing and re-starting webUI would've fixed that by instantiating the preload.py, but

Added error handling to avoid fully breaking even if arguments were not set up properly.

lol always a few minutes faster than me ;) was just about to push a nearly identical commit myself