Error: 'NoneType' object has no attribute 'enhance'

[glonlas]

Describe your environment

• GPU: [cuda/amd/mps/cpu] MPS and Cuda
• VRAM: [if known] N/A
• CPU arch: [x86/arm] ARM
• OS: [Linux/Windows/macOS] macOS
• Python: [Anaconda/miniconda/miniforge/pyenv/other (explain)] miniconda
• Branch: [if git status says anything other than "On branch main" paste it here] development
• Commit: [run git show and paste the line that starts with "Merge" here] bfb278127923fbd461c4549a4b7f2f2c1dd34b8c

Describe the bug Error 'NoneType' object has no attribute 'enhance' when using GFPGAN via the Web UI.

>> Error running RealESRGAN or GFPGAN. Your image was not upscaled.
'NoneType' object has no attribute 'enhance'

To Reproduce On MPS: Steps to reproduce the behavior:

1. Run the server with python scripts/dream.py --full_precision --web
2. Generate an image with Upscaling Level = x2|x3|x4 (GPFGAN Strength can be anything)

On CUDA Steps to reproduce the behavior:

1. Run the server with python scripts/dream.py --full_precision --web
2. Generate an image with Upscaling Level = x3 (GPFGAN Strength can be anything)

The terminal will return

>> Real-ESRGAN Upscaling seed:XXXX : scale:3x
>> Error running RealESRGAN or GFPGAN. Your image was not upscaled.
'NoneType' object has no attribute 'enhance'

Expected behavior The generation of the image works with GPFGAN and Upscaling without any trouble.

[glonlas]

I confirm the issue is visible in the current development branch. It happens because either it cannot load the Real-ESRGAN python Library (MPS) or cannot load a model for x3 upscaling on CUDA..

For MPS

This PR fix it for MacOS MPS because it loads Real-ESRGAN from the binary rather the Python Library. When I switch to this PR upscale work.

When on current development the terminal output is: (Failed)

>> Real-ESRGAN Upscaling seed:1586105133 : scale:2x
>> Error running RealESRGAN or GFPGAN. Your image was not upscaled.
'NoneType' object has no attribute 'enhance'

When I switched to the PR #424 the terminal output is: (Successful)

>> Real-ESRGAN Upscaling seed:1857161085 : scale:2x
127.0.0.1 - - [10/Sep/2022 23:18:09] "GET /outputs/img-samples/000008.1857161085.png HTTP/1.1" 200 -
[0 Apple M1]  queueC=0[1]  queueG=0[1]  queueT=0[1]
[0 Apple M1]  bugsbn1=0  bugbilz=97  bugcopc=0  bugihfa=0
[0 Apple M1]  fp16-p/s/a=1/1/1  int8-p/s/a=1/1/1
[0 Apple M1]  subgroup=32  basic=1  vote=1  ballot=1  shuffle=1
0.00%
11.11%
22.22%
33.33%
44.44%
55.56%
66.67%
77.78%
88.89%
>> Usage stats:
>>   1 image(s) generated in 30.51s
>>   Max VRAM used for this generation: 0.00G

[glonlas]

For System other than MPS

The issue happens because the code base will load a model for 2x and x4 scaling only.

To understand the error on CUDA.

The error happens in:

>> Real-ESRGAN Upscaling seed:3350596856 : scale:2x
Traceback (most recent call last):
  File "/ldm/generate.py", line 467, in upscale_and_reconstruct
    image = real_esrgan_upscale(
  File "ldm/gfpgan/gfpgan_tools.py", line 149, in real_esrgan_upscale
    output, img_mode = upsampler.enhance(
AttributeError: 'NoneType' object has no attribute 'enhance'

>> Error running RealESRGAN or GFPGAN. Your image was not upscaled.
'NoneType' object has no attribute 'enhance'

When we dive deep this happens because of the following code which is executed only on CUDA platform:

           model_path = {
                2: 'https://github.com/xinntao/Real-ESRGAN/releases/download/v0.2.1/RealESRGAN_x2plus.pth',
                4: 'https://github.com/xinntao/Real-ESRGAN/releases/download/v0.1.0/RealESRGAN_x4plus.pth',
            }

            if upsampler_scale not in model_path:
                return None

As we can see it load model for upscale at x2 and x4, but no model for x3 which returns None. This is what generates the 'NoneType' object has no attribute 'enhance'

[glonlas]

As of today the x3 cannot work with the implementation of the python lib of Real-ESRGAN. While it works if we use the Bin. @blessedcoolant you were 100% about the non support of x3 upscaling.

We need to revert this PR: https://github.com/lstein/stable-diffusion/pull/442 (I will do the revert)

cc: @lstein

[lstein]

@glonlas could you confirm that upscaling is now working on the Macintosh platform? Please check either "development" HEAD, or the tag release-candidate-1.14.2.

Thnx

[Any-Winter-4079]

For the time being it works (1.14.3). In the future, it may be worth looking into using the executables (there's versions for Linux/Windows/Mac) if these reports I found running it are not isolated to my machine. https://github.com/lstein/stable-diffusion/discussions/511#discussioncomment-3624882 (about quality and time). But anyway, it works