Add support for Spandrel Image-to-Image models (e.g. ESRGAN, Real-ESRGAN, Swin-IR, DAT, etc.)

RyanJDick commented 3 months ago

Summary

Add support for all spandrel image-to-image models - this is a collection of many popular super-resolution models (e.g. ESRGAN, Real-ESRGAN, SwinIR, DAT, etc.)

Examples of supported models:

DAT: https://drive.google.com/drive/folders/1iBdf_-LVZuz_PAbFtuxSKd_11RL1YKxM
SwinIR: https://github.com/JingyunLiang/SwinIR/releases
Any ESRGAN / Real-ESRGAN model

Related Issues

Closes #6394

QA Instructions

[x] Test that unsupported models still fail the probe (i.e. no false positive spandrel models)
[x] Test adding a few non-spandrel model types
[x] Test adding a handful of spandrel model types: ESRGAN, Real-ESRGAN, SwinIR, DAT
[x] Verify model size estimation for the model cache
[x] Test using the spandrel models in a practical image upscaling workflow

Merge Plan

[x] Get approval from @brandonrising and @maryhipp before merging - this PR has commercial implications.
[x] Merge #6571 and change the target branch to main

Checklist

[x] The PR has a short but descriptive title, suitable for a changelog
[x] Tests added / updated (if applicable)
[x] Documentation added / updated (if applicable)

RyanJDick commented 2 months ago

Questions for reviewers:

Currently, this new group of models is referred to as "Spandrel Image-to-Image Models" in the UI. Is there a better name? cc @hipsterusername
@psychedelicious , I'd appreciate a review from you on this since I copied a bunch of frontend boilerplate for the new model type without taking the time to fully understand it all. Of note, I didn't add any spandrel types to the workflow V1 and V2 schema versions - is that ok?
Are there any concerns with how we currently probe for this model type by loading the full model?

hipsterusername commented 2 months ago

@psychedelicious , I'd appreciate a review from you on this since I copied a bunch of frontend boilerplate for the new model type without taking the time to fully understand it all. Of note, I didn't add any spandrel types to the workflow V1 and V2 schema versions - is that ok?

Are there any concerns with how we currently probe for this model type by loading the full model?

It's a weird name. "Image Models", without naming spandrel?

RyanJDick commented 2 months ago

It's a weird name. "Image Models", without naming spandrel?

I renamed this class of models to Image-to-Image Models throughout the UI. (I assume that's what you meant?)

psychedelicious commented 2 months ago

@psychedelicious , I'd appreciate a review from you on this since I copied a bunch of frontend boilerplate for the new model type without taking the time to fully understand it all. Of note, I didn't add any spandrel types to the workflow V1 and V2 schema versions - is that ok?

Yep, that's fine. Those schemas exist for data migration purposes.

brandonrising commented 2 months ago

I was able to get it running and things were working great! I did try chaining 2 upscaling img2img nodes and the second one ran for 20 minutes and never finished. Maybe there's some kind of warning or limit we could set for max image size.

For this PR though, I think it's working really well!

RyanJDick commented 2 months ago

This PR is ready-to-go, but we are going to wait to merge it until we are confident that it can be released with low risk of rollback. It adds a new model type, which makes rollback non-trivial.

Also, https://github.com/invoke-ai/InvokeAI/pull/6594 should be merged as a fast-follow. They go hand-in-hand.

psychedelicious commented 2 months ago

Do you think we should deprecate the ESRGAN node?

RyanJDick commented 2 months ago

Do you think we should deprecate the ESRGAN node?

Yes, in a future PR that removes the use of that node from the starter workflows and for the upscale button.

invoke-ai / InvokeAI