search

chaiNNer-org / spandrel

Spandrel gives your project support for various PyTorch architectures meant for AI Super-Resolution, restoration, and inpainting. Based on the model support implemented in chaiNNer.
MIT License
134 stars 10 forks source link

Please add support for GRLIR Blind super resolution model #15

Closed zelenooki87 closed 9 months ago

zelenooki87 commented 1 year ago

Thank you devs for hard work and implementing SRFormer and DAT in nightly build. Could you consider implement https://github.com/ofsoundof/GRL-Image-Restoration GRL IR blind super resolution model? If it is possible? Thank you very much.

joeyballentine commented 1 year ago

On the one hand, this being a transformer model means I should be able to reuse some of my param detection code. On the other hand, it has a lot of parameters. So I could maybe add support for just the official models... maybe. I need a break after adding SRFormer and DAT though

zelenooki87 commented 1 year ago

I have tried vapoursynth repo of holywu for video clips and results are awesome for base model blind super resolution task of 4x.
when you have more spare time please add official model support. Thanks

zelenooki87 commented 1 year ago

Is it doable? please?

joeyballentine commented 1 year ago

It's probably doable but it's not a priority right now so I'll get to it when I get to it. It's generally a lot of work to add these and this one looks like it'll be a lot more work than most

zelenooki87 commented 11 months ago

Once again I ask you to implement this code. Results are superior than swinir and even the commercial apps like Gigapixel. thanks

joeyballentine commented 11 months ago

I've been very busy working on other stuff. This likely won't come any time soon, sorry. If you really can't wait, convert it to onnx outside of chaiNNer and then use onnx for it

zelenooki87 commented 9 months ago

Thank you great devs for implementing via spandrel, could you please add support for blind image super resolution model? bsr_grl_base.ckpt For that particular model I am only interested. Thanks

RunDevelopment commented 9 months ago

could you please add support for blind image super resolution model? bsr_grl_base.ckpt

Okay, I really don't like this model. Not GRLIR, just this particular model file. I don't like it for 2 reasons:

  1. For some reason, this model includes an omegaconf config. So we can't even add support for this model without adding omegaconf as a dependency to this project. Of course, the best thing about this config is that it's not actually used by the model :)
  2. This model is a checkpoint in the truest form. It contains a bunch of stuff only needed for training (e.g. the current iteration, the discriminator, and even VGG weights). This not only makes the file larger, it also makes it difficult to analyse the file. We detect models by analysing which weights are present in the file, so having a bunch of useless stuff really messes with that analysis.

So anyway, I hacked together some code to make this particular checkpoint load and then saved it as safetensors. You can get it here. Not only does this file load, since it doesn't contain a bunch of useless data it's also 66% smaller (77MB).

In summary, we already support GRLIR blind SR models. Use my .safetensors version since bsr_grl_base.ckpt is stupid.

That all being said, I'll add the hacky code I made to get bsr_grl_base.ckpt to load. I won't add omegaconf as a dependency though, so you'll have to install this pip package yourself.

zelenooki87 commented 9 months ago

Ough... What a awful results with that safetensors model. But thanks for trying. will post examples from vapoursynthed module from holywu... Comparations are extremely-

zelenooki87 commented 9 months ago

https://mega.nz/folder/yBV1wQJR#4UIlT6hYVH4AmD8l1SS1fg Some examples... randomly selected... Edit: will try again later with system python manually installed with all dependencies. Thanks

zelenooki87 commented 9 months ago

Even after manual installing system python to use and omegaconf and all dependiences results are the same with safetensors model you provided to me. Will wait for your heck codes to load original ckpt or .SF file support....natively :D Realy have not idea how holywu get it working from vapoursynth. By that I limmited to use it only for video files or individual frames... Can not understand original code of author of GIRL how to run it for photos. Thank you for effort.

zelenooki87 commented 9 months ago

could you please add support for blind image super resolution model? bsr_grl_base.ckpt

Okay, I really don't like this model. Not GRLIR, just this particular model file. I don't like it for 2 reasons:

  1. For some reason, this model includes an omegaconf config. So we can't even add support for this model without adding omegaconf as a dependency to this project. Of course, the best thing about this config is that it's not actually used by the model :)
  2. This model is a checkpoint in the truest form. It contains a bunch of stuff only needed for training (e.g. the current iteration, the discriminator, and even VGG weights). This not only makes the file larger, it also makes it difficult to analyse the file. We detect models by analysing which weights are present in the file, so having a bunch of useless stuff really messes with that analysis.

So anyway, I hacked together some code to make this particular checkpoint load and then saved it as safetensors. You can get it here. Not only does this file load, since it doesn't contain a bunch of useless data it's also 66% smaller (77MB).

In summary, we already support GRLIR blind SR models. Use my .safetensors version since bsr_grl_base.ckpt is stupid.

That all being said, I'll add the hacky code I made to get bsr_grl_base.ckpt to load. I won't add omegaconf as a dependency though, so you'll have to install this pip package yourself.

Any news for properly load model? Tnx

RunDevelopment commented 9 months ago

https://mega.nz/folder/yBV1wQJR#4UIlT6hYVH4AmD8l1SS1fg Some examples...

Those were really helpful. Thanks to your example, I was able to see the problem.

The issue with my parameter detection. GRLIR has a bunch of parameters that are different from model to model. Normally, you are just supposed to know these parameters for the model you're loading (e.g. the program you used hard-codes these parameters for BSR GRL model), but we can't do that since we want to support unofficial models (=models by the community) as well. So instead of hard-coding parameters, we try to detect parameters from the model file itself. This usually works very well, but it's not perfect. Unfortunately, some GRLIR parameters cannot be detected from the model file alone, so I just left them as the default value (e.g. 8 for window size). Those default values worked well enough to load models, but the output of the models was all wrong. This is what you saw.

So I added a heuristic to guess the undetectable parameters. This works well enough to get the correct output for BSR and SR models. Thanks to your example images, I was able to verify that the output is correct now.

Unfortunately, those heuristics do not work for 1x models. So all GRLIR denoise, deblur, jpeg, etc. models are unable right now. They will load fine, but the output will be all wrong.

Can not understand original code of author of GIRL how to run it for photos.

Same. That repo is a mess...

zelenooki87 commented 9 months ago

Yeah, repo is a mess but results I have got with Blind model through Selurs Hybrid are prety awesome... Even better than commercial products.. but not easy to deal with tiles for bigger inputs... if you need more examples, please let me know. I see you stated that output are now correct? How to test it in chaainner? Or you will update codes later? thanks

zelenooki87 commented 9 months ago

https://mega.nz/folder/yBV1wQJR#4UIlT6hYVH4AmD8l1SS1fg Some examples...

Those were really helpful. Thanks to your example, I was able to see the problem.

The issue with my parameter detection. GRLIR has a bunch of parameters that are different from model to model. Normally, you are just supposed to know these parameters for the model you're loading (e.g. the program you used hard-codes these parameters for BSR GRL model), but we can't do that since we want to support unofficial models (=models by the community) as well. So instead of hard-coding parameters, we try to detect parameters from the model file itself. This usually works very well, but it's not perfect. Unfortunately, some GRLIR parameters cannot be detected from the model file alone, so I just left them as the default value (e.g. 8 for window size). Those default values worked well enough to load models, but the output of the models was all wrong. This is what you saw.

So I added a heuristic to guess the undetectable parameters. This works well enough to get the correct output for BSR and SR models. Thanks to your example images, I was able to verify that the output is correct now.

Unfortunately, those heuristics do not work for 1x models. So all GRLIR denoise, deblur, jpeg, etc. models are unable right now. They will load fine, but the output will be all wrong.

Can not understand original code of author of GIRL how to run it for photos.

Same. That repo is a mess...

Maybe this repo would be helpful? https://github.com/HolyWu/vs-grlir/blob/master/vsgrlir/__init__.py