search

chaiNNer-org / spandrel

Spandrel gives your project support for various PyTorch architectures meant for AI Super-Resolution, restoration, and inpainting. Based on the model support implemented in chaiNNer.
MIT License
103 stars 7 forks source link

Add support for LUD-VAE #253

Open RunDevelopment opened 1 month ago

RunDevelopment commented 1 month ago

Not sure if this will be useful for spandrel to support. Inference tests don't work yet, but detection and loading does.

joeyballentine commented 1 month ago

Do you want help fixing the inference tests?

RunDevelopment commented 1 month ago

Sure, that would be nice. My main issue is that I don't know how LUD-VAE is supposed to be used and didn't have the time to look into that yet. Does our image -> image call API even make sense for LUD-VAE?

joeyballentine commented 1 month ago

I'm not really sure what LUD-VAE even is. Skimming the repo made it seem like a denoiser

RunDevelopment commented 1 month ago

It's the opposite: it adds noise. The arch is supposed to learn the noise of some input images to generated noisy datasets for training models.

joeyballentine commented 1 month ago

Interesting. Well if it's just clean -> noisy then our call API should work fine, no?

RunDevelopment commented 1 month ago

I saw some stuff suggesting that LUD-VAE is parameterized at inference time, so I'm not sure where image -> image will work. But again, I haven't looked into it yet.

joeyballentine commented 1 month ago

We might have to make a new model descriptor type for this