search

huggingface / swift-coreml-diffusers

Swift app demonstrating Core ML Stable Diffusion
Apache License 2.0
2.53k stars 214 forks source link

Latent Diffusion Upscaler #7

Open 3DTOPO opened 1 year ago

3DTOPO commented 1 year ago

I'd love to see the latent upscale model ported: https://github.com/Stability-AI/stablediffusion#image-upscaling-with-stable-diffusion

The Unet architecture in x4-upscaling.yaml looks very similar to the Stable Diffusion model (v2-inference.yaml) so should be pretty straightforward but getting all the way to inferencing in Swift is a bit beyond my skill set. I'm more than willing to help and learn though.

pcuenca commented 1 year ago

@3DTOPO That'd be very cool indeed! Let's see if someone in the community takes up the challenge to convert the model to Core ML!

3DTOPO commented 1 year ago

I could probably hack a conversion of the model to CoreML but what would take me a few days I bet someone could do in a few hours and I just don't want to work on something that might already be worked on.

I requested the model to be ported at the  repo.

MathisDetourbet commented 1 year ago

Hi there! Let me wish you a Happy new year first! I could be interested in working on that but my knowledge about Core ML and models is very limited. But, I would be happy to learn. On the other side, I have strong experience in Swift and I'm curious about Models and CoreML topics. Do you guys think it could be done without xp on that? Or it would be a waste of time?

3DTOPO commented 1 year ago

The upscaling model architecture is very similar to the diffusion U-net model that the  coreml conversion scripts currently convert so there is a good example to learn from.

If you feel like taking a stab, I'm happy to help if I can - just ask. I do have a lot of experience converting PyTorch models to MLModels.