search

liuliu / swift-diffusion

BSD 3-Clause "New" or "Revised" License
429 stars 33 forks source link

Memory Requirements and iOS Support? #2

Closed Lukas1h closed 2 years ago

Lukas1h commented 2 years ago

Dose this run on mobile devices? If so, what are the memory requierments?

liuliu commented 2 years ago

It should, haven't tried.

Consumes about 4GiB.

Lukas1h commented 2 years ago

Thanks. How would one go about converting the model to fp16?

liuliu commented 2 years ago

You can load f32 weights into a f16 model and then write the f16 model weights out. Current txt2img/main.swift example loads f32 weights into f16 model. You can then just add additional:

graph.openStore("xxx.ckpt") {
  $0.write("unet", model: unet)
  $0.write("decoder", model: decoder)
}

and the new file will be f16 weights.

Lukas1h commented 2 years ago

Would this decrease memory usage under 3gb? Thanks again for the help!

liuliu commented 2 years ago

No. Model already running at f16 with f16 weights. It just affects files stored on disk. FP16 already runs at around 3.3GiB including all the models (text model, unet and decoder). The extra 0.7 is from MPS implicit tensors which can be further reduced by just running one op at a time (slower).

Lukas1h commented 1 year ago

Have the memory requirements changed since being published on the App Store? This issue seems to lead me to believe so.