search

TensorStack-AI / OnnxStack

C# Stable Diffusion using ONNX Runtime
Apache License 2.0
222 stars 33 forks source link

Initial Float16 and BFloat16 onnx type support #31

Closed saddam213 closed 1 year ago

saddam213 commented 1 year ago

Add initial support for ONNX native Float16 and BFloat16 types Issue: 30

The main advantage of these types will be in native code during inference, we do little work with tensors outside inference, the schedulers are the most work, but its light and I would no expect too much gain on CPU using these types.

So this PR will use the NodeMetaData from the model to create the correct type native memory buffers, so all inference will be in the models native type, once inference is complete we cast back to float

In most cases we don't pass the output data into another session, TextEncoder, VaeEncode/Decoder just use the output and dispose, the scheduler steps however could benefit from staying in its native type, but that would require making the schedulers generic which would be quite a task, so I think perhaps measuring performance first may be a good option before getting too deep

TL;DR; Type support on native, cast back to float in managed

saddam213 commented 1 year ago

Tested: StableDiffusiom 1.5 LCM-Dreamshaper-V7 LCM-Dreamshaper-V7 f16 Photon

Amin456789 commented 1 year ago

nice! posted some more new fp16 olive optimized here if u need in testing them https://github.com/saddam213/OnnxStack/discussions/14

Amin456789 commented 1 year ago

it should fix @patientx problem too https://github.com/saddam213/OnnxStack/issues/8