search

vitoplantamura / OnnxStream

Lightweight inference library for ONNX files, written in C++. It can run Stable Diffusion XL 1.0 on a RPI Zero 2 (or in 298MB of RAM) but also Mistral 7B on desktops and servers. ARM, x86, WASM, RISC-V supported. Accelerated by XNNPACK.
https://yolo.vitoplantamura.com/
Other
1.86k stars 84 forks source link

Update onnxstream.cpp to use latest XNNPACK #80

Closed AeroX2 closed 4 months ago

AeroX2 commented 4 months ago

This is a small PR updating onnxstream.cpp to use the latest XNNPACK (commit: 1c8ee1b68f3a3e0847ec3c53c186c5909fa3fbd3)

Surprising this does result in a minor speedup for Windows (see test below)

On the Raspberry Pi however I saw very little speedup even with the new Kleidai framework (although maybe ARM smartphones will benefit a lot more from this since it was debuted with a Pixel 8 Pro)

Windows

Measure-Command { .\sd.exe --turbo --models-path .\stable-diffusion-xl-turbo-1.0-onnxstream-orig\ --steps 3 --prompt "a snowy night, cabin, nighttime, epic, beautiful, sombre" --output normal-xnnpack.png --seed 5 }
Days              : 0
Hours             : 0
Minutes           : 1
Seconds           : 42
Milliseconds      : 191
Ticks             : 1021918860
TotalDays         : 0.00118277645833333
TotalHours        : 0.028386635
TotalMinutes      : 1.7031981
TotalSeconds      : 102.191886
TotalMilliseconds : 102191.886

Measure-Command { .\sd-new-xnnpack.exe --turbo --models-path .\stable-diffusion-xl-turbo-1.0-onnxstream-orig\ --steps 3 --prompt "a snowy night, cabin, nighttime, epic, beautiful, sombre" --output modified-xnnpack.png --seed 5 }
Days              : 0
Hours             : 0
Minutes           : 1
Seconds           : 9
Milliseconds      : 163
Ticks             : 691631476
TotalDays         : 0.000800499393518519
TotalHours        : 0.0192119854444444
TotalMinutes      : 1.15271912666667
TotalSeconds      : 69.1631476
TotalMilliseconds : 69163.1476
vitoplantamura commented 4 months ago

Really cool!

I had the feeling that it would be a much more challenging job, since for a while the XNNPACK's function prototypes changed every day...

Just a little thing: could you update the commit ID in the Dockerfile too?

Thank you!, Vito

AeroX2 commented 4 months ago

I had the feeling that it would be a much more challenging job, since for a while the XNNPACK's function prototypes changed every day...

Yeah I think I tried a while ago and there was just too many conflicts, I think they've calmed down with the function prototypes changing everyday since some products like Tensorflow rely on the XNNPACK project

vitoplantamura commented 4 months ago

yes, the most significant changes happened soon after the original maintainer left the project. Now it seems that the API is more consolidated.

We may occasionally try to build with the most recent commit of XNNPACK and update the commit ID in OnnxStream, without making any changes to the code...

Thank you, Vito