Odroid N2+

[CarlFink]

This isn't really an "issue" per se, but ... I have an Odroid N2+ that isn't currently being used for anything. Would you (Vito?) find it useful if I attempted to install OnnxStream on the Odroid and reported on the results? Is there a standard benchmark you'd like me to run, assuming the install works?

N2+ specs: Cortex A-73 (4 cores) + Cortex A-53 (2 cores) + ARM Mali G-52. I have the version with 4GB of RAM.

Worth a try? It's certainly more powerful than the RPi Zero 2! From what you've written in other responses here, OnnxStream would use all six cores, correct?

This looks like a really cool project. Thanks for creating it.

[vitoplantamura]

please yes, because, from what I read online, the performance of your board should be similar to that of the RPI 5 :-)

Yes, OnnxStream uses all available CPU cores by default.

When launching the application, you should only specify the "--xl" parameter. If there is an error, add the "--rpi" parameter.

It would be interesting to know the total execution time, but the time per step, which is reported by the application, is also enough!

Thanks, Vito

[CarlFink]

I'll try to set it up tonight.

Will OnnxStream use the onboard ARM Mali GPU?

[vitoplantamura]

unfortunately not, but it should be able to use NEON...

Thanks, Vito

[CarlFink]

unfortunately not, but it should be able to use NEON... Thanks, Vito

Sorry about the long delay. I hadn't even powered on the Odroid N2+ for a year or so, and I had a lot of travel in the past month. And then it turned out my EMMC storage had a hardware failure, and then the thing wouldn't connect to the network any more ....

In any case, I finally got back to it today. Seemingly OnnxStream requires XNNPACK, but the XNNPACK page gives no instructions on how to build that. I'm sure it's obvious to a C++ dev, but that is not what I am. I could maybe muddle through, but you wouldn't happen to know of an actual howto?

[vitoplantamura]

hi,

instructions for building OnnxStream (and XNNPACK) are here:

https://github.com/vitoplantamura/OnnxStream#how-to-build-the-stable-diffusion-example-on-linuxmacwindowstermux

Thanks, Vito

[CarlFink]

You are 100% right. The instructions were right there and I somehow read right past them.

XNNPACK compiles fine with those instructions, but then trying to make OnnxStream fails with an include error. I'm going to quote starting 33% of the way through (according to make):

odroid@odroid:~/OnnxStream$ .
[ 33%] Building CXX object CMakeFiles/sd.dir/sd.cpp.o
/home/odroid/OnnxStream/OnnxStream/src/sd.cpp: In function ‘int main(int, char**)’:
/home/odroid/OnnxStream/OnnxStream/src/sd.cpp:1699:19: warning: format not a string literal and no format arguments [-Wformat-security]
 1699 |             printf(("Invalid command line argument: \"" + arg + "\".\n\n").c_str());
      |             ~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/home/odroid/OnnxStream/OnnxStream/src/sd.cpp:1724:23: warning: format not a string literal and no format arguments [-Wformat-security]
 1724 |                 printf(("Argument \"" + arg + "\" should be followed by a string.").c_str());
      |                 ~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
[ 66%] Building CXX object CMakeFiles/sd.dir/onnxstream.cpp.o
/home/odroid/OnnxStream/OnnxStream/src/onnxstream.cpp:20:10: fatal error: xnnpack.h: No such file or directory
   20 | #include <xnnpack.h>
      |          ^~~~~~~~~~~
compilation terminated.
gmake[2]: *** [CMakeFiles/sd.dir/build.make:90: CMakeFiles/sd.dir/onnxstream.cpp.o] Error 1
gmake[1]: *** [CMakeFiles/Makefile2:83: CMakeFiles/sd.dir/all] Error 2
gmake: *** [Makefile:91: all] Error 2

The makeonnxstream script is just a copy-paste of your code with a directory substituted:

#!/bin/bash
git clone https://github.com/vitoplantamura/OnnxStream.git
cd OnnxStream
cd src
mkdir build
cd build
cmake -DMAX_SPEED=ON -DXNNPACK_DIR=~odroid/XNNXPACK/XNNPACK ..
cmake --build . --config Release

xnnpack.h is right there:

odroid@odroid:~/OnnxStream$ find ../XNXXPACK/XNNPACK/ -name xnnpack.h
../XNXXPACK/XNNPACK/include/xnnpack.h

Puzzled, I am. Grateful for your help, too.

[vitoplantamura]

hi,

I think the path for XNNPACK_DIR should be "~/odroid/XNNXPACK/XNNPACK", i.e. with a tilde slash instead of just a tilde.

Vito

[CarlFink]

The Unix shell expands "~odroid" to "/home/odroid". That's the default account in the default image for the Odroid N2+. Changing that didn't affect the outcome of running the script.

What directory(s) does make search for xnnpack.h? It's in /home/odroid/XNXXPACK/XNNPACK/include.

[vitoplantamura]

xnnpack.h is loaded from the directory specified in XNNPACK_DIR with the string "/include" added.

So the correct value of XNNPACK_DIR should be "~/XNXXPACK/XNNPACK" or even "/home/odroid/XNXXPACK/XNNPACK", which are equivalent.

Vito

[vitoplantamura]

if the problem persists, try removing the "build" directory and rerunning the two "cmake" commands.

Vito

[CarlFink]

I renamed the XNNPACK directory, modified the script, and got ... a different error!

/home/odroid/XNNPACK/XNNPACK/include/xnnpack.h:15:10: fatal error: pthreadpool.h: No such file or directory
   15 | #include <pthreadpool.h>
      |          ^~~~~~~~~~~~~~~
compilation terminated.
gmake[2]: *** [CMakeFiles/sd.dir/build.make:90: CMakeFiles/sd.dir/onnxstream.cpp.o] Error 1
gmake[1]: *** [CMakeFiles/Makefile2:83: CMakeFiles/sd.dir/all] Error 2
gmake: *** [Makefile:91: all] Error 2

FTR:

odroid@odroid:~/OnnxStream$ find ~ -name "pthreadpool.h"
/home/odroid/old_OnnxStream/XNNPACK/build/pthreadpool-source/include/pthreadpool.h

[vitoplantamura]

please don't rename directories after running cmake :-)

You can restore the old directory name and try again, or repeat the entire build procedure with the new directory name.

Vito

[CarlFink]

I used rm -r to eliminate the XNNPACk and OnnxStream directories and restarted. This time, I ran your command in my home directory instead of subdirectories to shorten the final paths. XNNPACK build without issue. When I tried to build OnnxStream, this is what I got:

odroid@odroid:~/OnnxStream/src/build$ cmake --build . --config Release
[ 33%] Building CXX object CMakeFiles/sd.dir/sd.cpp.o
/home/odroid/OnnxStream/src/sd.cpp: In function ‘int main(int, char**)’:
/home/odroid/OnnxStream/src/sd.cpp:1739:19: warning: format not a string literal and no format arguments [-Wformat-security]
 1739 |             printf(("Invalid command line argument: \"" + arg + "\".\n\n").c_str());
      |             ~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/home/odroid/OnnxStream/src/sd.cpp:1765:23: warning: format not a string literal and no format arguments [-Wformat-security]
 1765 |                 printf(("Argument \"" + arg + "\" should be followed by a string.").c_str());
      |                 ~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
[ 66%] Building CXX object CMakeFiles/sd.dir/onnxstream.cpp.o
In file included from /usr/include/c++/11/bits/stl_algobase.h:64,
                 from /usr/include/c++/11/vector:60,
                 from /home/odroid/OnnxStream/src/onnxstream.h:4,
                 from /home/odroid/OnnxStream/src/onnxstream.cpp:1:
/usr/include/c++/11/bits/stl_pair.h: In instantiation of ‘constexpr std::pair<typename std::__strip_reference_wrapper<typename std::decay<_Tp>::type>::__type, typename std::__strip_reference_wrapper<typename std::decay<_Tp2>::type>::__type> std::make_pair(_T1&&, _T2&&) [with _T1 = float&; _T2 = float&; typename std::__strip_reference_wrapper<typename std::decay<_Tp2>::type>::__type = float; typename std::decay<_Tp2>::type = float; typename std::__strip_reference_wrapper<typename std::decay<_Tp>::type>::__type = float; typename std::decay<_Tp>::type = float]’:
/home/odroid/OnnxStream/src/onnxstream.cpp:2085:30:   required from here
/usr/include/c++/11/bits/stl_pair.h:567:5: note: parameter passing for argument of type ‘std::pair<float, float>’ when C++17 is enabled changed to match C++14 in GCC 10.1
  567 |     make_pair(_T1&& __x, _T2&& __y)
      |     ^~~~~~~~~
[100%] Linking CXX executable sd
/usr/bin/ld: cannot find ~odroid/XNNPACK//build/libXNNPACK.a: No such file or directory
/usr/bin/ld: cannot find ~odroid/XNNPACK//build/pthreadpool/libpthreadpool.a: No such file or directory
/usr/bin/ld: cannot find ~odroid/XNNPACK//build/cpuinfo/libcpuinfo.a: No such file or directory
collect2: error: ld returned 1 exit status
gmake[2]: *** [CMakeFiles/sd.dir/build.make:116: sd] Error 1
gmake[1]: *** [CMakeFiles/Makefile2:83: CMakeFiles/sd.dir/all] Error 2
gmake: *** [Makefile:91: all] Error 2

Note that:

odroid@odroid:~/OnnxStream/src/build$ ls ~odroid/XNNPACK/build/*.a
/home/odroid/XNNPACK/build/libXNNPACK.a
odroid@odroid:~/OnnxStream/src/build$ ls ~odroid/XNNPACK/build/pthreadpool/*.a
/home/odroid/XNNPACK/build/pthreadpool/libpthreadpool.a
odroid@odroid:~/OnnxStream/src/build$ ls ~odroid/XNNPACK/build/cpuinfo/
Makefile    cpuinfo-config.cmake  libcpuinfo.pc

So those files exist, but ld doesn't see them? Could the problem be that Ubuntu doesn't supply the version of ld your makefile expects? It comes from binutils 2.38. (I'm guessing, as I'm sure you could tell.)

[vitoplantamura]

I don't think it could be a toolchain problem. It is clear that ld does not find those libraries.

The value of XNNPACK_DIR should be set to "/home/odroid/XNNPACK" and I see no reason why it shouldn't work this way :-)

It should be enough to rerun the two cmake commands, specifying the correct XNNPACK_DIR in the first one.

Vito

[CarlFink]

Doing that worked, to my shock. For some reason, it never occurred to me that shell expansion wouldn't happen in that command. Why wouldn't the shell have changed "~odroid" to "/home/odroid", do you know?

Aaaand ... I can't find the example command to run. I know I saw it when I first read this Github repo's main page, but now it isn't there, that I can find. I have a proven record of not seeing stuff that is clearly right there, but I've just spend 10 minutes searching, and the word "example" and the string "./sd" are in short supply, only found in the section on importing a model. "./sd --help" results only in a list of options, but I do not even know the basic syntax.

If I put a prompt in input.txt and run "./sd < input.txt" it complains about being unable to open ./tokenizer/vocab.txt.

Thanks, Vito.

[vitoplantamura]

good :-)

You should now download the SDXL Base weights from Hugging Face. The instructions are here:

https://github.com/vitoplantamura/OnnxStream#how-to-build-the-stable-diffusion-example-on-linuxmacwindowstermux

Once this is done, you should run the application specifying the flags "--xl --rpi --models-path " where is the path where the weights downloaded from Hugging Face have been saved.

Vito

[CarlFink]

The first command in that section is "git lfs install". So, git responds, "git: 'lfs' is not a git command. See 'git --help'."

Not being totally helpless, I went ahead and installed git-lfs.

That done, it worked flawlessly. The Odroid is now processing my first prompt. The first two steps were 679 and 687 seconds, so it'll take a while. I probably won't see the result until tomorrow.

So ... anything special I can do to test this out for you? I remember you were interested because it should have performance comparable to a Raspberry Pi 5.

[vitoplantamura]

yes, I'm interested in the performance (seconds per step) and also whether it worked on that particular SBC.

So I have all the information I needed :-)

Thank you for your time,

Vito

[CarlFink]

Might I ask: is the method of using OnnxStream documented anywhere? I got the default astronaut on a horse, ignoring what I thought was my actual prompt. Presumably I just did it wrong (again), but in fairness to me there is no man page nor a guide to using it. Does it follow syntax documented in some predecessor project, e. g. ONNX?

Thanks again.

[CarlFink]

OK, Nick Bild gives some hints here: https://edgeimpulse.com/blog/a-big-leap-for-small-tech