search

Qengineering / Rock-5-image

Rock Pi 5 image with OpenCV, deep learning frameworks and NPU drivers
https://qengineering.eu/deep-learning-examples-on-raspberry-32-64-os.html
BSD 3-Clause "New" or "Revised" License
7 stars 0 forks source link

Rock 5 image on Orange pi 5 #3

Open FeAlias opened 1 year ago

FeAlias commented 1 year ago

Hi, just wondering if this would work on the Orange pi 5? Thanks.

Qengineering commented 1 year ago

@FeAlias ,

You can always try, but I don't think it will work. Undoubtedly there will be low level hardware differences which prevent the image to function properly.

FeAlias commented 1 year ago

Thanks for letting me know. Is there anything you could tell me that might help me get the NPU on the orange pi working?

On Fri, Aug 4, 2023 at 1:42 AM Q-engineering @.***> wrote:

@FeAlias https://github.com/FeAlias ,

You can always try, but I don't think it will work. Undoubtedly there will be low level hardware differences which prevent the image to function properly.

— Reply to this email directly, view it on GitHub https://github.com/Qengineering/Rock-5-image/issues/3#issuecomment-1665092067, or unsubscribe https://github.com/notifications/unsubscribe-auth/AFMBWXISP3HUTEGKZV6MXK3XTSKU7ANCNFSM6AAAAAA3DG3GYY . You are receiving this because you were mentioned.Message ID: @.***>

Qengineering commented 1 year ago

You need to install the RKNPU framework on your Orange Pi. It comes with examples.

To get home brewed DNN models working you need the RKNN Toolkit. It can be installed on a Ubuntu machine.

Note, the Toolkit will install/use Tensorflow, Pytorch, PaddlePaddle and other major frameworks to convert their models to NPU models. It means a massive load on your machine. Use a PC, not the Orange Pi if you want to use the Toolkit.

FeAlias commented 1 year ago

That is very helpful. Thanks! So, once I convert it to a an NPU model using my pc, I can load it on to the orange pi, run it in Python, and it should work? I’m on the fence about just buying a rock pi and using that. With the orange pi I ended up getting a SIGKILL error. If I can manage to get this working, I’m hoping it will handle this model “ togethercomputer/RedPajama-INCITE-7B-Chat” pretty smoothly. But, if the rock pi is better, I may just do that. Could you recommend one over the other?

On Fri, Aug 4, 2023 at 6:10 AM Q-engineering @.***> wrote:

You need to install the RKNPU framework https://github.com/rockchip-linux/rknpu2 on your Orange Pi. It comes with examples.

To get home brewed DNN models working you need the RKNN Toolkit https://github.com/rockchip-linux/rknn-toolkit2. It can be installed on a Ubuntu machine.

Note, the Toolkit will install/use Tensorflow, Pytorch, PaddlePaddle and other major frameworks to convert their models to NPU models. It means a massive load on your machine. Use a PC, not the Orange Pi if you want to use the Toolkit.

— Reply to this email directly, view it on GitHub https://github.com/Qengineering/Rock-5-image/issues/3#issuecomment-1665438388, or unsubscribe https://github.com/notifications/unsubscribe-auth/AFMBWXLPYYOEJWTIEBQZXSLXTTKBPANCNFSM6AAAAAA3DG3GYY . You are receiving this because you were mentioned.Message ID: @.***>

Qengineering commented 1 year ago

I think I must warn you. Don't set your expectations too high! Biggest problem with converting models are the supported operations. Not every operation can be done in a NPU. Second problem is the 8-bit conversion. NPUs are working fast with 8 or 16 bit integers. They are not so keen on floats. Not every model can be converted into another format without a penalty. (Loss of accurateness, for instance). You may try converting the model on a PC and see if it succeeds. If so, your lucky. If not , too bad. There is almost no difference between the Orange and the Rock. No need for a Rock, if you have the Orange.

FeAlias commented 1 year ago

Thanks again for your help. You are the first person I have been able to find to help me with this. Is the Orange Pi 5 able to run an ai model decently? Is there a way to turn the NPU and GPU on at a lower level, maybe in the BIOS, so they are just always running, instead of needing to convert a model?

On Fri, Aug 4, 2023 at 6:57 AM Q-engineering @.***> wrote:

I think I must warn you. Don't set your expectations too high! Biggest problem with converting models are the supported operations. Not every operation can be done in a NPU. Second problem is the 8-bit conversion. NPUs are working fast with 8 or 16 bit integers. They are not so keen on floats. Not every model can be converted into another format without a penalty. (Loss of accurateness, for instance). You may try converting the model on a PC and see if it succeeds. If so, your lucky. If not , too bad. There is almost no difference between the Orange and the Rock. No need for a Rock, if you have the Orange.

— Reply to this email directly, view it on GitHub https://github.com/Qengineering/Rock-5-image/issues/3#issuecomment-1665493601, or unsubscribe https://github.com/notifications/unsubscribe-auth/AFMBWXJFGXSM7NN644ZG25DXTTPSNANCNFSM6AAAAAA3DG3GYY . You are receiving this because you were mentioned.Message ID: @.***>

Qengineering commented 1 year ago

You (Python or C++) program must evoke the NPU. See it as a slave. It only execute calculations given by another "master" program, like your Python program. Don't try to modify low level drivers or software. The correct procedure goes as follows: 1) You get your model. 2) See if the model is supported by the RK toolkit (Tensorflow, Pytorch etc). 3) If not, see if you can convert it to ONNX. (Here, things can get complicated, as mentioned earlier) 4) Try to convert your model to RKNPU. 5) Transfer the RKNPU model to your Orange Pi. (So far you were working on the Ubuntu PC) 6) Write a Python program that uses the model. 7) Once you are happy, you can call the program during booting the Pi.

Qengineering commented 1 year ago

Showstopper. I just read that RedPajama-INCITE-7B-Chat is a LLM model. Most of these models consist over a billion parameters. That is far too much for a tiny device like Orange or Rock. The NPU hasn't the capacity to hold even a fraction of the parameters.

Better to use BERT base, BERT Squad, MobileBERT or DistilBERT. They are designed for edge devices.

FeAlias commented 1 year ago

That is good to know. Thank you again for your help!

On Sat, Aug 5, 2023 at 2:18 AM Q-engineering @.***> wrote:

Showstopper. I just read that RedPajama-INCITE-7B-Chat is a LLM model. Most of them consist over a billion parameters. That is far too much for a tiny edge device like Orange or Rock. The NPU hasn't the capacity to hold even a fraction of the parameters.

— Reply to this email directly, view it on GitHub https://github.com/Qengineering/Rock-5-image/issues/3#issuecomment-1666423369, or unsubscribe https://github.com/notifications/unsubscribe-auth/AFMBWXKGYROPHDTB7XMQFADXTXXULANCNFSM6AAAAAA3DG3GYY . You are receiving this because you were mentioned.Message ID: @.***>