search

PhotonVision / photonvision

PhotonVision is the free, fast, and easy-to-use computer vision solution for the FIRST Robotics Competition.
https://photonvision.org
GNU General Public License v3.0
262 stars 166 forks source link

Support Object Detection on non-RKNN platforms #1340

Open srimanachanta opened 1 month ago

srimanachanta commented 1 month ago

A transformed yolo model does Object Detection on RKNN, there is no reason we can't just support running this same model without the help of the NPU on other platforms, albeit, at worse performance. I would say the best way to handle this change would be in the photon-core JNI where that build will link the backend RKNN-toolkit and the JNI shouldn't be different for RKNN or non-RKNN.

One issue is that this building is done at build time, and we build binaries on platforms, not per coprocessor, so build action needs to be updated.

mcm001 commented 1 month ago

It's just an ONNX model right? What changes did rockchip make to YOLO to make it run on their NPU?

srimanachanta commented 1 month ago

It's just an ONNX model right? What changes did rockchip make to YOLO to make it run on their NPU?

They take a model (Caffe, TensorFlow, TensorFlow Lite, ONNX, or Darknet) and then converts it to an RKNN model which I assume just uses their native toolkit for BLAS vs the native CPU ops. The only difference between the converted model and the pre-converted one is that the converted one is capable of running on their NPU. This should be fairly trivial to implement this. The build issue is the only problem I see.

ramalamadingdong commented 1 month ago

I am also hoping to get NPU support on the board I am testing.

Without an NPU you might get some pretty bad performance, I think supporting many types of NPUs is really the move here.

I am hoping more and more companies build easier solutions for NPU support but they are all currently a bit of a nightmare to get drivers working.

I am imaging a solution where we autodetect OS / NPU support or ask user to pick then download requirments and run that on their drivers but that's not easy.

srimanachanta commented 1 month ago

I am also hoping to get NPU support on the board I am testing.

Without an NPU you might get some pretty bad performance, I think supporting many types of NPUs is really the move here.

I am hoping more and more companies build easier solutions for NPU support but they are all currently a bit of a nightmare to get drivers working.

I am imaging a solution where we autodetect OS / NPU support or ask user to pick then download requirments and run that on their drivers but that's not easy.

For FRC, what are the main NPUs we would have to target? I don't know of anything beyond RKNN NPU that is mainly used.

ramalamadingdong commented 1 month ago

RPi AI Kit is an example

I can't release what I am working on yet but with the RFP for the new control system coming out I expect a lot of companies will start releasing boards that have NPU support at a cheaper cost.

Alextopher commented 1 day ago

The RPi AI kit looks very promising and worth while supporting.

I haven't found an off the shelf runtime supporting it and the RK NPU.

Here's what I think we could look to support for 2025 (at least what I'm comfortable contributing).

Longer term maybe CUDA support for Jetsons - but I've got no experience with those devices.

Looking into onnx-runtime it advertises CPU and CUDA support, that might be a good starting place. https://onnxruntime.ai/docs/execution-providers/

mcm001 commented 1 day ago

A build of opencv with CUDA support and the DNN module should be all ya need for that. I've got opencv DNN cpu inference rotting in a branch somewhere already.