search

SWAMP-Blimps / CatchingBlimp

4 stars 3 forks source link

Get ML Pipeline running on NPU #28

Open sah4jpatel opened 2 weeks ago

sah4jpatel commented 2 weeks ago

Working with rknn-toolkit2 to deploy a vision model on NPU

sah4jpatel commented 3 days ago

any updates too add here? @Rhyme0730

Rhyme0730 commented 2 days ago

I directly converted our previous model best.pt --> best.onnx --> best.rknn. And then implementing it on npu but failed. Now trying to train a new test yolo model on npu.

sah4jpatel commented 2 days ago

Are you using the docker instance to run the rknn-toolkit2 converter? I was having trouble with that too with OP set incompatibility reasons. I'll be spending tonight on figuring out how to convert huggingface onnx models to rknn.

Rhyme0730 commented 2 days ago

No, I don't use docker. Instead, for converting model, I just install rknn-toolkit2 on PC and convert .onnx model to .rknn. Then use ssh scp -r command copy them to orangepi and run it on rknn-toolkit-lite2. The entire work flow is: Converting model on PC --> ssh scp to Opi --> Using rknn-toolkit-lite2 running model on Opi

sah4jpatel commented 2 days ago

I see. Let me know if you solve that issue you ran into.

Get Outlook for iOShttps://aka.ms/o0ukef

From: Rhyme-Rao @.> Sent: Wednesday, June 26, 2024 10:36:56 AM To: SWAMP-Blimps/CatchingBlimp @.> Cc: Sahaj Patel @.>; Assign @.> Subject: Re: [SWAMP-Blimps/CatchingBlimp] Get ML Pipeline running on NPU (Issue #28)

No, I don't use docker. Instead, for converting model, I just install rknn-toolkit2 on PC and convert .onnx model to .rknn. Then use ssh scp -r command copy them to orangepi and run it on rknn-toolkit-lite2. The entire work flow is: Converting model on PC --> ssh scp to Opi --> Using rknn-toolkit-lite2 running model on Opi

— Reply to this email directly, view it on GitHubhttps://github.com/SWAMP-Blimps/CatchingBlimp/issues/28#issuecomment-2192015071, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ARYGKYKPUB7GPPNMRYETHULZJLNZRAVCNFSM6AAAAABJLCQF76VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCOJSGAYTKMBXGE. You are receiving this because you were assigned.Message ID: @.***>

Rhyme0730 commented 2 days ago
  1. Training custom yolov5 model (Finish)
  2. Convert yolov5.pt into yolov5.onnx (Finish)
  3. Conver yolov5.onnx into yolov5.rknn (Official code not work, TBD)
  4. Implement yolov5.rknn on board using rknn-toolkit-lite2 (match input output size, TBD)
Rhyme0730 commented 2 days ago

@sah4jpatel Do you know how to convert .onnx to .rknn model for a custom yolo model? I am currently stuck in it.

sah4jpatel commented 2 days ago

I haven't tried that yet. I am about to start investigating that. I opted to use the dockerized version of RKNN-toolkit on Ubuntu20.04 because it seems like the best experience to get everything working. However, I was getting an error with incompatible OP sets but that's because I was trying to convert Florence2's weights (which might be too new or just incompatible). I found v10 onnx weights on huggingface that I downloaded, I'll try debugging diligently tonight.

Rhyme0730 commented 1 day ago

Ok, I see. The compability is an esstential part for us, we can document every tool's version after running successfully.

sah4jpatel commented 1 day ago

Yea I think that will be important. V10 had so many issues, V8 was almost flawless once I wrote the script to convert it.