search

airockchip / rknn_model_zoo

Apache License 2.0
1.05k stars 192 forks source link

Issue with Deploying custom yolov8m Model on RK3588 #152

Open aaquibniyama opened 3 months ago

aaquibniyama commented 3 months ago

Hello everyone,

I am currently working on deploying a yolov8m model on the Orange Pi 5 RK3588. I have followed the conversion steps from .pt to ONNX, and then from ONNX to RKNN model. However, I am encountering issues when trying to deploy the model on the board using RKNNLite and the yolov8 example from the RKNN Model Zoo GitHub repository.

I have modified the number of classes and class ID in the code, but I keep getting the following error:

`Model-yolov8m-640-640_rk3588.rknn is rknn model, starting val

E Catch exception when setting inputs.

E Traceback (most recent call last):

File "/usr/local/lib/python3.10/dist-packages/rknnlite/api/rknn_lite.py", line 201, in inference

self.rknn_runtime.set_inputs(inputs, data_type, data_format, inputs_pass_through=inputs_pass_through)

File "rknnlite/api/rknn_runtime.py", line 1082, in rknnlite.api.rknn_runtime.RKNNRuntime.set_inputs

Exception: The input[0] need 4dims input, but 3dims input buffer feed.

Traceback (most recent call last):

File "/home/orangepi/RKNN/rknn_model_zoo/examples/yolov8/python/yolov8.py", line 263, in

boxes, classes, scores = post_process(outputs)

File "/home/orangepi/RKNN/rknn_model_zoo/examples/yolov8/python/yolov8.py", line 120, in post_process

pair_per_branch = len(input_data)//defualt_branch

TypeError: object of type 'NoneType' has no len()`

I have tried various solutions from different forums and GitHub repositories, but I am still unable to successfully deploy the model. I also attempted using RKNNPU by changing the post-process, but I am not able to detect objects properly according to my model.

now next when we modified the code and print the outputs boxes scores and classes we were able to print this but after post process when we are printing the boxes scores and classes the output is showing None I am able to open the camera but there is no detections. Here is the output results

`(1, 255, 80, 80) input_data is ------ [[[[0.39955714 0.5797496 0.44656387 ... 0.5249084 0.48573616 0.4034744 ] [0.4818189 0.5249084 0.4191433 ... 0.552329 0.5053223 0.4230605 ] [0.517074 0.52882564 0.41522607 ... 0.57583237 0.5092395 0.4230605 ] ... [0.54449457 0.48965338 0.39563993 ... 0.56799793 0.5249084 0.4034744 ] [0.5092395 0.47398448 0.37997103 ... 0.5562462 0.5092395 0.42697775] [0.54057735 0.43872944 0.39563993 ... 0.5249084 0.5249084 0.42697775]]

[[0.54449457 0.47006723 0.45048112 ... 0.45048112 0.4622328 0.5797496 ] [0.5562462 0.60717016 0.5484118 ... 0.5562462 0.54057735 0.5092395 ] [0.517074 0.61108744 0.5954185 ... 0.57583237 0.57583237 0.47398448] ... [0.5562462 0.50140506 0.53274286 ... 0.5366601 0.552329 0.61108744] [0.4818189 0.4348122 0.42697775 ... 0.44656387 0.4622328 0.53274286] [0.52882564 0.4348122 0.4191433 ... 0.4073916 0.47398448 0.48573616]]

[[0.43872944 0.7638593 0.8539555 ... 0.8774589 0.7795282 0.5092395 ] [0.43872944 0.7638593 0.8852933 ... 0.90096223 0.7716937 0.517074 ] [0.44264665 0.7442731 0.8892105 ... 0.90487945 0.74819034 0.5092395 ] ... [0.45831555 0.7442731 0.8892105 ... 0.8892105 0.74819034 0.5053223 ] [0.43872944 0.7442731 0.8774589 ... 0.8774589 0.7677765 0.49748784] [0.43089497 0.6659286 0.79127985 ... 0.7991143 0.6933492 0.48573616]]

...

[[0.00391723 0.00391723 0.00391723 ... 0.00391723 0.00391723 0.00391723] [0.00391723 0.00391723 0.00391723 ... 0.00391723 0.00391723 0.00391723] [0.00391723 0.00391723 0.00391723 ... 0.00391723 0.00391723 0.00391723] ... [0.00391723 0.00391723 0.00391723 ... 0.00391723 0.00391723 0.00391723] [0.00391723 0.00391723 0.00391723 ... 0.00391723 0.00391723 0.00391723] [0.00391723 0.00391723 0.00391723 ... 0.00391723 0.00391723 0.00391723]]

[[0. 0. 0. ... 0. 0.

  1. ] [0. 0. 0. ... 0. 0.
  2. ] [0. 0. 0. ... 0. 0.00391723 0.00391723] ... [0. 0. 0. ... 0. 0.
  3. ] [0. 0. 0. ... 0. 0.00391723
  4. ] [0. 0. 0. ... 0. 0.

  5. ]]

    [[0.00391723 0.00391723 0.00391723 ... 0.00391723 0.00391723 0.00391723] [0.00391723 0.00391723 0.00391723 ... 0.00391723 0.00391723 0.00391723] [0.00391723 0.00391723 0.00391723 ... 0.00391723 0.00391723 0.00391723] ... [0.00391723 0.00391723 0.00391723 ... 0.00391723 0.00391723 0.00391723] [0.00391723 0.00391723 0.00391723 ... 0.00391723 0.00391723 0.00391723] [0.00391723 0.00391723 0.00391723 ... 0.00391723 0.00391723 0.00391723]]]] post process is ------ [array([[[[0.39955714, 0.5797496 , 0.44656387, ..., 0.5249084 , 0.48573616, 0.4034744 ], [0.4818189 , 0.5249084 , 0.4191433 , ..., 0.552329 , 0.5053223 , 0.4230605 ], [0.517074 , 0.52882564, 0.41522607, ..., 0.57583237, 0.5092395 , 0.4230605 ], ..., [0.54449457, 0.48965338, 0.39563993, ..., 0.56799793, 0.5249084 , 0.4034744 ], [0.5092395 , 0.47398448, 0.37997103, ..., 0.5562462 , 0.5092395 , 0.42697775], [0.54057735, 0.43872944, 0.39563993, ..., 0.5249084 , 0.5249084 , 0.42697775]],

    [[0.54449457, 0.47006723, 0.45048112, ..., 0.45048112,
  0.4622328 , 0.5797496 ],
 [0.5562462 , 0.60717016, 0.5484118 , ..., 0.5562462 ,
  0.54057735, 0.5092395 ],
 [0.517074  , 0.61108744, 0.5954185 , ..., 0.57583237,
  0.57583237, 0.47398448],
 ...,
 [0.5562462 , 0.50140506, 0.53274286, ..., 0.5366601 ,
  0.552329  , 0.61108744],
 [0.4818189 , 0.4348122 , 0.42697775, ..., 0.44656387,
  0.4622328 , 0.53274286],
 [0.52882564, 0.4348122 , 0.4191433 , ..., 0.4073916 ,
  0.47398448, 0.48573616]],

[[0.43872944, 0.7638593 , 0.8539555 , ..., 0.8774589 ,
  0.7795282 , 0.5092395 ],
 [0.43872944, 0.7638593 , 0.8852933 , ..., 0.90096223,
  0.7716937 , 0.517074  ],
 [0.44264665, 0.7442731 , 0.8892105 , ..., 0.90487945,
  0.74819034, 0.5092395 ],
 ...,
 [0.45831555, 0.7442731 , 0.8892105 , ..., 0.8892105 ,
  0.74819034, 0.5053223 ],
 [0.43872944, 0.7442731 , 0.8774589 , ..., 0.8774589 ,
  0.7677765 , 0.49748784],
 [0.43089497, 0.6659286 , 0.79127985, ..., 0.7991143 ,
  0.6933492 , 0.48573616]],

...,

[[0.        , 0.00391723, 0.        , ..., 0.00391723,
  0.        , 0.        ],
 [0.        , 0.00391723, 0.00391723, ..., 0.00391723,
  0.00391723, 0.00391723],
 [0.00391723, 0.00391723, 0.00391723, ..., 0.00391723,
  0.00391723, 0.00391723],
 ...,
 [0.        , 0.00391723, 0.00391723, ..., 0.00391723,
  0.00391723, 0.00391723],
 [0.        , 0.00391723, 0.00391723, ..., 0.00391723,
  0.00391723, 0.00391723],
 [0.        , 0.00391723, 0.00391723, ..., 0.00391723,
  0.00391723, 0.        ]],

[[0.        , 0.        , 0.        , ..., 0.        ,
  0.        , 0.        ],
 [0.        , 0.        , 0.        , ..., 0.        ,
  0.        , 0.        ],
 [0.        , 0.        , 0.        , ..., 0.        ,
  0.        , 0.        ],
 ...,
 [0.        , 0.        , 0.        , ..., 0.        ,
  0.        , 0.        ],
 [0.        , 0.        , 0.        , ..., 0.        ,
  0.        , 0.        ],
 [0.        , 0.        , 0.        , ..., 0.        ,
  0.        , 0.        ]],

[[0.00391723, 0.00391723, 0.00391723, ..., 0.00783445,
  0.00391723, 0.00391723],
 [0.00391723, 0.00391723, 0.00391723, ..., 0.00391723,
  0.00391723, 0.00391723],
 [0.00391723, 0.00391723, 0.00783445, ..., 0.00391723,
  0.00783445, 0.00783445],
 ...,
 [0.00391723, 0.00391723, 0.00391723, ..., 0.00391723,
  0.00391723, 0.00391723],
 [0.00391723, 0.00391723, 0.00391723, ..., 0.00391723,
  0.00391723, 0.00391723],
 [0.00391723, 0.00391723, 0.00783445, ..., 0.00391723,
  0.00783445, 0.00391723]]],

    [[[0.4348122 , 0.6189219 , 0.5092395 , ..., 0.5209912 , 0.4622328 , 0.36430213], [0.53274286, 0.60325295, 0.5484118 , ..., 0.552329 , 0.4622328 , 0.3760538 ], [0.57583237, 0.60325295, 0.5366601 , ..., 0.56799793, 0.47006723, 0.3760538 ], ..., [0.5915013 , 0.5562462 , 0.4935706 , ..., 0.5209912 , 0.46615002, 0.36038488], [0.5484118 , 0.53274286, 0.4779017 , ..., 0.53274286, 0.4622328 , 0.38780546], [0.5797496 , 0.4935706 , 0.47006723, ..., 0.52882564, 0.5249084 , 0.4230605 ]],

    [[0.56016344, 0.50140506, 0.4779017 , ..., 0.4779017 ,
  0.48573616, 0.5836668 ],
 [0.59933573, 0.64242524, 0.5875841 , ..., 0.5915013 ,
  0.57583237, 0.53274286],
 [0.54057735, 0.6189219 , 0.60325295, ..., 0.5875841 ,
  0.5797496 , 0.4818189 ],
 ...,
 [0.5249084 , 0.48573616, 0.5209912 , ..., 0.5131567 ,
  0.5249084 , 0.5915013 ],
 [0.42697775, 0.38780546, 0.38388824, ..., 0.39955714,
  0.41522607, 0.48965338],
 [0.47398448, 0.37997103, 0.37213656, ..., 0.36038488,
  0.4230605 , 0.43872944]],

[[0.39563993, 0.6189219 , 0.7638593 , ..., 0.80303156,
  0.638508  , 0.4191433 ],
 [0.36038488, 0.61500466, 0.7677765 , ..., 0.7991143 ,
  0.61108744, 0.4230605 ],
 [0.37997103, 0.61108744, 0.7638593 , ..., 0.79519707,
  0.60717016, 0.4191433 ],
 ...,
 [0.39955714, 0.60325295, 0.7834454 , ..., 0.810866  ,
  0.59933573, 0.41522607],
 [0.37213656, 0.5915013 , 0.7716937 , ..., 0.79519707,
  0.61108744, 0.4034744 ],
 [0.39563993, 0.5484118 , 0.7011836 , ..., 0.7442731 ,
  0.57583237, 0.39955714]],

...,

[[0.        , 0.00391723, 0.00391723, ..., 0.00391723,
  0.00391723, 0.00391723],
 [0.00391723, 0.00391723, 0.00391723, ..., 0.00391723,
  0.00391723, 0.00391723],
 [0.00391723, 0.00391723, 0.00391723, ..., 0.00391723,
  0.00391723, 0.00391723],
 ...,
 [0.00391723, 0.00391723, 0.00391723, ..., 0.00391723,
  0.00391723, 0.00391723],
 [0.00391723, 0.00391723, 0.00391723, ..., 0.00391723,
  0.00391723, 0.00391723],
 [0.00391723, 0.00391723, 0.00391723, ..., 0.00391723,
  0.00391723, 0.00391723]],

[[0.        , 0.        , 0.        , ..., 0.        ,
  0.        , 0.        ],
 [0.        , 0.        , 0.        , ..., 0.        ,
  0.        , 0.        ],
 [0.        , 0.        , 0.        , ..., 0.        ,
  0.        , 0.        ],
 ...,
 [0.        , 0.        , 0.        , ..., 0.        ,
  0.        , 0.        ],
 [0.        , 0.        , 0.        , ..., 0.        ,
  0.        , 0.        ],
 [0.        , 0.        , 0.        , ..., 0.        ,
  0.        , 0.        ]],

[[0.00391723, 0.00391723, 0.00391723, ..., 0.00391723,
  0.00391723, 0.00391723],
 [0.00391723, 0.00391723, 0.00391723, ..., 0.00391723,
  0.00391723, 0.00391723],
 [0.00391723, 0.00391723, 0.00783445, ..., 0.00391723,
  0.00783445, 0.00783445],
 ...,
 [0.00391723, 0.00391723, 0.00391723, ..., 0.00391723,
  0.00391723, 0.00391723],
 [0.00391723, 0.00391723, 0.00391723, ..., 0.00391723,
  0.00391723, 0.00391723],
 [0.00391723, 0.00391723, 0.00391723, ..., 0.00391723,
  0.00783445, 0.00391723]]],

    [[[0.43872944, 0.61500466, 0.5092395 , ..., 0.50140506, 0.43872944, 0.3486332 ], [0.54057735, 0.60325295, 0.56016344, ..., 0.5249084 , 0.4230605 , 0.34471598], [0.5797496 , 0.59933573, 0.54449457, ..., 0.5366601 , 0.43089497, 0.33688152], ..., [0.59933573, 0.552329 , 0.50140506, ..., 0.4779017 , 0.4230605 , 0.32121262], [0.5562462 , 0.53274286, 0.48965338, ..., 0.5053223 , 0.42697775, 0.35646766], [0.5875841 , 0.4935706 , 0.4779017 , ..., 0.5092395 , 0.50140506, 0.4034744 ]],

    [[0.56799793, 0.5092395 , 0.48573616, ..., 0.4818189 ,
  0.48965338, 0.5836668 ],
 [0.61500466, 0.6541769 , 0.5954185 , ..., 0.59933573,
  0.5875841 , 0.5366601 ],
 [0.54449457, 0.61500466, 0.59933573, ..., 0.5836668 ,
  0.57583237, 0.47398448],
 ...,
 [0.5092395 , 0.4779017 , 0.5092395 , ..., 0.50140506,
  0.5131567 , 0.57583237],
 [0.41522607, 0.3760538 , 0.3760538 , ..., 0.38780546,
  0.4034744 , 0.47398448],
 [0.4622328 , 0.37213656, 0.36038488, ..., 0.35255045,
  0.41130883, 0.42697775]],

[[0.29379204, 0.43089497, 0.5209912 , ..., 0.5562462 ,
  0.45048112, 0.31337816],
 [0.26637143, 0.4348122 , 0.54449457, ..., 0.5640807 ,
  0.44656387, 0.32512984],
 [0.28595757, 0.4348122 , 0.552329  , ..., 0.5640807 ,
  0.45048112, 0.33688152],
 ...,
 [0.29379204, 0.4230605 , 0.5562462 , ..., 0.57583237,
  0.44656387, 0.3329643 ],
 [0.28204036, 0.4191433 , 0.54449457, ..., 0.5562462 ,
  0.44656387, 0.3172954 ],
 [0.3172954 , 0.39563993, 0.49748784, ..., 0.5249084 ,
  0.4191433 , 0.31337816]],

...,

[[0.00391723, 0.00391723, 0.00391723, ..., 0.00391723,
  0.00391723, 0.00391723],
 [0.00391723, 0.00391723, 0.00391723, ..., 0.00391723,
  0.00391723, 0.00391723],
 [0.00391723, 0.00391723, 0.00391723, ..., 0.00391723,
  0.00391723, 0.00391723],
 ...,
 [0.00391723, 0.00391723, 0.00391723, ..., 0.00391723,
  0.00391723, 0.00391723],
 [0.00391723, 0.00391723, 0.00391723, ..., 0.00391723,
  0.00391723, 0.00391723],
 [0.00391723, 0.00391723, 0.00391723, ..., 0.00391723,
  0.00391723, 0.00391723]],

[[0.        , 0.        , 0.        , ..., 0.        ,
  0.        , 0.        ],
 [0.        , 0.        , 0.        , ..., 0.        ,
  0.        , 0.        ],
 [0.        , 0.        , 0.        , ..., 0.        ,
  0.00391723, 0.00391723],
 ...,
 [0.        , 0.        , 0.        , ..., 0.        ,
  0.        , 0.        ],
 [0.        , 0.        , 0.        , ..., 0.        ,
  0.00391723, 0.        ],
 [0.        , 0.        , 0.        , ..., 0.        ,
  0.        , 0.        ]],

[[0.00391723, 0.00391723, 0.00391723, ..., 0.00391723,
  0.00391723, 0.00391723],
 [0.00391723, 0.00391723, 0.00391723, ..., 0.00391723,
  0.00391723, 0.00391723],
 [0.00391723, 0.00391723, 0.00391723, ..., 0.00391723,
  0.00391723, 0.00391723],
 ...,
 [0.00391723, 0.00391723, 0.00391723, ..., 0.00391723,
  0.00391723, 0.00391723],
 [0.00391723, 0.00391723, 0.00391723, ..., 0.00391723,
  0.00391723, 0.00391723],
 [0.00391723, 0.00391723, 0.00391723, ..., 0.00391723,
  0.00391723, 0.00391723]]]], dtype=float32)]

    boxes [[-1.45675600e+00 9.58085716e-01 6.24258459e+00 8.46574050e+00] [ 1.60637307e+00 1.44266725e-01 2.49456136e+01 6.89788473e+00] [ 4.56022143e+00 4.75251079e-02 3.37298214e+01 6.36787063e+00] ... [ 6.01962923e+02 6.27135650e+02 6.38332731e+02 6.40145966e+02] [ 6.16427508e+02 6.28475995e+02 6.39617413e+02 6.40685871e+02] [ 6.27973979e+02 6.28190278e+02 6.40937154e+02 6.41473052e+02]] classes [[0.36038488 0.00391723 0.14885463 ... 0. 0. 0.00391723] [0.32512984 0.00391723 0.16844076 ... 0.00391723 0. 0.00391723] [0.27028868 0.00391723 0.20761304 ... 0. 0. 0.00391723] ... [0.2781231 0.01175168 0.09793068 ... 0.00391723 0. 0.00391723] [0.258537 0.01175168 0.09009622 ... 0.00391723 0. 0.00391723] [0.258537 0.01175168 0.086179 ... 0.00391723 0. 0.00391723]] scores [[0.] [0.] [0.] ... [0.] [0.] [0.]] __class_pos (array([], dtype=int64),) boxes [] classes [[0.36038488 0.00391723 0.14885463 ... 0. 0. 0.00391723] [0.32512984 0.00391723 0.16844076 ... 0.00391723 0. 0.00391723] [0.27028868 0.00391723 0.20761304 ... 0. 0. 0.00391723] ... [0.2781231 0.01175168 0.09793068 ... 0.00391723 0. 0.00391723] [0.258537 0.01175168 0.09009622 ... 0.00391723 0. 0.00391723] [0.258537 0.01175168 0.086179 ... 0.00391723 0. 0.00391723]] scores [] None None None processing time = 0.0685875415802002 FPS: 14.57722170090015`

I have attached the code and output numpy file for your reference

Code: https://drive.google.com/file/d/1YdnEI12c-Hup6Gq0otO2EoWGFSOr-gMg/view?usp=sharing Numpy File: https://drive.google.com/file/d/1SPixCOhCfUoMxWdLz0m9ucMfXrcJoNE6/view?usp=sharing

zen-xingle commented 3 months ago

rknnlite - problem

Exception: The input[0] need 4dims input, but 3dims input buffer feed. This means that the input should be 4D. For 4D NCHW input, rknn.inference([4D_data], data_format='nchw'). For 4D NHWC input, rknn.inference([4D_data], data_format='nhwc').

For the second problem, it is more likely that there are differences in the output layers defined in the model. Please check with the provided model and figure out whether the final layer op type is same.