search

LuckfoxTECH / luckfox-pico

luckfox-pico sdk
322 stars 136 forks source link

Yolo5 demo not working anymore - Regression? #183

Closed pottendo closed 1 month ago

pottendo commented 1 month ago

hi,

My HW: luckfox pico max + SC3336 camera

A few days ago (Oct. 13'th) I cloned the luckfox-pic SDK and built both for SDCard and NAND flash - images boot. I further cloned some demos: e.g.

# yolov5 object detection
git clone https://github.com/luckfox-eng29/luckfox_pico_rtsp_yolov5.git

following the instructions to run /oem/usr/bin/RkLunch-stop.sh and then running the demo works as expected.

Yesterday I forked the SDK repo, cloned my fork and using an Ubuntu docker the build went again flawlessly and boots from the NAND flash (https://github.com/pottendo/pottendo-luckfox-pico/tree/main/project)

I can stream to my host PC after start But when using /oem/usr/bin/RkLunch-stop.sh and then launching the Yolo5 demo some errors appear and the demo isn't working.

Here the console 

[root@luckfox demos]# [ 1153.902457] venc_release 51
[ 1185.950803] stream_cif_mipi_id0: s_power 1, entity use_count 1
[ 1185.951274] stream_cif_mipi_id0: s_power 0, entity use_count 0
[ 1186.100289] stream_cif_mipi_id0: s_power 1, entity use_count 1
[ 1186.101199] stream_cif_mipi_id0: open video, entity use_countt 2
[ 1186.101305] stream_cif_mipi_id1: open video, entity use_countt 1
[ 1186.109158] rkisp rkisp-vir0: first params buf queue
[ 1186.120776] vsys dev open 27
[ 1186.129883] stream_cif_mipi_id0: open video, entity use_countt 3
[ 1186.130047] stream_cif_mipi_id0: close video, entity use_count 2
[ 1186.131333] stream_cif_mipi_id0: open video, entity use_countt 3
[ 1186.133625] stream_cif_mipi_id0: close video, entity use_count 2
[ 1186.141804] stream_cif_mipi_id0: open video, entity use_countt 3
[ 1186.141889] stream_cif_mipi_id0: close video, entity use_count 2
[ 1186.154480] stream_cif_mipi_id0: open video, entity use_countt 3
[ 1186.154568] stream_cif_mipi_id0: close video, entity use_count 2
[ 1186.158278] rkisp queue buf done
[ 1186.159223] rkisp_hw ffa00000.rkisp: set isp clk = 198000000Hz
[ 1186.161161] rkcif-mipi-lvds: stream[0] start streaming
[ 1186.161287] rockchip-mipi-csi2 mipi0-csi2: stream on, src_sd: 46747aed, sd_name:rockchip-csi2-dphy0
[ 1186.161301] rockchip-mipi-csi2 mipi0-csi2: stream ON
[ 1186.161354] rockchip-csi2-dphy0: dphy0, data_rate_mbps 506
[ 1186.161388] rockchip-csi2-dphy csi2-dphy0: csi2_dphy_s_stream stream on:1, dphy0, ret 0
[ 1186.189431] mpp_vcodec: 45: num_chan = 0
[ 1186.189536] mpp_vcodec: 387: attr->max_width = 720, attr->max_height = 480
[ 1186.191163] mpp_vcodec: 103: create channel 0 handle 73df573b online 0
[ 1186.193357] 836: MPP_ENC_SET_CFG in
[ 1186.193388] 532: MPP_ENC_SET_RC_CFG bps 3888000 [3645000 : 4131000] fps [30:30] gop 60
[ 1186.193403] 335: MPP_ENC_SET_PREP_CFG w:h [720:480] stride [2160:480]
[ 1186.193418] 270: send header for set cfg change input/format
[ 1186.193576] 836: MPP_ENC_SET_CFG in
[ 1186.193592] 532: MPP_ENC_SET_RC_CFG bps 3072000 [2880000 : 3264000] fps [30:30] gop 1
[ 1186.193602] 270: send header for set cfg change rc_mode/fps/gop
[ 1186.216930] rga_mm: failed to calculating buffer size!
[ 1186.216955] rga_common: memory param: w = 0, h = 0, f = YCbCr420SP(0xa), size = 0
[ 1186.216963] rga_mm: rga_mm_map_buffer map dma_buf error!
[ 1186.217100] rga_mm: This handle[-22] is illegal.
[ 1186.217135] rga_mm: Image size cal error! width = 0, height = 0, format = YCbCr420SP
[ 1186.217144] rga_mm: Can't get src buffer info from handle!
[ 1186.217151] rga_mm: failed to get buffer from handle
[ 1186.217159] rga_job: rga_job_commit: failed to map job info
[ 1186.217178] rga_job: failed to commit job!
[ 1186.217188] rga: rga_mpi_commit, commit mpi job failed

and here the launch for the demo app:

[root@luckfox yolo5_demo]# ./luckfox_rtsp_yolov5 
  index=0, name=images, n_dims=4, dims=[1, 640, 640, 3], n_elems=1228800, size=1228800, fmt=NHWC, type=INT8, qnt_type=AFFINE, zp=-128, scale=0.003922
  index=0, name=output0, n_dims=4, dims=[1, 80, 80, 255], n_elems=1632000, size=1632000, fmt=NHWC, type=INT8, qnt_type=AFFINE, zp=-128, scale=0.003922
  index=1, name=286, n_dims=4, dims=[1, 40, 40, 255], n_elems=408000, size=408000, fmt=NHWC, type=INT8, qnt_type=AFFINE, zp=-128, scale=0.003922
  index=2, name=288, n_dims=4, dims=[1, 20, 20, 255], n_elems=102000, size=102000, fmt=NHWC, type=INT8, qnt_type=AFFINE, zp=-128, scale=0.003922
model is NHWC input fmt
model input height=640, width=640, channel=3
init rknn model success!
load lable ./model/coco_80_labels_list.txt
rkaiq log level ff1
XCORE:K:rk_aiq_init_lib, ISP HW ver: 32
XCORE:K:
************************** VERSION INFOS **************************
version release date: 2024-2-18
         AIQ:       AIQ v5.0x5.0

git logs:
25bd14e RV1106: smart_door: release rga/isp/mpp for V1.3.0
99d080a RV1106/RV1103: Battery/Doorbell: release kmpp/rockit-ko for V1.3.2
6458e2a RV1106/RV1103: Battery/Doorbell: release kmpp/rockit-ko for V1.3.1
e8dab39 RV1106: CVR: release rga/isp/mpp for V1.1.0
a021b32 RV1106/RV1103: Battery/Doorbell: release kmpp/rockit-ko for V1.3.0

************************ VERSION INFOS END ************************

ID: 0, sensor_name is m00_b_sc3336 4-0030, iqfiles is /etc/iqfiles
XCORE:K:rk_aiq_uapi_sysctl_preInit_devBufCnt: dev_ent:rkraw_rx, buf_cnt:2
XCORE:E:invalid main scene len!

IPCSERVER:K:cid[0] Process path:/tmp/UNIX.domain0, sockfd:20
XCORE:K:cid[0] rk_aiq_uapi_sysctl_init success. iq:/etc/iqfiles/sc3336_CMK-OT2119-PC1_30IRC-F16.json
XCORE:K:cid[0] rk_aiq_uapi_sysctl_prepare success. mode:0 
rk_aiq_uapi2_sysctl_init/prepare succeed
CAMHW:K:cid[0] start success. isGroup:0, isOnline:1, isMultiIsp:0, init_ens:0x1bfc0ffc4739
XCORE:K:cid[0] rk_aiq_uapi_sysctl_start success.
rk_aiq_uapi2_sysctl_start succeed
rockit log path (null), log_size = 0, can use export rt_log_path=, export rt_log_size= change
log_file = (nil) 
RTVersion        21:50:04-603 {dump              :064} ---------------------------------------------------------
RTVersion        21:50:04-604 {dump              :065} rockit version: git-4720dc10a Tue Apr 30 15:47:55 2024 +0800
RTVersion        21:50:04-605 {dump              :066} rockit building: built- 2024-04-30 15:53:43
RTVersion        21:50:04-606 {dump              :067} ---------------------------------------------------------
(null)           21:50:04-606 {log_level_init    :207} 

 please use echo name=level > /tmp/rt_log_level set log level 
    name: all cmpi mb sys vdec venc rgn vpss vgs tde avs wbc vo vi ai ao aenc adec
    log_level: 0 1 2 3 4 5 6 

rockit default level 6, can use export rt_log_level=x, x=0,1,2,3,4,5,6 change
(null)           21:50:04-607 {read_log_level    :098} text is all=4
(null)           21:50:04-608 {read_log_level    :100} module is all, log_level is 4
(null)           21:50:04-612 {monitor_log_level :149} #Start monitor_log_level thread, arg:(nil)
[INFO  rtsp_demo.c:280:rtsp_new_demo] rtsp server demo starting on port 554
[DEBUG rtsp_demo.c:480:rtsp_new_session] add session path: /live/0
rtsp streamq_alloc size is 1753484
vi_dev_init
RKViDev          21:50:04-617 {vi_set_dev_attr   :612} VI_DEV_ATTR_S all parameter reserved
RTIsp3x          21:50:04-619 {ispInitDevice     :591} sensor name = m00_b_sc3336 4-0030
RTDeviceSensorV4 21:50:04-621 {ispOpenDevice     :502} v4l2_open(/dev/video0), handler=53
RTDeviceSensorV4 21:50:04-621 {ispOpenDevice     :502} v4l2_open(/dev/v4l-subdev2), handler=55
RTDeviceSensorV4 21:50:04-622 {close             :112} do RTDeviceSensorV4L2 close
RTDeviceSensorV4 21:50:04-623 {close             :112} do RTDeviceSensorV4L2 close
RKViChn          21:50:04-624 {prepareRuntime    :354} ---------------------------------------------------------
RKViChn          21:50:04-624 {prepareRuntime    :356} vi version: 1.22, name:vvi
RKViChn          21:50:04-626 {prepareRuntime    :357} rockit-ko version: vmpi:28ee389dbc33
RKViChn          21:50:04-627 {prepareRuntime    :358} rockit-ko building: -2024-05-24-18:15:06
RKViChn          21:50:04-628 {prepareRuntime    :359} ---------------------------------------------------------
cmpi             21:50:04-629 {prepareRuntime    :378} mb pool create success, MBCnt= 2
RTIsp3x          21:50:04-630 {ispInitDevice     :591} sensor name = m00_b_sc3336 4-0030
cmpi             21:50:04-633 {createRuntime     :634} >>>>>>>>> Sensor link info dump: <<<<<<<<<
cmpi             21:50:04-633 {createRuntime     :636}   vicap media name           : /dev/media0
cmpi             21:50:04-633 {createRuntime     :637}   vicap model name           : rkcif-mipi-lvds
cmpi             21:50:04-634 {createRuntime     :638}   camera id                  : 0
cmpi             21:50:04-634 {createRuntime     :639}   sensor name                : m00_b_sc3336 4-0030
cmpi             21:50:04-635 {createRuntime     :641}   stream_cif_mipi_id0 video name: /dev/video0
cmpi             21:50:04-635 {createRuntime     :642}   stream_cif_mipi_id1 video name: /dev/video1
cmpi             21:50:04-635 {createRuntime     :643}   stream_cif_mipi_id2 video name: /dev/video2
cmpi             21:50:04-636 {createRuntime     :644}   stream_cif_mipi_id3 video name: /dev/video3
cmpi             21:50:04-636 {createRuntime     :646}   sensor raw width           : 2304
cmpi             21:50:04-636 {createRuntime     :647}   sensor raw height          : 1296
cmpi             21:50:04-637 {createRuntime     :648} ---------------------------------------------------------
cmpi             21:50:04-637 {createRuntime     :654}   isp media name           : /dev/media1
cmpi             21:50:04-638 {createRuntime     :655}   isp model name           : rkisp0
cmpi             21:50:04-638 {createRuntime     :656}   isp driver name          : rkisp-vir0
cmpi             21:50:04-638 {createRuntime     :657}   isp dev name             : /dev/v4l-subdev3
cmpi             21:50:04-639 {createRuntime     :659}   isp rawrd0_m_path        : /dev/video17
cmpi             21:50:04-639 {createRuntime     :660}   isp rawrd1_l_path        : 
cmpi             21:50:04-639 {createRuntime     :661}   isp rawrd2_s_path        : /dev/video18
cmpi             21:50:04-640 {createRuntime     :664}   mainpath video name      : /dev/video11
cmpi             21:50:04-640 {createRuntime     :665}   selfpath video name      : /dev/video12
cmpi             21:50:04-640 {createRuntime     :666}   bypasspath video name    : /dev/video13
cmpi             21:50:04-640 {createRuntime     :667}   sensor link type         : 2
cmpi             21:50:04-640 {createRuntime     :668}   sensor is united         : 0
cmpi             21:50:04-640 {createRuntime     :672} ---------------------------------------------------------
cmpi             21:50:04-640 {createRuntime     :711} [non-WRAP MODE]: buff size = 518400
RTIsp3x          21:50:04-643 {ispInitDevice     :591} sensor name = m00_b_sc3336 4-0030
RTDeviceV4L2     21:50:04-645 {open              :137} cameraIndex:0, entityName:rkisp_mainpath
RTDeviceV4L2     21:50:04-645 {open              :143} open video name(/dev/video11)
RKViChn          21:50:04-646 {setFrameRate      :1469} [vi] dev(0) ch(0) illegal param s32SrcFrameRate(0) s32DstFrameRate(0)
RTDeviceV4L2     21:50:04-646 {ispCameraInfo     :567} current device:/dev/video11 isn't compatible(cap:0x84201000) device,memoryType:4, retry:0
RTDeviceV4L2     21:50:04-647 {ispInitFormat     :767} ioctl VIDIOC_S_FMT OK
cmpi             21:50:04-648 {start             :1093} =========== vi Start startRuntime ===========
RTDeviceV4L2     21:50:04-648 {ispStreamOn       :458} do ispStreamOn start
CAMHW:K:camId:0, notify_isp_stream_status on
RTDeviceV4L2     21:50:04-667 {ispStreamOn       :511} do ispStreamOn done
vpss_init
rga_api version 1.10.0_[2]
=== 0 ===
====RK_MPI_SYS_Bind vi0 to vpss0====
venc_init
MpiAdapterEncode 21:50:04-678 {ma_encode_create  :1743} ringbuf buf_size 520192
MpiAdapterEncode 21:50:04-681 {ma_encode_create  :1782} mem_fd = 61
venc init success
=== 1 ===

When I connect from the host-PC, there's no further output.

I noticed that during build.sh lunch my forked repo build distiguishes the hardware variants Pro from Max:

pottendo@c3cd94f13217:/luckfox$ ./build.sh lunch
ls: cannot access 'BoardConfig*.mk': No such file or directory
You're building on Linux
  Lunch menu...pick the Luckfox Pico hardware version:
  选择 Luckfox Pico 硬件版本:
                [0] RV1103_Luckfox_Pico
                [1] RV1103_Luckfox_Pico_Mini_A
                [2] RV1103_Luckfox_Pico_Mini_B
                [3] RV1103_Luckfox_Pico_Plus
                [4] RV1106_Luckfox_Pico_Pro
                [5] RV1106_Luckfox_Pico_Max
                [6] RV1106_Luckfox_Pico_Ultra
                [7] RV1106_Luckfox_Pico_Ultra_W
                [8] custom
Which would you like? [0~7][default:0]: 5
  Lunch menu...pick the boot medium:
  选择启动媒介:
                [0] SD_CARD
                [1] SPI_NAND
Which would you like? [0~1][default:0]: 1
  Lunch menu...pick the system version:
  选择系统版本:
                [0] Buildroot(Support Rockchip official features) 
Which would you like? [0~1][default:0]: 0
[build.sh:info] Lunching for Default BoardConfig_IPC/BoardConfig-SPI_NAND-Buildroot-RV1106_Luckfox_Pico_Max-IPC.mk boards...
[build.sh:info] switching to board: /luckfox/project/cfg/BoardConfig_IPC/BoardConfig-SPI_NAND-Buildroot-RV1106_Luckfox_Pico_Max-IPC.mk
[build.sh:info] Running build_select_board succeeded.

this was not the case previously:

pottendo@hansi:/work/src/luckfox/luckfox-pico(main)$ ./build.sh lunch
You're building on Linux
  Lunch menu...pick the Luckfox Pico hardware version:
  选择 Luckfox Pico 硬件版本:
                [0] RV1103_Luckfox_Pico
                [1] RV1103_Luckfox_Pico_Mini_A
                [2] RV1103_Luckfox_Pico_Mini_B
                [3] RV1103_Luckfox_Pico_Plus
                [4] RV1106_Luckfox_Pico_Pro_Max
                [5] RV1106_Luckfox_Pico_Ultra
                [6] RV1106_Luckfox_Pico_Ultra_W
                [7] custom
Which would you like? [0~7][default:0]: 4
  Lunch menu...pick the boot medium:
  选择启动媒介:
                [0] SD_CARD
                [1] SPI_NAND
Which would you like? [0~1][default:0]: 1
  Lunch menu...pick the system version:
  选择系统版本:
                [0] Buildroot(Support Rockchip official features) 
Which would you like? [0~1][default:0]: 0
[build.sh:info] Lunching for Default BoardConfig_IPC/BoardConfig-SPI_NAND-Buildroot-RV1106_Luckfox_Pico_Pro_Max-IPC.mk boards...
[build.sh:info] Running build_select_board succeeded.

Can someone confirm that with the most recent builds, the Yolo5 demo still works? Maybe the log information tells what's wrong.

Running /oem/usr/bin/RkLunch.sh again makes the video-stream working.

thanks, pottendo

luckfox-eng33 commented 1 month ago

RGA version not updated, please re-pull https://github.com/LuckfoxTECH/luckfox_pico_rkmpi_example

pottendo commented 1 month ago

Thanks @luckfox-eng33, the demo works again with this update - also on my recent builds. happy hacking, pottendo