Can you make it support rockchip encoder/decoder?

[ywangwxd]

If I have a self-complied ffmpeg library which support other hardware acceleration such as Rockchip (which is quite popular now), can I compile your library on the top of this ffmpeg lib?

[chenxinfeng4]

I have tested rockchip rk3588 in ffmpegcv at cpu mode. There is no need to compile ffmpegcv. The ffmpegcv communicate the ffmpeg by running os.system('ffmpeg -c:v DECODER -i VIDEO -xxxx PIPE)", something like this. Make sure your ffmpeg is executive.

I didn't use rockchip specific encoder/decoder, but you can specify them by vid = ffmpegcv.VideoCapture(INPUT, codec=YOURCODEC), as wheel as VideoWriter. Then you can print(vid.ffmpeg_cmd) to see the full ffmpeg command.

[chenxinfeng4]

The ffmpeg has decoders for rockchip, but no encoder. Try vid = ffmpegcv.VideoCapture(INPUT, codec='h264_rkmpp').

(base) chenxinfeng@orangepi5:~$ ffmpeg -decoders 2>/dev/null  | grep mpp
 V..... h263_rkmpp           h263 (rkmpp) (codec h263)
 V..... h264_rkmpp           h264 (rkmpp) (codec h264)
 V..... hevc_rkmpp           hevc (rkmpp) (codec hevc)
 V..... mpeg1_rkmpp          mpeg1 (rkmpp) (codec mpeg1video)
 V..... mpeg2_rkmpp          mpeg2 (rkmpp) (codec mpeg2video)
 V..... mpeg4_rkmpp          mpeg4 (rkmpp) (codec mpeg4)
 V..... vp8_rkmpp            vp8 (rkmpp) (codec vp8)
 V..... vp9_rkmpp            vp9 (rkmpp) (codec vp9)

(base) chenxinfeng@orangepi5:~$ ffmpeg -encoders 2>/dev/null  | grep mpp
<NULL>

[ywangwxd]

The ffmpeg I compiled has encoder and decoder from MPP rockchip, as listed by ffmpeg -codecs. But not all of them.

 D.V.L. av1                  Alliance for Open Media AV1 (decoders: av1_rkmpp_decoder av1 )
 DEV.L. h263                 H.263 / H.263-1996, H.263+ / H.263-1998 / H.263 version 2 (decoders: h263_rkmpp_decoder h263 h263_v4l2m2m ) (encoders: h263 h263_v4l2m2m )
 DEV.LS h264                 H.264 / AVC / MPEG-4 AVC / MPEG-4 part 10 (decoders: h264_rkmpp_decoder h264 h264_v4l2m2m ) (encoders: h264_rkmpp_encoder h264_v4l2m2m )
 DEV.L. hevc                 H.265 / HEVC (High Efficiency Video Coding) (decoders: hevc_rkmpp_decoder hevc hevc_v4l2m2m ) (encoders: hevc_rkmpp_encoder hevc_v4l2m2m )
 DEV.L. mpeg1video           MPEG-1 video (decoders: mpeg1_rkmpp_decoder mpeg1video mpeg1_v4l2m2m )
 DEV.L. mpeg2video           MPEG-2 video (decoders: mpeg2_rkmpp_decoder mpeg2video mpegvideo mpeg2_v4l2m2m )
 DEV.L. mpeg4                MPEG-4 part 2 (decoders: mpeg4_rkmpp_decoder mpeg4 mpeg4_v4l2m2m ) (encoders: mpeg4 mpeg4_v4l2m2m )
 DEV.L. vp8                  On2 VP8 (decoders: vp8_rkmpp_decoder vp8 vp8_v4l2m2m ) (encoders: vp8_rkmpp_encoder vp8_v4l2m2m )
 D.V.L. vp9                  Google VP9 (decoders: vp9_rkmpp_decoder vp9 vp9_v4l2m2m )

[ywangwxd]

I have tested rockchip rk3588 in ffmpegcv at cpu mode. There is no need to compile ffmpegcv. The ffmpegcv communicate the ffmpeg by running os.system('ffmpeg -c:v DECODER -i VIDEO -xxxx PIPE)", something like this. Make sure your ffmpeg is executive.

I didn't use rockchip specific encoder/decoder, but you can specify them by vid = ffmpegcv.VideoCapture(INPUT, codec=YOURCODEC), as wheel as VideoWriter. Then you can print(vid.ffmpeg_cmd) to see the full ffmpeg command.

Will the use of pipe be inefficient, especially for multiple number of video stream process.

[ywangwxd]

As for the efficiency of linux pipe, I have done some google search and found this page helpful. How fast are Linux pipes anyway?

The author made the pipe 20x faster finally. I have not went into the details.

[chenxinfeng4]

Yeah, the pipe is not inefficient, sometimes would be the bottleneck. It has advantages using pipe.

• Easy to use of pipe. The ffmpeg is a c++ lib/program, it's great but complex. It's hard to embedding ffmpeg into naive python, out of my capability.
• Easy to install the ffmpegcv. Any compiling would make the ffmpegcv hard to install for users.
• Pipe is multiprocessing. It release the cpu usage of python, the video-encoding-decoding may be in other cpu core.

Will the use of pipe be inefficient, especially for multiple number of video stream process.

In your case, multiple number of video stream would not waste much performance.

[chenxinfeng4]

The efficiency of pipe maybe not caused by linux. The shell is very efficient, but not python.

[chenxinfeng4]

In my test, the pipe in ffmpeg>>python(ffmpegcv) is a bit slow, but the cpu usage is also slow down. You will have the some cpu usage in total.

[ywangwxd]

How should I config VideoCaptureStream object so that it can use the ffmpeg lib for decoder/encoder? I used default, but it show error message RuntimeError: The ffmpeg is not compiled with NVENC support.

What I suppose is this, ffmpegcv call ffmpeg, ffmpeg call MPP internally for h264, hevc.

[chenxinfeng4]

What do you mean of video stream. Would that be video files (ffmpegcv.VideoCapture), video cameras (ffmpegcv.VideoCaptureCAM) or IP camera stream (ffmpegcv.VideoCaptureStream).

Currently, the VideoCaptureStream didn't support codec as input, you can manually specify that is ffmpeg_cmd. Tips. You can use noblock to detach the multiple video processing task among cpu-multi-cores, and relieve the cpu usage in python. The noblock support ffmpegcv.VideoCapture(/NV) and VideoWriter(/NV) only.

[chenxinfeng4]

It's not the output of VideoCaptureStream, can you post your full code?

RuntimeError: The ffmpeg is not compiled with NVENC support.

[ywangwxd]

What do you mean of video stream. Would that be video files (ffmpegcv.VideoCapture), video cameras (ffmpegcv.VideoCaptureCAM) or IP camera stream (ffmpegcv.VideoCaptureStream).

Currently, the VideoCaptureStream didn't support codec as input, you can manually specify that is ffmpeg_cmd. Tips. You can use noblock to detach the multiple video processing task among cpu-multi-cores, and relieve the cpu usage in python. The noblock support ffmpegcv.VideoCapture(/NV) and VideoWriter(/NV) only.

I mean rtsp video stream

[ywangwxd]

For rtsp vdieo stream, I found only VideoCaptureStream can handle it, otherwise I need to give a file as parameter. If I do not do any config in the VideoCaptureStream object, its default will look for nvidia device, which will fail. My ffmpeg is on rk3588, without nvidia device. I removed all the nvidia stuff in the source code and retried. Then it showed error message:

mpp[1021876]: mpp_info: mpp version: ed377c99 author: Herman Chen   2023-12-14 fix[hal_enc]: Add encoder internal buffer sync
Traceback (most recent call last):
  File "/mnt/data/ffmpegcv/ffmpegcv-main/test.py", line 13, in <module>
    vidout.write(frame)
  File "/mnt/data/ffmpegcv/ffmpegcv-main/ffmpegcv/ffmpeg_writer.py", line 70, in write
    self.process.stdin.write(img)
BrokenPipeError: [Errno 32] Broken pipe

Here is the changed ffmpegcv_writer.py file as txt.

ffmpeg_writer_txt.txt

[ywangwxd]

the test program is simple:

import ffmpegcv

vidin = ffmpegcv.VideoCaptureStream("rtsp url xxxxx")

vfile_out = "ffmpegcv_test_out.h264"
vidout = ffmpegcv.VideoWriterNV(vfile_out, 'h264', vidin.fps)

with vidin, vidout:

    i=0
    for frame in vidin:
        #cv2.imshow('image', frame)
        vidout.write(frame)
        i=i+1
        if(i>1000):
           break

[chenxinfeng4]

replace the VideoWriterNV with VideoWriter

import ffmpegcv

vidin = ffmpegcv.VideoCapture("rtsp url xxxxx")

vfile_out = "ffmpegcv_test_out.h264"
vidout = ffmpegcv.VideoWriter(vfile_out, 'h264', vidin.fps)

with vidin, vidout:

    i=0
    for frame in vidin:
        #cv2.imshow('image', frame)
        vidout.write(frame)
        i=i+1
        if(i>1000):
           break

[ywangwxd]

replace the VideoWriterNV with VideoWriter

import ffmpegcv

vidin = ffmpegcv.VideoCapture("rtsp url xxxxx")

vfile_out = "ffmpegcv_test_out.h264"
vidout = ffmpegcv.VideoWriter(vfile_out, 'h264', vidin.fps)

with vidin, vidout:

    i=0
    for frame in vidin:
        #cv2.imshow('image', frame)
        vidout.write(frame)
        i=i+1
        if(i>1000):
           break

Sorry, I have not noticed I used VideoWriterNV actually. After change it to VideoWriter, it is running.

[ywangwxd]

Strang thing happened, but it is not related to the ffmpegcv lib. I picked out the ffmpeg command and execute it on bash. e.g,

ffmpeg -loglevel info -i rtsp:////Streaming/Channels/101 -pix_fmt bgr24 -f rawvideo pipe: > /dev/null

The string thing is, if I did not specify -codec hevc option, it will use mpp decoder automatically, as shown in the debug output:

[hevc_rkmpp_decoder @ 0x5557dda030] Picture format is nv12.
Stream mapping:
  Stream #0:0 -> #0:0 (hevc (hevc_rkmpp_decoder) -> rawvideo (native))

but the cpu load is 500%!, and it often complained delay.

If I specify -codec hevc, it did not show using mpp decoder, looks like it used cpu decoder, but the cpu load is only 300%.

[chenxinfeng4]

I update the VideoCaptureStream, now you can specify the decoder codec in args. Can you test it and give me some feedbacks. I'll release in days.

pip install -U git+https://github.com/chenxinfeng4/ffmpegcv.git

def VideoCaptureStream(
    stream_url,
    codec=None,
    pix_fmt="bgr24",
    crop_xywh=None,
    resize=None,
    resize_keepratio=True,
    resize_keepratioalign="center"
):

[ywangwxd]

Strang thing happened, but it is not related to the ffmpegcv lib. I picked out the ffmpeg command and execute it on bash. e.g,

ffmpeg -loglevel info -i rtsp:////Streaming/Channels/101 -pix_fmt bgr24 -f rawvideo pipe: > /dev/null

The string thing is, if I did not specify -codec hevc option, it will use mpp decoder automatically, as shown in the debug output:

[hevc_rkmpp_decoder @ 0x5557dda030] Picture format is nv12.
Stream mapping:
  Stream #0:0 -> #0:0 (hevc (hevc_rkmpp_decoder) -> rawvideo (native))

but the cpu load is 500%!, and it often complained delay.

If I specify -codec hevc, it did not show using mpp decoder, looks like it used cpu decoder, but the cpu load is only 300%.

by the way, if you are wondering if my ffmpeg is alright. I confirmed it was. If I transocded a rtsp video stream and saved to a video file, everything is good. CPU load is very low, speed is ok, no complain message shown up.

[chenxinfeng4]

In my experiment, the GPU many not work better than CPU. Sound strange, but it's truth. I've tested in Intel QSV gpu and NVIDIA gpu, them seems not accelerate much.

The ffmegcv GPU reader is a bit slower than CPU reader, but much faster when use ROI operations (crop, resize, pad).

One reason may be intensive data copy CPU==>frame-GPU==>frame-CPU. You waste time copy data when using GPU.

The second, the pix_fmt is wasting time of CPU, video====(decoder)====>YUV420p ===(CPU pix convert)====>BRG24. If you try pix_fmt="yuv420p" or "gray", you will get faster and small cpu usage. The pix convert is a huge burden, but cannot be done in GPU by ffmpeg.

[chenxinfeng4]

If you just want to save the stream to file, I strongly recommand you to use yuv420p as pix_fmt. Sadly, the VideoWriter didn't support yuv420p yet.

[chenxinfeng4]

You're picking out the ffmpeg command and testing. I wish you can low down the CPU load in naive ffmpeg command, and can share the code if you succeed.

but the cpu load is 500%!, and it often complained delay. If I specify -codec hevc, it did not show using mpp decoder, looks like it used cpu decoder, but the cpu load is only 300%.

[ywangwxd]

ffmpeg -loglevel info -i rtsp:////Streaming/Channels/101 -pix_fmt bgr24 -f rawvideo pipe: > /dev/null

Your guess is right. It is the problem of pix fmt. But the lowest cpu load is not using yuv420p, it is nv12. Using nv12, cpu load is 58%, using yuv420p cpu load is 220%. But the input video pix fmt is yuv420p, so here I still have questions.

[chenxinfeng4]

I'm not an expert in video coding neither. I can not tell the difference between nv12 yuv420p and yuvj420p. I use the gray and the rgb24 most.

[chenxinfeng4]

The IP cam stream is often delay, if you want the low latency, you can try the experimental ffmpeg.VideoCaptureStreamRT for realtime (low delay) reading.

from ffmpegcv.ffmpeg_noblock import ReadLiveLast, noblock
vid = ffmpegcv.VideoCaptureStreamRT(streamurl)   #will strange if main thread fail behind stream
vid = noblock(ffmpegcv.VideoCaptureStreamRT, streamurl) #multiprocessing, buffered frames
vid = ReadLiveLast(ffmpegcv.VideoCaptureStreamRT, streamurl) #multiprocessing, always pick last frame

with vid:
    for i in range(100):
        _, _ = vid.read()  #give off begin frames, aways noisy
    while True:
        ret, image = vid.read()

[ywangwxd]

The IP cam stream is often delay, if you want the low latency, you can try the experimental ffmpeg.VideoCaptureStreamRT for realtime (low delay) reading.

from ffmpegcv.ffmpeg_noblock import ReadLiveLast, noblock
vid = ffmpegcv.VideoCaptureStreamRT(streamurl)   #will strange if main thread fail behind stream
vid = noblock(ffmpegcv.VideoCaptureStreamRT, streamurl) #multiprocessing, buffered frames
vid = ReadLiveLast(ffmpegcv.VideoCaptureStreamRT, streamurl) #multiprocessing, always pick last frame

with vid:
    for i in range(100):
        _, _ = vid.read()  #give off begin frames, aways noisy
    while True:
        ret, image = vid.read()

The message about delay is not because the stream is delay, it is because the decoder is slower than the packet arriving speed.

Taking the scene of transcoding a rtsp stream as an example. Using ffmpeg transcoding single command is fast and only one cpu used, and its load is as low as 30%.

Use ffmpegcv, you get two cpu busy for two ffmpeg process (each 50%-80% load) and another python process (cpu load about 70%), so overall speaking, use pipe is not effienct, of course it is easy for coding.

[chenxinfeng4]

Did you find the way out?

[ywangwxd]

Did you find the way out?

I have stopped digging in this issue, since pipe mode is not what I want. Thank you anyway.

[ywangwxd]

I am looking for a well maintained c++ wrapper of ffmpeg so that I can develop down stream application easier. For real application, we still need c,c++.

[chenxinfeng4]

Great. I hope you can use the c++ to wrap the ffmpeg, and share with me then. I hope you can use cython to wrap your wrapper into python. We can discuss latter if you get progress.