OMX is not sending frames

[gamelaster]

SoC: S3

I have verified that camera works good.

# ffmpeg -s 640x480 -r 15 -i /dev/video0 -vcodec h264 out5.mp4

ffmpeg version 4.3.1 Copyright (c) 2000-2020 the FFmpeg developers
  built with gcc 9.3.0 (Buildroot 2020.08-951-g20ddd63292-dirty)
  configuration: --enable-cross-compile --cross-prefix=/home/gamiee/Desktop/buildroot/output/host/bin/arm-buildroot-linux-gnueabihf- --sysroot=/home/gamiee/Desktop/buildroot/output/host/arm-buildroot-linux-gnueabihf/sysroot --host-cc=/usr/bin/gcc --arch=arm --target-os=linux --disable-stripping --pkg-config=/home/gamiee/Desktop/buildroot/output/host/bin/pkg-config --disable-static --enable-shared --prefix=/usr --enable-avfilter --disable-version3 --enable-logging --enable-optimizations --disable-extra-warnings --enable-avdevice --enable-avcodec --enable-avformat --enable-network --disable-gray --enable-swscale-alpha --disable-small --enable-dct --enable-fft --enable-mdct --enable-rdft --disable-crystalhd --disable-dxva2 --enable-runtime-cpudetect --disable-hardcoded-tables --disable-mipsdsp --disable-mipsdspr2 --disable-msa --enable-hwaccels --disable-cuda --disable-cuvid --disable-nvenc --disable-avisynth --disable-frei0r --disable-libopencore-amrnb --disable-libopencore-amrwb --disable-libdc1394 --disable-libgsm --disable-libilbc --disable-libvo-amrwbenc --disable-symver --disable-doc --enable-gpl --enable-nonfree --enable-ffmpeg --disable-ffplay --disable-avresample --disable-ffprobe --disable-libxcb --disable-postproc --disable-swscale --enable-indevs --disable-alsa --enable-outdevs --enable-pthreads --disable-zlib --disable-bzlib --disable-libfdk-aac --disable-libcdio --disable-gnutls --disable-openssl --disable-libdrm --disable-libopenh264 --disable-vaapi --disable-vdpau --disable-mmal --disable-omx --disable-omx-rpi --disable-libopencv --disable-libopus --disable-libvpx --disable-libass --disable-libbluray --disable-libmfx --disable-librtmp --disable-libmp3lame --disable-libmodplug --disable-libspeex --disable-libtheora --disable-libwavpack --disable-iconv --disable-libfreetype --disable-fontconfig --disable-libopenjpeg --disable-libx264 --disable-libx265 --disable-libdav1d --disable-x86asm --disable-mmx --disable-sse --disable-sse2 --disable-sse3 --disable-ssse3 --disable-sse4 --disable-sse42 --disable-avx --disable-avx2 --enable-armv6 --enable-vfp --enable-neon --disable-altivec --extra-libs=-latomic --enable-pic --cpu=cortex-a7 --enable-omx
  libavutil      56. 51.100 / 56. 51.100
  libavcodec     58. 91.100 / 58. 91.100
  libavformat    58. 45.100 / 58. 45.100
  libavdevice    58. 10.100 / 58. 10.100
  libavfilter     7. 85.100 /  7. 85.100
  libswresample   3.  7.100 /  3.  7.100
[video4linux2,v4l2 @ 0x5b380] ioctl(VIDIOC_G_PARM): Inappropriate ioctl for device
[video4linux2,v4l2 @ 0x5b380] Time per frame unknown
Input #0, video4linux2,v4l2, from '/dev/video0':
  Duration: N/A, start: 117.711179, bitrate: N/A
    Stream #0:0: Video: rawvideo (I420 / 0x30323449), yuv420p, 640x480, 15 tbr, 1000k tbn, 1000k tbc
Stream mapping:
  Stream #0:0 -> #0:0 (rawvideo (native) -> h264 (h264_omx))
Press [q] to stop, [?] for help
[h264_omx @ 0x5f2e0] libOMX_Core.so not found
 Leaving OMX_GetComponentsOfRole
[h264_omx @ 0x5f2e0] Using OMX.allwinner.video.encoder.avc
debug  : omx_venc <__AwOmxVencComponentCreate:2501>: COMPONENT_CREATE
debug  : cedarc <AwOmxComponentInit:26>:OMXCORE: aw_omx_component_init 67098

debug  : omx_venc <__AwOmxVencInit:392>: COMPONENT_INIT
debug  : ionAlloc <__GetIonMemOpsS:985>:*** get __GetIonMemOpsS ***
debug  : ionAlloc <ion_alloc_open:134>:begin ion_alloc_open

debug  : cedarc <VeSetSpeed:1559>: *** set ve freq to 360 Mhz ***
debug  : cedarc <VeInitialize:1198>: ve init ok

debug  : ionAlloc <ion_alloc_open:175>:** phy offset = 40000000
debug  : cedarc <VeRelease:1253>: ve release ok

debug  : cedarc <CdcMessageQueueCreate:47>:nMessageSize = 20
debug  : cedarc <CdcMessageQueueCreate:47>:nMessageSize = 20
debug  : cedarc <updateOmxDebugFlag:238>:DEBUG_TYPE is (null).
debug  : cedarc <AwOmxComponentSetCallbacks:309>:OMXCORE: aw_omx_component_set_callbacks 67098, b6233a2c , 5f690

debug  : omx_venc <__AwOmxVencSetCallbacks:2317>:===== vdec set callbacks***************
debug  : omx_venc <__AwOmxVencGetParameter:953>:InPort definition: width = 0, height = 0
debug  : omx_venc <__AwOmxVencGetParameter:965>:OutPort definition: width = 176, height = 144
debug  : omx_venc <__AwOmxVencSetParameter:1335>:init_input_port: stride = 640, width = 640, height = 480
debug  : omx_venc <__AwOmxVencSetParameter:1341>:set video port format 0x13
debug  : omx_venc <__AwOmxVencGetParameter:953>:InPort definition: width = 640, height = 480
debug  : omx_venc <__AwOmxVencGetParameter:965>:OutPort definition: width = 176, height = 144
debug  : omx_venc <__AwOmxVencSetParameter:1405>:init_output_port: framerate: 983040(>>16) 15, output width 640 height 480.
debug  : omx_venc <__AwOmxVencGetParameter:965>:OutPort definition: width = 640, height = 480
debug  : omx_venc <__AwOmxVencAllocateBuffer:1963>:allocate buffer, nPortIndex: InPort, nSizeBytes: 460800
debug  : omx_venc <__AwOmxVencAllocateBuffer:1963>:allocate buffer, nPortIndex: InPort, nSizeBytes: 460800
debug  : omx_venc <__AwOmxVencAllocateBuffer:1963>:allocate buffer, nPortIndex: OutPort, nSizeBytes: 460800
debug  : omx_venc <__AwOmxVencAllocateBuffer:1963>:allocate buffer, nPortIndex: OutPort, nSizeBytes: 460800
debug  : omx_venc <__AwOmxVencAllocateBuffer:1963>:allocate buffer, nPortIndex: OutPort, nSizeBytes: 460800
debug  : omx_venc <__AwOmxVencAllocateBuffer:1963>:allocate buffer, nPortIndex: OutPort, nSizeBytes: 460800
debug  : omx_venc <processThreadCommand:3317>:x set state command, cmd = main_thread_cmd_set_state, pCmdData = OMX_StateIdle.
debug  : omx_venc <processThreadCommand:3317>:x set state command, cmd = main_thread_cmd_set_state, pCmdData = OMX_StateExecuting.
debug  : omx_venc <ComponentVencThread:4316>:(vdrvThread receive cmd[venc_cmd_open]
debug  : cedarc <LogVersionInfo:40>:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Cedar Codec <<<<<<<<<<<<<<<<<<<<<<<<<<<<
tag   : createBranch_customize_sylix
branch: CedarC-v1.1.9
commit: 4a182bd42c931bea2f77b0ff9a531d233df8541e
date  : Thu May 25 15:34:03 2017 +0800
author: jenkins8080
patch :
----------------------------------------------------------------------

debug  : cedarc <VeSetSpeed:1559>: *** set ve freq to 360 Mhz ***
debug  : cedarc <VeInitialize:1198>: ve init ok

debug  : ionAlloc <__GetIonMemOpsS:985>:*** get __GetIonMemOpsS ***
debug  : ionAlloc <ion_alloc_open:134>:begin ion_alloc_open

debug  : omx_venc <init_h264_param:2820>:profile-venc=66, profile-omx=1, frame_rate:983040, bit_rate:200000, idr:12, eColorFormat:00000013

debug  : omx_venc <init_h264_param:2946>:Enable VBR. QpMin: 20. QpMax: 45. Bitrate: 200000.
debug  : cedarc <determineVencColorFormat:167>:color format: VENC_PIXEL_YUV420P/I420
debug  : omx_venc <openVencDriver:3184>:omx_venc base_config info: src_wxh:640x480, dis_wxh:640x480, stride:640

debug  : ionAlloc <__GetIonMemOpsS:985>:*** get __GetIonMemOpsS ***
debug  : omx_venc <openVencDriver:3221>:AVC 4 byte interleave nalu Disable.
debug  : cedarc <VideoEncInit:195>: (f:VideoEncInit, l:195)
debug  : cedarc <VideoEncInit:210>: (f:VideoEncInit, l:210)
debug  : cedarc <BitStreamCreate:102>: BitStreamCreate OK

[petit-miner]

Yeah, I also have the same issue with an USB Webcam as Input. If you use ffmpegs builtin Test Video the Video engine produces encoded H.264 just fine. You can try for yourself: ffmpeg -f lavfi -i testsrc=duration=2:size=1280x720:rate=30 testsrc.h264 -y

[niyazFattahov]

Yeah, I also have the same issue with an USB Webcam as Input. If you use ffmpegs builtin Test Video the Video engine produces encoded H.264 just fine. You can try for yourself: ffmpeg -f lavfi -i testsrc=duration=2:size=1280x720:rate=30 testsrc.h264 -y

This works for me: ffmpeg -f v4l2 -r 30 -t 10 -s 640x480 -i /dev/video0 -pix_fmt yuv420p -c:v h264_omx output.avi -y

[gamelaster]

Hmm, I already abandoned idea of using their OMX driver, since it didn't worked both on FFMPEG but also GStreamer. @niyazFattahov I will probably try it again with your ffmpeg configuration.

[niyazFattahov]

@petit-miner what board you use with s3 chip? I also wanna try it for video encoding tests. I cannot find any board with this chip

[gamelaster]

@niyazFattahov PineCube have S3, and I'm testing everything on it.

[petit-miner]

@petit-miner what board you use with s3 chip? I also wanna try it for video encoding tests. I cannot find any board with this chip

I'm testing the Video encoder on a custom V3s board.

[niyazFattahov]

Hmm, I already abandoned idea of using their OMX driver, since it didn't worked both on FFMPEG but also GStreamer. @niyazFattahov I will probably try it again with your ffmpeg configuration.

do you try to use cedrus codec with v4l2_m2m ?

[petit-miner]

Yeah, I also have the same issue with an USB Webcam as Input. If you use ffmpegs builtin Test Video the Video engine produces encoded H.264 just fine. You can try for yourself: ffmpeg -f lavfi -i testsrc=duration=2:size=1280x720:rate=30 testsrc.h264 -y

This works for me: ffmpeg -f v4l2 -r 30 -t 10 -s 640x480 -i /dev/video0 -pix_fmt yuv420p -c:v h264_omx output.avi -y

Yeah, this works for me too, but I'm not very happy about the performance. I can encode a rawvideo with 720p however not in realtime. I get like ~8 FPs I tried baseline / medium / High but no difference in speed. The produced H264 files look fine and sharp and without any glitches. H264_omx hasn't much options to tweak: Encoder h264_omx [OpenMAX IL H.264 video encoder]: General capabilities: delay Threading capabilities: none Supported pixel formats: yuv420p h264_omx AVOptions: -omx_libname ED.V.... OpenMAX library name -omx_libprefix ED.V.... OpenMAX library prefix -zerocopy E..V.... Try to avoid copying input frames if possible (from 0 to 1) (default 0) -profile E..V.... Set the encoding profile (from -99 to 100) (default -99) baseline E..V.... main E..V.... high E..V.... Does anyone get any better performance?

[gamelaster]

@petit-miner there might be some problem in OMX driver. So far, when using raw CedarC library, the encoding is quite fast, but I seen some weird latencies too when testing. I need to finish my gstreamer plugin for libcedarc and that will show if it's capable of 720p live stream

[petit-miner]

okay, good to know. Where can I find this raw CedarC library? And how can I build it? I just need to capture 720p @ 30fps and encode it to H.264.

[gamelaster]

@petit-miner it's this repository 😊 OMX library is just additional wrapper for OMX support, which uses CedarC API

[petit-miner]

@petit-miner it's this repository 😊 OMX library is just additional wrapper for OMX support, which uses CedarC API

I know this is maybe a bit much to ask, but could you help me to compile the encoderdemo? I used these commands to compile: ./bootstrap ./configure --host=arm-linux-gnueabihf --enable-static make ARCH=arm Making all in memory make[2]: Entering directory '/home/mthuerm/libcedarc/memory' CC libMemAdapter_la-memoryAdapter.lo CC libMemAdapter_la-ionAlloc.lo CCLD libMemAdapter.la /usr/bin/ld: cannot find -lVE collect2: error: ld returned 1 exit status make[2]: [Makefile:441: libMemAdapter.la] Error 1 make[2]: Leaving directory '/home/mthuerm/libcedarc/memory' make[1]: [Makefile:398: all-recursive] Error 1 make[1]: Leaving directory '/home/mthuerm/libcedarc' make: *** [Makefile:330: all] Error 2

Furthermore it seems like the Makefile in /demo/vencoderDemo/ doesn't get updated to my build environment. It would be very nice if you could help me.

[gamelaster]

@petit-miner Do you have libVE.so in /usr/local/lib or /usr/lib directory?

[petit-miner]

Yeah I got that fixed now. If I use the encoderdemo in this repo the encoded video is super fast. I use a 17 second raw yuv420p video and convert it to H264 / MJPEG and I only get a video which is only a handful of frames long. Colours look fine in those few frames. The same happens with ffmpeg: (I also found that the bottleneck with ffmpeg was the generation of the testsrc video and not the encoder) ffmpeg -f lavfi -i testsrc=duration=10:size=1280x720:rate=30 -c:v rawvideo -pix_fmt yuv420p test.yuv and then pass it trough the encoder with ffmpeg: ffmpeg -re -f rawvideo -pix_fmt yuv420p -s:v 1280x720 -r 30 -i test.yuv output.h264

The video also seems speed up like crazy and is much shorter as my raw yuv420p testfile. I also verified that the testfile looks ok. I tried adding -re to ffmpeg to slow it down somewhat but it didn't help at all. Funny enough when we overloaded the CPU with generating the testsrc video directly, the encoded video file looked fine and I didn't have such issues. Maybe it has something to do with the fact that the encoder only received 14 frames per second when the CPU was bottlenecked? Do you have an idea?

[gamelaster]

@petit-miner about using encoding test, when you play raw .h264, there isn't information about frame rate, thus VLC / any media player picks how fast it will playback. About other things, I need to test it, but sadly, many things on my task list right now, but eventually I will get on it and will let you know about results, but mostly, the performance and quality was good.

[Unturned3]

@petit-miner I tried your method of generating a test vid first, then running the encoder. However, I'm still only getting about ~8 FPS. Is ffmpeg -re -f rawvideo -pix_fmt yuv420p -s:v 1280x720 -r 30 -i test.yuv output.h264 the correct command for invoking the encoder? Did you manage to get any good performances?

By the way, I'm only getting about 2.4 FPS when running the test vid generation command. This doesn't seem too normal, since I think you said that you were able to achieve ~8 FPS when generating the test vid AND encoding at the same time?

Many thanks.

[Unturned3]

By the way, how does the h264 encoding image quality look for you all? I tried to encode a 640x480 stream from a camera (ov5640) on the V3s, and the image quality appears to be quite poor, even with -profile 100 in the ffmpeg commands. Does this happen for anyone else?

This is the command I used ffmpeg -f v4l2 -r 30 -t 10 -s 640x480 -i /dev/video0 -pix_fmt yuv420p -c:v h264_omx output.avi -y, same as the one used by @niyazFattahov

Here's an example of the blurry images that I get in the encoded video:

[jlee167]

@Unturned3 Same here. Even simple testsrc gets messed up after encoding

[cameronelliott]

By the way, how does the h264 encoding image quality look for you all? I tried to encode a 640x480 stream from a camera (ov5640) on the V3s, and the image quality appears to be quite poor, even with -profile 100 in the ffmpeg commands. Does this happen for anyone else?

@Unturned3 Did you ever resolve the quality issue? Is this an issue in the H264 encoding core in the V3s, or maybe just the way you are using it? I'd like to try using the V3s for an IP camera, but it doesn't make sense if the encoding core produces very low quality encoding. Thanks.

[Unturned3]

@Unturned3 Did you ever resolve the quality issue? Is this an issue in the H264 encoding core in the V3s, or maybe just the way you are using it? I'd like to try using the V3s for an IP camera, but it doesn't make sense if the encoding core produces very low quality encoding. Thanks.

This was quite a while ago, but I did resolve the issue by manually editing the Qp (or something like that) parameter in the OMX source code & recompiling the whole thing. I've since abandoned using OMX + ffmpeg; it's just too buggy and limited.

Directly using V4L2 and the Cedar API gets me very good results.

See here: https://github.com/Unturned3/h264enc_demo

[tunerok]

By the way, how does the h264 encoding image quality look for you all? I tried to encode a 640x480 stream from a camera (ov5640) on the V3s, and the image quality appears to be quite poor, even with -profile 100 in the ffmpeg commands. Does this happen for anyone else?

@Unturned3 Did you ever resolve the quality issue? Is this an issue in the H264 encoding core in the V3s, or maybe just the way you are using it? I'd like to try using the V3s for an IP camera, but it doesn't make sense if the encoding core produces very low quality encoding. Thanks.

I had almost the same question. I needed to compress the video on the chip, and not get it from the camera. The quality of the output video was constantly disgusting

Added an encoder override in the omx_venc.c file

diff:

To get an mp4 container and .h264 encoding, you need to take the following two steps. First, we go through the encoder while saving the container. Then I change the container using ffmpeg without recoding the video.

ffmpeg -i test2.avi -c:v h264_omx outpui.avi -y
ffmpeg -i outpui.avi -c:v copy -c:a copy outpui.mp4

Added code mostly from https://github.com/Unturned3/h264enc_demo/blob/master/h264.c

An example:

Input video - mjpeg AVI 16mb 1024x656, 4838 kb/s, 10 fps Output video - h264 AVI 5mb 1024x656, 1913 kb/s, 10 fps

Next step - instant-time video conversion:

And done

[tunerok]

By the way, how does the h264 encoding image quality look for you all? I tried to encode a 640x480 stream from a camera (ov5640) on the V3s, and the image quality appears to be quite poor, even with -profile 100 in the ffmpeg commands. Does this happen for anyone else?

@Unturned3 Did you ever resolve the quality issue? Is this an issue in the H264 encoding core in the V3s, or maybe just the way you are using it? I'd like to try using the V3s for an IP camera, but it doesn't make sense if the encoding core produces very low quality encoding. Thanks.

Here fix for baudrate (default is 200kbs and less)

https://github.com/Unturned3/v3s3/issues/2#issuecomment-1319895068

After that you can specify br. I'm still searching about sending qp-parameters via ffmpeg to encoder

ffmpeg -i test2.avi -c:v h264_omx -b:v 4096k outpui_high.avi -y