search

motioneye-project / motioneyeos

A Video Surveillance OS For Single-board Computers
Other
7.9k stars 904 forks source link

ffmpeg 3.1 supports h264 hardware assist encoding #365

Closed caguas closed 7 years ago

caguas commented 8 years ago

to compile ffmpeg with hardware assist h264 encoding: ./configure --enable-mmal --enable-omx-rpi --enable-omx

if you have an RPI3 , you can use multiple cores to speed up compilation: make -j4

to test: ./ffmpeg -f v4l2 -input_format yuv420p -framerate 25 -video_size 640x480 -i /dev/video0 -frames 500 -an -c:v h264_omx test.mp4

ffmpeg will generate warnings just ignore them for now.

The omx h264 hardware assist encoding was ported from libav by Aman Gupta.

Amancio

ccrisan commented 8 years ago

And would that hardware assisted encoder be automatically used by motion when creating movies?

ccrisan commented 8 years ago

I managed to compile ffmpeg 3.1 with the options you have suggested and when specifying "mp4" as ffmpeg_video_codec to Mr Dave's motion version, it creates h264-encoded movies. However the movies have wrong frame rates (10k or higher!) and are therefore unplayable.

caguas commented 8 years ago

Hi,

Can you run this command with ffmpeg-3.1: ./ffmpeg -f v4l2 -input_format yuv420p -framerate 25 -video_size 1024x768 -i /dev/video0 -frames 500 -an -c:v h264_omx test.mp4

On July 3, 2016 at 9:04:53 AM, Calin Crisan (notifications@github.com) wrote:

I managed to compile ffmpeg 3.1 with the options you have suggested and when specifying "mp4" as ffmpeg_video_codec to Mr Dave's motion version, it creates h264-encoded movies. However the movies have wrong frame rates (10k or higher!) and are therefore unplayable.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/ccrisan/motioneyeos/issues/365#issuecomment-230160692, or mute the thread https://github.com/notifications/unsubscribe/AG9y6aVydHf_cgjhvND_l_U4pBO2IaL1ks5qR92fgaJpZM4I_BpU .

ccrisan commented 8 years ago

Which test.mp4? Can you give me a link where I can download the file from?

caguas commented 8 years ago

The command generates test.mp4

On July 3, 2016 at 9:15:40 AM, Calin Crisan (notifications@github.com) wrote:

Which test.mp4? Can you give me a link where I can download the file from?

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/ccrisan/motioneyeos/issues/365#issuecomment-230161228, or mute the thread https://github.com/notifications/unsubscribe/AG9y6Qby_oXKYADAdEEPUCXJUAzEO8UFks5qR-AsgaJpZM4I_BpU .

ccrisan commented 8 years ago

My apologies, I didn't read your command like carefully enough. Yes, it works and it creates a file with the correct frame rate. However when invoked from motion (through libavcodec I suppose) the movie has a wrong frame rate set.

caguas commented 8 years ago

Fine, I will let you play with motion/ffmpeg-3.1 for a while.

In the mean time, please post a pointer to the version of motion that you are using.

Last but not least there is now a ffmpeg-3.1.1. The guys worked on ffmpeg a little bit more.

Enjoy! Amancio

On July 3, 2016 at 9:41:22 AM, Calin Crisan (notifications@github.com) wrote:

My apologies, I didn't read your command like carefully enough. Yes, it works and it creates a file with the correct frame rate. However when invoked from motion (through libavcodec I suppose) the movie has a wrong frame rate set.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/ccrisan/motioneyeos/issues/365#issuecomment-230162456, or mute the thread https://github.com/notifications/unsubscribe/AG9y6UXvltcScAB0-X8OQntTowGzUEkVks5qR-YygaJpZM4I_BpU .

ccrisan commented 8 years ago

Actually I've been playing with ffmpeg-3.1.1. The motion version I'm using is the latest commit of Mr Dave's fork.

ccrisan commented 8 years ago

It turns out the framerate problem has nothing to do with enabling h264 hardware accelerated encoding. I disabled it and went the software libx264 way. The framerate problem is still there. However with older ffmpeg versions (2.8.x) I don't seem to be having this problem.

caguas commented 8 years ago

Cool, just report it to the ffmpeg bug tracker:

http://ffmpeg.org/bugreports.html

Cheers Amancio

On July 3, 2016 at 12:01:53 PM, Calin Crisan (notifications@github.com) wrote:

It turns out the framerate problem has nothing to do with enabling h264 hardware accelerated encoding. I disabled it and went the software libx264 way. The framerate problem is still there. However with older ffmpeg versions (2.8.x) I don't seem to be having this problem.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/ccrisan/motioneyeos/issues/365#issuecomment-230169435, or mute the thread https://github.com/notifications/unsubscribe/AG9y6Y3uF70JvWLsjhYmEF2H1MXO3ZPKks5qSAchgaJpZM4I_BpU .

caguas commented 8 years ago

Hi

Not sure if you solved the problem or not. If motion does not work , it is most likely due to an api change in ffmpeg which can be tracked by looking at ffmpeg-3.1.1/ffmpeg.c and ffmpeg-3.1.1/libvcodec/omx.c . The last one is the one that does the hardware assist encoding for the RPI. Perhaps the owner of the motion release can address the issue. Again I have no problem with hardware assist encoding using the “./ffmpeg” command.

If you are using an old version of ffmpeg it should not be too difficult to back port omx.c to to it . It should involved minor changes to configure, libavcodec/Makefile , libavcodec/allcodecs.c and adding libavcodec/omx.c.

Those minor changes can be easily discerned by looking at the current source release of ffmpeg.

Last but not least a static binary release of ffmpeg-3.1.1/ffmpeg can be used to do hardware assist for motion by invoking the command at run time. This option is kind of attractive because if ffmpeg’s api changes supposedly the ffmpeg group will incorporate the change into the command ‘ffmpeg’.

Amancio

On Jul 3, 2016, at 12:01 PM, Calin Crisan notifications@github.com wrote:

It turns out the framerate problem has nothing to do with enabling h264 hardware accelerated encoding. I disabled it and went the software libx264 way. The framerate problem is still there. However with older ffmpeg versions (2.8.x) I don't seem to be having this problem.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/ccrisan/motioneyeos/issues/365#issuecomment-230169435, or mute the thread https://github.com/notifications/unsubscribe/AG9y6Y3uF70JvWLsjhYmEF2H1MXO3ZPKks5qSAchgaJpZM4I_BpU.

jasaw commented 7 years ago

@ccrisan Have you played around with motion + ffmpeg 3.x combination? What is the status? I'm running motioneyeos latest release and running "ffmpeg -hwaccels" gives an empty list. Does that mean hardware accel is not currently supported? From the ffmpeg configuration, looks like you have --enable-hwaccels. Have you tried replacing "--enable-hwaccels" with "--enable-omx --enable-omx-rpi"?

ccrisan commented 7 years ago

@jasaw I'll look into it.

jasaw commented 7 years ago

Looks like to support hwaccel on the Pi, we need to add "--enable-omx --enable-omx-rpi --enable-mmal" to ffmpeg config via the buildroot xconfig. Also need to enable "bellagio" package, because ffmpeg omx depends on libomxil-bellagio.so. Buildroot at this point will complain about both rpi-userland and bellagio both providing libopenmax. I just removed libopenmax from rpi-userland Config.in and mk file just to proceed with buildroot compilation. I haven't got time to test the generated image yet.

cedricve commented 7 years ago

hey @jasaw @ccrisan, tried this before with @kerberos-io, everything compiled properly but when executing the ffmpeg command it was complaining about a couple of .so missing. What I did is moved them from /usr/bin to /opt/vc, this made the complaining going away but caused ffmpeg to crash when selecting the h264_omx encoder.

At the moment we are implementing OpenMAX directly into Kerberos.io/machiner, and we are able to stream MJPEG at 25FPS at 1280x720 and record at 30 FPS 1280x720, and 90 FPS at 640x480. Maybe integrating OpenMax out-of-the-box can also be a solution for motion.

ccrisan commented 7 years ago

I have managed to build ffmpeg with --enable-omx --enable-omx-rpi --enable-mmal --extra-cflags=-I../../staging/usr/include/IL/ and it does not crash. At compile time I get:

Enabled hwaccels:
h264_mmal    mpeg2_mmal    mpeg4_mmal    vc1_mmal

However, when I run ffmpeg -hwaccels on the PI, I get an empty list. I did not use bellagio but pointed the configure script instead to the headers provided by rpi-userland.

Any ideas?

cedricve commented 7 years ago

@ccrisan I didn't mean that it crashed at compiling. It did crash if I used the h264_omx codec on command line. What happens if you write ffmpeg --codecs.

jasaw commented 7 years ago

I managed to get ffmpeg to encode with hwaccel in isolation. This is very much a hack, but here's what I've done:

Next thing is to get motion to use hwaccel ffmpeg.

ccrisan commented 7 years ago

@cedricve my ffmpeg -codecs with the mentioned configuration: https://pastebin.com/xuwhAuD3

ccrisan commented 7 years ago

For the record, I ran both versions of motionEyeOS in parallel (on PI1 boards) and did not see any significant difference between the two. I set them up to encode h264 movies continuously via our good old motion. Either my hardware-accelerated ffmpeg isn't hardware accelerated at all or motion does not use it. I really don't know too much about this hardware accelerated stuff but anyways, these are my findings.

cedricve commented 7 years ago

Pretty sure motion doesn't use it, you should see the difference definitely (either in frame rate and CPU usage).

jasaw commented 7 years ago

Alright... I managed to get motion to record movie using ffmpeg with h264_omx encoder. I got 18 fps, whereas software encoder gave me 2 fps, but the h264_omx bitrate is not great because I haven't figured out how to get motion to set bitrate via ffmpeg's API.

Anyway, this is the interesting bit in my motion.log:

[0:ILCS_HOST] [INF] [ENC] ffmpeg_avcodec_log: OMX state changed to 2 [0:ILCS_HOST] [INF] [ENC] ffmpeg_avcodec_log: OMX state changed to 1

Here's what I've done. Again, this is all hacked up to prove that it can be done.

The proper way of supporting this is to get motion to specify which encoder to use when setting up a new movie. Motion current tells ffmpeg the output file extension is "mp4", then ffmpeg guesses which encoder to use. I don't know ffmpeg's API well enough to specify the desired encoder and bitrate. If I have some free time, I'll look into it. Maybe someone more familiar can point us to the right direction?

jasaw commented 7 years ago

Did another quick hack to improve the h264 video quality by setting the CRF (Constant Rate Factor). At best CRF quality, 800 x 600 resolution, I'm doing 25 fps 40% CPU load on a Pi Zero W. Woohoo ! This will do for me.

The CRF hack that I did, if anyone is interested. motion_h264_crf_best_quality.patch.txt

cedricve commented 7 years ago

@jasaw amazing! I'll try to integrate this within Kerbersos.io as well for USB and IP cameras. For Raspicamera's we'll be using OpenMax directly though as it is much more performant for our image processing. If I find a solution to build and link with buildroot, I'll share the solution @ccrisan

jasaw commented 7 years ago

@ccrisan I tried using rpi-userland's libopenmax but ffmpeg doesn't work with that. Looks like it needs bellagio's libopenmax, unless I made a mistake. You said you managed to get ffmpeg to use rpi-userland's libopenmax, do you mind sharing more details so I can replicate it?

I had a quick look into buildroot on how to enable bellagio package "properly", but not sure how to do it. Basically, both rpi-userland and bellagio pakages are providing libopenmax virtual package, and buildroot takes that as configuration error and doesn't proceed to compilation. Any ideas on how we should go about getting ffmpeg omx to work?

ffmpeg should have mpeg4_omx support as well, but not present on the version used by motioneyeos.

ccrisan commented 7 years ago

@jasaw I simply added the following extra configure options to ffmpeg:

--enable-omx --enable-omx-rpi --enable-mmal --extra-cflags=-I../../staging/usr/include/IL/

Having it look for headers in ../../staging/usr/include/IL/ made it detect the openmax support from rpi-userland.

jasaw commented 7 years ago

@ccrisan That's what I did as well, and managed to get ffmpeg to compile with rpi-userland headers, but when I ran ffmpeg, I get this:

[h264_omx @ 0x1d906f0] Using OMX.broadcom.video_encode [h264_omx @ 0x1d906f0] OMX_GetHandle(OMX.broadcom.video_encode) failed: 80001005

I didn't get this error when I used bellagio's libopenmax. I don't know what the differences are between the rpi-userland and bellagio.

cedricve commented 7 years ago

indeed @jasaw Ttat was exactly what I mean with previous comment.

jasaw commented 7 years ago

Apparently, there are 3 versions of libopenmaxil.so

  1. One is installed at /usr/lib/libopenmaxil.so, file size 6396 bytes. This one came from rpi-userland?
  2. Another one from rpi-firmware/hardfp/opt/vc/lib/libopenmaxil.so, file size 33208.
  3. Last one from rpi-firmware/opt/vc/lib/libopenmaxil.so, file size 32974.

Either number 2 or 3 is normally installed at /opt/vc/lib directory. ffmpeg seems to work when I use the /usr/lib version, but doesn't work with the rpi-firmware versions. I don't know why yet.

Anyway, there is no need to use bellagio's libopenmax.

jasaw commented 7 years ago

I've created pull request 998 as my first attempt to address this issue.

jasaw commented 7 years ago

It appears that ffmpeg occasionally locks up when encoding certain resolutions. I've been running 800 x 600 resolution and it has been working flawlessly, but I'm testing with other resolutions and ffmpeg sometimes locks up when calling avcodec_send_frame.

I've been monitoring GPU memory usage and it seems to be doing OK (no OOM problem). CPU usage is low too, never reach full throttle.

Anyone has any clue to what might be happening?

Note: I'm running ffmpeg version 3.2.3 with h264_omx encoder, latest motion master (git hash 280141f4178f2d656a2af4c5b3d3c430f41646a1) with hw_accel patch. I'm seeing this problem on Pi B, Pi2 B, and Pi Zero W. Haven't tested with other Pis.

Edit: ffmpeg version 3.3.2 (latest release) has the same issue.

jasaw commented 7 years ago

After fixing recorded movie playback bug on motion, I'm finally able to really test the performance of ffmpeg h264_omx encoder.

At 800 x 600 resolution, all models of Raspberry Pis have no issue running higher than 20 fps but at 1920 x 1080 resolution, I only managed to get 3 fps on a Pi2 B. From what I've read, the bottleneck appears to be memory bandwidth between ARM and GPU. With motion software, we currently have this pipeline: Pi Camera (GPU) --> motion detection & adding text to image (CPU) --> h264 encoding (GPU) --> save to file (ARM). I don't know how to check memory bandwidth to verify this. With raspivid type programs, we have this pipeline: Pi Camera (GPU) --> h264 encoding (GPU) --> save to file (ARM), so it's no surprise that it's very efficient. As for Pi1 B with 256MB of RAM, there's no chance of running motion at 1080p resolution because it just doesn't have enough memory. I can't test 720p resolution because ffmpeg sometimes locks up when calling avcodec_send_frame.

@cedricve You mentioned you're able to achieve high frame rates on a Pi.

At the moment we are implementing OpenMAX directly into Kerberos.io/machiner, and we are able to stream MJPEG at 25FPS at 1280x720 and record at 30 FPS 1280x720

Would you be able to share with us what kind of set up you have to get these frame rates? I only know your software is talking to OpenMax directly. What was your pipeline configuration? Pi camera --> ARM CPU processing --> GPU encoding --> ARM CPU file storage?

@ccrisan Have you tried getting vcdbg to work? It's a Pi GPU debugging tool, but I can't get it to work with motioneyeos because vcdbg is provided as a binary.

jasaw commented 7 years ago

Looks like the avcodec_send_frame problem is actually caused by OMX_EmptyThisBuffer locking up, and that's in the Raspberry Pi GPU firmware territory. I don't know why OMX_EmptyThisBuffer is locking up. If anyone has any idea, bright or stupid idea, please suggest.

cedricve commented 7 years ago

@jasaw we've seen this also in the Kerberos.io project, when recording at a higher resolution we get locks and Kerberos.io freezes. I confirm having this myself after 9days running 24/7 at a resolution of 1280x960. This might be caused because the os isn't able to read frames fast enough, did you tried to lower FPS on the higher resolution?

jasaw commented 7 years ago

@cedricve Yes, I've tried with 3 fps 1280 x 720, and it's still locking up. Weird thing is, I haven't seen 30 fps 1920 x 1080 lock up, even though it's only able to encode 5 fps. Although I haven't tested 1920 x 1080 for more than 1 day.

With 1280 x 720, it locks up very quickly, easy to reproduce, that's why I'm using it to debug.

cedricve commented 7 years ago

Hmm well this is a different use case then. Although we're using OMX directly, not in combo with ffmpeg. Not sure how many FPS you can achieve with a RPI Camera on motioneyeos? @ccrisan can you answer?

I'm trying to integrate your PR into KiOS and see if I can simulate with ip cameras. My approach will be different:

IP camera -> OpenCV -> extract frames and do images processing -> when motion fireup an ffmpeg command with the h264_omx encoder specified.

ccrisan commented 7 years ago

@cedricve That's a broad question. The best that I have seen on a RPi 3 at the lowest resolution possible is somewhere about 20fps. With the current motionEyeOS setup you're obviously bound by the CPU as the hardware acceleration is not used at all.

cedricve commented 7 years ago

thanks @ccrisan for answering, I think we should use OpenMAX directly as well. I confirm recording 30FPS HD on a Pi Zero. But that might need some more changes to motion.

@jasaw do you have Skype to discuss this further, don't want to spam this github issue too much. We can report progress in here.

ccrisan commented 7 years ago

Having motion use openmax when talking to the PI cam either via ffmpeg or directly, would be a great thing to have. I believe it's worth everyone's time and effort, a lot more than with any other motionEye features.

jasaw commented 7 years ago

@cedricve I have thought about modifying motion to talk to OpenMAX directly, but wanted to get a better understanding of what the current problems are, where the bottlenecks are, in the hope that our implementation can avoid those issues.

My other concern is code maintenance: ideally, we do a pull-request to merge into Mr-Dave's motion master so other developers can help maintain the code.

Let's discuss further. I'll email you directly.

jasaw commented 7 years ago

I tried running motion with extpipe to see what is causing the lock up, and this is what I found.

Encode via extpipe

Running motion with extpipe to ffmpeg h264_omx is stable at various resolutions. motion --> extpipe --> ffmpeg (h264_omx encoder)

This configuration also gave me higher frame rate:

extpipe config:

use_extpipe on
extpipe ffmpeg -y -f rawvideo -pix_fmt yuv420p -video_size %wx%h -framerate %fps -i pipe:0 -c:v h264_omx -profile:v high -b:v 3000000 -f mp4 %f.mp4

Encode via C API

Running motion encoding via ffmpeg C API hangs at 1280 x 720 resolution. Exact same encoder configuration (bitrate, profile, etc...) as extpipe version. motion --> ffmpeg API (h264_omx encoder)

This configuration gave me lower frame rate:

At this stage, I can only conclude that motion is doing something wrong.

IP camera -> OpenCV -> extract frames and do images processing -> when motion fireup an ffmpeg command with the h264_omx encoder specified.

@cedricve Your use case should be OK.

jasaw commented 7 years ago

Hardware accelerated video encoding is now supported in the pre-release 20171008.

cedricve commented 7 years ago

@jasaw sorry this is an old thread, but I'm curious what the issue was with the lockup at 1280x720. I also get it locked with other resolutions like 640x480. Thanks for your answer @jasaw.

jasaw commented 7 years ago

@cedricve The lockup only happens when passing a GPU/ARM shared buffer back to the GPU with zero copy. Zero copy is the important bit here.

To enable zero copy, the resolution width must be divisible by 32 and height divisible by 16, so image can be handed over to GPU without copying. Resolutions 640x480 and 1280x720 both are divisible by 32 width and 16 height, so you get a lockup. To workaround zero copy lockup, you'll need to do a copy regardless of the resolution.

6by9 thinks it's caused by the Linux kernel memory mapping/management. I am reluctant to dive deep into the kernel memory management code. I've reported this issue here: https://github.com/raspberrypi/firmware/issues/851.

I hacked up motion software to support zero copy, i.e. taking images straight from MMAL camera (height & width divisible by 32 &16), and let motion do its thing, and pass the images to OMX encoder, all with zero copy. Curiously, it sort of works on my Raspbian, but locks up on motionEyeOS. I guess both devices had different versions of kernel at that time. It "sort of works" because it was encoding at 1920x1088 resolution 30 fps for a few seconds, and paused for a few seconds, and continued at 30 fps. I didn't have time to investigate further and gave up on making motion zero copy because it requires too much code changes and it's impossible to get it merged upstream.

tmm1 commented 6 years ago

I'm trying to use ffmpeg with --enable-omx --enable-omx-rpi on an rpi3 and hitting the same error as referenced above:

[h264_omx @ 0x151a020] Using OMX.broadcom.video_encode
[h264_omx @ 0x151a020] OMX_GetHandle(OMX.broadcom.video_encode) failed: 80001005

Can someone advise as to which OMX headers are required to build a working binary?

jasaw commented 6 years ago

Apparently, there are 3 versions of libopenmaxil.so

  1. One is installed at /usr/lib/libopenmaxil.so, file size 6396 bytes. This one came from rpi-userland?
  2. Another one from rpi-firmware/hardfp/opt/vc/lib/libopenmaxil.so, file size 33208.
  3. Last one from rpi-firmware/opt/vc/lib/libopenmaxil.so, file size 32974.

Either number 2 or 3 is normally installed at /opt/vc/lib directory. ffmpeg seems to work when I use the /usr/lib version, but doesn't work with the rpi-firmware versions.

Make sure you have the right version of libraries.

tmm1 commented 6 years ago

ffmpeg seems to work when I use the /usr/lib version, but doesn't work with the rpi-firmware versions.

Where the lib in /usr come from? I don't see that on raspbian, only the files in /opt/vc

tmm1 commented 6 years ago

I upgraded from jessie to stretch and it fixed the issue.

cmsmith81 commented 6 years ago

@jasaw the patch file you provided above appears to not work with the latest source. Can you tell me which checkin/version of the source that patch should work with? I really would like to be able to enable crf with the hardware encoder. Thank you!!

jasaw commented 6 years ago

@cmsmith81 Just FYI, h264_omx encoder only accepts bitrate option, not crf option. I'm not sure what you're trying to do, but MotionEyeOS already has all my h264_omx work merged in.

If you are trying to enable omx on your own project, check out the latest motion master branch. All my motion patches have been merged upstream. You'll also need to apply this patch to enable it.