use openh264 in ffmpeg

[shenrh]

how to use open264 in ffmpeg?

[ethanhugg]

I don't know about ffmpeg, but the folks at Ericsson made a GStreamer plugin for it, does that help you? http://cgit.freedesktop.org/gstreamer/gst-plugins-bad/tree/ext/openh264

[mstorsjo]

It's a bit tricky to add general code to libavcodec to use openh264 since the API changes constantly and the headers don't even expose defines with an API version number, so unless the code is intended to be used with one single version of openh264 and the calling code be upgraded in sync with openh264, it's pretty hard. If there would be some sort of api version numbers in the headers, and some sort of forward-compatibility source-wise, it'd be much easier.

[mstorsjo]

Now that #1603 has been merged, code for using it can probably be added to libavcodec after the next release of openh264 (or earlier, if it is backported to the 1.2 release branch).

[mstorsjo]

libav and ffmpeg (in the latest git versions) now support linking to OpenH264.

[sijchen]

thank you very much!

[BarkyTheDog]

I'd like to try using OpenH264 with ffmpeg...is there any specific info on how this can be done (a few lines of example code, what steps are necessary, etc), before I have to resort to digging through source code? :-) Thanks! Apologies if I'm just missing something obvious here...

[mstorsjo]

Build and install openh264 (make install PREFIX=/some/path). Configure ffmpeg (configure --enable-libopenh264), make sure that pkg-config is available and looks in the right directory (if /some/path is a default path like /usr or /usr/local it may be picked up by default, otherwise set export PKG_CONFIG_PATH=/some/path/lib/pkgconfig). For other details with pkg-config in e.g. cross-building setups, refer to ffmpeg.

[BarkyTheDog]

Thanks much!

Ahh...how about this for another question: I'm assuming that using configure to enable openh264 will compile in that library...what if I need to go the route of using downloaded binaries, to take advantage of the provider pre-paying the license royalties/fees?

[mstorsjo]

If you've only built/installed a static library, it will compile it in, but if you built a .so/.dll, it will only link to it, not compile it in. (For MSVC builds, replace /lib/openh264.lib with /lib/openh264_dll.lib, to have it link to it as a DLL instead.) Then if you leave out the openh264 shared library, you can replace it with one downloaded at runtime (just make sure you've built it from the same release branch as the binary you're using.)

A libavcodec library built this way won't load or run at all until you actually provide the right openh264.so (or provide a fake dummy one which doesn't contain any implementations) though. Support for loading it at runtime via dlopen isn't available (yet at least).

[BarkyTheDog]

OK. That makes sense. So, the openh264 shared library would simply need to be located where the runtime linker can find it. Support for dlopen, etc. would be a wee bit more convenient, but not a big deal at this point :-) Thanks!

[BarkyTheDog]

I'm attempting to make use of openh264 with FFmpeg, but for now am not dealing with the "downloaded shared library" issue -- I'm just compiling both from their latest git repositories.

I have some code that encodes "raw" RGB data (via conversion to YUV) nicely, when used with x264. I can switch to openh264 using the configure args you pointed out. However, with openh264 I get a number of somewhat ominous-sounding warnings, and then end up with "EncodeFrame failed" after just a few frames...

I expect some of the warnings might be addressed by some additional settings in my use of the FFmpeg API, and these may also be contributing to the encoding failure. On the other hand, it's also possible I'm just unfortunately running into a bug in openh264.

I understand completely that openh264 is relatively new, and its use with FFmpeg even newer, so these sorts of things are expected. Unfortunately, I don't have enough experience with FFmpeg, or knowledge of its internals, to be effective in debugging my way through these issues.

I'd certainly like to see FFmpeg working with openh264, but about all I can offer in terms of furthering this goal are specifics on the warnings, as well as the code that triggers them. But, I'm not sure the comments section of this issue is the best way to deal with this. If you or someone else would care to investigate, please let me know the best way we can communicate.

Thanks!

[sijchen]

Hi, could you please paste the warning printed out by encoder here so that we can take a look?

[BarkyTheDog]

OK. Here are the warnings I get. The first one seems it's just that I've not set appropriate values in AVCodecContext (frame_skip_threshold, etc?), but I'm not sure, and don't know appropriate values. The second warning's probably from FFmpeg itself, but is just a deprecation warning. The third seems rather serious, as I get it twice, and then the "EncodeFrame failed" for every frame after that...

By the way, the app code in question is from here: http://www.imc-store.com.au/Articles.asp?ID=276

[OpenH264] this = 0x0x608000054cd0, Warning:bEnableFrameSkip = 0,bitrate can't be controlled for RC_QUALITY_MODE,RC_BITRATE_MODE and RC_TIMESTAMP_MODE without enabling skip frame.

[mp4 @ 0x10584f600] Using AVStream.codec.time_base as a timebase hint to the muxer is deprecated. Set AVStream.time_base instead.

[OpenH264] this = 0x0x608000054cd0, Warning:WelsEncoderEncodeExt()MinCr Checking,codec bitstream size is larger than Level limitation

[libopenh264 @ 0x105806a00] EncodeFrame failed

[sijchen]

Hi, from the warning it did hit a problem of "bitstream size is larger than Level limitation", where the Level limitation is MinCr=2 or 4, given that we had a known issue of "Encoder errors when compressed frame size exceeds half uncompressed size". HOWEVER, there should be very low probability to hit this with ordinary video 420 input. Here are the parts we need more info: 1, what is the Encoder return error you get? 2, could you also paste the encoder input parameters? such as the following or the info trace from the encoder? m_AVIMOV_WIDTH=Width; //Movie width m_AVIMOV_HEIGHT=Height; //Movie height m_AVIMOV_FPS=FPS; //Movie frames per second m_AVIMOV_GOB=GOB; //I frames per no of P frames, see note below! m_AVIMOV_BPS=BitPerSecond; //Bits per second, if this is too low then movie will become garbled

[BarkyTheDog]

The description of your "known issue" led me to consider the test data that's used by the encoding example code that I linked to...For testing purposes, they synthesize the RGB data per frame: each pixel has the value "char(rand()*255)". Of course this isn't going to compress well at all...

I changed it so all pixels get a constant value, and the error no longer appears...which I guess is what one might expect. However, no matter what value I assign the pixels, the output frames are all black...I'll have to check my testing methodology here, though... :-)

The encoder input is this: MovieEncoderClass->SetupVideo("/tmp/test.mov",640,480,30,10,40000000); So:

• width = 640
• height = 480
• fps = 30
• gob = 10
• bitrate = 40000000

Still wondering what I need to do to get rid of the first warning I reported...any advice appreciated :-)

I'll try to synthesize something more reasonable for the input frame data...and let you know the outcome...

[BarkyTheDog]

Hmm...changed the synthesized frame data so that all pixels have RGB values each equal to the frame number mod 255 (so the movie just fades from black to white). When using x264, I get the desired output; but with openh264, all frames are black. Could this be due to that first warning I'm getting:

[OpenH264] this = 0x0x608000054cd0, Warning:bEnableFrameSkip = 0,bitrate can't be controlled for RC_QUALITY_MODE,RC_BITRATE_MODE and RC_TIMESTAMP_MODE without enabling skip frame.

[sijchen]

Yes if you give such a random image as input, the compression ratio will not be high; that's why I mentioned "ordinary video input". Note that openh264 encoder does not do color space conversion, it only takes YUV420 as input. The skip-frame setting should not result in all black frames.

[BarkyTheDog]

Yes, similarly to x264...The original data is RGB, but is converted to YUV420:

    sws_ctx = sws_getContext(c->width, c->height, AV_PIX_FMT_RGB24,
                       c->width, c->height, AV_PIX_FMT_YUV420P,
                       sws_flags, NULL, NULL, NULL);

sws_scale(sws_ctx,
          m_src_picture.data, m_src_picture.linesize,
          0, c->height, m_dst_picture.data, m_dst_picture.linesize);

With x264, I get the output I expect, but not with open264; in both cases it's converted to YUV420...

I'm assuming that openh264 works, in general, so it is likely that some other missing/incorrect setting is to blame, but it's not clear to me what it might be. I only mentioned the skip-frame settings because I am getting a warning about it, but you are right...it does not seem like it should be a problem...

[sijchen]

Hi, I am not sure how "EncodeFrame (const SSourcePicture* kpSrcPic, SFrameBSInfo* pBsInfo)" is called in the codes you are using here, but two things may be checked: 1, the input parameter: if you can switch the trace level to INFO and copy the input param here so that we can check. 2, the size info in "SSourcePicture* kpSrcPic" 3, the frame type returned in "SFrameBSInfo* pBsInfo"

[BarkyTheDog]

Since I'm using openh264 through FFmpeg, I don't think I have direct access to a function that sets the tracing level. I could force it in the openh264 sources, if you could tell me where to modify the code. But in any case, I can stop in the debugger at EncodeFrame, and these are the values at return time. I tracked the data pointer over the course of generating what should be a fade over time from black to white, and pData[0] seems to have the expected value at each frame...

For the frame type, I see an instance of videoFrameTypeIDR, followed by 9 or 10 videoFrameTypeP, repeating...

kpSrcPic

• iColorFormat: 23
• iStride: [640, 320, 320, 0]
• pData[0] : "~" (repeating)
• pData[1]: "\x80" (repeating)
• pData[2]: "\x80" (repeating)
• pData[3]: NULL
• iPicWidth: 640
• iPicHeight: 480
• uTimeStamp: 0

pBsInfo:

• iTemporalId: 0
• iSubSeqId: 3
• iLayerNum: 1
• sLayerInfo
• eFrameType: (EVideoFrameType) videoFrameTypeP
• iFrameSizeInBytes: 1382
• uTimeStamp

kiEncoderReturn: 0

I'm not sure if this reveals anything, though...

Here's why: with x264, I can create avi, mov, and mp4 files; with openh264, mov and mp4 files seem to be all black frames, but avi files are correctly generated. At this point, I'm guessing that the FFmpeg code may not be quite sufficiently setting up the openh264 encoder. The x264 encoder, on the other hand, seems to have been built and tested with FFmpeg, and so its defaults & logic work with what FFmpeg (and the example code) are doing, as-is.

So, it seems the game now may be to try to figure out what's missing from how FFmpeg is talking to the openh264 codec... :-(

[mstorsjo]

I tried this, and I can actually reproduce your issue, but I had to guess and fill in some of the details left out. You're using the almost-latest git master branch of OpenH264, right?

OpenH264 has changed some behaviour on the master branch wrt to what NAL units are output. (When writing to MP4 files, the libavcodec/libopenh264 wrapper tries to strip out the SPS/PPS NAL units from keyframes, since these are passed as extradata. The way this currently is done seems to drop too much data when using the master branch.)

Note that the libavcodec integration only currently supports the 1.3 release branch, not the master branch in general (which can have large changes back and forth, including changes to API and ABI - the libavcodec code doesn't even build successfully against today's master of OpenH264, when the bEnableSpsPpsIdAddition field has been renamed).

I'll try to have a look at what needs to be changed in libavcodec to support this, but since there probably may be other changes coming up (and the API version defines in OpenH264 haven't been bumped yet, they still say 1.3 even though it's API incompatible with 1.3), at latest when the next release is finalized (probably in a few months). So TL;DR, stick to the releases of OpenH264.

[mstorsjo]

Hmm, or I'm not sure if it's the same issue I'm seeing as you're running into - when using the latest git master of OpenH264, encoded to MP4 files, I get decoding errors when trying to play it back (no picture at all, which doesn't necessarily sound like the same as you with black image - depending on the player of course). However, I only get this with versions of OpenH264 that don't even compile cleanly with libavcodec (since the bEnableSpsPpsIdAddition field has been renamed) - every version on master after the 1.3 branch that does compile cleanly with libavcodec seems to work fine when encoding to an MP4 file; for versions after that, bEnableSpsPpsIdAddition = 0 needs to be changed into eSpsPpsIdStrategy = CONSTANT_ID.

But if it built fine for you (while using the correct OpenH264 headers matching your lib) without touching libavcodec/libopenh264.c, this probably isn't your issue...

[BarkyTheDog]

I'm on the master branch for both FFmpeg and OpenH264. My understanding was that the master branch was required for FFmpeg; I just assumed the master for OpenH264 would be a good choice :-) I guess I should have suspected something when I saw that bEnableSpsPpsIdAddition caused a compilation failure... :-(

So, I should switch to OpenH264v1.3, and it should work?

In answer to your "not sure if its the same issue": I get the black frames in QuickTime Player, but I get no picture at all in VLC (it "plays", but no window is shown)...so, probably the same issue...

[mstorsjo]

Oh, then it's indeed the same issue.

Yes, generally staying on the master branch probably is a good idea, but here, it's a bit trickier, since there are incompatible API changes regularly. In this case, commenting out bEnableSpsPpsIdAddition doesn't work (but setting the new field eSpsPpsIdStrategy to CONSTANT_ID does work) - I hadn't reflected over this before either, but for the MP4 output mode, this mode is cruicial for the encoder. Otherwise, for every keyframe, the encoder outputs a new SPS/PPS (which the libopenh264 wrapper drops, since for MP4 you're supposed to only have one SPS/PPS in the file header, not one pair prepended in each keyframe) and all new frames refer to this one. By setting it to "constant", all the new SPS/PPS pairs that are prepended in keyframes (which the wrapper drops) have the same ID as the one written in the MP4 file header, and decoding works as intended.

[BarkyTheDog]

OK, so I made the change to change bEnableSpsPpsIdAddition = 0 to eSpsPpsIdStrategy = CONSTANT_ID and the code now works. I'm dreadfully sorry I forgot I hacked that variable out...probably would have saved a lot of typing into this comment section :-(

So, I suppose it just would be a better idea to use the OpenH264v1.3 branch?

[mstorsjo]

Well, for now, with this changed, the master branch should work just as well (I don't know of any other current incompatibilities), but I wouldn't recommend tracking/updating the master branch for use with libavcodec unless you're willing to hack/debug it. So for now, either the 1.3 branch or master (with this fixed) should be equally good. But the 1.3 branch is what the libavcodec officially claims to support right now.

[BarkyTheDog]

Thanks ever so much for your efforts in regard to this issue, and again my apologies for basically causing it :-(

[neilpa]

@BarkyTheDog You weren't the only one with this problem. I've been struggling the last couple of days trying to get this working w/ ffmpeg. The most recent comments finally got me unstuck.

[sijchen]

sorry about making things complicated by adding new strategies.... we will update the version info as soon as possible.

[BarkyTheDog]

Not sure if I should open another thread on this...but here goes...

I finally got around to trying to use openh264 with ffmpeg on some real image data (previously, I'd just synthesized some animated ramp images for testing). The overall quality is fairly bad...as well, the file size is quite a bit smaller than expected (compared against h.264 output using AVFoundation on the Mac, for example).

I'm assuming I've just not got the necessary settings communicated to openh264 via the ffmpeg API. I've tried the most "obvious" settings (GOP size set to 1, very high bitrate) but still the overall quality is not good.

Any suggestions about what/how to set for high quality output would be appreciated...

[mstorsjo]

I'm not sure if my definition of high quality is the same as yours, but with enough bitrate (and either with GOP size 1 or larger) it does look decent to me. If you'd post specific samples and the command lines you've tried, it'd be easier to compare.

[BarkyTheDog]

Ahh...no command line...doing this via the ffmpeg API. So it's especially difficult to figure out how to get settings into any codec. My data is raw RGB frames, which are the result of 3D rendering. For a nearly full-screen image (2516x1388), I set the bitrate as high as 800,000,000 (and at a wide range of values less than that), and still it looks very undersampled. Anything I can inspect with the debugger, to determine how the codec's being driven?

Update: OK, this seems to be an entirely ffmpeg API issue -- the sample rate is actually unaffected by the bitrate setting...gotta dig around and see how to talk to ffmpeg API correctly :-(

Further Update: Yep, definitely "just" an FFmpeg API issue. Turns out that much/most of the control over quality-related parameters lies in the codec-specific "private" dictionary. For example:

    AVCodecContext *c = <your context>;
    av_opt_set(c->priv_data, "profile", "baseline", AV_OPT_SEARCH_CHILDREN);
    av_opt_set(c->priv_data, "level", "3.0", AV_OPT_SEARCH_CHILDREN);
    av_opt_set(c->priv_data, "preset", "slow", AV_OPT_SEARCH_CHILDREN);
    av_opt_set(c->priv_data, "crf",  "18", AV_OPT_SEARCH_CHILDREN);

Judicious tweaking of these (and other) params gets me the control over quality I was searching for.

Hope this helps anyone else who's looking for info...

[Jobq]

Hi folks, I don't know whether I should open a new thread for this or not, but I would like to use openh264 decoder in ffmpeg (not jsut the encoder). Would this be possible with the current available version of ffmpeg on git? Maybe I am not sure what I am asking here. I understand that if I compile ffmepg with openh264, I'd be able to encode with ffmpeg using openH264. Would I also be able to decode my encoded file with th eopenh264 decoder through the use of ffmpeg ?

[mstorsjo]

No, only the encoder is hooked up in libavcodec for now - I don't see much point in using the decoder via libavcodec other than for testing it.

Also, can someone close this issue, so people can open new issues if they have other questions, instead of spamming everybody who have ever taken part in this discussion?

[huylv95]

OS: Ubuntu 18.04

1. Install package nessessary: sudo apt install build-essential -y && sudo apt install nasm -y && sudo apt install pkg-config && sudo apt install frei0r-plugins-dev

2. Clone source: git clone https://github.com/stoyanovgeorge/ffmpeg.git

3. Comment package unnessesary and just install ffmpeg and openh264 vi ffmpeg/compilation.sh

4. Run script by this command: sudo sh ./ffmpeg/compilation.sh

5. cd ffmpeg/downloads/ffmpeg-4.2 ./configure --enable-libopenh264

6. Next step, we run script install ffmpeg again, sudo sh ./ffmpeg/compilation.sh

7. Check version by this command: root@huylv:~# ffmpeg -version ffmpeg version 4.2 Copyright (c) 2000-2019 the FFmpeg developers built with gcc 7 (Ubuntu 7.5.0-3ubuntu1~18.04) configuration: --enable-libopenh264 libavutil 56. 31.100 / 56. 31.100 libavcodec 58. 54.100 / 58. 54.100 libavformat 58. 29.100 / 58. 29.100 libavdevice 58. 8.100 / 58. 8.100 libavfilter 7. 57.100 / 7. 57.100 libswscale 5. 5.100 / 5. 5.100 libswresample 3. 5.100 / 3. 5.100

[Ghayas-cyber]

@lehuy2012 Can you run this successfully.

[Ghayas-cyber]

Hi folks, I don't know whether I should open a new thread for this or not, but I would like to use openh264 decoder in ffmpeg (not jsut the encoder). Would this be possible with the current available version of ffmpeg on git? Maybe I am not sure what I am asking here. I understand that if I compile ffmepg with openh264, I'd be able to encode with ffmpeg using openH264. Would I also be able to decode my encoded file with th eopenh264 decoder through the use of ffmpeg ?

Are you able to run openh264 over ffmpeg. Thanks