Hardware decode on Android

[allbabel]

Does QtAV support HW decode on Android? If so, how is this configured?

[wang-bin]

No. I plan to use OMX IL. But not sure when to start.

[Ligverd]

Moved in #471 @Wang have you ever tried to collect ffmpeg with --enable-libstagefright-h264 As I understand this is not enough for hardware decoding. You probably have already seen but just in case, throw off the link.

https://vec.io/posts/use-android-hardware-decoder-with-omxcodec-in-ndk

[wang-bin]

--enable-libstagefright-h264 I tried 1 year ago. It's hard to build and always crash at runtime

[Ligverd]

Now there will be many letters.

Build ffmpeg 2.6.2 with stagefright with libraries cm10.1

The build process

Get source ffmpeg-2.6.2 unpack, cd source however, it took a bit to tweak the build script Naturally to change on their own NDK_PATH

diff --git a/libavcodec/libstagefright.cpp b/libavcodec/libstagefright.cpp
index 44d7ea1..d5c02e4 100644
--- a/libavcodec/libstagefright.cpp
+++ b/libavcodec/libstagefright.cpp
@@ -280,7 +280,7 @@ static av_cold int Stagefright_init(AVCodecContext *avctx)
     memcpy(s->orig_extradata, avctx->extradata, avctx->extradata_size);

     meta = new MetaData;
-    if (!meta) {
+    if (meta==NULL) {
         ret = AVERROR(ENOMEM);
         goto fail;
     }
diff --git a/tools/build_libstagefright b/tools/build_libstagefright
old mode 100644
new mode 100755
index 22bb712..2da1049
--- a/tools/build_libstagefright
+++ b/tools/build_libstagefright
@@ -1,30 +1,55 @@
 #!/bin/bash

+#NDK_PATH=${HOME}/android-ndk
+NDK_PATH=/home/dev/Android/ndk
+
 if [ "$NDK" = "" ]; then
-    echo NDK variable not set, assuming ${HOME}/android-ndk
-    export NDK=${HOME}/android-ndk
+    echo NDK variable not set, assuming ${NDK_PATH}
+    export NDK=${NDK_PATH}
 fi

+
+
 echo "Fetching Android system headers"
-git clone --depth=1 --branch gingerbread-release https://github.com/CyanogenMod/android_frameworks_base.git ../android-source/frameworks/base
-git clone --depth=1 --branch gingerbread-release https://github.com/CyanogenMod/android_system_core.git ../android-source/system/core
+#branch="gingerbread-release"
+#branch="jellybean-stable"
+branch="cm-10.1"
+
+git clone --depth=1 --branch ${branch} https://github.com/CyanogenMod/android_frameworks_base.git      ../android-source/frameworks/base
+git clone --depth=1 --branch ${branch} https://github.com/CyanogenMod/android_system_core.git          ../android-source/system/core
+git clone --depth=1 --branch ${branch} https://github.com/CyanogenMod/android_frameworks_native.git    ../android-source/frameworks/native
+git clone --depth=1 --branch ${branch} https://github.com/CyanogenMod/android_frameworks_av.git        ../android-source/frameworks/av
+git clone --depth=1 --branch ${branch} https://github.com/CyanogenMod/android_hardware_libhardware.git ../android-source/hardware/libhardware
+
+

 echo "Fetching Android libraries for linking"
 # Libraries from any froyo/gingerbread device/emulator should work
 # fine, since the symbols used should be available on most of them.
+
+#CM="update-cm-7.0.3-N1-signed.zip"
+#CM="update-cm-7.1.0-N1-signed.zip"
+CM="cm-10.1.3-p5110.zip"
+
+GETCM="http://download.cyanogenmod.com/get/jenkins/42541/"
+#GETCM="ftp://mirror.i3d.net/cyanogenmod/stable"
+#GETCM="http://download.cyanogenmod.com/get/stable"
+
+
 if [ ! -d "../android-libs" ]; then
-    if [ ! -f "../update-cm-7.0.3-N1-signed.zip" ]; then
-        wget http://download.cyanogenmod.com/get/update-cm-7.0.3-N1-signed.zip -P../
+    if [ ! -f "../$CM" ]; then
+        wget $GETCM/$CM -P../ || exit 1
     fi
-    unzip ../update-cm-7.0.3-N1-signed.zip system/lib/* -d../
+    unzip ../$CM system/lib/* -d../
     mv ../system/lib ../android-libs
     rmdir ../system
 fi

-SYSROOT=$NDK/platforms/android-9/arch-arm
+GCCVERSION=4.8
+SYSROOT=$NDK/platforms/android-19/arch-arm
 # Expand the prebuilt/* path into the correct one
-TOOLCHAIN=`echo $NDK/toolchains/arm-linux-androideabi-4.4.3/prebuilt/*-x86`
+TOOLCHAIN=`echo $NDK/toolchains/arm-linux-androideabi-${GCCVERSION}/prebuilt/*-x86`
 export PATH=$TOOLCHAIN/bin:$PATH
 ANDROID_SOURCE=../android-source
 ANDROID_LIBS=../android-libs
@@ -34,25 +59,33 @@ rm -rf ../build/stagefright
 mkdir -p ../build/stagefright

 DEST=../build/stagefright
-FLAGS="--target-os=linux --cross-prefix=arm-linux-androideabi- --arch=arm --cpu=armv7-a"
+FLAGS="--target-os=android --cross-prefix=arm-linux-androideabi- --arch=arm --cpu=armv7-a"
 FLAGS="$FLAGS --sysroot=$SYSROOT"
 FLAGS="$FLAGS --disable-avdevice --disable-decoder=h264 --disable-decoder=h264_vdpau --enable-libstagefright-h264"

-EXTRA_CFLAGS="-I$ANDROID_SOURCE/frameworks/base/include -I$ANDROID_SOURCE/system/core/include"
-EXTRA_CFLAGS="$EXTRA_CFLAGS -I$ANDROID_SOURCE/frameworks/base/media/libstagefright"
-EXTRA_CFLAGS="$EXTRA_CFLAGS -I$ANDROID_SOURCE/frameworks/base/include/media/stagefright/openmax"
-EXTRA_CFLAGS="$EXTRA_CFLAGS -I$NDK/sources/cxx-stl/gnu-libstdc++/include -I$NDK/sources/cxx-stl/gnu-libstdc++/libs/$ABI/include"
+
+EXTRA_CFLAGS=""
+EXTRA_CFLAGS="$EXTRA_CFLAGS -I$ANDROID_SOURCE/frameworks/base/include"
+EXTRA_CFLAGS="$EXTRA_CFLAGS -I$ANDROID_SOURCE/system/core/include"
+EXTRA_CFLAGS="$EXTRA_CFLAGS -I$ANDROID_SOURCE/frameworks/av/include"
+EXTRA_CFLAGS="$EXTRA_CFLAGS -I$ANDROID_SOURCE/hardware/libhardware/include"
+EXTRA_CFLAGS="$EXTRA_CFLAGS -I$ANDROID_SOURCE/frameworks/av/include/media/stagefright"
+EXTRA_CFLAGS="$EXTRA_CFLAGS -I$ANDROID_SOURCE/frameworks/native/include"
+EXTRA_CFLAGS="$EXTRA_CFLAGS -I$ANDROID_SOURCE/frameworks/native/include/media/openmax"
+
+EXTRA_CFLAGS="$EXTRA_CFLAGS -I$NDK/sources/cxx-stl/gnu-libstdc++/${GCCVERSION}/include -I$NDK/sources/cxx-stl/gnu-libstdc++/${GCCVERSION}/libs/$ABI/include"

 EXTRA_CFLAGS="$EXTRA_CFLAGS -march=armv7-a -mfloat-abi=softfp -mfpu=neon"
-EXTRA_LDFLAGS="-Wl,--fix-cortex-a8 -L$ANDROID_LIBS -Wl,-rpath-link,$ANDROID_LIBS -L$NDK/sources/cxx-stl/gnu-libstdc++/libs/$ABI"
+EXTRA_LDFLAGS="-Wl,--fix-cortex-a8 -L$ANDROID_LIBS -Wl,-rpath-link,$ANDROID_LIBS -L$NDK/sources/cxx-stl/gnu-libstdc++/${GCCVERSION}/libs/$ABI"
 EXTRA_CXXFLAGS="-Wno-multichar -fno-exceptions -fno-rtti"
 DEST="$DEST/$ABI"
-FLAGS="$FLAGS --prefix=$DEST"
+FLAGS="$FLAGS --prefix=$DEST --enable-shared --disable-static"

 mkdir -p $DEST

 echo $FLAGS --extra-cflags="$EXTRA_CFLAGS" --extra-ldflags="$EXTRA_LDFLAGS" --extra-cxxflags="$EXTRA_CXXFLAGS" > $DEST/info.txt
-./configure $FLAGS --extra-cflags="$EXTRA_CFLAGS" --extra-ldflags="$EXTRA_LDFLAGS" --extra-cxxflags="$EXTRA_CXXFLAGS" | tee $DEST/configuration.txt
+../configure $FLAGS --extra-cflags="$EXTRA_CFLAGS" --extra-ldflags="$EXTRA_LDFLAGS" --extra-cxxflags="$EXTRA_CXXFLAGS" | tee $DEST/configuration.txt
 [ $PIPESTATUS == 0 ] || exit 1
 make clean
-make -j4 || exit 1
+make -j2 || exit 1
+make install || exit 1

and start build_libstagefright

copy ffmpeg libs and headers to qt

and I even copied some libraries from cm10.1 the question is what should the library's copy? I copied libstagefright*.so and QtCreator put them all in APK After the rebuild QtAV and run sampleplayer mpeg2 works , but video h264 and start build_libstagefright

copy ffmpeg libs and headers to qt

and I even copied some libraries from cm10.1 the question is what should the library's copy? I copied libstagefright*.so and QtCreator put them all in APK After the rebuild QtAV and run sampleplayer, video mpeg2 works , video h264 not shows black screen.

D/libsimpleplayer.so(15979): /home/kid/projects/QtAV/src/QtAV_Global.cpp:225 (void QtAV::qtav_ffmpeg_log_callback(void*, int, const char*, va_list)): "[FFmpeg:mpegts] parser not found for codec h264, packets or times may be invalid."
D/libsimpleplayer.so(15979): /home/kid/projects/QtAV/src/QtAV_Global.cpp:225 (void QtAV::qtav_ffmpeg_log_callback(void*, int, const char*, va_list)): "[FFmpeg:mpegts] max_analyze_duration 5000000 reached at 5000000 microseconds"
W/libsimpleplayer.so(15979): /home/kid/projects/QtAV/src/QtAV_Global.cpp:227 (void QtAV::qtav_ffmpeg_log_callback(void*, int, const char*, va_list)): "[FFmpeg:mpegts] Could not find codec parameters for stream 0 (Video: h264 ([27][0][0][0] / 0x001B), none): unspecified size
W/libsimpleplayer.so(15979): Consider increasing the value for the 'analyzeduration' and 'probesize' options"
...
W/libsimpleplayer.so(15979): /home/kid/projects/QtAV/src/QtAV_Global.cpp:227 (void QtAV::qtav_ffmpeg_log_callback(void*, int, const char*, va_list)): "[FFmpeg:?] Error avcodec_open2(d.codec_ctx, codec, d.options.isEmpty() ? __null : &d.dict) @94 /home/kid/projects/QtAV/src/codec/AVDecoder.cpp: (0xffffffff) Operation not permitted"
W/libsimpleplayer.so(15979): /home/kid/projects/QtAV/src/AVPlayerPrivate.cpp:455 (bool QtAV::AVPlayer::Private::setupVideoThread(QtAV::AVPlayer*)): "Video codec not found"

The sound when this is playing. This somehow cause the decoder ffmpeg need for special or some libraries missing? Or I do all wrong doing?

[wang-bin]

libstagefright is a system library I think. So no copy is required. are you sure mpeg2 works? I think you have to do some changes in QtAV to specify the codec name.

[Ligverd]

mpeg2 works, but probably not by stagefright can full debug log to show.

I'll check availability on the tablet stagefright libraries

[Ligverd]

QtCeator includes APK only libstagefright.so libstagefright_avc_common.so libstagefright_enc_common.so libstagefright_foundation.so libstagefright_omx.so libstagefright_yuv.so

this direct dependence libavcodec

Not on my tablet no stagefright :(

[wang-bin]

I plan to support pure OMX in the next next release (the next release is coming soon). Maybe no libstagefright dependency is required

[Ligverd]

Now looked Nexus 5 and p5110 (samsung) , there stagefright and lots of other have. but they error when playing h264

D/libtest_qtav.so(11165): /home/kid/projects/QtAV/src/QtAV_Global.cpp:225 (void QtAV::qtav_ffmpeg_log_callback(void*, int, const char*, va_list)): "[FFmpeg:mpegts] parser not found for codec h264, packets or times may be invalid.

the impression that was not in the list of codec. I'll deal with libstagefright

[Ligverd]

@wang-bin

in AVDecoder::setCodecContext(void *codecCtx) call AV_ENSURE_OK(avcodec_copy_context(d.codec_ctx, ctx)); and get this error

D/libtest_qtav.so(28007): /home/kid/projects/QtAV/src/AVPlayerPrivate.cpp:437 (bool QtAV::AVPlayer::Private::setupVideoThread(QtAV::AVPlayer*)): "**********trying video decoder: FFmpeg..."
D/libtest_qtav.so(28007): /home/kid/projects/QtAV/src/AVPlayerPrivate.cpp:442 (bool QtAV::AVPlayer::Private::setupVideoThread(QtAV::AVPlayer*)): "===! setCodec..."
D/libtest_qtav.so(28007): /home/kid/projects/QtAV/src/codec/AVDecoder.cpp:137 (void QtAV::AVDecoder::setCodecContext(void*)): "===! codec_src: 28, codec_dst: 0"
D/libtest_qtav.so(28007): /home/kid/projects/QtAV/src/codec/AVDecoder.cpp:138 (void QtAV::AVDecoder::setCodecContext(void*)): "===! 1"
D/libtest_qtav.so(28007): /home/kid/projects/QtAV/src/codec/AVDecoder.cpp:141 (void QtAV::AVDecoder::setCodecContext(void*)): "===! 2"
D/libtest_qtav.so(28007): /home/kid/projects/QtAV/src/codec/AVDecoder.cpp:147 (void QtAV::AVDecoder::setCodecContext(void*)): "===! 3"
D/libtest_qtav.so(28007): /home/kid/projects/QtAV/src/codec/AVDecoder.cpp:153 (void QtAV::AVDecoder::setCodecContext(void*)): "===! 4"
D/libtest_qtav.so(28007): /home/kid/projects/QtAV/src/codec/AVDecoder.cpp:156 (void QtAV::AVDecoder::setCodecContext(void*)): "===! 5"
D/libtest_qtav.so(28007): /home/kid/projects/QtAV/src/codec/AVDecoder.cpp:161 (void QtAV::AVDecoder::setCodecContext(void*)): "===! 6"
W/libtest_qtav.so(28007): /home/kid/projects/QtAV/src/QtAV_Global.cpp:227 (void QtAV::qtav_ffmpeg_log_callback(void*, int, const char*, va_list)): "[FFmpeg:?] Error avcodec_open2(d.codec_ctx, codec, d.options.isEmpty() ? __null : &d.dict) @94 /home/kid/projects/QtAV/src/codec/AVDecoder.cpp: (0xffffffff) Operation not permitted"

codec_id:28 => h264

My thoughts ended. :( Tomorrow I will try to trace but I think will not help. You've talked to small changes in QtAV can give a tip?

[wang-bin]

what did you do?

You can try to force a FFmpeg codec "libstagefright_h264" to use libstagefright

var opt = player.videoCodecOptions
var ffopt = opt["FFmpeg"]
ffopt["codecName"]  = "libstagefright_h264"
opt["FFmpeg"] = ffopt
player.videoCodecOptions = opt

[Ligverd]

Explicit substitution failed too. when you compile ffmpeg with the keys --enable-libstagefright-h264 --disable-decoder=h264 This error appears with and without the lookup codec libstagefright_h264 When compiling with --enable-libstagefright-h264 --enable-decoder=h264 plays the default h264 , explicit substitution goes the exact same error.

W/libKurskOnline.so(30498): /home/kid/projects/QtAV/src/QtAV_Global.cpp:227 (void QtAV::qtav_ffmpeg_log_callback(void*, int, const char*, va_list)): "[FFmpeg:?] Error avcodec_open2(d.codec_ctx, codec, d.options.isEmpty() ? __null : &d.dict) @102 /home/kid/projects/QtAV/src/codec/AVDecoder.cpp: (0xffffffff) Operation not permitted"
W/libKurskOnline.so(30498): /home/kid/projects/QtAV/src/AVPlayerPrivate.cpp:468 (bool QtAV::AVPlayer::Private::setupVideoThread(QtAV::AVPlayer*)): "Video codec not found"
D/libKurskOnline.so(30498): /home/kid/projects/QtAV/src/AVPlayer.cpp:1149 (void QtAV::AVPlayer::playInternal()): "auto select clock: audio > external"

Think to find for precompiled ffmpeg with stagefright :(

[wang-bin]

Why do you disable h264? You will not be able to decode h264. What you need to do is setting the codec name "libstagefright_h264" at runtime because the default codec is "h264".

[Ligverd]

The fact is that in the ffmpeg sources/tools/build_libstagefright the default is assembled as --disable-decoder=h264 --disable-decoder=h264_vdpau --enable-libstagefright-h264

So I tried both options --disable-decoder=h264 --enable-libstagefright-h264 and --enable-decoder=h264 --enable-libstagefright-h264

But even when I compile with --enable-decoder=h264 --enable-libstagefright-h264 and substitute codec "libstagefright_h264" comes out this error

W/libKurskOnline.so(30498): /home/kid/projects/QtAV/src/QtAV_Global.cpp:227 (void QtAV::qtav_ffmpeg_log_callback(void*, int, const char*, va_list)): "[FFmpeg:?] Error avcodec_open2(d.codec_ctx, codec, d.options.isEmpty() ? __null : &d.dict) @102 /home/kid/projects/QtAV/src/codec/AVDecoder.cpp: (0xffffffff) Operation not permitted"

I have such library, maybe something is missing on this and the codec is not connected?

./libOMX.TI.DUCATI1.MISC.SAMPLE.so ./libOMX.TI.DUCATI1.VIDEO.CAMERA.so ./libOMX.TI.DUCATI1.VIDEO.DECODER.secure.so ./libOMX.TI.DUCATI1.VIDEO.DECODER.so ./libOMX.TI.DUCATI1.VIDEO.H264E.so ./libOMX.TI.DUCATI1.VIDEO.MPEG4E.so ./libOMX_Core.so

./libstagefright.so ./libstagefright_amrnb_common.so ./libstagefright_avc_common.so ./libstagefright_chromium_http.so ./libstagefright_enc_common.so ./libstagefright_foundation.so ./libstagefright_omx.so ./libstagefright_soft_aacdec.so ./libstagefright_soft_aacenc.so ./libstagefright_soft_amrdec.so ./libstagefright_soft_amrnbenc.so ./libstagefright_soft_amrwbenc.so ./libstagefright_soft_flacenc.so ./libstagefright_soft_g711dec.so ./libstagefright_soft_h264dec.so ./libstagefright_soft_h264enc.so ./libstagefright_soft_mp3dec.so ./libstagefright_soft_mpeg4dec.so ./libstagefright_soft_mpeg4enc.so ./libstagefright_soft_rawdec.so ./libstagefright_soft_vorbisdec.so ./libstagefright_soft_vpxdec.so ./libstagefright_wfd.so ./libstagefright_yuv.so ./libstagefrighthw.so

Sorry to intrude , if I distract you just tell. :)

[Ligverd]

@wang-bin You didn't think to use MediaCodec for HW decoding?

[wang-bin]

@Ligverd I prefer to use OMX

[forlayo]

Just as an addition,

libstagefright have been removed from ffmpeg, because it creates confusions for Android developers looking for how to do hardware acceleration as this option is not the proper one.

avcodec: Remove libstagefright

It serves absolutely no purpose other than to confuse potentional Android developers about how to use hardware acceleration properly on the the platform. The stagefright "API" is not public, and the MediaCodec API is the proper way to do this.

Furthermore, stagefright support in avcodec needs a series of magic incantations and version-specific stuff, such that using it actually provides downsides compared just using the actual Android frameworks properly, in that it is a lot more work and confusion to get it even running. It also leads to a lot of misinformation, like these sorts of comments (in [1]) that are absolutely incorrect.

[1] http://stackoverflow.com/a/29362353/3115956

Signed-off-by: Derek Buitenhuis derek.buitenhuis@gmail.com

https://github.com/FFmpeg/FFmpeg/commit/72673ad7eae2d4f685f3c0a895558502bfe07c8e

[wang-bin]

The only solution now is mediacodec because since android7.0 app can not dl open system libraries. mediacodec currently is used in qtav.

[forlayo]

I'm using ffmpeg with MediaCodec and I'm getting an error while trying to open the codec

    avcodec_register_all();
    _codec = avcodec_find_decoder(AV_CODEC_ID_H264);
    _codecContext = avcodec_alloc_context3(_codec);
    _videoDecoder = QtAV::VideoDecoder::create("MediaCodec");

    if (_videoDecoder == nullptr) {
            qDebug() << Q_FUNC_INFO << "Error HW Decode initialize";
            return false;
    }else{
        _codecContext->pix_fmt = AV_PIX_FMT_YUV420P;
        _videoDecoder->setCodecContext(_codecContext);

        if (!_videoDecoder->open()) {
            qWarning() << Q_FUNC_INFO << "Unable to load codec";
            return false;
        }
    }

Then while calling ->open() I'm getting this error

void QtAV::qtav_ffmpeg_log_callback(void*, int, const char*, va_list)): [FFmpeg:?] Error avcodec_open2(d.codec_ctx, codec, d.options.isEmpty() ? __null : &d.dict) @116 /home/mike/workspace/testingh264Android/QtAV/src/codec/AVDecoder.cpp: (0xffffffff) Operation not permitted

someone knows what's happening here? Thanks a lot.

[wang-bin]

@forlayo You have to copy codec parameters from stream

[wang-bin]

The latest code using my private mediacodec library can get the best performance.