audio function does not work

[pangguoming]

hey @vshcryabets, did you find that audio function does not work? did you find a way to fix it? if you did it please let me know. thanks.,

[danman81]

Have you gotten this fixed, @pangguoming? If you take a look near the bottom of this thread:

https://github.com/youtube/yt-watchme/issues/16

You can see what might be wrong with the audio is a depricated call to avcodec_encode_audio in avcodec.h, which should be calling avcodec_encode_audio2 instead, although this will require a re-write of those calls and it doesn't look trivial. I've tried compiling ffmpeg on my own, as well as using precompiled libraries provided by @garyjamessilva at the above thread. Hope this helps u!

[Lightartist]

Hey everybody, is someone able to solve the audio problem? If it is complicated or takes too much time, we will pay for your service! Thanks in advance for your message!

[pangguoming]

@danman81 thanks a lot for your suggestion, i will try to fix it now.

if i have any progress , i will tell you @Lightartist

[danman81]

thanks, @pangguoming! let us know how it goes!

[danman81]

hey, @pangguoming have you made any progress? let me know if you run into errors so i can help you debug!

[pangguoming]

@danman81 sorry, i am occupied in other important work, i did not fix the audio function yet. i will continue to try to fix it in my free time

[danman81]

Thank you for the update, @pangguoming ! please let us know if you make more progress! and feel free to post any errors you run into!

[GavinCaptain]

@pangguoming , @danman81 : I got the audio part worked! The points are as follows:

1.

In the file "ffmpeg-jni.c", in the function 'Java_xxx_Ffmpeg_init'. Check this code: audio_codec_ctx->channels = 2;

If your device only support mono, you need to set this to 1. Also, in the file "AudioFrameGrabber.java", in the method "recordThread()",

Check this code: int channelConfiguration = AudioFormat.CHANNEL_CONFIGURATION_STEREO; you need to set it to AudioFormat.CHANNEL_CONFIGURATION_MONO

2.

In the file "ffmpeg-jni.c", in the function 'Java_xxx_Ffmpeg_init'. Check this code: audio_codec_ctx->sample_fmt = AV_SAMPLE_FMT_FLT; This is the format which ffmpeg's native aac supports. However,after checking the file AudioFrameGrabber.java, I found that it sent out 'AudioFormat.ENCODING_PCM_16BIT' data .

So, I changed the code to: audio_codec_ctx->sample_fmt = AV_SAMPLE_FMT_S16; However, this audio format needs libfdk_aac when compiling ffmpeg to *.a So I added the argument "--enable-libfdk_aac" to build.sh. By the way, here's my build.sh when I compiled ffmpeg2.3.6. I think it may be useful. (referenced from http://magiclen.org/):

NDK=$HOME/ffmpeg2.3.6/android-ndk-r10d SYSROOT=$NDK/platforms/android-19/arch-arm/ TOOLCHAIN=$NDK/toolchains/arm-linux-androideabi-4.8/prebuilt/linux-x86_64

PREFIX=""

function build_one { PREFIX="$HOME/ffmpeg_build" PKG_CONFIG_PATH="$HOME/ffmpeg_build/lib/pkgconfig/"

./configure \
--enable-static \
--prefix="$HOME/ffmpeg_build" \
--enable-avresample \
--enable-libfdk_aac \
--enable-gpl \
--enable-nonfree \
--enable-libx264 \
--disable-doc \
--disable-ffplay \
--disable-ffprobe \
--disable-ffserver \
--disable-avdevice \
--disable-doc \
--disable-symver \
--cross-prefix=$TOOLCHAIN/bin/arm-linux-androideabi- \
--target-os=linux \
--arch=arm \
--enable-cross-compile \
--sysroot=$SYSROOT \
--extra-cflags="-I$HOME/ffmpeg_build/include" \
--extra-ldflags="-L$HOME/ffmpeg_build/lib" \
$ADDITIONAL_CONFIGURE_FLAG

make clean make -j4 make install }

-----For neon------

CPU=arm_neon

ADDI_CFLAGS="-marm -march=armv7-a -mfpu=neon -mfloat-abi=softfp -mvectorize-with-neon-quad"

ADDI_LDFLAGS="-Wl,--fix-cortex-a8"

build_one

sudo mv $PREFIX/bin/ffmpeg $PREFIX/bin/libffmpeg.so

-----For general-----

CPU=arm

ADDI_CFLAGS="-marm"

ADDI_LDFLAGS=""

build_one

sudo mv $PREFIX/bin/ffmpeg $PREFIX/bin/libffmpeg.so

---------------------

sudo chmod 777 -R $(pwd)

3.

In the file "ffmpeg-jni.c", in the function 'Java_xxx_Ffmpeg_init'. Add this code:

if (fmt_context->oformat->flags & AVFMT_GLOBALHEADER){
                audio_codec_ctx->flags |= CODEC_FLAG_GLOBAL_HEADER;
}

before the line: open_res = avcodec_open2(audio_codec_ctx, audio_codec, &opts);

Or you may get this error (in your log file): Malformed AAC bitstream detected: use audio bitstream filter 'aac_adtstoasc' to fix it ('-bsf:a aac_adtstoasc' option with ffmpeg)

4.

In the file "ffmpeg-jni.c", in the function 'Java_xxx_Ffmpeg_init'. Replace this code: av_dict_set(&opts, "strict", "experimental", 0);

to this code: av_dict_set(&opts, "-bsf:a", "aac_adtstoasc", 0);

However, I am not sure if this step is a MUST. I did it and step 3 as the same time to fix the "Malformed AAC bitstream detected" error. So, maybe you could try taking this step off and see if it still works if you're interested.

[danman81]

@gavincaptain I love you from the bottom of my heart! I will try this tonite and let u know how it goes!

[pangguoming]

@gavincaptain i love you deeply, did you get married?

[pangguoming]

@gavincaptain chould you please share your whole project ?

[danman81]

@GavinCaptain just wanted to update everyone on my progress. when I try your optional suggestion #4, I get the following error:

The encoder 'aac' is experimental but experimental codecs are not enabled, add '-strict -2' if you want to use it.

But this was resolved when I added "av_dict_set(&opts, "strict", "experimental", 0);" back in along with "av_dict_set(&opts, "-bsf:a", "aac_adtstoasc", 0);" instead of replacing.

Also, when I try suggestion #2, with or without #4, I get the following error during runtime:

Specified sample format s16 is invalid or not supported

i'm running this on windows, so I did not add "--enable-libfdk_aac" to build.sh. but I can recompile libffmpeg.so without problems. i also have libfdk-aac.a built. anyways, when I google the error, looks like someone was able to resolve it by downgrading ffmpeg:

https://bbs.archlinux.org/viewtopic.php?pid=1245854#p1245854

I think i'm using ffmpeg-2.5.3, so maybe that is the problem? maybe i'll try downgrading to ffmpeg2.3.6 to see if that helps.

Either way, this is much farther than I've gotten, so thanks so much again!!

[GavinCaptain]

@danman81 :

1.

Yes, the level of ffmpeg DOES matter. I also failed when I used higher level ffmpeg. (but I forgot the error message...^^U)

2.

"Specified sample format s16 is invalid or not supported" -> Yes, if you don't add '--enable-libfdk_aac', you will encounter this problem.

I think the ffmpeg module is a "big machine", and all the arguments is to tell the machine which 'component' should be used in each part during the compiling process. Each component may support different audio format.

So, it's ok for compliing with no error, if you do not use ''--enable-libfdk_aac'. Because you will get the default ffmpeg's audio format support with that "*.a" So, if you add '--enable-libfdk_aac' and want to set: audio_codec_ctx->sample_fmt = AV_SAMPLE_FMT_FLT; You will get some error message like "not supported..", too.

3.

Don't know if this helps, my whole process is as follows:

1. compile x264, aac module to get .a in Ubuntu
2. compile lame module to get .a in Ubuntu (but I don't think this will be used, because I did not add '--enable-libmp3lame' argument in step 3)
3. compile ffmpeg module to get .a in Ubuntu
4. copy all the .a to the "watchme" project folder
5. Use Eclipse to compile whole project(including the libffmpeg.so) on Window7 [Ps.] I use ndk-r10d both on Windows and Ubuntu

Hope you could make audio work!

[GavinCaptain]

@pangguoming : Sorry, I cannot share the whole project. Because this is my company's project, not mine. I am afraid of the copyright issue if I do so. But I will keep answering the problem as possible as I could.

[pangguoming]

@GavinCaptain it does not matter. i will try to do it as you descripted thanks a lot

[sybercoda]

I'm having issues getting this app to work as well. I just don't understand how so many changes are required to get it to work when it was just uploaded to github 5 months ago.

[garyjamessilva]

Well you will understand what is required to get this working if you start with only what Google posted to Github originally, they didn't post any of the ffmpeg & associated libraries (or various other required parts), just a note saying -

'Include cross-platform compiled streaming libraries. Either libffmpeg.so under src/main/jniLibs/armeabi, or another streaming library with modifying VideoStreamingInterface'

Originally Google did not include the JNI code either, but did subsequently after it was asked for.

In one comment someone said this was like being given a car with no engine, I thought that was about right as you had to find, assemble (& fix) all those parts with only few mostly out of date instructions.

On Wed, Apr 22, 2015 at 11:40 PM, sybercoda notifications@github.com wrote:

I'm having issues getting this app to work as well. I just don't understand how so many changes are required to get it to work when it was just uploaded to github 5 months ago.

— Reply to this email directly or view it on GitHub https://github.com/youtube/yt-watchme/issues/23#issuecomment-95181081.

[sybercoda]

@garyjamessilva I compiled ffmpeg for android, rmtpdump, openssl after jumping through hoops. Im now trying to compile the app with the libraries via Android Studio but I'm getting errors so I think I'm going to try eclipse instead. I hope things are smoother porting this to iOS but that is probably a whole new can of worms. FYI, look Into open source streaming solutions for drones as a viable substitute for this project. Thanks again for your reply.

[pangguoming]

i am doing it with windows7 , there is a lot of questions. i had better to do it with ubuntu

[sybercoda]

@pangguoming I tried it with windows/android studio at first with no luck as well. Then tried with Ubuntu and was able to compile, much easier! Now I'm going to use eclipse on ubuntu because Android Studio is becoming troublesome. I think Eclipse on Ubuntu is your best bet. Command line compile as much as possible. I still may have to go back and recompile again if audio isn't working, per the above procedure.

[pangguoming]

@sybercoda thanks a lot for your suggestion

[pangguoming]

@GavinCaptain could you please share your libffmpeg.so file ? so that i don't have to compile it with windows .

[GavinCaptain]

@pangguoming : Here you are. Try it~ https://onedrive.live.com/?cid=eaf2ffc47d5c37b0&id=EAF2FFC47D5C37B0%21720&ithint=file,so&authkey=!AEfiV8h2FkG92Xs

[GavinCaptain]

By the way, I found that compiling ffmpeg to .a on Windows is not so easy. Lots of error happened..., and I did not succeed. However, compiling ffmpeg to .a on Ubuntu is much easier.

[pangguoming]

@GavinCaptain i want to use ubuntu, but my project is created with windows,so.... you konw.

with your libffmpeg.so file , google returned 403 error: stream is inactive. and ffmpeg-ini init function return false. Could not open file /data/data/com.google.android.apps.watchme/files/outputStreamFile.flv

[GavinCaptain]

@pangguoming : Oh, you need to change the "ffmpeg-jni.c" to meet your own need. It seems that only share libFFmpeg.so does not work, because it includes all my customized settings.

However, I am not so sure what do you mean "my project is created with windows". In my case, I import the project 'yt-watchme' with Eclipse on Window7, too. I only compile ffmpeg on Ubuntu, for getting all *.a which the project 'yt-watchme' needs. Then I finish all other works with Eclipse on Window7. (Of course, you need to install Cygwin and NDK on Windows first)

[sybercoda]

Is there any particular reason or benefit to using ffmpeg for android for live streaming to YouTube vs using the MediaCodec class ? It is to my understanding that MediaCodec offers hardware encoding and is much faster.

[pangguoming]

@GavinCaptain yes , sir, you are right , i mean i need install JDK, eclipse, ADT NDK softer ware etc to an ubuntu computer. i am so lazy.

[sybercoda]

@pangguoming You will also need libssl library on Ubuntu or you will receive an error

[GavinCaptain]

@sybercoda : yes, I am trying to use MediaCodec encoder now, and got stuck.... In short, Android only provides video/audio encoder and muxer for mp4. However, if we want to live stream to Youtube, we need to produce .flv stream. So,I'm trying to use ffmpeg for producing .flv with MediaCodec encoder. But no luck......

[JavierCabanas]

@GavinCaptain: As far as I know, MediaCodec encoder doesn't write the information to make the video readable until the closure of the file. So you won't be able to provide ffmpeg the output data provided by MediaCodec while recording the video

[pangguoming]

@GavinCaptain Thans a lot for your persevere answer.

finally, i have only one question ,when i compile libffmpeg.so file. it shows the Error: The encoder 'aac' is experimental but experimental codecs are not enabled, add '-strict -2' if you want to use it.

According to https://trac.ffmpeg.org/wiki/Encode/AAC, i konw '-strict -2' is a command parameter Do you know how to solve that problem ?.

[pangguoming]

if i did it , i will share my whole project . to show everybody my gratitude . LOL

[danman81]

Thanks again @GavinCaptain ! tonight, I will try to set the --enable-libfdk_aac flag as you suggest somewhere else in android studio. it's my ndk-build command that is not picking up that flag. I see a bunch of other ffmpeg flags in config.h, but i don't think they're being picked up -- i'm sure it's just somewhere else I need to set them. if that does not work, i will take your suggestion and use linux to compile. i will also try the .so file you graciously uploaded.

@pangguoming you need to add both lines: "av_dict_set(&opts, "strict", "experimental", 0);" and "av_dict_set(&opts, "-bsf:a", "aac_adtstoasc", 0);" to get rid of that error. hope that helps!

[pangguoming]

@danman81 i modified the code as you said, but get the error: I/ffmpeg-jni(7551): Specified sample format s16 is invalid or not supported I/ffmpeg-jni(7551): audio frame size: 0 I/ffmpeg-jni(7551): Error opening audio codec: -22 it is the same problem with yours, how did you solve that problem? my ffmpeg version is too low ?

[danman81]

hey @pangguoming , you need to set the --enable-libfdk_aac flag just like i need to. i don't think it's a problem with the version, at least not yet... i probably won't get a chance to look at this today myself, but i will keep u posted!

[pangguoming]

@danman81 yes ,sir, i added --enable-libfdk_aac flag to config.h file. but i still get that error :s16 is invalid or not supported oh, i forget it , config.h is Automatically generated, i should not modify it

[danman81]

very interesting, @pangguoming .. where did you set --enable-libfdk_aac? it's possible it won't fix the same error for me either then....

[pangguoming]

@danman81 first of all , i am not very familiar with ffmpeg , i noticed that a lot of compile flags are in config.h file. (line 3:#define FFMPEG_CONFIGURATION ......, it is a demo file here:https://github.com/mooncatventures-group/ffmpegDecoder/blob/master/FFmpegDecoder/config.h).

but i know that file is automatically generated when we compile the whole ffmpeg libiary.
so i think we need to modify Android.mk

[danman81]

thanks, @pangguoming ! i will try your suggestions tomorrow!

[GavinCaptain]

@pangguoming , @danman81 : I have never compile the *.a on Windows, so the following may only work on Ubuntu.

1. The config.h is auto-generated. It's generated by the command "./configure" with many arguments, like "--enable-libfdk_aac"...etc.
2. After doing step 1, the needed config.h is generated. Then we can use the command 'make' to compile. Theoriertically, modify config.h and then do 'make' should also work, too. But. maybe Windows is NOT FRIENDLY AT ALL environment to do this work....
3. For the convenience, it's usually merge step 1 and 2 to a script file. In my case, I made a file "build.sh". The usage is './build.sh'. Btw, when you compile on Ubuntu, maybe you will got some errors like 'cannot execute'. You could do ''chmod *.sh 777" or "chmod configure 777" to get the correct right to execute...
4. Btw, I did not modify Android.mk when I do the above steps. Of course, I don't know what is the correct way on Windows....

[pangguoming]

@GavinCaptain i got it ,thanks

[sybercoda]

Anyone know why I'm receiving the errors below when I try to run the app? Note that line 153 of the 012v.c file is as follows " .long_name = NULL_IF_CONFIG_SMALL("Uncompressed 4:2:2 10-bit"), " and I'm using -std=c99 mode in the Android.mk. Also, I'm using the MainActivity build supplied by @garyjamessilva.

C:\WatchMe\MainActivity\mainActivity\src\main\jni\libavcodec\012v.c Error:(153, 5) error: initializer element is not constant Error:(153, 5) error: (near initialization for 'ff_zero12v_decoder.long_name')

[danman81]

okay, so i did some reading and found that this ./configure stuff might not be supported in android.mk or something along those lines so i'm building with linux as suggested.. ran into "ERROR: libfdk_aac" not found when i run ./configure, so i did some more reading and found that the libfdk_aac license is not compatible with the gpl, which is why it's not included -- do i need to worry about licensing issues? here is some more info:

https://trac.ffmpeg.org/wiki/Encode/AAC

[sybercoda]

@danman81 Good catch! I also found the discussion below. Sounds like you're prohibited from compiling it with ffmpeg but would instead have to reference it as externally shared library. https://ffmpeg.zeranoe.com/forum/viewtopic.php?t=877

[GavinCaptain]

@danman81 : Have you tried "--enable-nonfree" argument?

[danman81]

@GavinCaptain yes, i have tried the --enable-nonfree argument -- the problem, however, was that I did not install libfdk_aac... however, the real problem is that i cannot use fdk_aac because it's not free, which would make my code non-distributable, which really defeats the purpose of using ffmpeg in the first place. i think my solution is going to be to not use fdk_aac, or in other words, not use the AV_SAMPLE_FMT_S16 audio codec, so i will modify AudioFrameGrabber.java instead to not output AudioFormat.ENCODING_PCM_16BIT.

that being said, @iulukaya, i'm going to go out on a limb and say that i really think Google dropped the ball here -- they should really be helping us to debug their own code... please!

[garyjamessilva]

If Google was seriously interested in us using this live streaming facility properly it would be a one line call from within the Android SDK. On Apr 28, 2015 12:35 PM, "danman81" notifications@github.com wrote:

@GavinCaptain https://github.com/GavinCaptain yes, i have tried the --enable-nonfree argument -- the problem, however, was that I did not install libfdk_aac... however, the real problem is that i cannot use fdk_aac because it's not free, which would make my code non-distributable, which really defeats the purpose of using ffmpeg in the first place. i think my solution is going to be to not use fdk_aac, or in other words, not use the AV_SAMPLE_FMT_S16 audio codec, so i will modify AudioFrameGrabber.java instead to not output AudioFormat.ENCODING_PCM_16BIT.

that being said, @iulukaya https://github.com/iulukaya, i'm going to go out on a limb and say that i really think Google dropped the ball here -- they should really be helping us to debug their own code... please!

— Reply to this email directly or view it on GitHub https://github.com/youtube/yt-watchme/issues/23#issuecomment-96889154.

[danman81]

hey friends, so it looks like android.media.AudioRecord from AudioFrameGrabber.java only supports ENCODING_PCM_16BIT, as talked about here:

http://stackoverflow.com/questions/4781781/why-does-audiorecord-getminbuffersize-return-error-bad-value-2

so, it looks like audio_codec_ctx->sample_fmt must be set to a 16bit format in ffmpeg-jni.c as @GavinCaptain says -- this is a limitation of android.media.AudioRecord. there has to be a way to hack this together without using non-free components of ffmpeg! fix this google!!!

[danman81]

hey guys, so i had some thoughts about this while i was sleeping and i realized it may be possible to convert the 16BIT PCM audio stream to 8BIT, programatically. does anyone else agree with this approach?