FFmpegFrameRecorder has no support for audio

[GoogleCodeExporter]

hi, samuel, 
    I am new to Android, android I want to use javacv/ffmpeg to encode camera data into .flv(h.264/aac), can javacv/ffmpeg be used in this way?
    Thank you very much!
What steps will reproduce the problem?
1.I have got data from Android camera(in onPreviewFrame(byte[] data, Camera 
camera))
2.I want to use JavaCV/FFmpeg to encode the data into flv(h.264/aac)
3.Can javacv/ffmpeg be used in this way?

What is the expected output? What do you see instead?
I just want to know that can javacv/ffmpeg be used in this way? And it would be 
better there is a demo, ha.

What version of the product are you using? On what operating system?
andriod 2.3, lenovo S2005A

Please provide any additional information below.
I would very much appreciate if you help me, thank you.

Original issue reported on code.google.com by zhangqia...@gmail.com on 27 Feb 2012 at 3:22

[GoogleCodeExporter]

Sure it's possible. It requires some coding, you'll find what you need to know 
around the site and the mailing list, but I'll keep this issue opened as a RFE 
for a sample... Let me know if you would like to contribute your code if you 
get it working, thank you.

Original comment by samuel.a...@gmail.com on 27 Feb 2012 at 3:41

• Added labels: Type-Enhancement
• Removed labels: Type-Defect

[GoogleCodeExporter]

[deleted comment]

[GoogleCodeExporter]

[deleted comment]

[GoogleCodeExporter]

No, I think the easiest way would be to use FFmpegFrameRecorder. We can copy 
the YUV byte[] data into an IplImage, call cvCvtColor() on that, and write the 
output image using the recorder, e.g.:

FrameRecorder recorder = new FFmpegFrameRecorder("video.mp4", imageWidth, 
imageHeight); 
recorder.setCodecID(CODEC_ID_MPEG4);
recorder.setFormat("mp4");
recorder.setPixelFormat(PIX_FMT_YUV420P);
recorder.start();
// loop ...
cvCvtColor(yuvimage, bgrimage, CV_YUV420sp2BGR)
recorder.record(bgrimage);

I guess though if we are going to write YUV420P video data anyway, we should 
allow YUV images as input to FFmpegFrameRecorder and skip the RGB conversions 
... 

Or if you do no wish the OpenCV dependency, then you can always copy 
FFmpegFrameRecorder and hack your own mod. I am sure someone has done something 
like that. You should ask on the mailing list...

Original comment by samuel.a...@gmail.com on 28 Feb 2012 at 4:50

[GoogleCodeExporter]

[deleted comment]

[GoogleCodeExporter]

[deleted comment]

[GoogleCodeExporter]

[deleted comment]

[GoogleCodeExporter]

[deleted comment]

[GoogleCodeExporter]

Issue 168 has been merged into this issue.

Original comment by samuel.a...@gmail.com on 7 Mar 2012 at 5:55

[GoogleCodeExporter]

BTW, I modified FFmpegFrameRecorder a bit to accept IplImage with nChannels == 
2 as having PIX_FMT_YUV420P .. This way, we can simply 
    IplImage.getByteBuffer().put(data)
    FFmpegFrameRecorder.record(IplImage) 
without having to do any conversion!

Not sure if it actually works, have not tested it, but would be great if you 
could, thank you!

Original comment by samuel.a...@gmail.com on 17 Mar 2012 at 5:34

Attachments:

• FFmpegFrameRecorder.java

[GoogleCodeExporter]

Oh wait a minute, we should put that to PIX_FMT_NV21, but I'm sure you get the 
idea. Let me know if it works after changing PIX_FMT_YUV420P to PIX_FMT_NV21 
inside the attached file above, thanks

Original comment by samuel.a...@gmail.com on 17 Mar 2012 at 7:43

[GoogleCodeExporter]

Am I possible to use this to merge images to a video?

Original comment by booker0108@gmail.com on 18 Mar 2012 at 12:15

[GoogleCodeExporter]

[deleted comment]

[GoogleCodeExporter]

[deleted comment]

[GoogleCodeExporter]

No, look inside FFmpegFrameRecorder.java, search for PIX_FMT_YUV420P and 
replace that with PIX_FMT_NV21

Original comment by samuel.a...@gmail.com on 19 Mar 2012 at 9:22

[GoogleCodeExporter]

[deleted comment]

[GoogleCodeExporter]

It works, but it's slow.. It's faster to convert NV21 to YUV420P directly.

Original comment by samuel.a...@gmail.com on 19 Mar 2012 at 10:00

[GoogleCodeExporter]

[deleted comment]

[GoogleCodeExporter]

Hi, samuel. now I am trying to transfer the recorded data with rtmp, but can 
not find the way, could you help me? thank you!

Original comment by zhangqia...@gmail.com on 21 Mar 2012 at 9:49

[GoogleCodeExporter]

I know that someone on the mailing list has been working on that. You could ask 
there and see if you get any answers...

Original comment by samuel.a...@gmail.com on 22 Mar 2012 at 6:51

[GoogleCodeExporter]

Thank god. I find the answer: if you have a rtmp server, you can easily do it 
with this code:
recorder = new 
FFmpegFrameRecorder("rtmp://192.168.1.27:1935/livestream/12070660", width, 
height);

Original comment by zhangqia...@gmail.com on 22 Mar 2012 at 9:16

[GoogleCodeExporter]

now the important thing is how to add the audio, it seems that I should modify 
FFmpegFrameRecorder for audio recording, right?

Original comment by zhangqia...@gmail.com on 22 Mar 2012 at 9:20

[GoogleCodeExporter]

Ah, for the client side, yes, but we cannot /send/ the data this way, and I 
have not coded a FFmpegFrameServer or something...

Original comment by samuel.a...@gmail.com on 22 Mar 2012 at 9:21

[GoogleCodeExporter]

Yes, let me know if you add audio support and want to make a contribution..

Original comment by samuel.a...@gmail.com on 22 Mar 2012 at 9:22

[GoogleCodeExporter]

but it does work, now I can play the stream on my computer. 

Original comment by zhangqia...@gmail.com on 22 Mar 2012 at 9:22

[GoogleCodeExporter]

[deleted comment]

[GoogleCodeExporter]

Wait a minute, I read this wrong. You could actually use FFmpegFrameRecorder to 
open a server socket?? Wow, interesting

Original comment by samuel.a...@gmail.com on 22 Mar 2012 at 9:24

[GoogleCodeExporter]

[deleted comment]

[GoogleCodeExporter]

yes, I just take place of "videoFile" with 
"rtmp://192.168.1.27:1935/livestream/12070660" in
recorder = new 
FFmpegFrameRecorder("rtmp://192.168.1.27:1935/livestream/12070660", width, 
height);

and it works, I also can't believe at first.

Original comment by zhangqia...@gmail.com on 22 Mar 2012 at 9:28

[GoogleCodeExporter]

my Technical Director said that the ffmpeg includes a lib which can transfer 
stream with rtmp, rtsp, etc. but not a socket server.

Original comment by zhangqia...@gmail.com on 22 Mar 2012 at 9:35

[GoogleCodeExporter]

Right, but it's still interesting! Thanks for letting us know.

BTW, the new FFmpegFrameRecorder I am attaching should work better at encoding 
NV21 data directly, but I haven't tested it. Let me know if it gives more 
interesting results than before.. Creating IplImage this way should work:
    IplImage image = IplImage.create(width, height, IPL_DEPTH_8U, 2);
    image.getByteBuffer().put(data)
    theFFmpegFrameRecorder.record(image) 
Let me know, thanks!

Original comment by samuel.a...@gmail.com on 22 Mar 2012 at 10:32

Attachments:

• FFmpegFrameRecorder.java

[GoogleCodeExporter]

[deleted comment]

[GoogleCodeExporter]

[deleted comment]

[GoogleCodeExporter]

Oh, I know how to solve it, just keep the value of Camera.Parameters.FrameRate 
same as the one of FFmpegFrameRecorder. eg:

        Camera.Parameters camParams = mCamera.getParameters();
        camParams.setPreviewFrameRate(20);
and:
        public VideoRecorder(String filename, int imageWidth, int imageHeight) {
        /* initialize libavcodec, and register all codecs and formats */
        av_register_all();

        this.frameRate   = 20; //30
    }

Original comment by zhangqia...@gmail.com on 23 Mar 2012 at 2:52

[GoogleCodeExporter]

It works with raw data captured from the camera with Android? Hurray! :) Thanks 
for testing

Is the time OK if you call this before start()?
    theFFmpegFrameRecorder.setFrameRate(20);

Original comment by samuel.a...@gmail.com on 23 Mar 2012 at 4:49

[GoogleCodeExporter]

it still seems a little fast, but very close.

Original comment by zhangqia...@gmail.com on 23 Mar 2012 at 5:56

[GoogleCodeExporter]

[deleted comment]

[GoogleCodeExporter]

setFrameRate() merely sets the frameRate field, just like you explain in 
comment #34, so if it produces different results, we've got a bug in Java 
here...

Yes, we need to modify the interface obviously

Original comment by samuel.a...@gmail.com on 23 Mar 2012 at 7:39

[GoogleCodeExporter]

Hi,samuel,I rewrite the FrameRecorder.java and FFmpegFrameRecorder.java。After 

     recorder.setCodecAudioID(CODEC_ID_AAC);
I try to find audio encoder through:
     AVCodec codec_audio= avcodec_find_encoder(codecAudioID);
I got the error:encoder not find
Can you help me ?Thank you！

Original comment by fujiaoji...@gmail.com on 26 Mar 2012 at 3:53

Attachments:

• FrameRecorder.java

[GoogleCodeExporter]

I've added the new FFmpegFrameGrabber in the latest release, thanks for testing!

As for audio support, I am afraid I do not have any experience with that. 
Please open another issue about audio if this is an ongoing concern of yours. 
Maybe someone else who knows more will help out there... thanks!

Original comment by samuel.a...@gmail.com on 29 Mar 2012 at 12:41

• Changed state: Fixed

[GoogleCodeExporter]

Issue 181 has been merged into this issue.

Original comment by samuel.a...@gmail.com on 4 Apr 2012 at 1:33

[GoogleCodeExporter]

Hi, samuel
     I set the recorder as: recorder.setCodecVideoID(CODEC_ID_H264); but
the LogCat said"codec not found", is CODEC_ID_H264 not included?

Original comment by zhangqia...@gmail.com on 8 Apr 2012 at 8:39

[GoogleCodeExporter]

Nope, we would require libx264 for that, and I have not ported/compiled that :( 
Some people seem to have had success though, e.g.:
http://stackoverflow.com/questions/8812827/build-ffmpeg-with-x264-for-android

Original comment by samuel.a...@gmail.com on 8 Apr 2012 at 1:34

[GoogleCodeExporter]

What version of FFmpeg and x264 should I use? (ffmpeg-0.7.11?)

Original comment by zhangqia...@gmail.com on 9 Apr 2012 at 2:33

[GoogleCodeExporter]

FFmpeg 0.7.x yes, but as for the version of x264 that requires, you should 
check its documentation

Original comment by samuel.a...@gmail.com on 9 Apr 2012 at 1:42

[GoogleCodeExporter]

I have finished compiling x264, but did not work, and my ffmpeg config.sh
like this:

#!/bin/bash

PREBUILT=/home/liang/android-ndk-r5/toolchains/arm-eabi-4.4.0/prebuilt/linux-x86
PLATFORM=/home/liang/android-ndk-r5/platforms/android-8/arch-arm

./configure --target-os=linux \
--arch=arm \
--enable-version3 \
--enable-gpl \
    --enable-libx264 \
--enable-nonfree \
--disable-stripping \
--disable-ffmpeg \
--disable-ffplay \
--disable-ffserver \
--disable-ffprobe \
--disable-encoders \
--disable-muxers \
--disable-devices \
--disable-protocols \
--enable-protocol=file \
--enable-avfilter \
--disable-network \
--disable-avdevice \
--enable-cross-compile \
--cc=$PREBUILT/bin/arm-eabi-gcc \
--cross-prefix=$PREBUILT/bin/arm-eabi- \
--nm=$PREBUILT/bin/arm-eabi-nm \
--extra-cflags="-fPIC -DANDROID" \
--disable-asm \
--enable-neon \
--enable-armv5te \
--extra-ldflags="-Wl,-T,$PREBUILT/arm-eabi/lib/ldscripts/armelf.x
-Wl,-rpath-link=$PLATFORM/usr/lib -L$PLATFORM/usr/lib -nostdlib
$PREBUILT/lib/gcc/arm-eabi/4.4.0/crtbegin.o
$PREBUILT/lib/gcc/arm-eabi/4.4.0/crtend.o -lc -lm -ldl" \
--extra-cflags="-I$PLATFORM/usr/include"

what config.sh should be like to match javacv?

Original comment by zhangqia...@gmail.com on 10 Apr 2012 at 10:11

[GoogleCodeExporter]

Apply the patch in this package:
http://code.google.com/p/javacv/downloads/detail?name=ffmpeg-0.7.11-android-arm.
zip
That should do all that you need

Original comment by samuel.a...@gmail.com on 10 Apr 2012 at 1:00

[GoogleCodeExporter]

[deleted comment]

[GoogleCodeExporter]

a good news: it can load library already, but also a bad news, see the encoded 
file

Original comment by zhangqia...@gmail.com on 12 Apr 2012 at 8:14

Attachments:

• stream.flv

[GoogleCodeExporter]

Looks alright, just that your CPU seems too slow to encode in real time :) BTW, 
if you could provide the details of how you compiled and linked x264, it could 
help others who would like to do the same, thanks! And there seems to be audio 
as well, no? You figured this out too?

Original comment by samuel.a...@gmail.com on 15 Apr 2012 at 2:09