search

lotheren / javacv

Automatically exported from code.google.com/p/javacv
0 stars 0 forks source link

How to change sample rate and number of channels? #388

Closed GoogleCodeExporter closed 8 years ago

GoogleCodeExporter commented 8 years ago 
 I want to add sound from audio file into video file with javacv. But final sound is built from several audio tracks. Each track has its own sample rate. I need make resampling before record.
 How can I make reasmpling with javacv?

Original issue reported on code.google.com by fortinab...@gmail.com on 25 Nov 2013 at 6:19

GoogleCodeExporter commented 8 years ago 
libavfilter does that, but it's not exposed by a high-level API like 
FFmpegFrameRecorder.

If you or anybody have any suggestions about how such a feature should be 
implemented, please post them here, thank you.

Original comment by samuel.a...@gmail.com on 26 Nov 2013 at 6:06

GoogleCodeExporter commented 8 years ago 
The typical program of recording video and audio with javacv looks this way:

FFmpegFrameGrabber grabberv = new FFmpegFrameGrabber(in_video.mp4);
FFmpegFrameGrabber grabbera = new FFmpegFrameGrabber(in_audio.mp3);

grabberv.start();
grabbera.start();

FrameRecorder recorder =
 new FFmpegFrameRecorder(out_video.mp4, grabberv.getImageWidth(),
    grabberv.getImageHeight(),
    grabbera.getAudioChannels() );

recorder.setFormat("mp4");
recorder.setFrameRate(grabbera.getFrameRate());
recorder.setVideoBitrate(bitrate_value);

recorder.setSampleRate(grabbera.getSampleRate()); // line 1

recorder.setAudioCodec(avcodec.AV_CODEC_ID_AAC);
recorder.setVideoCodec(avcodec.AV_CODEC_ID_MPEG4);
recorder.setPixelFormat(AV_PIX_FMT_YUV420P);

recorder.start();

//Then the loop of record follows.

i.e. I create a grabber and a recorder, set recorder parameters and call the 
recorder.start().

Notes.
1. If I write line 1 as
   recorder.setSampleRate(some_value);
   I can get a distorted sound.
2. If I don't write line 1 at all, sample rate may set up automatically.

Conclusion. For resampling I suggest using the special function, for example
 FrameGrabber.setResample(boolean param);
 if param==true, resampling,if it is needed, will be performed during loop of record and value sample rate is set with the recorder.setSampleRate() function.

Another possible way is to use the special constructor, for example
 FFmpegFrameRecorder(File file, int imageWidth, int imageHeight, int audioChannels, boolean resampling);

Original comment by fortinab...@gmail.com on 28 Nov 2013 at 11:14

GoogleCodeExporter commented 8 years ago 
It's going to need to be a bit more complicated than that, because we need to 
know the sampling rate of the input data...

Original comment by samuel.a...@gmail.com on 1 Dec 2013 at 12:29

GoogleCodeExporter commented 8 years ago 
I can offer the following function FrameRecorder.setAudioOption("ar", "44100") 
(as a possible variant). But  there are already two functions 
FrameRecorder.setSampleRate() and FrameGrabber.setSampleRate(). I think, all 
three functions should work together and not contradict each other. (By the 
way, why are there already two functions FrameRecorder.setSampleRate() and 
FrameGrabber.setSampleRate()?)
What result will be got, if I set FrameRecorder.setSampleRate(22050) and
 FrameRecorder.setAudioOption("ar", "44100")?

Original comment by fortinab...@gmail.com on 4 Dec 2013 at 5:56

GoogleCodeExporter commented 8 years ago 
setSampleRate() is the audio equivalent of setFrameRate(): We don't need any 
conversion to interpret the same data differently.

But say someone wants to mix two or more audio streams together. What would we 
do? There are frameworks out there that provide that kind of functionality, but 
they are pretty complicated to use I find. The task of mixing streams and 
resampling streams are common operations. There may be more of these operations 
that are performed frequently. I think we should start by making a list of 
those, and then figuring out what kind of easy-to-use API would support these 
operations. What do you think?

So, apart from mixing and resampling, what kind of operations do you feel are 
frequently in demand?

Original comment by samuel.a...@gmail.com on 15 Dec 2013 at 6:54

GoogleCodeExporter commented 8 years ago 
BTW, I'm not sure exactly myself what all the options of FFmpeg should do, so 
please do test them out and let me know if they make sense :) Thanks

Original comment by samuel.a...@gmail.com on 15 Dec 2013 at 6:58

GoogleCodeExporter commented 8 years ago 
Wait a minute, I don't think we need libavfilter just for resampling. 
libswresample should do just fine, and FFmpegFrameRecorder already uses 
libswresample, so it should not be that hard. Let me try to fix this up :)

Original comment by samuel.a...@gmail.com on 21 Dec 2013 at 12:22

GoogleCodeExporter commented 8 years ago 
Ok, I made the changes in this this revision:
http://code.google.com/p/javacv/source/detail?r=5500f7a78268ddf9f118df3cd475991d
3ac389a4
The initial sample code you posted on comment #2 with "Notes. 1." should now 
work as is! Let me know that this works for you, thanks.

Original comment by samuel.a...@gmail.com on 21 Dec 2013 at 4:08

GoogleCodeExporter commented 8 years ago 
The changes are in the newly released version 0.7. Please let me know if they 
do not work for you, thanks!

Original comment by samuel.a...@gmail.com on 7 Jan 2014 at 12:53

GoogleCodeExporter commented 8 years ago 
The resampling works but still there are the problems.

My program of recording video and audio looks approximately like this:

FFmpegFrameGrabber grabberv =
  new FFmpegFrameGrabber(in_video.mp4); //video without audio

FFmpegFrameGrabber grabbera = new FFmpegFrameGrabber(in_audio.mp3);

grabberv.start();
grabbera.start();

FrameRecorder recorder =
 new FFmpegFrameRecorder(out_video.mp4, grabberv.getImageWidth(),
    grabberv.getImageHeight(), 2 ); //line 1

recorder.setFormat("mp4");
recorder.setFrameRate(grabbera.getFrameRate());
recorder.setVideoBitrate(bitrate_value);

recorder.setSampleRate(grabbera.getSampleRate()); // line 2 without resampling
//recorder.setSampleRate(44100);                  // line 3 with resampling

recorder.setAudioCodec(avcodec.AV_CODEC_ID_AAC);
recorder.setVideoCodec(avcodec.AV_CODEC_ID_MPEG4);
recorder.setPixelFormat(AV_PIX_FMT_YUV420P);

recorder.setSampleFormat(grabbera.getSampleFormat());
recorder.setAudioBitrate(64000);

recorder.start();

long tsv = 0;

//record video
while(1)
{
    Frame frame = grabberv.grabFrame();

    if(frame == null)
    {
       break;
    }

    frame.samples = null;

    tsv = grabberv.getTimestamp();
    recorder.record(frame);
}

long tsa = 0;

//record audio
for(int i = 0; tsa <= tsv; ++i)
{
    Frame frame = grabbera.grabFrame();

    if(frame == null)
    {
       break;
    }

    frame.image = null;
    tsa = grabbera.getTimestamp();
    recorder.setTimestamp(tsa);
    recorder.record(frame);
    Log.d(TAG, "Frame is " + i);    //line 4
}

I tested four formats: mp3, ogg, 3gp and wav and got the following results:
  1. mp3 and ogg haven't any problems.
  2. 3gp is recorded fully if I use line 2 (without resampling) but only 5s 179ms are
     written if I use line 3 (with resampling) but the line 4 returns the same
     number of frames for both line 2 and line 3.
  3. The resampling for wav was successful if a source wav file contained two
     channels. The sound was distorted when it contained one channel. I tried to set
     one channel in line 1 but without success. The target mp4 had two channels.

  Could you please advise me how I can solve the problems with 3gp and wav.

  Thanks.

Original comment by fortinab...@gmail.com on 10 Jan 2014 at 9:09

GoogleCodeExporter commented 8 years ago 
Ok, I'll look into that, but if you know what we need to change in 
FFmpegFrameRecorder, please let me know, and I'll fix it right away! Thanks

Original comment by samuel.a...@gmail.com on 10 Jan 2014 at 1:26

GoogleCodeExporter commented 8 years ago 
Well, I can't reproduce the problem here with the data I have:
2. 3GP files use AAC, and I do not have any problems reencoding AAC files back 
into AAC.
3. MP4 files also use AAC, and I have no problems encoding a 1-channel WAV 
file: The output file has exactly one channel.

So, if you would like me to take a look at that, you're going to need to 
provide some (as small as possible) data, thanks!

Original comment by samuel.a...@gmail.com on 11 Jan 2014 at 11:23

GoogleCodeExporter commented 8 years ago 
My 3gp is 3gpp. Codec is AMR. The file contains the voice recorded by a 
microphone (MediaRecorder from Android SDK).
I use

FFmpegFrameRecorder recorder =
 new FFmpegFrameRecorder(out_video.mp4, width, heigh, grabbera.getAudioChannels());

and I always get good result. But I get distorted sound when I use

FFmpegFrameRecorder recorder =
 new FFmpegFrameRecorder(out_video.mp4, width, heigh, 2);

for a source audio with one channel.

My purpose is to create soundtrack from several parts independently from their 
format, sample rate and number of channel. I have to reduce these parts to 
common denominator. I convert these parts to mp4 with two channels, with 
resampling and without video (I don't known how to conver to mp3) than write in 
the out_video.mp4. If I use the source audio part with one channel, either I 
don't hear this part, or I get SIGSEGV during writing this part to the output 
mp4 file.
 Possible that all formats with one channel give distorted sound if I set two channels.
Thanks

Original comment by fortinab...@gmail.com on 14 Jan 2014 at 10:13

Attachments:

GoogleCodeExporter commented 8 years ago 
So, are you saying that you have problems only when the number of channels 
don't match?

If you keep the same number of channels, does it always work correctly?

Original comment by samuel.a...@gmail.com on 14 Jan 2014 at 10:35

GoogleCodeExporter commented 8 years ago 
I have the following results:

1. I can use without any problems the files with two channels and the files 
with one channel together if output file should have only one channel.
2. I have to use the sources audio files with two channels if the output file 
should have two channels.
3. I get duration only 5s 179ms after resampling 3gpp file.

Original comment by fortinab...@gmail.com on 14 Jan 2014 at 12:30

GoogleCodeExporter commented 8 years ago 
Ok, I think I've identified the causes. This revision should fix everything:
http://code.google.com/p/javacv/source/detail?r=e6450930b63e3f4c76695627bb671b36
08c44557
Let me know how it goes! thanks

Original comment by samuel.a...@gmail.com on 15 Feb 2014 at 10:58

GoogleCodeExporter commented 8 years ago 
The fix has been included in the latest release of JavaCV 0.8. There are a lot 
of other changes with this release, so please make sure to read the news 
release here:
    http://bytedeco.org/release/2014/04/28/first-release.html

Incidentally, the project is now hosted on GitHub:
    https://github.com/bytedeco/javacv

And please let me know if you still have problems concerning this issue.

Thanks for reporting!

Original comment by samuel.a...@gmail.com on 29 Apr 2014 at 1:11