Playing audio - Githubissues

pgodino commented 12 years ago

When I play a movie with audio, no audio is played. I checked the code and actually it seems that audio is considered only when writing to file.

BradLarson commented 12 years ago

You're right, there's nothing in the framework to support audio playback from movies right now. I don't do much work with movie playback myself, so I haven't spent any time looking into this.

I do welcome any contributions you or others might make to add this functionality, though.

pgodino commented 12 years ago

I'm trying to implement audio playing but I'm stuck. Since AVAssetReader reads both audio and video I thought to implement audio playing in this way:

add to AVAssetReader also the audio track
in readNextAudioSampleFromOutput adding a call to a new method processAudioFrame after calling processAudioBuffer (that I think handles writing the audio component to a file)
in the new method using CMSampleBufferGetAudioBufferListWithRetainedBlockBuffer to get the AudioBufferList which contains the audio samples
paying audio samples using RemoteIO audio units (I think that a low level audio playing is required in order to get opmimal performances) as described for example here: http://atastypixel.com/blog/using-remoteio-audio-unit/.

In this way I get only noise. I think that possible problems are (coding problems. I configured AVAssetReaderTrackOutput with this dictionary of options (in processAsset):

NSMutableDictionary *audioOutputSettings = [NSMutableDictionary dictionary]; [audioOutputSettings setObject:[NSNumber numberWithInt:kAudioFormatLinearPCM] forKey:AVFormatIDKey]; [audioOutputSettings setObject:[NSNumber numberWithInt:44100] forKey:AVSampleRateKey]; [audioOutputSettings setObject:[NSNumber numberWithInt:16] forKey:AVLinearPCMBitDepthKey]; [audioOutputSettings setObject:[NSNumber numberWithBool:NO] forKey:AVLinearPCMIsBigEndianKey]; [audioOutputSettings setObject:[NSNumber numberWithBool:NO] forKey:AVLinearPCMIsFloatKey]; [audioOutputSettings setObject:[NSNumber numberWithBool:NO] forKey:AVLinearPCMIsNonInterleaved];

    // This might need to be extended to handle movies with more than one audio track
    AVAssetTrack* audioTrack = [audioTracks objectAtIndex:0];
    readerAudioTrackOutput = [AVAssetReaderTrackOutput assetReaderTrackOutputWithTrack:audioTrack outputSettings:audioOutputSettings];
    [reader addOutput:readerAudioTrackOutput];

And in this way audio samples in the buffer I get should be signed int 16 bits.

The second problem I see is the synchronization between reader (the callback required by remoteIO audio unit) ad the writer of the audio samples. I'm working assuming that this is the problem building a buffer queue.

Some help from someone with some experience with playing the audio from an AVAssetReader could be very useful.

pgodino commented 12 years ago

OK I'm almost there. I'm using https://github.com/michaeltyson/TPCircularBuffer for the consumer/producer buffer for AVAssetReader and RemoteIO audio unit.

I have still 2 problems:

I'm trying with an 8 secs video and audio processing is much faster that the video one. It seems that the audio processing is not realtime but faster. This has the effect that the circular buffer should be very big in order to store all the audio samples. And I get some acceleration of the audio at the end of the video. This means that it could be worse and worse as the video size increases.
I get some noise at the end of the of the audio queue but I think it could be related to the first problem.

I hope to be able to solve this problems next week in order to submit a pull request (but I don't know exactly how to do it, is there some documentations?)

pgodino commented 11 years ago

Actually I wasn't able to solve the two problems I wrote before so I took a completely different direction (I had quite short sequences so I play the image sequence frame by frame especially in order to start and stop the "video" fast on iPad1). Anyway if you wish I can share the code I wrote to play audio but I don't know how I can do it.

gordemota commented 11 years ago

you can copy paste the audio code section and take out the rest.... and put it at https://gist.github.com/

however, i dont have short sequence since my video is 3-4 minutes at least

pgodino commented 11 years ago

Sorry if I did't answer before. I put the complete modified GPUImageMoview with the audio part. Here you have the link: https://gist.github.com/3819907

I hope it could be useful as a starting point.

linktoming commented 11 years ago

I also need this feature. I'm considering adopting a simpler work around. Play the audio of the origin file of the GPUImageMovie. Probable the sync between video and audio will be a problem since GPUImageMovie may not play at the original speed.

scottmcdonnell commented 11 years ago

Great library! I also need audio playback and sync to video.

I have also looked at Mo DeJong’s library AVAnimator which seems to have solved similar issues syncing a CAF file to the video images as they decode and adjusting the playback frame to stay in sync.

Do you think any of this code could be applied? I unfortunately cannot get my head around doing the mash up.

darcy-rayner commented 11 years ago

There was an issue with the timestep of movie playback in GPUImageMovie. Movie's gradually loose time, which causes audio to desync. Anyway, I modified GPUImageMovie to support basic audio playback, (using an AVAudioPlayer). See here . Play a movie file with audio, and make sure to set the playSound property to true. The code tries to sync the movie playback to the audio. If the movie can't be decoded in realtime or better, than it will still probably fall out of sync. This code hasn't been thoroughly tested, so take it with a grain of salt.

snofla commented 11 years ago

@darcy-rayner playing audio during recording freezes up the video camera. any idea why?

darcy-rayner commented 11 years ago

I've only tested it for playing back audio from a movie file. When playSound is turned on, the timestep of the video is tied to the audio track. My guess is it is trying to sync to an audio track with doesn't exist.

snofla commented 11 years ago

@darcy-rayner Fixed it It's just an audio session setup problem:

[[AVAudioSession sharedInstance] setCategory:AVAudioSessionCategoryPlayback error:nil];
UInt32 enabled = 1;
AudioSessionSetProperty (kAudioSessionProperty_OverrideCategoryMixWithOthers, sizeof(enabled), &enabled);
[[AVAudioSession sharedInstance] setActive:YES error:nil];

notedit commented 11 years ago

maybe AudioQueue is the right choice. i know a little about the AudioQueue, does anyone who know AudioQueue want to work on this?

JacobDev commented 11 years ago

Hi. I need some help. I integrated the audio playback following the changes mentioned in the previous comments. I am facing a problem: audio playback is working fine in the simulator but when running my application on a device, it is not working. No error is logged, so I have no idea why this is happening. Has someone encountered the same issue?

davidgloves commented 11 years ago

@snofla Hi,I want to know where to add your code. I also have the problem. When adding a filter to a sample movie file, the audio goes normally while the video freezes. these are my codes: movieFile.playSound = YES; movieWriter.shouldPassthroughAudio = NO; movieFile.audioEncodingTarget = movieWriter; [movieFile enableSynchronizedEncodingUsingMovieWriter:movieWriter]; My purpose is processing a movie file with a filter. At beging the result has no sound, and the video has no sound too when processing. But now the video freezes. Thanks! Please help me out of this.

jasaiher commented 11 years ago

@darcy-rayner I try you GPUImageMovie with the last framework code. It´s working for the .m4v file in the example but I cant do for other video formats. Are you still working on this?

davidgloves commented 11 years ago

@jasaiher hey, it isn't m4v it's mp4. You can see the profile of the file, it's mp4 file. I think that you can change other file to the format of mp4.

eluzix commented 11 years ago

as we encountered the same problem we decided to add audio support to GPUImageMovie using AudioUnits. You can grab our code at: https://github.com/pheed/GPUImage

To use link your project with AudioToolbox and CoreAudio frameworks than just set the playSound to YES on the GPUImageMovie.

few issues:

we rewritten the playAtActualSpeed logic as the old one was out of sync once we added the audio.
only one of playSound/audioEncodingTarget is supported.
the audio player doesn't support pause/play

We are planning to issue a pull request sometime in the future after we test it further.

oltis commented 10 years ago

Have you tested it? It is safe to use or there are some issues we should be aware of if we intend to use it?

Thanks

eluzix commented 10 years ago

@oltis we tested it relative to our needs and it seems to be working fine.

We did further changes to the GPUImage project to fit our needs but it shouldn't effect how it behaves under normal circumstances, if it does just grab the sources under GPUImage/Sources/Audio and GPUImageMovie.

shruthimr commented 10 years ago

@eluzix - I did the following still I don't see any audio being written out with the video file.

movieFile.playSound = YES; movieWriter.shouldPassthroughAudio = NO; movieFile.audioEncodingTarget = movieWriter; [movieFile enableSynchronizedEncodingUsingMovieWriter:movieWriter];

I see you comment in GPUImageMovie.h . /* This determines whether audio should be played. Can't be set to work with video writing. Defaults to NO. / @property(readwrite, nonatomic) BOOL playSound;

So does that mean I cannot add an audio track to my file ?

Thanks, Shruthi.

tuo commented 10 years ago

@shruthimr make sure that this line

movieWriter.shouldPassthroughAudio = NO;

is set to YES rather than NO

If you set it to NO, there is not gonna have AVAssetWriterInput of AVMediaTypeAudio initialized, then of course there is no sound/audio be written to file

shruthimr commented 10 years ago

movieFile.playSound = YES; movieWriter.shouldPassthroughAudio = YES; movieFile.audioEncodingTarget = movieWriter; [movieFile enableSynchronizedEncodingUsingMovieWriter:movieWriter]

Just plays the audio file music even before the actual file has been encoded. And the final file does not have any audio track.

eluzix commented 10 years ago

hi @shruthimr as i mentioned in my original post only one of playSound and audioEncodingTarget will work. This behaviour is fine for our needs so we didn't try to extend it.

if you wish to play with it your self, take a look at GPUImageMovie readNextAudioSampleFromOutput: thats where the decision is made, you should change:

} else if (self.audioEncodingTarget != nil && !audioEncodingIsFinished){
    [self.audioEncodingTarget processAudioBuffer:audioSampleBufferRef];
}

to:

} 

if (self.audioEncodingTarget != nil && !audioEncodingIsFinished){
    [self.audioEncodingTarget processAudioBuffer:audioSampleBufferRef];
}

removing the else so both the audio player and the encoding target gets the buffer.

I haven't tried it myself and have no idea what might go wrong if you try but i suspect it should go fairly well.

aiwo commented 10 years ago

Hi @eluzix. I think I've faced the same problem. After processing a video file in movie writer there's no audio track at all. Your solution above making a crash with EXC_BAD_ACCESS because of invalidation of audioBuffer. Trying to fix it right now.

nikhilcoolstuff commented 10 years ago

Hi @shruthimr ..we're you able to add support for audio encoding target with @eluzix changes? I tried making the changes as @eluzix suggested in his last post but it doesn't work. Still I get no audio after encoding and the sound plays much before the encoding actually starts. Request any pointers...Thanks!

huize commented 9 years ago

init: currentGPUImageMovie=[[GPUImageMovie alloc] initWithPlayerItem:playerItem]; player=[[AVPlayer alloc] initWithPlayerItem:playerItem]; currentGPUImageMovieWriter=[[GPUImageMovieWriter alloc] initWithMovieURL:tempMoiveURL size:_myVideoView.bounds.size];

[currentGPUImageMovie enableSynchronizedEncodingUsingMovieWriter:currentGPUImageMovieWriter]; currentGPUImageMovie.audioEncodingTarget=currentGPUImageMovieWriter; currentGPUImageMovieWriter.shouldPassthroughAudio=YES; [currentGPUImageMovieWriter setHasAudioTrack:YES];

start write: [player play]; [currentGPUImageMovieWriter startRecording]; [currentGPUImageMovie startProcessing];

why the final file not contain audio track ,i use the lastest version GPUImage ,please help

jcampbell05 commented 9 years ago

@eluzix I just added audioFormat.mSampleRate = asbd->mSampleRate; to the bit where you update the buffer format type and it plays correctly.

eluzix commented 9 years ago

@jcampbell05 thanks, as i no longer working with Pheed i can't really update the code. Why wont you fork it and make the update available?

amayers commented 9 years ago

To play audio all I had to do was:

NSURL mediaURL = ... AVPlayer mainPlayer = [AVPlayer alloc] init]; AVPlayerItem playerItem = [[AVPlayerItem alloc] initWithURL:mediaUrl]; [mainPlayer replaceCurrentItemWithPlayerItem:playerItem];

GPUImageMovie *currentMovieFile = [[GPUImageMovie alloc] initWithPlayerItem:playerItem];

// start playback [currentMovieFile startProcessing] [mainPlayer play]

jcampbell05 commented 9 years ago

@eluzix I have a fork available here.

https://github.com/Unii-Ltd/GPUImage

danniss commented 9 years ago

@jcampbell05
your fork breaks a lot of my existing code as it removes some fields from GPUImageMovie.h. what's the intention with it? may be something wrong with the code merge?

tspecht commented 9 years ago

Any news on this issue? I need to make sure the audio and the video is synced and I'm not yet sure if GPUImage is the way for me to go

UriChan commented 8 years ago

@huize ,hi huize, i met the same problem, have you solved it yet?

jcampbell05 commented 8 years ago

@UriChan I made a fork above which worked :)

UriChan commented 8 years ago

@jcampbell05, thanks, i check you fork, it works good for local file, but it does't work for a remote stream, "GPUImageMovie initWithPlayerItem" works with remote stream, but the GPUImageMovieWriter result file does't contain audio track. I have bothered by this problem for quite a long time, any idea how to solve it?

jcampbell05 commented 8 years ago

Not sure, in my case I downloaded the remote file and then played it. Didn't do it via a stream

Alfarache commented 8 years ago

@jcampbell05 I am getting a 404 on your fork link above. Is it still available?

jcampbell05 commented 8 years ago

@Alfarache Ah I think my old company may have shut their repo down - I can ask if someone has a copy. @UriChan would you care to upload it if you still have it ?