Audio.AddFrame(AudioData) throw invalid argument exception

[hosyhosyhosy]

Hi Sorry, but I didn't find a document about adding audio to video. Use Audio.AddFrame(AudioData) throw invalid argument exception: FFMediaToolkit.FFmpegException:“Cannot send a frame to the encoder. Error code: -22 : Invalid argument”. Please help, what do I do?

[IsaMorphic]

This feature of the library is quite fiddly. Verify that your values for SamplesPerFrame, SampleRate, and other parameters are supported by the codec you are using. You can find this type of information on the FFMPEG wiki. In the future perhaps FFMediaToolkit can provide presets for certain codecs or at least some better validation errors.

[dianejason]

I'm having a similar problem, I'm trying to separate the audio from a video, but it's reporting an error

[IsaMorphic]

@dianejason please refer to my previous response. Especially because you are transcoding you may have to implement some kind of buffering logic that puts all the sample data into frame buffers of the correct size for your output codec.

[Lee858]

@IsaMorphic Would it be possible to post a working example of reading an mp3 then adding it to a video?

[IsaMorphic]

I will work on this today and post a Gist link here as soon as its done. Cheers!

[IsaMorphic]

Here it is. If you have any questions, let me know :) https://gist.github.com/IsaMorphic/0bbc913905afc553e3a21964cd5bb49a

[Lee858]

Exactly what I needed. The comments were definitely helpful.

I'm placing audio clips at various positions in the video with silence remaining in the other areas. I had to fill in the gaps in the audio track to get it to work. It doesn't look like there's any way to create an AudioData object, other than using a MediaFile. I created an mp3 that started with silence then used it to create one AudioData frame. I then used that 'audioDataEmpty' frame to fill in the gaps.

There's one odd bug that I can't figure out. It only works if the video frame rate is 15. At other rates that I've tried, the audio track filler frame ends up being written ahead of the video frame. I'm looping through a list and repeating the process. The frame collision happens at different points in the list depending on the frame rate. With a frame rate of 15, I'm able to process 100 records.

Cannot add a frame that occurs chronologically before the most recently written frame!

`

bool stillMoreAudio = true;
TimeSpan audioInPosition = new TimeSpan(0);

for (int i = 0; i < 10; i++)      // Audio gap before
    outputMediaFile.Video.AddFrame(bitmapDataWords);

while (outputMediaFile.Audio.CurrentDuration < outputMediaFile.Video.CurrentDuration)
    outputMediaFile.Audio.AddFrame(audioDataEmpty);  //  Fill auto track up to this point in video track 
while (stillMoreAudio)
{
    outputMediaFile.Video.AddFrame(bitmapDataWords);
    var videoInPosition = outputMediaFile.Video.CurrentDuration;
    if (audioInPosition.Ticks == 0)
        audioInPosition = outputMediaFile.Video.CurrentDuration;

    while ((audioInPosition.Add(inputAudioFile.Audio.Position) < videoInPosition) && (stillMoreAudio = inputAudioFile.Audio.TryGetNextFrame(out AudioData audioData)))
        outputMediaFile.Audio.AddFrame(audioData, audioInPosition.Add(inputAudioFile.Audio.Position));
    frameCount++;
}

for (int i = 0; i < 10; i++)      // Audio gap after 
    outputMediaFile.Video.AddFrame(bitmapDataWords);

`

[IsaMorphic]

@Lee858 you're welcome, glad my example helped. Your problem now is that you are not actually adding silence to the audio track when you think you are. You can pass a float[][] to the AddFrame method just as well as you can an AudioData instance. Use this to explicitly add the silence frames to your output file.

Cheers!

Edit: I just realized that you said in your comment that you're adding the silence. The "bug" you mention is a feature of FFmpeg, and an error in your code. Best advice I can offer is to use a different mechanism instead of incrementing your timestamp variable. This is the limit of the help I can offer, as that is specific to your application rather than to FFMediaToolkit. Best of luck 🤞

[Lee858]

I changed the logic and resolved a few bugs. Code snippet for anybody that needs it.

` bool stillMoreAudio = true; TimeSpan audioTicksPerFrame = new TimeSpan(240000); var inputAudioSource = MediaFile.Open("C:\Path\To\Your\Audio.mp3", new MediaOptions { StreamsToLoad = MediaMode.Audio });

for (int i = 0; i < 5; i++) {
outputMediaFile.Video.AddFrame(frameTitle); // Audio Gap // Fill audio track up to the video track while ((outputMediaFile.Audio.CurrentDuration + audioTicksPerFrame) <= outputMediaFile.Video.CurrentDuration) outputMediaFile.Audio.AddFrame(getEmptyFrame());
}

stillMoreAudio = true; while (stillMoreAudio) { outputMediaFile.Video.AddFrame(frameTitle); while (((outputMediaFile.Audio.CurrentDuration + audioTicksPerFrame) <= outputMediaFile.Video.CurrentDuration) && (stillMoreAudio = inputAudioSource.Audio.TryGetNextFrame(out AudioData audioData))) outputMediaFile.Audio.AddFrame(audioData); } // Fill audio track up to the video track // The the last mp3 frame might not end at the end of the video frame while ((outputMediaFile.Audio.CurrentDuration + audioTicksPerFrame) <= outputMediaFile.Video.CurrentDuration) outputMediaFile.Audio.AddFrame(getEmptyFrame());

for (int i = 0; i < 5; i++) {
outputMediaFile.Video.AddFrame(frameTitle); // Audio Gap // Fill audio track up to the video track while ((outputMediaFile.Audio.CurrentDuration + audioTicksPerFrame) <= outputMediaFile.Video.CurrentDuration) outputMediaFile.Audio.AddFrame(getEmptyFrame());
} `

[hosyhosyhosy]

Thanks IsaMorphic, your example helped me