search

AOMediaCodec / libiamf

Reference Software for IAMF
BSD 3-Clause Clear License
40 stars 13 forks source link

Support decoding MP4 when IAMF is not the first track #124

Closed jwcullen closed 1 month ago

jwcullen commented 1 month ago

Current behavior (correct)

iamfdec works fine when decoding MP4 files where IAMF is the first track.

Current behavior (buggy)

iamfdec fails to decode MP4 files when IAMF is a track other than the first track

It would be better if iamfdec could find and decode the IAMF track. The rest the bug report describes how to generate files which show the difference between the two cases. I would expect both cases to decode the same if everything was working. I can provide actual files if there are any issues reproducing the issue.

System Setup

The rest of the bug report uses ffmpeg @504c1ff to generate some sample files.

All commands below are run from a checked out version and built version of AOMEdiaCodec/libiamf@3bbb1844.

Create sample files

Create simple video content to use when muxing below. ffmpeg -f lavfi -i color=c=green:s=1920x1080:d=0.500 sample_video.mp4

Create an MP4 file with IAMF as the first track and video as the second track

ffmpeg -i tests/sine_1000_48khz.wav -i sample_video.mp4 \
-filter_complex "[0:a]channelmap=0|1:stereo[FRONT]" -map "[FRONT]" -stream_group "type=iamf_audio_element:id=1:st=0:audio_element_type=channel,layer=ch_layout=stereo" -stream_group "type=iamf_mix_presentation:id=3:stg=0:,submix=parameter_id=100:parameter_rate=48000:default_mix_gain=0.0|element=stg=0:headphones_rendering_mode=binaural:parameter_id=101:parameter_rate=48000:default_mix_gain=0.0|layout=sound_system=stereo:integrated_loudness=0.0" \
-map 0:a:0 -c:a libopus -ar 48000 -map 1:v:0 -y iamf_then_video.mp4

Rearrange the final line to create an MP4 file with video as the first track and IAMF as the second track:

ffmpeg -i tests/sine_1000_48khz.wav -i sample_video.mp4 \
-filter_complex "[0:a]channelmap=0|1:stereo[FRONT]" -map "[FRONT]" -stream_group "type=iamf_audio_element:id=1:st=0:audio_element_type=channel,layer=ch_layout=stereo" -stream_group "type=iamf_mix_presentation:id=3:stg=0:,submix=parameter_id=100:parameter_rate=48000:default_mix_gain=0.0|element=stg=0:headphones_rendering_mode=binaural:parameter_id=101:parameter_rate=48000:default_mix_gain=0.0|layout=sound_system=stereo:integrated_loudness=0.0" \
 -map 1:v:0 -map 0:a:0 -c:a libopus -ar 48000 -y video_then_iamf.mp4

Decode sample files

# Decoding works correctly when IAMF is first.
code/test/tools/iamfdec/iamfdec -i1 iamf_then_video.mp4 -s0   

# Decoding fails when IAMF is not first.
code/test/tools/iamfdec/iamfdec -i1 video_then_iamf.mp4 -s0

Summary

Since the first file generated decodes successfully, this implies there is no problem with the IAMF content of the file.

yilun-zhangs commented 1 month ago

@jwcullen Thanks for putting forward this issue, the initial idea of iamfdec is used to parse mp4 with only audio track. But it is good suggestion to find correct audio track in mp4 to decode, we will try this.

Thanks.

yilun-zhangs commented 1 month ago

125 , in this PR, we have supported decoding MP4 when IAMF is not the first track,

could please try it.

Thanks.

jwcullen commented 1 month ago

@yilun-zhangs, Great! That PR works for me.