pdeljanov / Symphonia

Pure Rust multimedia format demuxing, tag reading, and audio decoding library
Mozilla Public License 2.0
2.29k stars 135 forks source link

isomp4: `CodecType` and `extra_data` missing from `avc1` track #271

Open torokati44 opened 6 months ago

torokati44 commented 6 months ago

When running the following code:

use symphonia_core::{
    formats::{FormatOptions, FormatReader},
    io::MediaSourceStream,
};
use symphonia_format_isomp4::IsoMp4Reader;

fn main() {
    let stream = std::fs::File::open("BigBuckBunny_1s.mp4").unwrap();
    let source = MediaSourceStream::new(Box::new(stream), Default::default());
    let reader = IsoMp4Reader::try_new(source, &FormatOptions::default()).unwrap();
    dbg!(reader.tracks());
}

I get this as (part of) the output:

    Track {
        id: 0,
        codec_params: CodecParameters {
            codec: CodecType(
                0,
            ),
            sample_rate: None,
            time_base: Some(
                TimeBase {
                    numer: 1,
                    denom: 90000,
                },
            ),
            n_frames: Some(
                105000,
            ),
            start_ts: 0,
            sample_format: None,
            bits_per_sample: None,
            bits_per_coded_sample: None,
            channels: None,
            channel_layout: None,
            delay: None,
            padding: None,
            max_frames_per_packet: None,
            packet_data_integrity: false,
            verification_check: None,
            frames_per_block: None,
            extra_data: None,
        },
        language: None,
    },

mp4box.js correctly identifies the codec as avc1.42421f, and the file is also valid otherwise.

The missing extra_data may not be an actual bug, as H.264 has two bitstream formats ("Annex B", and "AVCC"), and in one of them (albeit, the one geared more towards streaming, not storing in files), the extradata is in-band, transmitted as regular data packets (or, samples, or call it whatever you want).

The file for testing: https://github.com/pdeljanov/Symphonia/assets/288816/a92343a0-46f4-411e-9c72-a2328e70d78a

pdeljanov commented 6 months ago

This isn't really a bug so much as support for non-audio tracks is very weak right now. A big reason for this is that the current interfaces are very audio specific (e.g., Packet making no distinction between decode and presentation timestamps, CodecParameters being audio only, etc.). I plan to improve this in the dev-0.6 branch which I've just started pushing changes to.

While I don't think I'll be writing any video decoders soon, a project goal is to allow Symphonia demuxers to work with wrapped third-party video decoders like libvpx, etc.