scottlamb / retina

High-level RTSP multimedia streaming library, in Rust
https://crates.io/crates/retina
Apache License 2.0
244 stars 48 forks source link

mux MJPEG into `.mp4` if possible #101

Closed scottlamb closed 3 months ago

scottlamb commented 7 months ago

@zanshi

It'd be nice for the mp4 example app to mux MJPEG into .mp4 files, and for a new VideoParameters::sample_entry to create a suitable VisualSampleEntry for this for other applications (e.g. Moonfire NVR) to use for writing .mp4s without codec-specific logic.

mediamtx appears to have logic for doing this here: https://github.com/bluenviron/mediamtx/blob/3d69edb533a94b55bf3a98fb5391268ee324825c/internal/playback/mp4/track.go#L645-L694

ffmpeg also seems to do this. Just ffmpeg -i in.mjpeg -c copy out.mp4 produces something that plays with apps such as ffplay, VLC, and Apple QuickTime Player and still is reported by ffprobe as MJPEG.

I'm confused though. The VisualSampleEntry type is mp4v which I thought was MPEG-4 video (ISO/IEC 14496-2)? Is that standard backward compatible with MJPEG? I'm not actually sure what it's writing is correct even though it does play.

scottlamb commented 7 months ago

Ahh, this bit of mediamtx source was helpful, a ways above what I quoted:

// Specification: ISO 14496-1, Table 5
const (
    objectTypeIndicationVisualISO14496part2    = 0x20
    objectTypeIndicationAudioISO14496part3     = 0x40
    objectTypeIndicationVisualISO1318part2Main = 0x61
    objectTypeIndicationAudioISO11172part3     = 0x6B
    objectTypeIndicationVisualISO10918part1    = 0x6C
)

ISO 10918-1 is JPEG; wikipedia says:

In 1987, ISO TC 97 became ISO/IEC JTC 1 and, in 1992, CCITT became ITU-T. Currently on the JTC1 side, JPEG is one of two sub-groups of ISO/IEC Joint Technical Committee 1, Subcommittee 29, Working Group 1 (ISO/IEC JTC 1/SC 29/WG 1) – titled as Coding of still pictures.[14][15][16] On the ITU-T side, ITU-T SG16 is the respective body. The original JPEG Group was organized in 1986,[17] issuing the first JPEG standard in 1992, which was approved in September 1992 as ITU-T Recommendation T.81[18] and, in 1994, as ISO/IEC 10918-1.

so I guess there is a standard way to embed JPEG into elemental streams and thus the mp4v thing is valid. And that means we also can say it has an RFC 6381 codec of mp4v.6C. I'm not sure any web browser claims support for that and will play it, though.