Support http-mp4 file stream, fmp4 stream.

[winlinvip]

https://github.com/winlinvip/simple-rtmp-server/issues/174

TRANS_BY_GPT3

[winlinvip]

MP4 itself is a box within a box, not a streaming mode. Later, someone improved the box structure of MP4, similar to turning the box into an FLV tag or stacking the boxes side by side. If the client supports parsing and playing the boxes simultaneously, it can achieve similar functionality to HTTP-FLV.

It seems that Silverlight, the deceased Silverlight, and the MP4 stream directly with IIS are examples of this approach.

TRANS_BY_GPT3

[winlinvip]

WeChat Mini Program supports playing HLS live streaming and on-demand streaming, as well as HTTP-MP4 on-demand files. Let's see if it also supports HTTP MP4 live streaming and how much the latency can be reduced.

Microsoft's Live Smooth Streaming uses fragmented MP4: https://technet.microsoft.com/en-us/library/7d5f47fa-886b-4321-8367-60b1a126d4dd

This article specifically discusses fragmented MP4: http://blog.csdn.net/yu_yuan_1314/article/details/9289827

mp4box can generate this type of file.

MP4Box -dash 500000 -frag 50000 -rap file_name.mp4

GPAC (mp4box) official website: https://gpac.wp.mines-telecom.fr/mp4box/

GPAC on GitHub: https://github.com/gpac/gpac

There is an article discussing the replacement of HTTP with RTMP: http://blog.streamingmedia.com/2016/09/flash-rtmp-streaming-dying-out.html. It does not propose a fundamental solution to support RTMP-level latency.

Here is someone who tried fmp4 and said that H5 does not support it: https://my.oschina.net/langhuihui/blog/519901.

You can convert the SRS instance avatar.flv to mp4.

ffmpeg -i doc/source.200kbps.768x320.flv -c copy -y avatar.mp4 

Use mp4box to view information.

winlin:mp4 winlin$ mp4box -info avatar.mp4 
* Movie Info *
    Timescale 1000 - 2 tracks
    Computed Duration 00:03:30.652 - Indicated Duration 00:03:30.652
    Fragmented File: no
    File Brand isom - version 512
    Created: UNKNOWN DATE   Modified: UNKNOWN DATE
File has no MPEG4 IOD/OD

iTunes Info:
    Encoder Software: Lavf55.19.104
1 UDTA types: meta (1) 

Track # 1 Info - TrackID 1 - TimeScale 16000
Media Duration 00:03:30.560 - Indicated Duration 00:03:30.560
Track has 2 edit lists: track duration is 00:03:30.606
Media Info: Language "Undetermined (und)" - Type "vide:avc1" - 5264 samples
Visual Track layout: x=0 y=0 width=768 height=320
MPEG-4 Config: Visual Stream - ObjectTypeIndication 0x21
AVC/H264 Video - Visual Size 768 x 320
    AVC Info: 1 SPS - 1 PPS - Profile High @ Level 3.2
    NAL Unit length bits: 32
    Pixel Aspect Ratio 1:1 - Indicated track size 768 x 320
    Chroma format 0 - Luma bit depth 8 - chroma bit depth 8
    SPS#1 hash: 3031FBB2E6D922C701D45CCE28FFC6743FCA7D2A
    PPS#1 hash: 6BBE60BFBE383CE9144FA78B782534A704912778
Self-synchronized
    RFC6381 Codec Parameters: avc1.640020
    Average GOP length: 48 samples

Track # 2 Info - TrackID 2 - TimeScale 44100
Media Duration 00:03:30.651 - Indicated Duration 00:03:30.651
Track has 1 edit lists: track duration is 00:03:30.652
Media Info: Language "Undetermined (und)" - Type "soun:mp4a" - 9072 samples
MPEG-4 Config: Audio Stream - ObjectTypeIndication 0x40
MPEG-4 Audio AAC LC - 2 Channel(s) - SampleRate 44100
Synchronized on stream 1
    RFC6381 Codec Parameters: mp4a.40.2
Alternate Group ID 1
    All samples are sync

mp4box parameters:

winlin:mp4 winlin$ mp4box -h all
General Options:
 -frag time_in_ms     fragments file (track fragments of time_in_ms)
                       * Note: Always disables interleaving
 -out filename        specifies output file name
                       * Note: By default input (MP4,3GP) file is overwritten

DASH Options:
 -mpd m3u8            converts HLS manifest (local or remote http) to MPD 
                       Note: not compatible with other DASH options (except -out and -tmp) and does not convert associated segments
 -dash dur            enables DASH-ing of the file(s) with a segment duration of DUR ms
                       Note: the duration of a fragment (subsegment) is set
                                using the -frag switch.
                       Note: for onDemand profile, sets duration of a subsegment

Convert regular mp4 to fmp4:

mp4box -frag 3000 avatar.mp4 -out avatar_f.mp4

You can use mp4info to see a small mp4 fragment every 3 seconds:

However, with WeChat mini program, it can only play the first 3 seconds, which means it does not have complete support for fragment mp4.

The loading speed and the speed of starting playback are both fast, but unfortunately it does not continue to parse the remaining moof box...

The two MP4 links are as follows. Regular MP4: http://ossrs.net/weixin/avatar.mp4 Fragmented MP4, with a duration of 3 seconds per fragment: http://ossrs.net/weixin/avatar_f.mp4

For other live streaming methods on WeChat, please refer to: http://blog.csdn.net/win_lin/article/details/54410876

TRANS_BY_GPT3

[winlinvip]

In addition, the browsers on PC for fmp4 are as follows:

• [ ] Chrome cannot play fmp4.
• [ ] Safari can play fmp4, but it is very slow.
• [x] Firefox has relatively good support, continuously downloading the following fragments, and the duration keeps increasing.

Consider putting this unreliable protocol into consideration for SRS4+.

TRANS_BY_GPT3

[jswh]

You can try to support MPEG-TS and then use jsmpeg to achieve playback on regular HTML5 pages and within WeChat. If you use Canvas, it will also work fine in mini programs, although it may not be as efficient as WebGL.

---

It seems that it would be better to directly raise an issue for this.

TRANS_BY_GPT3

[wony-zheng]

flv.js is used in the frontend to convert flv to fmp4, why is it no longer supported?

TRANS_BY_GPT3

[winlinvip]

@wony-zheng MSE is supported, but playing MPEG4 streams directly in the browser is a different framework from MSE.

TRANS_BY_GPT3

[winlinvip]

The conclusion is that MP4 streaming is unreliable.

TRANS_BY_GPT3

[jackyyf]

Is it possible to reconsider this decision? Currently, a fragmented MP4 has been generated based on the method described in Comment#3.

$ ffmpeg -i source.200kbps.768x320.flv -c copy -y avatar.mp4
$ MP4Box -frag 1000 avatar.mp4 -out fmp4.mp4
$ ffprobe fmp4.mp4 
ffprobe version 5.1.2 Copyright (c) 2007-2022 the FFmpeg developers
<snip>
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'fmp4.mp4':
  Metadata:
    major_brand     : isom
    minor_version   : 1
    compatible_brands: isomiso2avc1mp41
    creation_time   : 2022-11-18T18:29:54.000000Z
    encoder         : GPAC-2.0-rev
  Duration: 00:03:30.65, start: 0.000000, bitrate: 246 kb/s
  Stream #0:0[0x1](und): Video: h264 (High) (avc1 / 0x31637661), yuv420p(progressive), 768x320 [SAR 1:1 DAR 12:5], 212 kb/s, 25 fps, 25 tbr, 16k tbn (default)
    Metadata:
      handler_name    : VideoHandler
      vendor_id       : [0][0][0][0]
  Stream #0:1[0x2](und): Audio: aac (LC) (mp4a / 0x6134706D), 44100 Hz, stereo, fltp, 30 kb/s (default)
    Metadata:
      handler_name    : SoundHandler
      vendor_id       : [0][0][0][0]

$ MP4Box -info fmp4.mp4 
# Movie Info - 2 tracks - TimeScale 1000
Duration 00:00:00.000
Fragmented: yes - duration 00:03:30.652
211 fragments - 0 SegmentIndexes
Progressive (moov before mdat)
Major Brand isom - version 1 - compatible brands: isom iso2 avc1 mp41
Created: GMT Fri Nov 18 18:29:54 2022

File has root IOD (9 bytes)
Scene PL 0xff - Graphics PL 0xff - OD PL 0xff
Visual PL: AVC/H264 Profile (0x7f)
Audio PL: AAC Profile @ Level 2 (0x29)
No streams included in root OD

iTunes Info:
    tool: GPAC-2.0-rev

# Track 1 Info - ID 1 - TimeScale 16000
Media Duration 00:00:00.000 
Track has 1 edits: track duration is 00:00:00.000
Track flags: Enabled In Movie
Media Info: Language "Undetermined (und)" - Type "vide:avc1" - 0 samples
Fragmented track: 5264 samples - Media Duration 00:03:30.560
Fragment sample defaults: duration 640 size 0 stsd 1 sync 0 padding 0 degradation_priority 0
Visual Sample Entry Info: width=768 height=320 (depth=24 bits)
Visual Track layout: x=0 y=0 width=768 height=320
AVC/H264 Video - Visual Size 768 x 320
    AVC Info: 1 SPS - 1 PPS - Profile High @ Level 3.2
    NAL Unit length bits: 32
    Pixel Aspect Ratio 1:1 - Indicated track size 768 x 320
    Chroma format YUV 4:2:0 - Luma bit depth 8 - chroma bit depth 8
    SPS#1 hash: 3031FBB2E6D922C701D45CCE28FFC6743FCA7D2A
    PPS#1 hash: 6BBE60BFBE383CE9144FA78B782534A704912778
    RFC6381 Codec Parameters: avc1.640020
    All samples are sync
    Max sample duration: 0 / 16000

# Track 2 Info - ID 2 - TimeScale 44100
Media Duration 00:00:00.000 
Track flags: Enabled In Movie
Media Info: Language "Undetermined (und)" - Type "soun:mp4a" - 0 samples
Fragmented track: 9072 samples - Media Duration 00:03:30.651
Fragment sample defaults: duration 1024 size 0 stsd 1 sync 1 padding 0 degradation_priority 0
Alternate Group ID 1
MPEG-4 Audio AAC LC (AOT=2 implicit) - 2 Channel(s) - SampleRate 44100
    RFC6381 Codec Parameters: mp4a.40.2
    All samples are sync
    Max sample duration: 0 / 44100

Then I set up a local file server and created a very simple webpage. <video src="fmp4.mp4" controls="true" />

After accessing, it can play normally.

Chrome version:

Google Chrome | 107.0.5304.110 (Official Build) (64-bit)
Revision | 2a558545ab7e6fb8177002bf44d4fc1717cb2998-refs/branch-heads/5304@{#1202}
OS | Linux
JavaScript | V8 10.7.193.22
User Agent | Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/107.0.0.0 Safari/537.36

When playing, there are a few drawbacks:

1. Although controls=true, it is not possible to fast forward or rewind, only adjust play/pause, volume, and fullscreen.
2. As shown in the screenshot, during playback, the total length of the file is not known (the file is loaded in segments).

The above two drawbacks have little impact in live streaming scenarios. Considering that Adobe has already made it clear that FLV will not be given Codec IDs for H265/VP9/AV1, fmp4 is currently the best alternative for FLV on webpages. It can be played without introducing JavaScript and using MSE, unlike FLV.

I hope you can reconsider this decision. fMP4 could be a very good solution for low-latency live streaming :)

TRANS_BY_GPT3