Creation of Fragmented MP4 to support streaming

[MaMazav]

I'm interested to use this library with a long mpegts streaming (rather than segmented one, as in HLS).

I guess I have to follow this spec: https://w3c.github.io/media-source/isobmff-byte-stream-format.html

I guess the following stages should be taken:

• Change all STTS, STSC, STCO to be zero-length.
• Add a traf boxes contain TFDT boxes.
• Use relative addressing.
• Using TRUN box.
• Pre-create an init segment which will be followed by the data segments. Data segments should consists MOOF and MDAT boxes.

The first stage seems simple, but I don't know how to do the other stages. I would be happy to contribute but I'm not very experienced with the internal structure of MP4, so any help is appreciated.

Thanks

MOTIVATION:

I would like to show live video with extra-low latency. I'm using WebSocket to send data to the client as soon as possible (I'm aware of other alternatives like MPEG-DASH, but it is out of scope now).

By pre-devision of the TS into chunks of full TS and PES packets, I succeeded to bypass the problem of jBinary streaming mentioned in this issue: https://github.com/jDataView/jBinary/issues/41

In addition I changed the mpegts library to support streaming context (index.js file), so future calls to mpegts can use previous segments passed to mpegts.

Now every TS segment is converted into MP4 complete file. The problem is that now the beginning of each segment missing some informations (e.g. first frames do not have key frames to refer). By creating fragmented MP4 I will be able to feed MSE segment-by-segment.

[RReverser]

This is pretty interesting task but requires time and money investments which I don't currently have.

[zoltan-fedor]

Hi @MaMazav , I would be interested in the same (currently using vlc browser add-in to play mpeg-ts in the browser, but that plugin is going away). Any success with mpeg-ts low-latency realtime video displaying?

[zoltan-fedor]

Hi @MaMazav , Actually I ended up trying your branch https://github.com/MaMazav/mpegts/tree/Streaming-ts-by-WebSocket and was able to recreated the UDP to WebSocket streaming of mpeg-ts. Unfortunately I managed to max out the CPU on the client side with the conversion so much, that the video screen doesn't display, while the original version of non-realtime HLS streaming by @RReverser does play (90% CPU), but obviously that is very far from realtime. I guess for now I will have to stay with vlc then...

[MaMazav]

This branch was only a try to make a Proof Of Concept of that. I stopped working on it when understood that until the following issue in Chrmium will not be solved the minimal latency will be the difference between key frames: https://code.google.com/p/chromium/issues/detail?id=229412

[zoltan-fedor]

Thanks. I see that the resolution of this issue of MSE has been moved to Chrome version 45. I guess we will return to it then.

[RReverser]

does play (90% CPU)

That's weird - never got more that 10% on my PC.

[zoltan-fedor]

When I wrote 90% CPU, I meant 90% of a single core. Also, this is a 1920x1080 .H264 mpeg-ts feed coming from a Logitech C920 webcam which can do .H264 encoding on hardware, so I can get 1920x1080 pixel video feed out of it with no problem.

Could this be this high resolution casing the CPU to go up to 90% (and the screen become sluggish)

[RReverser]

Dunno, maybe due to high resolution. Also depends on CPU.

How much are you getting with demo on Github Pages?

[zoltan-fedor]

You are right, I get the same high CPU with the demo page too (http://rreverser.github.io/mpegts/). It basically maxes out one of the CPU and the screen is choppy. Using Chrome v43 on a Windows 7 x64.

Also tried it on a different compure (i7) in Chrome v43 on Linux Mint - the result is the same. A single core of the CPU is maxed out and the video is choppy, although less choppy than the Windows one (but this machine has a better CPU, so maybe that's why)

[RReverser]

That's really weird. Tried right now - getting 6% avg. in Firefox, 9% avg. in Chrome, both on Windows x64 machine with i7.

"choppy" is a known issue (see README) ant completely unrelated to performance, it's just how browsers handle beginning of chunk (basically same problem as with MSE).

[zoltan-fedor]

I have checked it again. The "choppy"-nes I was referring to is not the one between the chunks, but the one caused by the CPU being maxed out (it is more frequent, every second or so). Looking at the JS console, when I start playing the demo video it starts by converting 21 chunks - while the CPU maxes out - this is when there is choppiness. Then when it finishes converting the 21 chunks the video plays smooth (CPU usage is down to 5-10%), then when it runs out of chunks and need new ones then the same thing happens. It looks that converting the new chunks happens with max CPU speed - maxing out the CPU and causing the choppiness while the converting is happening.

[RReverser]

then when it runs out of chunks and need new ones then the same thing happens

Conversion never stops - initially, yes, for bufferization purposes and making playback as smooth as possible, a lot of chunks is being scheduled for conversion (although obviously only one is always converted at a time, no multithreading here except as for splitting conversion process from UI).

[RReverser]

But in any case, I do agree that scheduling could be done more wisely and I'd be totally open to any improvements here via PR.

[zoltan-fedor]

I think the question more is whether we could limit the "speed" of conversion, so it is not taking up 100% of the CPU core. I would guess, that if it is only taking up 80%, then that slight "choppiness" wouldn't be visible.

[RReverser]

It's really not about speed of conversion itself but purely about scheduling - if it wouldn't try to download & schedule conversion of 20 chunks at once, but would limit count of them to, say, 3, then CPU wouldn't be highloaded that much. As I said above, I'll be happy to accept any PRs in this area.

[zoltan-fedor]

Wouldn't the CPU load be the same, except that it would last less long, eg. converting 3 chunks instead of 20. I think it would still max out the CPU core, just for 3/20th of the time of how it is now.

[RReverser]

Nope, it won't be. As I already said, pure conversion is exactly 5-10% that you're seeing after the initial buffering (see "Conversion never stops" comment above - it's still converting under this load; 80% is just due to worker being also busy with loading a lot of binary files besides the conversion itself).

[modest]

You should expect some choppiness / increased load every time a "moov" atom (beginning of an .mp4) is appended, as this causes the H.264 decoder to reinitialize. It's necessary if you are switching video resolutions, but avoidable if you are continuing at the same quality level.

[RReverser]

@modest Interesting thought. Do you mean comparing quality with previously saved, and if it matches, omit repeated moov atom?

[modest]

@RReverser Yeah, that's usually how MPEG-DASH and Smooth Streaming are parsed.

If you stream 2 segments at 720p then 2 segments at 1080p, the buffer looks like:

[ftyp] [moov for 720p] [moof][mdat] [moof][mdat] [moov for 1080p] [moof][mdat] [moof][mdat]

[RReverser]

Sounds good. Willing to try & PR?