How to process output.mp4 to m4s for dash.js client to play?

[howfars]

I'm trying to do research based on your paper and code.And I have a question.

I noticed that the code first merge the .m4s file and the segment_init.mp4 to an input.mp4 file,then using DNN model to process the input.mp4 to output.mp4 which cannot directly played by a dash.js client.

Can we directly extract frames from the .m4s file,then use DNN model to do super resolution,finally encode the HR frame to an "HR" .m4s file?I can't find any solution to do that on the Internet.If we can do like that,I think it will be more convenient to play the processed video chunk using dash.js.

Or how can I directly play the output.mp4 file using a dash.js client?When playing video using dash.js as client,what steps should be done to the output.mp4 file?I wonder how you solve this problem.I will appreciate it if you can give some help.

[jaykim305]

You cannot directly extract frames from .m4s file because the file lacks decoding information. The segment_init.mp4 contains the decoding information and that's why you need to merge them together before doing any extraction or decoding. You can think of segment_init.mp4 as a header file that contains information for codec.

As you said, ordinary mp4 file cannot be directly played using dash.js. That's why we added extra movflags option when encoding(see below), to make the mp4 file playable as a dash chunk.

https://github.com/kaist-ina/NAS_public/blob/0f791796d822f232a79f1f43eab76ea9ec7d4420/process.py#L347

In addition, it requires one more thing to play the chunk in dash.js. You need to change the decoding start time in the output file header to sequentially play the chunk one after another. We didn't include this to our public source code but it will not be that hard if you refer to this github issue. https://github.com/axiomatic-systems/Bento4/issues/251

[howfars]

Thank you very much for answering my question.I will try it.

[howfars]

Hi,I have tried to edit the tfdt box in the mp4 file and it works.The start time is changed by doing so. But I still don't know how to play those mp4 files using a dash.js client?How can I generate the mpd file for those mp4 files?

[jaykim305]

Sorry for the late reply. Let me briefly explain how NAS works on top of dash.js player.

• First, dash.js downloads the video chunks and keeps them in the buffer.
• Then, we send downloaded chunks (that have not been played yet) to the local SR inference server.
• After the inference, the server encodes the output SR images into video chunk with extra encoding options and changes the header (which I mentioned above) to make it playable in dash.js.
• Next, the server sends video chunk back to the dash.js browser.
• Finally, dash.js player replaces the received SR video chunk with the original one in the buffer if not yet has been played.

We modified the dash.js code to enable the process. In particular, we mostly changed the modules below in dash.js.

ScheduleController: schedules next video FragmentController, BufferController: manages the buffer

We cannot provide the entire source code but I hope this helps.