Implement Faster Encoder/Decoder

[josephernest]

Congrats for the project!

Import

Adding a MP4 file is very slow: I tried with a 12 MB .mp4 file.

After 2 minutes I am still at "6.45% loading...", so it would require 30+ minutes to load a small 12 MB MP4 file! What is in your opinion the reason of this slow processing?

Which browser API / algorithm do you use to import MP4 files ? What does it do exactly? Do you think there is room for improvement? Maybe ffmpeg using WASM?

Suggestion: with this method: https://stackoverflow.com/questions/4429440/html5-display-video-inside-canvas you can import a HTML5 <video> into a <canvas> with no loading time at all.

Export

Same question for export: which exporting/encoding technique do you use? Is there a ffmpeg or a browser API under the hood?

Or do you use a technique like https://webrtc.github.io/samples/src/content/capture/canvas-record/? They use here the MediaRecorder API: https://github.com/webrtc/samples/blob/gh-pages/src/content/capture/canvas-record/js/main.js

Tested platform: Windows, Chrome 101

[bwasti]

Surprisingly, Chrome is much slower than other browsers on this front (from what I've found).

The approach used is extremely naive and using FFMpeg (or similar) is a good idea.

For import, I manually pull frames in one-by-one: https://github.com/bwasti/mebm/blob/a1c131649b113a060eecf632be2059e46899dbbf/script.js#L648-L665

Export uses the native MediaRecorder as you mention (similar to frame-by-frame by it's native at least) https://github.com/bwasti/mebm/blob/a1c131649b113a060eecf632be2059e46899dbbf/script.js#L1677

I am going to go ahead and relabel this as a feature request and look into compiling a simple encoder/decoder into a small wasm blob. I'd prefer to avoid full ffmpeg as it's quite large last I checked.

[josephernest]

Thanks for your answer!

For import, I manually pull frames in one-by-one:

Is it correct that you are playing the video, and storing each frame in RAM?

Naive question, but is it really required to store each frame in RAM? I think it might not be mandatory:

• when you change the cursor timestamp and want to display the relevant frame (fixed image, not playing), you can just seek in the <video>, and ctx.drawImage(...) from the video element.
• when you want to play the video, then you can pull the frames at this time with https://stackoverflow.com/questions/4429440/html5-display-video-inside-canvas

Is it really mandatory to pull everything when adding the video?

Example: imagine the video is 5 minute long, there is no need for a 5-minute initial loading time. You can just pull the frames when they are needed for playback, just-in-time-style ;)

[bwasti]

You're 100% correct that a just-in-time solution would save a lot of compute here. I decided to use pure raw frames in RAM for a couple reasons

1. on Safari (primary testing), you can seek faster than playing the video so it wasn't a huge issue (unlike chrome, where seek seems to be slower than playing)
2. seek + display at playhead was presenting weird issues on some browsers (partial frames or just the wrong frames, I forget exactly)
3. if I wanted to edit the video at a lower level (e.g. editing colors or cropping in weird ways), I'd want a RAW layout as its a bit more intuitive to use

I think this whole thing deserves a refactor. Ideally it'd use a format that presents itself as "RAW" (individually addressable pixels at each frame) yet is actually stored in a compressed way.