search

google / wuffs

Wrangling Untrusted File Formats Safely
Other
4.18k stars 131 forks source link

Audio and video codecs #76

Open DemiMarie opened 2 years ago

DemiMarie commented 2 years ago

Being able to decode audio and video would be nice. Audio and video decoders are a well-known source of security holes.

One problem is that many existing implementations resort to hand-written assembler code. I am not sure if Wuffs can hope to match their performance.

nigeltao commented 2 years ago

Audio would be nice. FLAC and MP3 is already listed on https://github.com/google/wuffs/blob/main/doc/roadmap.md

Video is a lot more work, but, yeah, it would be nice to have something like ffmpeg with much more memory safety guarantees.

As for hand-written assembler code, Wuffs' PNG decoder already uses CPU-arch specific intrinsics, which is roughly equivalent.

DemiMarie commented 2 years ago

As for hand-written assembler code, Wuffs' PNG decoder already uses CPU-arch specific intrinsics, which is roughly equivalent.

Depends on the situation; there are cases where the compiler will do a bad job at stuff like instruction scheduling or register allocation, but I agree that asm should be a last resort. That said, making assembler code safe would be amazing, if perhaps out of scope.

ratchetfreak commented 2 years ago

Has there been any consideration for the API for these decoders.

Audio should be pretty simple, a config for communicating the sample rate, amount of channels, and bit depth. And perhaps an extension on the IO buffers to let them be used as ring buffers.

For video I don't think it's as simple as the image API augmented with audio. But I have no real experience with movie clip APIs to know what one should expect from them. Especially when you want to be able to use GPU acceleration for playback.

DemiMarie commented 2 years ago

For video I don't think it's as simple as the image API augmented with audio. But I have no real experience with movie clip APIs to know what one should expect from them. Especially when you want to be able to use GPU acceleration for playback.

I consider GPU acceleration to be out of scope for this.

nigeltao commented 2 years ago

Has there been any consideration for the API for these decoders.

I haven't done any API design work yet. I was going to do it at the same time I started on an actual FLAC or MP3 decoder, but there's other work to do first.

fire commented 2 years ago

In my research on video codex's I have found mention of this NUT mux format from ffmpeg. Maybe it is a small enough problem to be useful.

Other possible candidates for video is https://www.jonolick.com/home/mpeg-video-writer (256 lines of code!)

References

https://github.com/godotengine/godot-proposals/issues/4710 (has some nut details at the end)