Add multimedia support (audio and video playback and capture)

[kkoreilly]

There are various libraries that support some of this functionality on some platforms (like faiface/beep), but a unified wrapper that works on all platforms for the aforementioned functionalities would be extremely helpful and enable the creation of more advanced apps, as mentioned in #464.

[kkoreilly]

For audio, hajimehoshi/oto probably makes the most sense, as it has good cross-platform support and is maintained. For video, using a wrapper on ffmpeg (like u2takey/ffmpeg-go) might be the best option. Also, there is a Vulkan Video API, but it isn't supported on macOS and iOS. Finally, we can also support GIFs easily through the builtin image/gif package.

[c1ngular]

How about gstreamer ? glimagesink maybe ?

[rohrlich]

I used hajimehoshi/oto in gospeech and found the author very responsive to resolving issues. The audio playback code in gospeech is out of date now I expect but I think the library is a good choice. I used ffmpeg for a lot of audio processing and it is what is used in Audacity. I found it to work well.

[kkoreilly]

The development of this has moved to https://github.com/goki/video, but I will leave this issue here as a v2 milestone item.

[rcoreilly]

direct rendering video working well now. in terms of basic playback, we just need a version of vgpu.Drawer Scale that also does simple rotations, and need to detect rotations in videos (or at least have manual setting).

It seems very fast and doesn't register much on my CPU meter on the mac -- should check on other platforms.

[rcoreilly]

also need some way of stopping the audio playback if we stop the video.

[rcoreilly]

For those who might know about these different options: the key thing we need is the ability to decode the video directly into something we can integrate with the rest of the gui elements. running an external player is not going to work. It looks like several of the available options are of that sort, in particular I looked at https://github.com/adrg/libvlc-go for libvlc and it seems to just be pulling up a player.

Several other options we looked at previously were of this form: calling executables via command line and scraping video frames from the output of the command is not a viable solution.

The current lib we're using at least has the direct api in Go wrapped around libc to get the raw video frames and audio so we can use them.

Vulkan these days has a direct hardware decoding API which would be great but isn't supported in our existing Go vulkan wrapper. We are also considering WebGPU: might it have a hardware video decoding component?

[c1ngular]

@kkoreilly @rcoreilly

as far as i know :

libmpv : it has integratabtle APIs

libvlc: it seems offer integratabtle APIs in 4.0 , but it's been delayed a long time , 3.0 is what we got so far .

ffmpeg: go-astiav is the most complete and maintained golang binding we got so far , i believe this is the place to start if we are going ffmpeg/libav way .

P.S. there are many compatibility problems to consider across different platforms if we want start from scratch , but great challenge come with great achievement and flexibility i guess .

EDIT

here's my two cents:

if we are going fine-grained controllable ffmpeg/libav way , i would suggest making composable blocks in addition to a compact video widget :

1. mediaInput package
2. decoder widget/package
3. encoder widget/package
4. render/speaker widget/package (using filter/scale)
5. mediaOutput package
6. filter package
7. scale package

with these configurable blocks as a composable pipeline for multimedia support , would introduce maximum compatibility/flexibility/performance not only to the video widget , but for other use cases such as you mentioned in yesterday's blog : a video editor for instance .

pipeline1: NewMediaInput(path , options)=>NewMediaDecoder(options,callbacks)=>NewMediaRender(options,events)=> ...

pipeline2: images/pcm data =>NewFramebuffer(options)=>NewMediaEncoder(options,callbacks)=>NewMediaOutput(path , options)

based on go-astiav and what you already have done this project , it won't be too much hassle to make a proof-of-concept experiment .

[gedw99]

https://github.com/zergon321/reisen

It might help . Gio video works well with it

I do t know why it’s discontinued but it still works well as a player and frame extractor

[rcoreilly]

reisen is the one we're currently using. go-astiav looks like a good option. @c1ngular that proposal looks reasonable to me on first glance.

[c1ngular]

I have done quite some study and experiments on this rabbit hole (golang gui and multimedia) some time , there is no easy way to make it happen or good, especially for a cross platform gui lib .

i have tried gstreamer, vlc, libmpv, libav golang bindings , IMHO ,none existing i would call it seriously GUI / general purpose/ cross platform oriented, most are just for cli app like using ffmpeg the executable in golang fashion.