Multimedia support for fyne

[spatocode]

Providing multimedia support for fyne such as playing back audios and videos which may take advantage of the available codecs in the platform. This requires a media engine or similar. May require #225

[MarcoVdE]

I suggest relying on packages that already exist: Video: https://awesome-go.com/#video I'd suggest https://github.com/3d0c/gmf as it uses the ffmpeg libraries. Audio: https://awesome-go.com/#audio-and-music specifically I'd probably go with: https://github.com/gen2brain/malgo as it is a (unofficial) port of the miniaudio project: https://github.com/dr-soft/miniaudio

EDIT: Note https://github.com/fyne-io/fyne/projects/4#card-14426093 , think we should add this issue to the card @andydotxyz

[c1ngular]

Preferably with support of streaming protocols , such as http/rtmp/rtsp...

[Bluebugs]

Video support is problematic for license reason. Every dependencies that provide video/audio decoding is under lgpl. If we want to continue providing one binary with no dependencies, the resulting application can only be gpl. We might want to have an extra repository that provide this feature independently. Also the simplest and most efficient API to use for just a player would be libmpv in my opinion (mpv.io).

[c1ngular]

@Bluebugs true , media player support require lots of consideration and work indeed .
though libmvp stated on its github page :

mpv's focus is not on power-efficient playback on embedded or integrated GPUs (for example, hardware decoding is not even enabled by default). Low power GPUs may cause issues like tearing, stutter, etc. The main video output uses shaders for video rendering and scaling, rather than GPU fixed function hardware.

i believe fyne has support for embedded hardware/systems such as Raspberry Pi ? i am really interested to try this combination out on Raspberry Pi 4B whenever available , hardware accelerated and not .

[MarcoVdE]

Video support is problematic for license reason. Every dependencies that provide video/audio decoding is under lgpl. If we want to continue providing one binary with no dependencies, the resulting application can only be gpl. We might want to have an extra repository that provide this feature independently. Also the simplest and most efficient API to use for just a player would be libmpv in my opinion (mpv.io).

I'd go the dependency route on another project rather than building a custom solution, using e.g. gmf would allow for this repo not having to manage keeping up besides adding the latest ffmpeg library.

[Bluebugs]

I am not sure gmf answer or need after a quick look at it (might be wrong). I see two problems with it.

First it depends on ffmpeg being installed on the system as specified with the pkg-config of ffmpeg when the application has been build. This might be a problem in term of portability and will require some work to be addressed.

The second problem I see is that it doesn't seems to be designed to integrate efficiently with a canvas, but more designed for video manipulation. This is not too surprising as there isn't many UI framework in go. This might mean an important amount of work.

Also it isn't clear to me how portable a solution it is.

I think the way to go at it, is to list our requirement first and see if we can find something that help. So what I believe we need:

• portable solution (Linux, Windows, Mac, iOS and Android)
• single binary possible if the application accept to be GPL, relocatable dependencies if not a single binary for the application
• integration into an open gl canvas easily
• would be great if we can get some hardware acceleration on some platform (looking at how hard it is for Firefox to do so, I don't put that as a hard requirement)

What do you think? Any think I missed?

[c1ngular]

@Bluebugs is it a doable option to develop a new opengl integrated/optimized media player based on ffmpeg libraries ?
ffmpeg has a very solid infrastructures with a strong community/plugins/patches , it enables hardware acceleration on many platforms (even on Raspberry Pi ), so you/we don't have to start everything from scratch . IMHO, ingredients needed for such a media player is already out there , main work would be integration to fyne ?

[Bluebugs]

The difficulty is in the pipeline to manage having it setup the right way everywhere. Ffmpeg is the building block for it, but using it is non trivial. That's why things like libmpv (vlc, gstreamer, upipe, ...) exist and encapsulate it. The warning on their website is something that I do not think would be a problem for us as fyne already use shader and not a fixed pipeline. Also I would strongly recommend to use an existing abstracting layer on top of ffmpeg as it is a really hard job to get right.

[c1ngular]

@Bluebugs i am not an expert , and your point makes sense , it would be a huge boost if a solid "existing abstracting layer on top of ffmpeg" could be found and integrated .

is this work already in progress ?

[Uldiniad]

are animated gifs currently supported or would these need to be handled as well?

[andydotxyz]

Animated gifs are not fully supported, but they can be shown in Fyne with a little application code. This is possible because we can draw a single GIF frame in the normal image way. We will be adding them in fully as a result of the animation code I’m working on I hope. I’m just not sure how to expose the Loop option for them.

[zdypro888]

what's about IjkPlayer?

[andydotxyz]

what's about IjkPlayer?

Sadly no use, their GPL license would require that all our software be licensed with a more restricted open source license which we won't do.

[andydotxyz]

There is some work underway using the reisen wrapper library. We can delegate to system libraries to work around licensing issues. This is only Windows macOS and Linux at the moment, but it's a start. Simple app added over the top at https://github.com/FyshOS/movies

[beoran]

Here again, the best solution would be to port one or more liberally licensed codecs from C to Go and use that. Like https://github.com/cisco/openh264. Of course this is a considerable effort but the C to Go conversion tools are getting better.

[Bluebugs]

Software decompression is done with assembly as it is otherwise very slow. Also the logic to handle all the file format is very complex. Gstreamer or vlc have been at it for decades and are still at it. The difficulty of doing any multimedia framework is huge and should be avoided at all cost in my opinion. This is one of the case where we better use what exist already.

[beoran]

Well, but it seems someone already did it for MPEG(1). While outdated, the patents are expired and the source is very liberally licensed: https://github.com/gen2brain/mpeg

[andydotxyz]

Nice find. But these days it's not really a commonly used codec is it? If we controlled the video source as well it would be great, but I think this is more a ticket about showing user or streaming external content?

[beoran]

Well, yes, it is not so commonly used anymore because the compression is less than for modern formats. But for cases where we can control the input or record the video ourselves it might be useful to have a player that supports mpeg1.

Another approach would be to port the liberally licensed Java library jcodec from Java to Go. Not a trivial effort, but perhaps more feasible than porting C++. https://github.com/jcodec/jcodec

[Bluebugs]

I really think multi-media framework are a huge amount of work and require a lot more than just codec. I will point out at the huge and permanent effort done by project like gstreamer, ffmpeg, vlc and many other over decades literally. That I would really recommend fyne to not step into it. There is a raster object that allow for uploading pixels directly and can be used for this purpose.

The only thing that could make sense in my opinion for fyne to support are accelerated camera rendering for os/ hardware where that is possible and for h264 accelerated frame rendering again for os/hardware that support it. If the os/ hardware doesn't support it, the user of the api would have to implement its fallback strategy, maybe switching to software and raster or just saying that it isn't supported on that platform.

[hq-zhonger]

Trapped in fyne without multimedia support for a long time, the gpl protocol of ffmpeg opcv and the binary dependency free portability are also very troublesome. I have implemented a fyne video playback before. The scheme I adopted is to extract frames from mp4 files and then display them by cyclic updating of pictures, just like gif can realize video playback The problem is a lack of fluency but maybe it's a method that the author can think about feasibility

[andydotxyz]

If we can find a decoder with a suitable license we could certainly consider it.

[Bluebugs]

@hq-zhonger I would think you would be better using the raster object to have better synchronization and only push the image that need to be pushed.

[aynakeya]

I'm currently working on a project that uses mpv to play videos, and it works without any problems on both Linux and Windows platforms. (I'm not sure about mobile platforms.)

Although I need to apply a patch on fyne first. (so I can pass window handle to mpv)

example code here:

package main

import (
    "fmt"
    "fyne.io/fyne/v2"
    "fyne.io/fyne/v2/app"
    "fyne.io/fyne/v2/theme"
    "github.com/aynakeya/go-mpv"
)

type WindowHandleMagic interface {
    Wid() uintptr
}

// need to apply patch first
func GetWindowHandle(window fyne.Window) uintptr {
    x, ok := window.(WindowHandleMagic)
    if ok {
        return x.Wid()
    }
    return 0
}

func main() {
    a := app.NewWithID("io.fyne.mpvdemo")
    a.SetIcon(theme.FyneLogo())
    window := a.NewWindow("Fyne MPV Player Demo")
    var libmpv *mpv.Mpv = nil
    window.Resize(fyne.NewSize(1080, 720))
    go func() {
        wid := GetWindowHandle(window)
        for wid == 0 {
            wid = GetWindowHandle(window)
        }
        fmt.Printf("got windows handle %d\n", wid)
        libmpv = mpv.Create()
        err := libmpv.Initialize()
        if err != nil {
            panic(err)
        }
        fmt.Println(libmpv.SetOptionString("wid", fmt.Sprintf("%d", wid)))
        fmt.Println(libmpv.Command([]string{"loadfile", "/home/aynakeya/Videos/ymca.mp4"}))
    }()
    window.ShowAndRun()
    if libmpv != nil {
        libmpv.Destroy()
    }
}

patches

internal/driver/glfw/window_windows.go

func (w *window) Wid() uintptr {
    w.viewLock.RLock()
    defer w.viewLock.RUnlock()

    if w.closing {
        return 0
    }
    if (w.viewport == nil) {
        return uintptr(0)
    }
    return uintptr(w.viewport.GetWin32Window())
}

internal/driver/glfw/window_linux.go

func (w *window) Wid() uintptr {
    w.viewLock.RLock()
    defer w.viewLock.RUnlock()

    if w.closing {
        return 0
    }
    if (w.viewport == nil) {
        return uintptr(0)
    }
    return uintptr(w.viewport.GetX11Window())
}

internal/driver/glfw/window_darwin.go

package glfw

import "fyne.io/fyne/v2"

func (w *window) Wid() uintptr {
    w.viewLock.RLock()
    defer w.viewLock.RUnlock()

    if w.closing {
        return 0
    }
    if (w.viewport == nil) {
        return uintptr(0)
    }
    return uintptr(w.viewport.GetCocoaWindow())
}

[c1ngular]

@aynakeya will work on macOS too ?

[aynakeya]

@aynakeya will work on macOS too ?

theoretically yes, as long as you have proper mpv dependency installed. But I can't really tell yet, since I doesn't have a macos computer.

edit, actually, you may need to apply another patch for macos. since macos doesn't use x11.

check glfwGetCocoaWindow

[c1ngular]

@aynakeya thanks. i would love to test it if you could publish your patch . i was looking for a more natural/efficient way to play video using libvlc/libmpv with fyne, i.e play multiple video at same window same time/customize the player UI , still could not figure it out .

[aynakeya]

@aynakeya thanks. i would love to test it if you could publish your patch . i was looking for a more natural/efficient way to play video using libvlc/libmpv with fyne, i.e play multiple video at same window same time/customize the player UI , still could not figure it out .

I've updated macos patch on my previous comment. haven't test yet. you can try it to see if it works.

im not sure about whether mpv allow you to play multiple videos in a single mpv instance. but with multiple instance, it's definitely doable.

about ui part. you can control mpv video using mpv commands. here is my project for reference

[itsmontoya]

Are there any updates on this by chance?

[andydotxyz]

Are there any updates on this by chance?

Nothing beyond what is posted above. If more happens on this subject it will be added or linked to this issue.

[itsmontoya]

A lot of the discussion on this thread is about video codecs in general. It would be lovely to have a 2d rendering container where I could pipe in a video feed. More specifically, feeds directly from a camera. In addition, this container could be used for multimedia playback when the other issues are figured out and resolved.

[andydotxyz]

How can you feed directly an encoded video feed without having to consider the codec? Decoding video is the most complex (and license heavy) part of a multimedia project.

[bjorndm]

I was able to combine Ebitengine with the Go language mpeg1 decoder I mentioned above and it worked great. At least that should be easy enough to integrate with fyne as well, if we also include a sound library like Oto or such.

[andydotxyz]

Right, but the intermediate will be a series of frames yes? Matching the image.Image format? Or was some image stream type introduced to make that possible?

[bjorndm]

https://github.com/gen2brain/mpeg-examples/blob/main/player-eb/main.go

This is the example for Ebitengine. The image frames are relatively easy to get.