Integrate video saving with GStreamer

[boulabiar]

Describe the project you are working on: I am testing using Godot as a video animation tool. I put everything in place but I want to export the stream as a video (or live stream). A tablet can be connected to the camera and physically use its position to move into the scene and select the view.

Describe the problem or limitation you are having in your project: Godot now can't live stream the video without exporting frame by frame in gdscript which takes a lot of resources.

Describe the feature / enhancement and how it helps to overcome the problem or limitation: Having a plugin exporting the video into gstreamer will help remove these limitations.

Describe how your proposal will work, with code, pseudocode, mockups, and/or diagrams: Godot only need to send the stream into gstreamer, which maybe can be done through a gstreamer plugin in godot. Once the stream is sent to gstreamer, it can be easily modified or mixed or saved. gstreamer also uses hardware acceleration for encoding/decoding available on any recent CPU. here is a gstreamer bash command on linux to save the screen into a file, and it uses Intel hardware acceleration so the CPU usage keeps being low. gst-launch-1.0 ximagesrc use-damage=0 ! video/x-raw,framerate=30/1 ! videoscale method=0 ! video/x-raw,width=1920,height=1080 ! videoconvert ! vaapih264enc ! matroskamux ! filesink location=record.mkv

If this enhancement will not be used often, can it be worked around with a few lines of script?: gscript exporting may results in huge usage in resources if exported frame by frame.

Is there a reason why this should be core and not an add-on in the asset library?: I am not a godot developer, but maybe it's more efficient to work on it from the renderer directly instread of copying the frame multiple times between the modules until reaching the desired output. If the device allows it, the saved video stream can be done in resolutions higher that what the screen supports.

[groud]

Please fill in the issue template, it is mandatory to ensure the quality of the proposals in this repository. I will reopen when the issue template is correctly filled in.

[boulabiar]

I filled the issue as requested, it's clear from the number of sentences. Why you think it's not? because there are no titles?

[boulabiar]

I added the titles.

[groud]

Why you think it's not? because there are no titles?

Yes, the titles should be kept. Thanks :)

[Calinou]

We can't integrate GStreamer into Godot's core since it's licensed under the LGPL. This license doesn't allow static linking when used in proprietary software, which is problematic for Godot-exported projects as they are exported into a single binary. Moreover, iOS doesn't even provide a form of dynamic linking.

Therefore, this must be done as a third-party module or fork somehow.

[Sslaxx]

Are there no non-(L)GPL alternatives to GStreamer?

[boulabiar]

It's not a problem that GStreamer itself is LGPL. What is needed to be done is a gstreamer plugin, which can be plugged into godot dynamically, and that creates the frame and send it to the gstreamer pipeline (then gstreamer will handle everything else)

Many ways to be done are possible for this, and example source plugins can be taken as a reference. their name look like this: src https://gstreamer.freedesktop.org/documentation/additional/design/element-source.html

• This one takes a creates an OpenGL context and sends the frame into gstreamer: Doc: https://gstreamer.freedesktop.org/documentation/opengl/gltestsrc.html Code: https://github.com/GStreamer/gst-plugins-base/blob/master/ext/gl/gltestsrc.c

• This one do the same but without OpenGL: Doc: https://gstreamer.freedesktop.org/documentation/videotestsrc/index.html Code: https://github.com/GStreamer/gst-plugins-base/blob/master/gst/videotestsrc/gstvideotestsrc.c

• There is also applicationsrc, which I haven't tested yet (I think I should because it's the way to go). From an Application, this creates frames and send them into gstreamer Documentation: https://gstreamer.freedesktop.org/documentation/app/appsrc.html Code: https://github.com/GStreamer/gst-plugins-base/blob/master/gst-libs/gst/app/gstappsrc.c

If you look into documentation, there is some sort of inheritance so you inherit from gstappsrc, and you only modify the function that fills the frame with information and that's it.

[boulabiar]

Most car industry uses gstreamer for cars media, Nvidia also found a ways to plug into it to create their deepstream sdk: https://github.com/NVIDIA-AI-IOT/redaction_with_deepstream So it has already a large set of video acceleration plugins that connect to embedded or classic hardware gpu/cpu/... either to encode the video using hardware codecs then either to save a file or to send the video into network. OpenCV is BSD and connects to gstreamer too https://github.com/opencv/opencv/blob/master/modules/videoio/src/cap_gstreamer.cpp

[Calinou]

What is needed to be done is a gstreamer plugin, which can be plugged into godot dynamically, and that creates the frame and send it to the gstreamer pipeline (then gstreamer will handle everything else)

It sounds like this is a job for a third-party C++ module or GDNative add-on :slightly_smiling_face:

Therefore, I'll close this proposal since there hasn't been much support for it.

Edit (February 2022): This may be related: https://www.reddit.com/r/godot/comments/siwxhg/ffmpeg_streaming_frames_to_godot_textures_through/