Export Godot Engine movies to ffmpeg pipe (nut server)

[fire]

Describe the project you are working on

A 3d game that could benefit from streaming and recording movies.

Describe the problem or limitation you are having in your project

I want to stream the movie player output for multiple uses.

See https://github.com/godotengine/godot/pull/62122.

Describe the feature / enhancement and how it helps to overcome the problem or limitation

1. Save the game to the disk for movie editing.
2. Stream the game directly to Twitch or Youtube.
3. Avoid temporary pngs / avis' that aren't immediately ready for editing / watching

Describe how your proposal will work, with code, pseudo-code, mock-ups, and/or diagrams

Open to implementers, let the thread know if you're working on this.

TL;DR We write nut frames from Godot Engine to a ffmpeg stdin. The ffmpeg saves to the disk or ffmpeg saves to a rtsp / rtp server.

• [ ] Use binary stdin to send to ffmpeg. https://stackoverflow.com/a/21036020
• [ ] Research the nut header.
• [ ] Get the video frame from Godot Engine.
• [ ] Get the audio frame from Godot Engine.
• [ ] Add a nut header.
• [ ] Encode the video frame for nut
• [ ] Encode the audio frame for nut
• [ ] Find ffmpeg on the system via a NodePath.

  # pacman -Su mingw-w64-x86_64-ffmpeg

• [ ] Add Ref<FileAccess>> OS::exec_pipe(cmdline) api.

A future option is to convert rgba directly to yuv + a 10bit in Godot Engine. Needs profiling to determine the slow spots.

The follow options below are pretending that godot is a nut server and describes the ffmpeg saving flows.

NVIDIA Edition: Streaming video format h264 on mkv. FAST!

ffmpeg \
`# Input webm` -i sample-30s.webm \
`# Setup format` -f nut -pix_fmt rgba \
`# Pipe to stdout` - \
| \
ffmpeg \
`# Overwrite the last file` -y \
`# Use yuv4mpegpipe to recieve the formats` -f nut \
`# Read a pipe` -i - \
`# rgb8 can only be represented by yuv at 10 bit without precision loss.` \
`# https://docs.microsoft.com/en-us/windows/win32/medfound/10-bit-and-16-bit-yuv-video-formats` \
`# Encode h264` -f matroska -c:v h264_nvenc -pix_fmt yuv422p \
`# Hide banner` -hide_banner \
`# Output file` sample-30s.mkv

Intermediate video editor format dnxhd

ffmpeg \
`# Input webm` -i sample-30s.webm \
`# Setup format` -f nut -pix_fmt rgba \
`# nut supports alpha` \
`# Pipe to stdout` - \
| \
ffmpeg \
`# Overwrite the last file` -y \
`# Use yuv4mpegpipe to recieve the formats` -f nut \
`# Through a pipe` -i - \
`# rgb8 can only be represented by yuv at 10 bit without precision loss.` \
`# https://docs.microsoft.com/en-us/windows/win32/medfound/10-bit-and-16-bit-yuv-video-formats` \
`# dnxhd in ffmpeg does not support alpha` \
`# Encode dnxhd` -c:v dnxhd -vf "scale=3840:2160,fps=30/1,format=yuv422p10le" -profile:v dnxhr_hqx \
`# Hide banner` -hide_banner \
`# Output file` sample-30s.mxf

See also https://dovidenko.com/2019/999/ffmpeg-dnxhd-dnxhr-mxf-proxies-and-optimized-media.html.

Streaming video format vp9 on mkv. Too slow for most computers.

This sample is missing the 4k options

ffmpeg \
`# Input webm` -i sample-30s.webm -pix_fmt yuv422p \
`# Setup format` -f nut \
`# Pipe to stdout` - \
| \
ffmpeg \
`# Overwrite the last file` -y \
`# Use yuv4mpegpipe to recieve the formats` -f nut \
`# Read a pipe` -i - \
`# rgb8 can only be represented by yuv at 10 bit without precision loss.` \
`# https://docs.microsoft.com/en-us/windows/win32/medfound/10-bit-and-16-bit-yuv-video-formats` \
`# Encode vp9` -f matroska -c:v libvpx-vp9 -pix_fmt yuv422p  \
`# Hide banner` -hide_banner \
`# Output file` sample-30s.mkv

If this enhancement will not be used often, can it be worked around with a few lines of script?

It has soft-realtime performance constraints and cannot be gdscript.

Is there a reason why this should be core and not an add-on in the asset library?

The ffmpeg pipe can be gd extension but I wanted to type a design for review.

References

• https://ffmpeg.org/nut.html
• https://github.com/retailnext/gonut

This is the only link to the spec. https://ffmpeg.org/~michael/nut.txt

nut.txt

[fire]

Godot MIT version compatible nut was found here. https://github.com/lu-zero/nut

[jamie-pate]

Some notes:

1. Ideally as you said the output is yuv. This would reduce the cpu load for transcoding significantly if it can be generated directly by the gpu and efficiency sent back to the cpu. Even better would be some time efficient frame compression? Need to research how shadowplay/gamebar do this, but they induce nearly 0 overhead on a decent gaming machine.
2. I'm guessing the requirements for the game itself to have integrated streaming preclude the use of the windows gamebar or nvidia shadow play instead?
3. H264 isn't comparable with vp9... Should compare with vp8 for encoding speed since h264 and vp8 are from the same generation.

[fire]

I found the https://en.wikipedia.org/wiki/Network_Device_Interface spec which covers this too

[fire]

See also https://code.videolan.org/rist/librist

[patrickelectric]

any update on this ?

[Calinou]

any update on this ?

To my knowledge, nobody is currently working on implementing this.

[fire]

There's a ffmpeg c++ module from https://github.com/EIRTeam/EIRTeam.FFmpeg, but only for playback. You may be able to patch it to do export.