Using nvh264enc As Video Encoder

[tunahanertekin]

Hi there,

I intend to use nvh264enc when creating video pipeline for Neko. I tried to pass my pipeline using NEKO_VIDEO environment variable, which is:

ximagesrc display-name=%s show-pointer=true use-damage=false ! video/x-raw,framerate=30/1 ! videoconvert ! queue ! video/x-raw,format=NV12 ! nvh264enc ! video/x-h264,stream-format=byte-stream (and started Neko with --h264 argument)

I configured my environment and installed gstreamer NVENC plugin, and activated it using CUDA. I tested plugin and it worked well in my environment. However, when tried to pass this pipeline to Neko, my browser output was black screen. I used nload to see network when I open Neko on browser and saw that it uses network as it can stream properly.

How can I start Neko to use nvh264enc as video encoder? What should be my custom pipeline and other configurations? Would be appreciated if you have any suggestions on this.

[m1k1o]

Have you set correct codec in WebRTC by NEKO_H264=true? It needs to negotiate that value with browsers. Could you share your modified Dockerfile that adds NVENC plugins? I would like to try it out.

[tunahanertekin]

I only updated runtime image, currently using nvidia/opengl:1.2-glvnd-runtime-ubuntu20.04. Since it's in experimental state, I did not modified Dockerfile much and installed required packages manually (did not dockerize these steps). Here's my steps:

# configuration for my container

# nvenc autogen and build dependencies
apt-get install -y \
        software-properties-common \
    wget git make \
    build-essential \
    autogen \
    autoconf \
    autopoint \
    libtool \
    gtk-doc-tools \
    libgtk-3-dev \
    libgtop2-dev \
    librsvg2-dev \
    gstreamer-1.0 \
    libgstreamer1.0-dev \
    libgstreamer-plugins-base1.0-dev \
    libssl-dev \
        dirmngr \
        gstreamer1.0-tools

# nvidia driver installation (in my case it's nvidia 470.141.03)
# ...

# cuda
wget https://developer.download.nvidia.com/compute/cuda/11.4.0/local_installers/cuda_11.4.0_470.42.01_linux.run
sudo sh cuda_11.4.0_470.42.01_linux.run # choosing only cuda, not driver

# gstreamer nvenc plugin
cd ~
git clone --single-branch -b 1.16.2 git://anongit.freedesktop.org/gstreamer/gst-plugins-bad
cd gst-plugins-bad
./autogen.sh --with-cuda-prefix="/usr/local/cuda"
cd ./sys/nvenc
make
cp .libs/libgstnvenc.so /usr/lib/x86_64-linux-gnu/gstreamer-1.0/

# testing plugin
gst-inspect-1.0 nvenc
gst-inspect-1.0 nvh264enc

# successful x capture with nvenc
gst-launch-1.0 \
  ximagesrc remote=1 use-damage=0 ! \
  videoconvert ! \
  video/x-raw,format=NV12,framerate=60/1 ! \
  nvh264enc ! \
  filesink location=test.h264

I start all services inside Supervisor (dbus, pulseaudio etc.) except Neko. Then I initiate Neko like this:

# I prefer to build server again inside this container since it compiles C++ dependencies.
export NEKO_VIDEO="ximagesrc display-name=%s show-pointer=true use-damage=false ! video/x-raw,framerate=30/1 ! videoconvert ! queue ! video/x-raw,format=NV12 ! nvh264enc ! video/x-h264,stream-format=byte-stream"
export NEKO_H264="true"

# After starting an X session (in my case, it's ubuntu mate desktop)
/usr/bin/neko serve -d --static "/var/www" --bind "0.0.0.0:$NEKO_BIND"   --epr "$NEKO_UDP_PORT" --display ":0" --h264

When I try to use openh264enc, x264enc or vp8enc, it works properly. Strange thing is nvh264enc seems to be properly configured, but I cannot see nothing but a black screen on browser.

And here's my logs:

ser@new-neko-1:/src$ ./bin/neko serve -d --static "/var/www" --bind "0.0.0.0:$NEKO_BIND"   --epr "$NEKO_UDP_PORT" --display ":0"

    _   __     __
   / | / /__  / /______   \    /\
  /  |/ / _ \/ //_/ __ \   )  ( ')
 / /|  /  __/ ,< / /_/ /  (  /  )
/_/ |_/\___/_/|_|\____/    \(__)|
 nurdism/m1k1o server v2.6.0 dev
11:58AM WRN preflight complete without config file config= debug=true logging=false
11:58AM INF starting neko server service=neko
11:58AM WRN invalid screen option 1280x720@30 module=remote
11:58AM INF webrtc starting ephemeral_port_range=59011-59011 ice_lite=false ice_servers="[{URLs:[stun:stun.l.google.com:19302] Username: Credential:<nil> CredentialType:password}]" module=webrtc nat_ips=44.210.16.3
11:58AM WRN http listening on 0.0.0.0:58011 module=http
11:58AM INF neko ready service=neko
11:58AM DBG request complete (304) module=http req={"agent":"Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:103.0) Gecko/20100101 Firefox/103.0","id":"new-neko-1/ah41Eikht3-000001","method":"GET","proto":"HTTP/1.1","remote":"37.202.48.107:35754","scheme":"http","uri":"http://44.210.16.3:58011/"} res={"bytes":0,"elapsed":0.111523,"status":304,"time":"Tue, 16 Aug 2022 11:58:28 UTC"}
11:58AM DBG request complete (200) module=http req={"agent":"Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:103.0) Gecko/20100101 Firefox/103.0","id":"new-neko-1/ah41Eikht3-000002","method":"GET","proto":"HTTP/1.1","remote":"37.202.48.107:35754","scheme":"http","uri":"http://44.210.16.3:58011/keyboard_layouts.json"} res={"bytes":1657,"elapsed":3.433456,"status":200,"time":"Tue, 16 Aug 2022 11:58:28 UTC"}
11:58AM DBG request complete (200) module=http req={"agent":"Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:103.0) Gecko/20100101 Firefox/103.0","id":"new-neko-1/ah41Eikht3-000003","method":"GET","proto":"HTTP/1.1","remote":"37.202.48.107:35752","scheme":"http","uri":"http://44.210.16.3:58011/emoji.json"} res={"bytes":107528,"elapsed":164.328966,"status":200,"time":"Tue, 16 Aug 2022 11:58:28 UTC"}
11:58AM DBG attempting to upgrade connection module=websocket
11:58AM INF Pipelines starting... audio_codec=Opus audio_device=auto_null.monitor audio_pipeline_src="pulsesrc device=auto_null.monitor ! audio/x-raw,channels=2 ! audioconvert ! opusenc inband-fec=true bitrate=128000 ! appsink name=appsink" module=remote screen_resolution=1280x720@30 video_codec=H264 video_display=:0 video_pipeline_src="ximagesrc display-name=:0 show-pointer=true use-damage=false ! video/x-raw,framerate=30/1 ! videoconvert ! queue ! video/x-raw,format=NV12 ! nvh264enc ! video/x-h264,stream-format=byte-stream ! appsink name=appsink"
error: XDG_RUNTIME_DIR not set in the environment.
11:58AM INF signaling state changed to have-local-offer module=webrtc subsystem=pc
11:58AM DBG sending message to client address=37.202.48.107:35756 module=websocket raw="{\"event\":\"signal/provide\",\"id\":\"umww-Pllx5EbIBEEL9avBj6Ft3c_M9wY\",\"sdp\":\"v=0\\r\\no=- 4000312261513551189 1660651108 IN IP4 0.0.0.0\\r\\ns=-\\r\\nt=0 0\\r\\na=fingerprint:sha-256 76:5B:E8:2C:E7:1E:8C:62:C4:7D:F5:A5:2C:EF:9F:25:6A:C7:A9:12:A4:42:A4:BD:78:C8:B2:78:49:20:83:FC\\r\\na=extmap-allow-mixed\\r\\na=group:BUNDLE 0 1 2\\r\\nm=video 9 UDP/TLS/RTP/SAVPF 102\\r\\nc=IN IP4 0.0.0.0\\r\\na=setup:actpass\\r\\na=mid:0\\r\\na=ice-ufrag:SrhpVniqpcRJotrl\\r\\na=ice-pwd:quodRLvOxOXbLnJRAuAaKydOWnqvDiSB\\r\\na=rtcp-mux\\r\\na=rtcp-rsize\\r\\na=rtpmap:102 H264/90000\\r\\na=fmtp:102 level-asymmetry-allowed=1;packetization-mode=1;profile-level-id=42e01f\\r\\na=rtcp-fb:102 nack \\r\\na=rtcp-fb:102 nack pli\\r\\na=rtcp-fb:102 transport-cc \\r\\na=extmap:1 http://www.ietf.org/id/draft-holmer-rmcat-transport-wide-cc-extensions-01\\r\\na=ssrc:619207984 cname:stream\\r\\na=ssrc:619207984 msid:stream video\\r\\na=ssrc:619207984 mslabel:stream\\r\\na=ssrc:619207984 label:video\\r\\na=msid:stream video\\r\\na=sendrecv\\r\\nm=audio 9 UDP/TLS/RTP/SAVPF 111\\r\\nc=IN IP4 0.0.0.0\\r\\na=setup:actpass\\r\\na=mid:1\\r\\na=ice-ufrag:SrhpVniqpcRJotrl\\r\\na=ice-pwd:quodRLvOxOXbLnJRAuAaKydOWnqvDiSB\\r\\na=rtcp-mux\\r\\na=rtcp-rsize\\r\\na=rtpmap:111 opus/48000/2\\r\\na=fmtp:111 useinbandfec=1\\r\\na=rtcp-fb:111 transport-cc \\r\\na=extmap:1 http://www.ietf.org/id/draft-holmer-rmcat-transport-wide-cc-extensions-01\\r\\na=ssrc:2548456624 cname:stream\\r\\na=ssrc:2548456624 msid:stream audio\\r\\na=ssrc:2548456624 mslabel:stream\\r\\na=ssrc:2548456624 label:audio\\r\\na=msid:stream audio\\r\\na=sendrecv\\r\\nm=application 9 UDP/DTLS/SCTP webrtc-datachannel\\r\\nc=IN IP4 0.0.0.0\\r\\na=setup:actpass\\r\\na=mid:2\\r\\na=sendrecv\\r\\na=sctp-port:5000\\r\\na=ice-ufrag:SrhpVniqpcRJotrl\\r\\na=ice-pwd:quodRLvOxOXbLnJRAuAaKydOWnqvDiSB\\r\\n\",\"lite\":false,\"ice\":[{\"credentialType\":\"password\",\"urls\":[\"stun:stun.l.google.com:19302\"]}]}" session=umww-Pllx5EbIBEEL9avBj6Ft3c_M9wY
11:58AM DBG sending message to client address=37.202.48.107:35756 module=websocket raw="{\"event\":\"system/init\",\"implicit_hosting\":false,\"locks\":{}}" session=umww-Pllx5EbIBEEL9avBj6Ft3c_M9wY
11:58AM DBG sending message to client address=37.202.48.107:35756 module=websocket raw="{\"event\":\"signal/candidate\",\"data\":\"{\\\"candidate\\\":\\\"candidate:1604336667 1 udp 2130706431 44.210.16.3 59011 typ host\\\",\\\"sdpMid\\\":\\\"\\\",\\\"sdpMLineIndex\\\":0,\\\"usernameFragment\\\":null}\"}" session=umww-Pllx5EbIBEEL9avBj6Ft3c_M9wY
11:58AM DBG session created id=umww-Pllx5EbIBEEL9avBj6Ft3c_M9wY module=websocket
11:58AM DBG new connection created address=37.202.48.107:35756 module=websocket session=umww-Pllx5EbIBEEL9avBj6Ft3c_M9wY
11:58AM WRN could not get server reflexive address udp6 stun:stun.l.google.com:19302: write udp6 [::]:59011->[2607:f8b0:4002:c03::7f]:19302: sendto: cannot assign requested address module=webrtc subsystem=ice
11:58AM DBG failed to listen :59011: listen udp4 :59011: bind: address already in use module=webrtc subsystem=ice
11:58AM WRN Conn is not allocated (Failed to listen for 142.250.9.127:19302: invalid port) module=webrtc subsystem=ice
11:58AM INF sent all ICECandidates module=webrtc
11:58AM DBG received message from client address=37.202.48.107:35756 module=websocket raw="{\"event\":\"signal/answer\",\"sdp\":\"v=0\\r\\no=mozilla...THIS_IS_SDPARTA-99.0 4828060222489780978 0 IN IP4 0.0.0.0\\r\\ns=-\\r\\nt=0 0\\r\\na=fingerprint:sha-256 37:85:36:7A:F6:EE:09:B3:0B:64:FD:5D:8F:DE:1A:71:72:6D:45:03:95:51:14:D1:F2:9A:92:4B:CC:A3:E9:4C\\r\\na=group:BUNDLE 0 1 2\\r\\na=ice-options:trickle\\r\\na=msid-semantic:WMS *\\r\\nm=video 9 UDP/TLS/RTP/SAVPF 102\\r\\nc=IN IP4 0.0.0.0\\r\\na=recvonly\\r\\na=extmap:1 http://www.ietf.org/id/draft-holmer-rmcat-transport-wide-cc-extensions-01\\r\\na=fmtp:102 profile-level-id=42e01f;level-asymmetry-allowed=1;packetization-mode=1\\r\\na=ice-pwd:61a2da024951387f9b5eee7e56f150df\\r\\na=ice-ufrag:d4bc7505\\r\\na=mid:0\\r\\na=rtcp-fb:102 nack\\r\\na=rtcp-fb:102 nack pli\\r\\na=rtcp-fb:102 transport-cc\\r\\na=rtcp-mux\\r\\na=rtcp-rsize\\r\\na=rtpmap:102 H264/90000\\r\\na=setup:active\\r\\na=ssrc:2632103539 cname:{7d7b85b4-dd8b-4ae5-a75a-9e754937a1c8}\\r\\nm=audio 9 UDP/TLS/RTP/SAVPF 111\\r\\nc=IN IP4 0.0.0.0\\r\\na=recvonly\\r\\na=fmtp:111 maxplaybackrate=48000;stereo=1;useinbandfec=1\\r\\na=ice-pwd:61a2da024951387f9b5eee7e56f150df\\r\\na=ice-ufrag:d4bc7505\\r\\na=mid:1\\r\\na=rtcp-mux\\r\\na=rtpmap:111 opus/48000/2\\r\\na=setup:active\\r\\na=ssrc:3549603747 cname:{7d7b85b4-dd8b-4ae5-a75a-9e754937a1c8}\\r\\nm=application 9 UDP/DTLS/SCTP webrtc-datachannel\\r\\nc=IN IP4 0.0.0.0\\r\\na=sendrecv\\r\\na=ice-pwd:61a2da024951387f9b5eee7e56f150df\\r\\na=ice-ufrag:d4bc7505\\r\\na=mid:2\\r\\na=setup:active\\r\\na=sctp-port:5000\\r\\na=max-message-size:1073741823\\r\\n\",\"displayname\":\"neko\"}" session=umww-Pllx5EbIBEEL9avBj6Ft3c_M9wY
11:58AM INF signal update - RemoteAnswer id=umww-Pllx5EbIBEEL9avBj6Ft3c_M9wY module=session
11:58AM INF signaling state changed to stable module=webrtc subsystem=pc
11:58AM DBG Started agent: isControlling? true, remoteUfrag: "d4bc7505", remotePwd: "61a2da024951387f9b5eee7e56f150df" module=webrtc subsystem=ice
11:58AM INF Setting new connection state: Checking module=webrtc subsystem=ice
11:58AM DBG adding a new peer-reflexive candidate: 37.202.48.107:54996  module=webrtc subsystem=ice
11:58AM INF ICE connection state changed: checking module=webrtc subsystem=pc
11:58AM INF connection state has changed connection_state=checking module=webrtc
11:58AM INF Setting new connection state: Connected module=webrtc subsystem=ice
11:58AM INF ICE connection state changed: connected module=webrtc subsystem=pc
11:58AM INF connection state has changed connection_state=connected module=webrtc
11:58AM DBG CipherSuite not initialized, queuing packet module=webrtc subsystem=dtls
11:58AM DBG received packet of next epoch, queuing packet module=webrtc subsystem=dtls
11:58AM INF peer connection state changed: connected module=webrtc subsystem=pc
11:58AM INF peer connected id=umww-Pllx5EbIBEEL9avBj6Ft3c_M9wY module=webrtc
11:58AM DBG sending message to client address=37.202.48.107:35756 module=websocket raw="{\"event\":\"member/list\",\"members\":[{\"id\":\"umww-Pllx5EbIBEEL9avBj6Ft3c_M9wY\",\"displayname\":\"neko\",\"admin\":false,\"muted\":false}]}" session=umww-Pllx5EbIBEEL9avBj6Ft3c_M9wY
11:58AM DBG sending message to client address=37.202.48.107:35756 module=websocket raw="{\"event\":\"screen/resolution\",\"width\":1920,\"height\":1080,\"rate\":50}" session=umww-Pllx5EbIBEEL9avBj6Ft3c_M9wY
11:58AM DBG sending message to client address=37.202.48.107:35756 module=websocket raw="{\"event\":\"member/connected\",\"id\":\"umww-Pllx5EbIBEEL9avBj6Ft3c_M9wY\",\"displayname\":\"neko\",\"admin\":false,\"muted\":false}" session=umww-Pllx5EbIBEEL9avBj6Ft3c_M9wY
11:58AM DBG session connected id=umww-Pllx5EbIBEEL9avBj6Ft3c_M9wY module=websocket

I can share if you need any other output.

[tunahanertekin]

Also I can monitor Neko process when I run nvidia-smi on my container, so it uses GPU when encoding video but cannot get desktop on browser.

[m1k1o]

I now remember having the same issue when trying it, switching screen size while logged in and watching fixed it. It was just workaround, I did not have chance to find the real cause of that issue.

[tunahanertekin]

I would like to try this, debug the situation and share if I can understand the problem. What do you mean by switching screen size, is it something I should do after starting Neko process? I would be glad if you can elaborate it a little more.

[m1k1o]

I mean this switch. That restarts the pipeline, and it only seems to be working, when pipeline (re)starts after webrtc listener is provided. Maybe it sends some initialization (key) frames, that browser needs to have.

[tunahanertekin]

When I build current code or run one of the current images (m1k1o/neko:xfce for example), I couldn't see resolution options on the top right of the screen. Isn't this feature available in latest versions?

[m1k1o]

No, it's been there for more than 2 years. But it is only visible for admins. Default user password is neko and admin password is admin.

[tunahanertekin]

It worked well, thank you so much! I'm closing this issue for now since my custom pipeline that uses nvh264enc worked. I will post here if I discover the reason of this issue.

[nums]

Hello @tunahanertekin, could you please share your custom pipeline, I am facing the same issue, thanks !

[tunahanertekin]

My custom pipeline is:

gst-launch-1.0 \
ximagesrc display-name=%s show-pointer=true use-damage=false ! \
video/x-raw,framerate=25/1 ! \
videoconvert ! \
queue ! \
video/x-raw,format=NV12 ! \
nvh264enc bitrate=%d rc-mode=1 preset=0 ! \
video/x-h264,stream-format=byte-stream ! \
appsink name=appsink

[tunahanertekin]

Black screen at the start is still a problem. I think that the problem is related with XGetScreenConfigurations() function in xorg.c. When I try to generate and use an xorg.conf with nvidia-xconfig command to make Xorg use GPU, it causes some kind of an override and this function doesn't return resolutions' framerates right. (probably an indexing problem) So I solved it with a workaround on code as below.

First, I updated screen.go. I made setting screen resolution possible for non-admin users.

func (h *MessageHandler) screenSet(id string, session types.Session, payload *message.ScreenResolution) error {

        // ***Admin control is removed.***

    if err := h.remote.ChangeResolution(payload.Width, payload.Height, payload.Rate); err != nil {
        h.logger.Warn().Err(err).Msgf("unable to change screen size")
        return err
    }

    if err := h.sessions.Broadcast(
        message.ScreenResolution{
            Event:  event.SCREEN_RESOLUTION,
            ID:     id,
            Width:  payload.Width,
            Height: payload.Height,
            Rate:   payload.Rate,
        }, nil); err != nil {
        h.logger.Warn().Err(err).Msgf("broadcasting event %s has failed", event.SCREEN_RESOLUTION)
        return err
    }

    return nil
}

Then I updated session.go. I replaced screenResolution function with screenSet and fixed a resolution at start. (This will probably break Neko's --screen argument)

func (h *MessageHandler) SessionConnected(id string, session types.Session) error {
    // send list of members to session
    if err := session.Send(message.MembersList{
        Event:    event.MEMBER_LIST,
        Memebers: h.sessions.Members(),
    }); err != nil {
        h.logger.Warn().Str("id", id).Err(err).Msgf("sending event %s has failed", event.MEMBER_LIST)
        return err
    }

    // send screen current resolution
    // ***replaced with screen/set instead of screen/resolution because of the nvidia black screen problems***
    if err := h.screenSet(id, session, &message.ScreenResolution{
        Event:  event.SCREEN_RESOLUTION,
        ID:     id,
        Width:  1920,
        Height: 1080,
        Rate:   50,
    }); err != nil {
        return err
    }

    // tell session there is a host
    host, ok := h.sessions.GetHost()
    if ok {
        if err := session.Send(message.Control{
            Event: event.CONTROL_LOCKED,
            ID:    host.ID(),
        }); err != nil {
            h.logger.Warn().Str("id", id).Err(err).Msgf("sending event %s has failed", event.CONTROL_LOCKED)
            return err
        }
    }

    // let everyone know there is a new session
    if err := h.sessions.Broadcast(
        message.Member{
            Event:  event.MEMBER_CONNECTED,
            Member: session.Member(),
        }, nil); err != nil {
        h.logger.Warn().Err(err).Msgf("broadcasting event %s has failed", event.CONTROL_RELEASE)
        return err
    }

    return nil
}

Finally, I updated video.ts. I disabled filtering for framerates. (since they come broken when using Nvidia)

    for (const i of Object.keys(configurations)) {
      const { width, height, rates } = configurations[i]
      if (width >= 600 && height >= 300) {
        for (const j of Object.keys(rates)) {
          const rate = rates[j]
            // ***filtering framerates disabled
            data.push({
              width,
              height,
              rate,
            })
        }
      }
    }

ps. I will add XGetScreenConfigurations() function's output at xorg.c when I will be able to try it.

[nums]

Many thanks @tunahanertekin I will test today, if I make some progress about Screen dimensions I will do some update here.

[m1k1o]

@tunahanertekin I still cannot fix the blackscreen problem. It has actually nothing to do with XGetScreenConfigurations you only need to restart gstreamer:

• when gstreamer pipeline is running and somebody joins that -> blackscreen
• when somebody joins stopped pipeline and then gstreamer starts -> works
• when somebody joins started pipeline and then changes screen size, it restarts gstreamer pipeline -> works

This is working properly with h264 and VP8 CPU encoding, only for nvenc it causes problems. Usually when user joins and keyframe distance is big, he sees blacksceen until the first keyframe arrives. I checked for nvenc, it sends properly keyframes but it does not start.

Do you have any news for this topic?

[tunahanertekin]

@m1k1o yes. I have confirmed that it works if the cases you mentioned on the bullet list happen. I was passively looking for a workaround that will restart the gstreamer, digging the server code. However, the server still sends incorrect configurations (resolutions and fps) to the client as far as I see. For example, it sends 30 possible configurations but only those screen configurations seems to work when using an xorg.conf that includes NVIDIA:

• 2048x1152@50
• 1600x1200@50

Other than that, I tried to update client as it'll send screen/set after every connection (member/connected) as a workaround but sometimes I cannot get screen/resolution after connection, so it is not a verified method too.

I will try the solution you mentioned here and share my experience here. Thanks for the update!

[tunahanertekin]

Hi again!

I've managed to run Neko using nvenc as the video encoder using the option --hwenc. Thanks for the support!

Additionally, I tried to configure xorg.conf that uses dummy objects to make it use NVIDIA GPU device as well. However, I couldn't manage to run X with hardware acceleration (the only option was generating xorg.conf after running the container with nvidia-xconfig, and that option breaks the resolutions). Naturally, most of the GUI application I launched couldn't use GPU by default because of that.

Do you have any suggestions about configuring xorg.conf such that X uses GPU device?

Thanks in advance. Have a good day!