search

adamrehn / pixel-streaming-linux

Issue tracker repository for Pixel Streaming for Linux
https://adamrehn.com/articles/pixel-streaming-in-linux-containers/
22 stars 7 forks source link

turnserver implementation, nvenc stream limitation #40

Closed comsyspro closed 3 years ago

comsyspro commented 3 years ago

turnserver implementation i have also another question concerning the turn server. in windows there is normally included a bundeled turnserver.exe in the project package. what would be the right way for the linux version. can you use here a coturn setup? if yes, what have to be configured inside the turnserver.conf so it can work together with the signalling server, are there any tutorials available?

nvenc stream limitation normally geforce cards like the gtx 1080 ti, rtx 2080 or rtx 3080 have a limited amount of nvenc streams (~3 per gpu). does this limitation also exist when using the pixel streaming container, or can you achieve with the containers theoretically unlimited nvenc streams?

gpu selection if you have a multi gpu server will it then be possible to select the gpu which is used for creating the nvenc stream? i saw there is a parameter in the docker run command docker run --rm --gpus=all ... but could you select here also e.g. docker run --rm --gpus=1 ... (meaning select gpu in pcie slot #1) if you have more gpus available in one server?

lcdlv commented 3 years ago

Hi @comsyspro. To solve the TURN server problem, just use this AMI in case you are on AWS: https://aws.amazon.com/marketplace/pp/Amazon-Web-Services-AWS-Deep-Learning-AMI-Ubuntu-1/B07Y43P7X5

lukehb commented 3 years ago

  1. TURN server. We have not ported the turn server that ships with the Engine to work with Linux, our recommendation would be to use Coturn as you have suggested. We have not had time to try this yet. If you do try this, I would SUPER APPRECIATE if you could share your writeup here about getting it working!

  2. NVENC limit. The container just uses the same underlying card and drivers as the host. This means the container has the exact same nvenc stream limit as the host. I'll mention that if you are using a QUADRO card (which is fairly common in the cloud) @ImmortalEmperor told me that these do not have any NVENC stream limit imposed and it is simply a matter of seeing what the ceiling really is for total number of streams (we have not had time to investigate this). But again, we would be very appreciative if you want to go and test that for us and share.

  3. I'll leave the GPU selection in a container question for @adamrehn.

lukehb commented 3 years ago

I just did some reading from the Docker docs w.r.t GPU selection. https://docs.docker.com/engine/reference/commandline/run/#access-an-nvidia-gpu

Use the device option to specify GPUs. The example below exposes a specific GPU.

docker run -it --rm --gpus device=GPU-3a23c669-1f69-c64e-cf85-44e9b07e7a2a ubuntu nvidia-smi

The example below exposes the first and third GPUs.

docker run -it --rm --gpus device=0,2 nvidia-smi

So it looks like you can easily select which GPU you want per container in Linux.

dai-martov commented 3 years ago

Hello, related to this, as I notice that there are 2 Linux binaries for stunserver and turnserver in this directory Engine/Source/ThirdParty/WebRTC/rev.24472/bin/Linux/x86_64-unknown-linux-gnu/Release

Could this be your Linux implementation that works with pixel streaming? I tried running and got the following parameters, but i'm not sure how to use it:

$ ./turnserver 
usage: turnserver int-addr ext-ip realm auth-file
lukehb commented 3 years ago

Those are not built by us, but are the old stun and turn server bundled with WebRTC M70. They probably will work, I haven't tried them. The bundled stun and turn server that ship with WebRTC, are, in my opinion only intended for example usage. I would recommend using coturn as your stun/turn server because it is built as a production-ready solution. Coturn should have plenty of docs to get you started.

https://github.com/coturn/coturn

dai-martov commented 3 years ago

Hello, I tried the following TURN server in cirrus.js, which is a public TURN server that also passes the ICE trickle test:

let clientConfig = {type: 'config', 
                    peerConnectionOptions: { 
                        "offerExtmapAllowMixed": false, 
                        iceServers: [{
                                urls: ['turn:numb.viagenie.ca'],
                                credential: 'muazkh',
                                username: 'webrtc@live.com'
                        }]
                    }}

and get the following response from the pixel streaming application:

[2021.05.31-20.48.37:821][  0]LogInit: Adding external input plugin.
[2021.05.31-20.48.37:822][  0]LogContentStreaming: Texture pool size now 1000 MB
[2021.05.31-20.48.37:827][  1]LogPixelStreamingSS: Connected to SS
[2021.05.31-20.48.37:827][  1]LogPixelStreamingSS: <- SS: {"type":"config","peerConnectionOptions":{"offerExtmapAllowMixed":false,"iceServers":[{"urls":["turn:numb.viagenie.ca"],"credential":"muazkh","username":"webrtc@live.com"}]}}
[2021.05.31-20.48.41:388][215]LogPixelStreamingSS: <- SS: {"type":"offer","sdp":"v=0\r\no=- 8374836114675864778 2 IN IP4 127.0.0.1\r\ns=-\r\nt=0 0\r\na=group:BUNDLE 0 1 2\r\na=msid-semantic: WMS\r\nm=audio 9 UDP/TLS/RTP/SAVPF 111 103 104 9 0 8 106 105 13 110 112 113 126\r\nc=IN IP4 0.0.0.0\r\na=rtcp:9 IN IP4 0.0.0.0\r\na=ice-ufrag:o8Y0\r\na=ice-pwd:aN8tJsS3u5O1klrzA+xefmsP\r\na=ice-options:trickle\r\na=fingerprint:sha-256 66:C5:66:A4:AF:CD:4C:C8:4F:66:7A:96:5F:D6:3F:93:31:1A:3D:0E:B7:63:B0:A8:7C:DD:16:55:63:87:7C:03\r\na=setup:actpass\r\na=mid:0\r\na=extmap:1 urn:ietf:params:rtp-hdrext:ssrc-audio-level\r\na=extmap:2 http://www.webrtc.org/experiments/rtp-hdrext/abs-send-time\r\na=extmap:3 http://www.ietf.org/id/draft-holmer-rmcat-transport-wide-cc-extensions-01\r\na=extmap:4 urn:ietf:params:rtp-hdrext:sdes:mid\r\na=extmap:5 urn:ietf:params:rtp-hdrext:sdes:rtp-stream-id\r\na=extmap:6 urn:ietf:params:rtp-hdrext:sdes:repaired-rtp-stream-id\r\na=recvonly\r\na=rtcp-mux\r\na=rtpmap:111 opus/48000/2\r\na=rtcp-fb:111 transport-cc\r\na=fmtp:111 minptime=10;useinbandfec=1\r\na=rtpmap:103 ISAC/16000\r\na=rtpmap:104 ISAC/32000\r\na=rtpmap:9 G722/8000\r\na=rtpmap:0 PCMU/8000\r\na=rtpmap:8 PCMA/8000\r\na=rtpmap:106 CN/32000\r\na=rtpmap:105 CN/16000\r\na=rtpmap:13 CN/8000\r\na=rtpmap:110 telephone-event/48000\r\na=rtpmap:112 telephone-event/32000\r\na=rtpmap:113 telephone-event/16000\r\na=rtpmap:126 telephone-event/8000\r\nm=video 9 UDP/TLS/RTP/SAVPF 96 97 98 99 100 101 122 102 121 127 120 125 107 108 109 35 36 124 119 123 118 114 115 116 37\r\nc=IN IP4 0.0.0.0\r\na=rtcp:9 IN IP4 0.0.0.0\r\na=ice-ufrag:o8Y0\r\na=ice-pwd:aN8tJsS3u5O1klrzA+xefmsP\r\na=ice-options:trickle\r\na=fingerprint:sha-256 66:C5:66:A4:AF:CD:4C:C8:4F:66:7A:96:5F:D6:3F:93:31:1A:3D:0E:B7:63:B0:A8:7C:DD:16:55:63:87:7C:03\r\na=setup:actpass\r\na=mid:1\r\na=extmap:14 urn:ietf:params:rtp-hdrext:toffset\r\na=extmap:2 http://www.webrtc.org/experiments/rtp-hdrext/abs-send-time\r\na=extmap:13 urn:3gpp:video-orientation\r\na=extmap:3 http://www.ietf.org/id/draft-holmer-rmcat-transport-wide-cc-extensions-01\r\na=extmap:12 http://www.webrtc.org/experiments/rtp-hdrext/playout-delay\r\na=extmap:11 http://www.webrtc.org/experiments/rtp-hdrext/video-content-type\r\na=extmap:7 http://www.webrtc.org/experiments/rtp-hdrext/video-timing\r\na=extmap:8 http://www.webrtc.org/experiments/rtp-hdrext/color-space\r\na=extmap:4 urn:ietf:params:rtp-hdrext:sdes:mid\r\na=extmap:5 urn:ietf:params:rtp-hdrext:sdes:rtp-stream-id\r\na=extmap:6 urn:ietf:params:rtp-hdrext:sdes:repaired-rtp-stream-id\r\na=recvonly\r\na=rtcp-mux\r\na=rtcp-rsize\r\na=rtpmap:96 VP8/90000\r\na=rtcp-fb:96 goog-remb\r\na=rtcp-fb:96 transport-cc\r\na=rtcp-fb:96 ccm fir\r\na=rtcp-fb:96 nack\r\na=rtcp-fb:96 nack pli\r\na=rtpmap:97 rtx/90000\r\na=fmtp:97 apt=96\r\na=rtpmap:98 VP9/90000\r\na=rtcp-fb:98 goog-remb\r\na=rtcp-fb:98 transport-cc\r\na=rtcp-fb:98 ccm fir\r\na=rtcp-fb:98 nack\r\na=rtcp-fb:98 nack pli\r\na=fmtp:98 profile-id=0\r\na=rtpmap:99 rtx/90000\r\na=fmtp:99 apt=98\r\na=rtpmap:100 VP9/90000\r\na=rtcp-fb:100 goog-remb\r\na=rtcp-fb:100 transport-cc\r\na=rtcp-fb:100 ccm fir\r\na=rtcp-fb:100 nack\r\na=rtcp-fb:100 nack pli\r\na=fmtp:100 profile-id=2\r\na=rtpmap:101 rtx/90000\r\na=fmtp:101 apt=100\r\na=rtpmap:122 VP9/90000\r\na=rtcp-fb:122 goog-remb\r\na=rtcp-fb:122 transport-cc\r\na=rtcp-fb:122 ccm fir\r\na=rtcp-fb:122 nack\r\na=rtcp-fb:122 nack pli\r\na=fmtp:122 profile-id=1\r\na=rtpmap:102 H264/90000\r\na=rtcp-fb:102 goog-remb\r\na=rtcp-fb:102 transport-cc\r\na=rtcp-fb:102 ccm fir\r\na=rtcp-fb:102 nack\r\na=rtcp-fb:102 nack pli\r\na=fmtp:102 level-asymmetry-allowed=1;packetization-mode=1;profile-level-id=42001f;x-google-start-bitrate=10000;x-google-max-bitrate=20000\r\na=rtpmap:121 rtx/90000\r\na=fmtp:121 apt=102\r\na=rtpmap:127 H264/90000\r\na=rtcp-fb:127 goog-remb\r\na=rtcp-fb:127 transport-cc\r\na=rtcp-fb:127 ccm fir\r\na=rtcp-fb:127 nack\r\na=rtcp-fb:127 nack pli\r\na=fmtp:127 level-asymmetry-allowed=1;packetization-mode=0;profile-level-id=42001f;x-google-start-bitrate=10000;x-google-max-bitrate=20000\r\na=rtpmap:120 rtx/90000\r\na=fmtp:120 apt=127\r\na=rtpmap:125 H264/90000\r\na=rtcp-fb:125 goog-remb\r\na=rtcp-fb:125 transport-cc\r\na=rtcp-fb:125 ccm fir\r\na=rtcp-fb:125 nack\r\na=rtcp-fb:125 nack pli\r\na=fmtp:125 level-asymmetry-allowed=1;packetization-mode=1;profile-level-id=42e01f;x-google-start-bitrate=10000;x-google-max-bitrate=20000\r\na=rtpmap:107 rtx/90000\r\na=fmtp:107 apt=125\r\na=rtpmap:108 H264/90000\r\na=rtcp-fb:108 goog-remb\r\na=rtcp-fb:108 transport-cc\r\na=rtcp-fb:108 ccm fir\r\na=rtcp-fb:108 nack\r\na=rtcp-fb:108 nack pli\r\na=fmtp:108 level-asymmetry-allowed=1;packetization-mode=0;profile-level-id=42e01f;x-google-start-bitrate=10000;x-google-max-bitrate=20000\r\na=rtpmap:109 rtx/90000\r\na=fmtp:109 apt=108\r\na=rtpmap:35 AV1X/90000\r\na=rtcp-fb:35 goog-remb\r\na=rtcp-fb:35 transport-cc\r\na=rtcp-fb:35 ccm fir\r\na=rtcp-fb:35 nack\r\na=rtcp-fb:35 nack pli\r\na=rtpmap:36 rtx/90000\r\na=fmtp:36 apt=35\r\na=rtpmap:124 H264/90000\r\na=rtcp-fb:124 goog-remb\r\na=rtcp-fb:124 transport-cc\r\na=rtcp-fb:124 ccm fir\r\na=rtcp-fb:124 nack\r\na=rtcp-fb:124 nack pli\r\na=fmtp:124 level-asymmetry-allowed=1;packetization-mode=1;profile-level-id=4d001f;x-google-start-bitrate=10000;x-google-max-bitrate=20000\r\na=rtpmap:119 rtx/90000\r\na=fmtp:119 apt=124\r\na=rtpmap:123 H264/90000\r\na=rtcp-fb:123 goog-remb\r\na=rtcp-fb:123 transport-cc\r\na=rtcp-fb:123 ccm fir\r\na=rtcp-fb:123 nack\r\na=rtcp-fb:123 nack pli\r\na=fmtp:123 level-asymmetry-allowed=1;packetization-mode=1;profile-level-id=64001f;x-google-start-bitrate=10000;x-google-max-bitrate=20000\r\na=rtpmap:118 rtx/90000\r\na=fmtp:118 apt=123\r\na=rtpmap:114 red/90000\r\na=rtpmap:115 rtx/90000\r\na=fmtp:115 apt=114\r\na=rtpmap:116 ulpfec/90000\r\na=rtpmap:37 flexfec-03/90000\r\na=rtcp-fb:37 goog-remb\r\na=rtcp-fb:37 transport-cc\r\na=fmtp:37 repair-window=10000000\r\nm=application 9 UDP/DTLS/SCTP webrtc-datachannel\r\nc=IN IP4 0.0.0.0\r\na=ice-ufrag:o8Y0\r\na=ice-pwd:aN8tJsS3u5O1klrzA+xefmsP\r\na=ice-options:trickle\r\na=fingerprint:sha-256 66:C5:66:A4:AF:CD:4C:C8:4F:66:7A:96:5F:D6:3F:93:31:1A:3D:0E:B7:63:B0:A8:7C:DD:16:55:63:87:7C:03\r\na=setup:actpass\r\na=mid:2\r\na=sctp-port:5000\r\na=max-message-size:262144\r\n","playerId":102}
[2021.05.31-20.48.41:390][215]PixelStreamer: FPlayerSession::FPlayerSession: PlayerId=102, quality controller: 1
[2021.05.31-20.48.41:393][215]PixelStreamer: FPlayerSession::OnRenegotiationNeeded : PlayerId=102
[2021.05.31-20.48.41:393][215]PixelStreamer: FPlayerSession::OnRenegotiationNeeded : PlayerId=102
[2021.05.31-20.48.41:393][215]PixelStreamer: FPlayerSession::OnSignalingChange : PlayerId=102, NewState=HaveRemoteOffer
[2021.05.31-20.48.41:393][215]LogPixelStreamingSS: <- SS: {"type":"iceCandidate","candidate":{"candidate":"candidate:3988902457 1 udp 2113937151 1c2e3150-cbbd-49c3-a846-fd679042ce88.local 58765 typ host generation 0 ufrag o8Y0 network-cost 999","sdpMid":"0","sdpMLineIndex":0},"playerId":102}
[2021.05.31-20.48.41:393][215]LogPixelStreamingSS: <- SS: {"type":"iceCandidate","candidate":{"candidate":"candidate:3988902457 1 udp 2113937151 1c2e3150-cbbd-49c3-a846-fd679042ce88.local 58767 typ host generation 0 ufrag o8Y0 network-cost 999","sdpMid":"1","sdpMLineIndex":1},"playerId":102}
[2021.05.31-20.48.41:395][215]PixelStreamer: FPlayerSession::OnSignalingChange : PlayerId=102, NewState=Stable
[2021.05.31-20.48.41:395][215]PixelStreamer: FPlayerSession::OnIceConnectionChange : PlayerId=102, NewState=IceConnectionChecking
[2021.05.31-20.48.41:395][215]LogPixelStreamingSS: <- SS: {"type":"iceCandidate","candidate":{"candidate":"candidate:3988902457 1 udp 2113937151 1c2e3150-cbbd-49c3-a846-fd679042ce88.local 58769 typ host generation 0 ufrag o8Y0 network-cost 999","sdpMid":"2","sdpMLineIndex":2},"playerId":102}
[2021.05.31-20.48.41:396][215]PixelStreamer: FPlayerSession::OnIceGatheringChange : PlayerId=102, NewState=IceGatheringGathering
[2021.05.31-20.48.41:396][215]PixelStreamer: FPlayerSession::OnIceCandidate : PlayerId=102
[2021.05.31-20.48.41:421][217]PixelStreamer: WebRTC VideoEncoder created, quality controller
[2021.05.31-20.48.41:421][217]LogCudaVideoEncoder: Buffer #0 (0) dropped
[2021.05.31-20.48.41:438][218]LogCudaVideoEncoder: Buffer #1 (0) dropped
[2021.05.31-20.48.41:455][219]LogCudaVideoEncoder: Buffer #2 (0) dropped
[2021.05.31-20.48.41:471][220]LogCudaVideoEncoder: Buffer #3 (0) dropped
[2021.05.31-20.48.41:488][221]LogCudaVideoEncoder: Buffer #4 (0) dropped
[2021.05.31-20.48.41:505][222]LogCudaVideoEncoder: Buffer #5 (0) dropped
[2021.05.31-20.48.41:505][222]PixelStreamer: FPlayerSession::OnIceCandidate : PlayerId=102
[2021.05.31-20.48.41:521][223]LogCudaVideoEncoder: Buffer #6 (0) dropped
[2021.05.31-20.48.41:538][224]LogCudaVideoEncoder: Buffer #7 (0) dropped
[2021.05.31-20.48.41:555][225]LogCudaVideoEncoder: Buffer #8 (0) dropped
[2021.05.31-20.48.41:571][226]LogCudaVideoEncoder: Buffer #9 (0) dropped
[2021.05.31-20.48.41:588][227]LogCudaVideoEncoder: Buffer #10 (0) dropped
[2021.05.31-20.48.41:605][228]LogCudaVideoEncoder: Buffer #11 (0) dropped
[2021.05.31-20.48.41:621][229]LogCudaVideoEncoder: Buffer #12 (0) dropped
[2021.05.31-20.48.41:638][230]LogCudaVideoEncoder: Buffer #13 (0) dropped
[2021.05.31-20.48.41:655][231]LogCudaVideoEncoder: Buffer #14 (0) dropped
[2021.05.31-20.48.41:671][232]LogCudaVideoEncoder: Buffer #15 (0) dropped
[2021.05.31-20.48.41:688][233]LogCudaVideoEncoder: Buffer #16 (0) dropped
[2021.05.31-20.48.41:705][234]LogCudaVideoEncoder: Buffer #17 (0) dropped
[2021.05.31-20.48.41:721][235]LogCudaVideoEncoder: Buffer #18 (0) dropped
[2021.05.31-20.48.41:738][236]LogCudaVideoEncoder: Buffer #19 (0) dropped
[2021.05.31-20.48.41:755][237]LogCudaVideoEncoder: Buffer #20 (0) dropped
[2021.05.31-20.48.41:771][238]LogCudaVideoEncoder: Buffer #21 (0) dropped

adding a STUN server to the urls list solve the issue:

let clientConfig = {type: 'config',
                                        peerConnectionOptions: {
                                                "offerExtmapAllowMixed": false,
                                                iceServers: [{
                                                        urls: ['turn:numb.viagenie.ca', 'stun:stun.l.google.com:19302'],
                                                        credential: 'muazkh',
                                                        username: 'webrtc@live.com'
                                                }]
                                        }}

Does it mean that it can not work with the TURN server alone? I'm not sure if it's even using the TURN server because it would still work fine with just a STUN server