search

AirenSoft / OvenMediaEngine

OvenMediaEngine (OME) is a Sub-Second Latency Live Streaming Server with Large-Scale and High-Definition. #WebRTC #LLHLS
https://OvenMediaEngine.com/ome
GNU Affero General Public License v3.0
2.54k stars 1.06k forks source link

Load Balancer OME with nginx, maybe a OME BUG #597

Closed alnux closed 2 years ago

alnux commented 2 years ago

I am trying to use nginx as a load balancer for OME (webrtc) but ovenplayer tells me it tells me it has failed

WebRTCLoader.js: 622 WebSocket connection to 'ws: //sub.domain.com: 8081 / conectar / endirecto / stream_1280' failed

The question is that I already have it configured and I do not know what the problem is, or nginx does not work with OME

nginx.conf

http {
.
.
.
.

upstream servers_webrtc {
        server origen01.domain.com:3333 max_conns=200;
       server edge01.domain.com:3333 max_conns=200;
    #i've trayed with ip too
}
.
.
.
server {
    listen      <public ip>:8081;
    server_name sub.domain.com ;

    #modsecurity on;
    #modsecurity_rules_file /etc/nginx/modsec/main.conf;

location /conectar/ {
            # prevents 502 bad gateway error
            proxy_buffers 8 32k;
            proxy_buffer_size 64k;
            proxy_pass http://servers_webrtc;

            # enables WS support
            proxy_http_version 1.1;
            proxy_set_header Upgrade $http_upgrade;
            proxy_set_header Connection "upgrade";
            proxy_read_timeout 999999999;
        }

}

my origin webrtc is ws://origen01.domain.com:3333/endirecto/stream_1280/ ws://origen01.domain.com:3333/endirecto/stream_640/ and my edge is ws://edge01.domain.com:3333/endirecto/stream_1280/ ws://edge01.domain.com:3333/endirecto/stream_640/

i will apreciate some of help, and if nginx does not work with OME, which peasy to use Load Balancer would you recommend me with a small example .. thank you very much.

as well with hls,

really thanks

dbContext commented 2 years ago

@alnux You're connecting to port 3333/TCP although your nginx configuration is binding/listening to 8081/TCP

alnux commented 2 years ago

@dbContext hi, THE upstream Is going to 3333, I'm using 8081 becuse y have nginx production on 80 and apache on 8080 . THE thing Is that Is working now but udp (10000 port) does not proxy i think that THE way Is using stream directiva, mmmm something with lights on this?? Thanks

getroot commented 2 years ago

After the player has negotiated with a specific server in the signaling phase, the player and the selected server directly exchange stream data after that.

More specifically, in the signaling phase (WebSocket), the server delivers its Candidate (Server's IP/Port) to the Player, and the Player communicates directly with the corresponding IP/Port. (The settings are IceCandidate and TcpRelay (optional) in Server.xml.)

Therefore, you only need to load balancing at the signaling stage. Now you are normal. (Note that RTP cannot be proxyed by nginx (only HTTP), and even if that is possible, it is a worse structure if all traffic goes through nginx.)

alnux commented 2 years ago

@getroot I understand, but how do I make OME connect with the player because my nginx configuration is the following and it connects but the return message from the browser in the socket is 404

"code": 404, "error": "Cannot create offer"}

as you can see on screenshot

https://prnt.sc/23aqgt7

but the header of this return the 101 code status, as you can see screenshot too

https://prnt.sc/23ar3uk

My settings of IceCandidate and TcpRelay are the same as default installation ( by the way the doc url does not work https://prnt.sc/23arpji )

<IceCandidates>
                                        <!-- 
                                                If you want to stream WebRTC over TCP, specify IP:Port for TURN server.
                                                This uses the TURN protocol, which delivers the stream from the built-in TURN server to the player'$
$-publishing#webrtc-over-tcp
                                        -->
                                        <TcpRelay>*:3478</TcpRelay>
                                        <!-- TcpForce is an option to force the use of TCP rather than UDP in WebRTC streaming. (You can omit ?tran$
                                        <TcpForce>true</TcpForce>

                                        <IceCandidate>*:10000/udp</IceCandidate>
                                        <TcpRelayWorkerCount>1</TcpRelayWorkerCount>
                                </IceCandidates>

and my nginx is this very basic and connect

..
..
..
upstream servers_webrtc {
        server origen01.nubehits.com:3333 max_conns=200; #one server for the example
        }
..
...
..
..

location /conectar/ {
         # enables WS support
        proxy_pass http://servers_webrtc;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
        }

...
...
..
dbContext commented 2 years ago

@alnux The initial WSS negotiation through your nginx load balancer looks to be working as expected, it seems to be unable to establish a WebRTC connection directly to the streaming server.

Depending on the interface/NIC configuration on the server that you're running OME on, it may be unable to get a public IPv4 address. I'd suggest replacing <IceCandidate>*:10000/udp</IceCandidate> with <IceCandidate>xxx.xxx.xxx.xxx:10000/udp</IceCandidate> (xxx.xxx.xxx.xxx being the public facing IPv4 address of the OME server.)

If this is within an automated deployment, you can setup a bash script on startup that get's the public facing address via a IP API service, then update the config with pattern matching, and finally restart the ovenmediaengine service.

dbContext commented 2 years ago

@alnux Also, as you're forcing TCP, I'd do the same to the TcpRelay binding as well.

getroot commented 2 years ago

The screenshot you posted is not showing the screenshot (probably an error).

Anyway, "code": 404, "error": "Cannot create offer" is an error that occurs when OME cannot find the stream you requested. A websocket connection has been established (101). A 404 error is an error code that the OME application responds to.

Check what url nginx will call to OME. That's probably the problem.

alnux commented 2 years ago

thanks @dbContext the server has public ip i did the changes that you say but the problem is still there (can you see the screenshots that @getroot cant?).

Well as you can saw on my last config the real url (all in ws ) of my OME is

ws://origen01.nubehits.com:3333/endirecto/stream_1280/

an the nginx load balancer will be and (im using) it is

ws://ws.nubehits.com/conectar/endirecto/stream_1280/

and the OME log says that cant find virtualHost, why??

[2021-12-17 11:32:22.659] I [SPRtcSig-T3333:9205] Signalling | rtc_signalling_server.cpp:201  | New client is connected: <ClientSocket: 0x7f49580010f0, #28, Connected, TCP, Nonblocking, 207.244.240.88:35684>
[2021-12-17 11:32:22.659] W [SPRtcSig-T3333:9205] Orchestrator | orchestrator.cpp:266  | Could not find VirtualHost for domain: servers_webrtc
[2021-12-17 11:32:22.921] E [SPRtcSig-T3333:9205] AccessController | access_controller.cpp:119  | Could not resolve the domain: servers_webrtc
[2021-12-17 11:32:22.921] E [SPRtcSig-T3333:9205] Signalling | rtc_signalling_server.cpp:292  | Cannot find stream [##conectar/endirecto]
[2021-12-17 11:32:22.921] I [SPRtcSig-T3333:9205] Signalling | rtc_signalling_server.cpp:339  | Client is disconnected: <WebSocketClient: 0x7f49580038f0, <ClientSocket: 0x7f49580010f0, #28, Closed, TCP, Nonblocking, 207.244.240.88:35684>> (##conectar / endirecto, ufrag: local: (N/A), remote: (N/A))

is is traying to get load balancer domain (servers_webrtc)?? but it is sending to origin01.nubehits.com (and it works perfect) across proxy_pass

here a pease of my virtualhost Server.xml config

<VirtualHost>
                        <Name>origen01.nubehits.com</Name>
                        <!--Distribution is a value that can be used when grouping the same vhost distributed across multiple servers. This value i$
                        <Distribution>nubehits.com</Distribution>

                        <!-- Settings for multi ip/domain and TLS -->
                        <Host>
                                <Names>
                                        <Name>origen01.nubehits.com</Name>
                                        <Name> ws.nubehits.com</Name> <!-- this i add for the error log, but does noot work too -->
                                </Names>
                                <TLS>
                                        <CertPath>/home/nubehitsorigen01/conf/web/origen01.nubehits.com/ssl/origen01.nubehits.com.crt</CertPath>
                                        <KeyPath>/home/nubehitsorigen01/conf/web/origen01.nubehits.com/ssl/origen01.nubehits.com.key</KeyPath>
                                        <ChainCertPath>/home/nubehitsorigen01/conf/web/origen01.nubehits.com/ssl/origen01.nubehits.com.crt</ChainCe$
                                </TLS>

                        </Host>

....
<aplication>
....
....
...
</virtualHost>

regards

dbContext commented 2 years ago

@alnux You're not forwarding the Host Header when sending the proxy_pass, add proxy_set_header Host $host; to the location directive below proxy_pass http://servers_webrtc;

alnux commented 2 years ago

@dbContext yes you are.

Well for those who want to implement nginx as load balancer they have to have the following configuration.

Server.xml

....
.....
.......
<VirtualHost>
                        <Name>omeserver.domain.com</Name>
                        <!--Distribution is a value that can be used when grouping the same vhost distributed across multiple servers. This value i$
                        <Distribution>nubehits.com</Distribution>

                        <!-- Settings for multi ip/domain and TLS -->
                        <Host>
                                <Names>
                                        <Name>omeserver.domain.com</Name>
                                        <Name>loadbalancer.domain.com</Name>
                                </Names>
                                <!--
                                <TLS>
                                        <CertPath>path/to/file.crt</CertPath>
                                        <KeyPath>path/to/file.key</KeyPath>
                                        <ChainCertPath>path/to/file.crt</ChainCertPath>
                                </TLS>
                                -->
                        </Host>
....
.....
....
.....

then the load balancer has to look like this

nginx.conf

http {
.....
.....
.....

upstream servers_webrtc {
      upstream origen01.nubehits.com {
            server omeserver.domain.com:3333 max_conns=200;
            server edge.domain.com:3333 max_conns=200;
      }

server {
    ........
     ........
    location /webrtc/ {
         # enables WS support
        rewrite ^/webrtc(.*)$ $1 break;
        proxy_pass http://servers_webrtc;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
        proxy_set_header Host $host;
        }

    .........
   ..........
    .........

}

thanks to @dbContext and @getroot

alnux commented 2 years ago

dbContext hi there again, on SSL OME does not keep the signal or connect an disconnect, the OME log is

[2021-12-17 14:35:48.207] I [SPRtcSig-T3333:9490] Signalling | rtc_signalling_server.cpp:201  | New client is connected: <ClientSocket: 0x7f2088021ae0, #17, Connected, TCP, Nonblocking, 207.244.240.88:42802>
[2021-12-17 14:35:48.348] I [SPRtcSig-T3333:9490] Signalling | rtc_signalling_server.cpp:339  | Client is disconnected: <WebSocketClient: 0x7f2088001050, <ClientSocket: 0x7f2088021ae0, #17, Disconnected, TCP, Nonblocking, 207.244.240.88:42802>> (#edge01.nubehits.com#endirecto / stream_1280, ufrag: local: (N/A), remote: (N/A))

the thing is i just put SSL on domain and call it by wss, the outup on browser just finish with fail websocket and the request headers

Browser output WebRTCLoader.js:622 WebSocket connection to 'wss://nubehits.com/webrtc/endirecto/stream_1280/' failed:

Request header

Provisional headers are shown
Learn more
Accept-Encoding: gzip, deflate, br
Accept-Language: es,en-US;q=0.9,en;q=0.8
authorization: basic YWNjZXNvLWEtYXBpLW92ZW5lbWRpYWVuZ2luZQ==
Cache-Control: no-cache
Connection: Upgrade
Cookie: wp-settings-time-1=1638318030; wp-settings-1=libraryContent%3Dbrowse; _pk_id.4.2895=48d9905a02c764d4.1638462414.
Host: nubehits.com
Origin: https://nubehits.com
Pragma: no-cache
Sec-WebSocket-Extensions: permessage-deflate; client_max_window_bits
Sec-WebSocket-Key: mxOQBTAsFzrN/sRwvrGUQQ==
Sec-WebSocket-Version: 13
Upgrade: websocket
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/94.0.4606.85 YaBrowser/21.11.0.2054 Yowser/2.5 Safari/537.36
dbContext commented 2 years ago

@alnux how have you gone about terminating the SSL connection? Please provide configuration examples of both the nginx LB and OME service.

alnux commented 2 years ago

of course @dbContext , here my nginx.conf

upstream servers_webrtc {   
      Server origen01.nubehits.com:3334 max_cons=200; #here is going to wss  
      server edge01.nubehits.com:3333 max_conns=200; #here is going to ws
        }

server {
    listen      207.244.240.88:443 ssl http2;
    server_name nubehits.com www.nubehits.com;
    ssl_certificate      /home/nubehits/conf/web/nubehits.com/ssl/nubehits.com.pem;
    ssl_certificate_key  /home/nubehits/conf/web/nubehits.com/ssl/nubehits.com.key;
    ssl_stapling on;
    ssl_stapling_verify on;
    error_log  /var/log/apache2/domains/nubehits.com.error.log error;

    modsecurity on;
    modsecurity_rules_file /etc/nginx/modsec/main.conf;

    include /home/nubehits/conf/web/nubehits.com/nginx.hsts.conf*;

   ....
   ....
  .....
   .....
   .....

    proxy_hide_header Upgrade;

    location /webrtc/ { #here is the OME passing
        modsecurity off;
        rewrite ^/webrtc(.*)$ $1 break;
        proxy_pass http://servers_webrtc;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
        proxy_set_header Host $host;
        }

here you have Origin Server the name of this is origin01.nubehits.com

<VirtualHosts>
                <!-- You can use wildcard like this to include multiple XMLs -->
                <VirtualHost include="VHost*.xml" />
                <VirtualHost>
                        <Name>origen01.nubehits.com</Name>
                        <!--Distribution is a value that can be used when grouping the same vhost distributed across multiple servers. This value i$
                        <Distribution>nubehits.com</Distribution>

                        <!-- Settings for multi ip/domain and TLS -->
                        <Host>
                                <Names>
                                         <Name>origen01.nubehits.com</Name>
                                        <Name>ws.nubehits.com</Name>
                                        <Name>nubehits.com</Name>
                                </Names>

                                <TLS>
                                        <CertPath>/home/nubehitsorigen01/conf/web/origen01.nubehits.com/ssl/origen01.nubehits.com.crt</CertPath>
                                        <KeyPath>/home/nubehitsorigen01/conf/web/origen01.nubehits.com/ssl/origen01.nubehits.com.key</KeyPath>
                                        <ChainCertPath>/home/nubehitsorigen01/conf/web/origen01.nubehits.com/ssl/origen01.nubehits.com.crt</ChainCe$
                                </TLS>

                        </Host>

and here you have edge Name edge01.nubehits.com

 <VirtualHost>
                        <Name>edge01.nubehits.com</Name>
                        <!--Distribution is a value that can be used when grouping the same vhost distributed across multiple servers. This value i$
                        <Distribution>nubehits.com</Distribution>

                        <!-- Settings for multi ip/domain and TLS -->
                        <Host>
                                <Names>
                                        <Name>*.nubehits.com</Name>
                                </Names>
                        </Host>

the certificate on origin is for origen01.nubehits.com domain, and on nubehits is load balancer

Client ->nubehits.com(wss://) -> proxy (origin01.nubehits.com:3334)-> origin01.nubehits.com(wss) Client ->nubehits.com(wss://) -> proxy (edge01.nubehits.com:3333)-> edge01.nubehits.com(ws)

both show me the OME error that i write here

edge

[2021-12-17 14:35:48.207] I [SPRtcSig-T3333:9490] Signalling | rtc_signalling_server.cpp:201  | New client is connected: <ClientSocket: 0x7f2088021ae0, #17, Connected, TCP, Nonblocking, 207.244.240.88:42802>
[2021-12-17 14:35:48.348] I [SPRtcSig-T3333:9490] Signalling | rtc_signalling_server.cpp:339  | Client is disconnected: <WebSocketClient: 0x7f2088001050, <ClientSocket: 0x7f2088021ae0, #17, Disconnected, TCP, Nonblocking, 207.244.240.88:42802>> (#edge01.nubehits.com#endirecto / stream_1280, ufrag: local: (N/A), remote: (N/A))

origin

[2021-12-17 16:05:53.351] I [SPRtcSig-T3333:6879] Signalling | rtc_signalling_server.cpp:201  | New client is connected: <ClientSocket: 0x7fe65c002bf0, #28, Connected, TCP, Nonblocking, 207.244.240.88:39574>
[2021-12-17 16:05:53.494] I [SPRtcSig-T3333:6879] Signalling | rtc_signalling_server.cpp:339  | Client is disconnected: <WebSocketClient: 0x7fe66c0124e0, <ClientSocket: 0x7fe65c002bf0, #28, Disconnected, TCP, Nonblocking, 207.244.240.88:39574>> (#origen01.nubehits.com#endirecto / stream_1280, ufrag: local: (N/A), remote: (N/A))
dbContext commented 2 years ago

@alnux I'm slightly confused with your load balancing configuration, you're distributing traffic to both edge and origin at the same entry point? Are you using your origin (RTMP ingest?) as an edge server as well?

There are inconsistencies with your TLS configuration on origin / edge servers, yet you're distributing to them equally so you're going to get different results based on what server it chooses send the traffic too.

You're also still sending proxy_pass via http:// which isn't really an issue, if you're just wanting to terminate the TLS connection on the load balancer, however you're then configuring TLS on the OME servers.

alnux commented 2 years ago

hi @getroot, I was breaking my head seeing why WEBRTC does not work over SSL because in non-secure it works normally. so I went to HLS to do the same, configure it over SSL and it was not problematic at all, it worked at the first time. Returning to WEBRTC, it connects with OME but instantly disconnected and the worst case is that the log does not release much information about what happens, it only connects and disconnects, here the log.

[2021-12-18 15:44:32.920] I [SPRtcSig-T3334:11525] Signalling | rtc_signalling_server.cpp:201  | New client is connected: <ClientSocket: 0x7f6ea4025cc0, #27, Connected, TCP, Nonblocking, 207.244.240.88:34764>
[2021-12-18 15:44:33.067] I [SPRtcSig-T3334:11525] Signalling | rtc_signalling_server.cpp:339  | Client is disconnected: <WebSocketClient: 0x7f6ef0016cd0, <ClientSocket: 0x7f6ea4025cc0, #27, Disconnected, TCP, Nonblocking, 207.244.240.88:34764>> (#origen01.nubehits.com#endirecto / stream_1280, ufrag: local: (N/A), remote: (N/A))

and here the HLS logs

[2021-12-18 15:44:33.324] I [SegWorker:11532] HTTPPublisher | segment_publisher.cpp:188  | [#origen01.nubehits.com#endirecto/stream_1280] Segment requested 1613.ts from 207.244.240.88:46710 : Segment number : 1613 Duration : 5
[2021-12-18 15:44:33.324] I [SegWorker:11532] Monitor | stream_metrics.cpp:116  | A new session has started playing #origen01.nubehits.com#endirecto/stream on the HLS publisher. HLS(3)/Stream total(3)/App total(3)
[2021-12-18 15:44:34.723] I [SegWorker:11532] HTTPPublisher | segment_publisher.cpp:188  | [#origen01.nubehits.com#endirecto/stream_1280] Segment requested 1614.ts from 207.244.240.88:46714 : Segment number : 1614 Duration : 5
[2021-12-18 15:44:35.491] I [SegWorker:11532] HTTPPublisher | segment_publisher.cpp:188  | [#origen01.nubehits.com#endirecto/stream_1280] Segment requested 1615.ts from 207.244.240.88:46718 : Segment number : 1615 Duration : 5
[2021-12-18 15:44:36.956] I [SegWorker:11532] HTTPPublisher | segment_publisher.cpp:188  | [#origen01.nubehits.com#endirecto/stream_1280] Segment requested 1616.ts from 207.244.240.88:46722 : Segment number : 1616 Duration : 5

it make me thing that is a bug, maybe (the server are in USA maybe delay)

if helps here a SSL url player of wss and hls https://nubehits.com/player/player.extend.demo.php, please looks with dev tools becouse it autogoes to HLS

and here a normal URL of WS of the same signal frome same OME origin Server http://ws.nubehits.com/player/player.extendido.php

the Server.xml OME conf is next

<VirtualHost include="VHost*.xml" />
                <VirtualHost>
                        <Name>origen01.nubehits.com</Name>
                        <!--Distribution is a value that can be used when grouping the same vhost distributed across multiple servers. This value i$
                        <Distribution>nubehits.com</Distribution>

                        <!-- Settings for multi ip/domain and TLS -->
                        <Host>
                                <Names>
                                        <Name>origen01.nubehits.com</Name>
                                        <Name>ws.nubehits.com</Name>
                                        <Name>nubehits.com</Name>

                                </Names>
                                <TLS>
                                        <CertPath>/home/nubehitsorigen01/conf/web/origen01.nubehits.com/ssl/origen01.nubehits.com.crt</CertPath>
                                        <KeyPath>/home/nubehitsorigen01/conf/web/origen01.nubehits.com/ssl/origen01.nubehits.com.key</KeyPath>
                                        <ChainCertPath>/home/nubehitsorigen01/conf/web/origen01.nubehits.com/ssl/origen01.nubehits.com.crt</ChainCe$
                                </TLS>

                        </Host>

now as I told you, in nginx I tried various configurations even with the proxy_ssl directives, but with HLS it worked at the first time so that leads me to say maybe it is a bug. As I told you and I showed you the information in the log it does not say much

if helps here my nginx config

SECURE (SSL) WSS and HLS

location /webrtc/ {
        modsecurity off;
        rewrite ^/webrtc(.*)$ $1 break;
        proxy_pass https://servers_webrtc;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
        proxy_set_header Host $host;
        proxy_ssl_server_name      on;
        proxy_ssl_name $host;
        proxy_ssl_certificate      /home/nubehits/conf/web/nubehits.com/ssl/nubehits.com.pem;
        proxy_ssl_certificate_key  /home/nubehits/conf/web/nubehits.com/ssl/nubehits.com.key;

        }
location /hls/ {
        modsecurity off;
        rewrite ^/hls(.*)$ $1 break;
        proxy_pass https://servers_hls;
        proxy_set_header Host $host;
        }

The http ws Nginx Config Works normally

location /conectar/ {
         # enables WS support
        rewrite ^/conectar(.*)$ $1 break;
#       proxy_pass http://origen01.nubehits.com;
        proxy_pass http://servers_webrtc_http;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
        proxy_set_header Host $host;
        }

the Nginx Load Balancer config

upstream servers_webrtc_http {
        server origen01.nubehits.com:3333 max_conns=200;
#       server edge01.nubehits.com:3334 max_conns=200;
        }

upstream servers_webrtc {
        server origen01.nubehits.com:3334 max_conns=200;
#       server edge01.nubehits.com:3334 max_conns=200;
        }
upstream servers_hls{
        server origen01.nubehits.com:4443 max_conns=200;
#       server edge01.nubehits.com:4443 max_conns=200;
}

right now the urls that i paste here are working with the config files that i share, and @dbContext thanks for your help too. sorry about my bad spelling.

dbContext commented 2 years ago

@alnux You're not understanding how the protocols / connections work.

You shouldn't be load balancing the WebRTC (RTC) connection, You should only be load balancing the initial negotiation which you have named "conectar" for the web socket connection, once a web socket connection has been established then OME will negotiate a RTP connection with the browser, and the browser will connect directly to the OME server for a persistent RTP connection.

If you're load balancing the RTP connection, then it won't persist to a single server, and will not establish a full connection.

so keep the "connectar" directive, bin the webrtc one and use TLS as you'd expect for WSS, and you can keep the hls one if you wish as that's just normal http/https.

alnux commented 2 years ago

@dbContext i know @getroot told me that before.

Therefore, you only need to load balancing at the signaling stage. Now you are normal. (Note that RTP cannot be proxyed by nginx (only HTTP), and even if that is possible, it is a worse structure if all traffic goes through nginx.)

the thing is the on https://domain.com the wss:// WebRtc does not work. you tell me keep "conectar" directive, but if i do that show the next error

[2021-12-18 19:00:34.149] E [SPRtcSig-T3334:11525] OpenSSL | tls.cpp:193  | An error occurred while accept SSL connection: [OpenSSL] error:0A00009C:SSL routines::http request (167772316)
[2021-12-18 19:00:34.149] E [SPRtcSig-T3334:11525] OpenSSL | tls_server_data.cpp:108  | An error occurred while accept TLS connection: error code: 1

that is the way i put https:// on proxy_pass directive 

location /webrtc/ {
        modsecurity off;
        rewrite ^/webrtc(.*)$ $1 break;
        proxy_pass https://servers_webrtc;       #     <--------- HERE
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
        proxy_set_header Host $host;
#       proxy_ssl_server_name      on;  
#       proxy_ssl_name $host;
#       proxy_ssl_certificate      /home/nubehits/conf/web/nubehits.com/ssl/nubehits.com.pem;
#       proxy_ssl_certificate_key  /home/nubehits/conf/web/nubehits.com/ssl/nubehits.com.key;

        }
[2021-12-18 19:09:53.106] I [SPRtcSig-T3334:11525] Signalling | rtc_signalling_server.cpp:201  | New client is connected: <ClientSocket: 0x7f6edc023810, #28, Connected, TCP, Nonblocking, 207.244.240.88:40880>
[2021-12-18 19:09:53.349] I [SPRtcSig-T3334:11525] Signalling | rtc_signalling_server.cpp:339  | Client is disconnected: <WebSocketClient: 0x7f6ef0034550, <ClientSocket: 0x7f6edc023810, #28, Disconnected, TCP, Nonblocking, 207.244.240.88:40880>> (#origen01.nubehits.com#endirecto / stream_1280, ufrag: local: (N/A), remote: (N/A))

just two lines of log :(

there is no more differences, from "conectar" directive as you tell me (use it), that resume the error that i put before by log, that is the way that i tell that maybe is a bug, It is very frustrating, but thank you very much for your interest @dbContext . I hope you do not get tired before finding a solution, or someone else here has already used a load balancer in nginx with OME?

dbContext commented 2 years ago

@alnux Configure the load balancer so it is terminating the TLS connection, as you already are, but do NOT configure TLS/SSL on the OME server, or forward on the proxy as https, simply send it all as http traffic, as the load balancer is already terminating the TLS, there is no real need to re-encrypt the traffic, causing potential issues when it is being sent.

Also, what is the deal with you sending the traffic to both origin01 and edge01? this makes no sense, unless the name is just "incorrect" and they both serve as edges. You only want to be load balancing servers that will serve traffic to the browser/client on the WSS side of things.

getroot commented 2 years ago

@alnux I'm not familiar with NGINX. I usually distribute traffic on the edge via a DNS server (GSLB).

alnux commented 2 years ago

@dbConnect I think we are both kicking stones, as I told you as http: // it works without problems but if I call it from https: // (i have to use necessarily) first i have to call to wss:// you know if i call just ws:// the browser tell is insecure bla bla bla

As you can see in the previous messages, the load balancer has both types of configurations but on https:// url it does not work, with or without configuring load balancer with SSL / TLS ... just HLS That is why I say that maybe it is a bug that was not detected, it would be interesting to check it; Unfortunately the log only shows me those two connection and disconnection lines.

that is the way that on docs just recommend use round robin DNS :( :( :(

but any way is a great software

dbContext commented 2 years ago

@alnux I think there is some miscommunication here with what I'm asking, I've used NGINX with OME before without issue.

Send me your entire configs for NGINX and OME, and I'll re-configure them to work and send back..

alnux commented 2 years ago

hi @dbContext Sending you all my nginx configuration is a bit difficult it is distributed in different places on my server (hestiacp).

Now the reason why I call origin01.nubehits.com which you call edge is because I use it as rtmp origin and as edge, I will use it only as origen after I get more servers You say that previously you used it without problems so it may be a version problem so I will describe the versions that I have

nginx version: nginx/1.21.4 csf: v14.15 (generic) OvenMediaEngine v0.12.9 OpenSSL 1.1.1 11 Sep 2018

well nginx.conf

# Server globals
user                    www-data;
worker_processes        auto;
worker_rlimit_nofile    65535;
error_log               /var/log/nginx/error.log;
pid                     /var/run/nginx.pid;

include /etc/nginx/modules-enabled/*.conf;

# Worker config
events {
        worker_connections  1024;
        use                 epoll;
        multi_accept        on;
}

http {
    # Main settings
    sendfile                        on;
    tcp_nopush                      on;
    tcp_nodelay                     on;
    client_header_timeout           180s;
    client_body_timeout             180s;
    client_header_buffer_size       2k;
    client_body_buffer_size         256k;
    client_max_body_size            256m;
    large_client_header_buffers     4 8k;
    send_timeout                    60s;
    keepalive_timeout               30s;
    keepalive_requests              100000;
    reset_timedout_connection       on;
    server_tokens                   off;
    server_name_in_redirect         off;
    server_names_hash_max_size      512;
    server_names_hash_bucket_size   512;
    charset                         utf-8;

    # FastCGI settings
    fastcgi_buffers                 4 256k;
    fastcgi_buffer_size             256k;
    fastcgi_busy_buffers_size       256k;
    fastcgi_temp_file_write_size    256k;
    fastcgi_connect_timeout         30s;
    fastcgi_read_timeout            300s;
    fastcgi_send_timeout            180s;
    fastcgi_cache_lock              on;
    fastcgi_cache_lock_timeout      5s;
    fastcgi_cache_background_update on;
    fastcgi_cache_revalidate        on;
# Proxy settings
    proxy_redirect                  off;
    proxy_set_header                Host $host;
    proxy_set_header                X-Real-IP $remote_addr;
    proxy_set_header                X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_pass_header               Set-Cookie;
    proxy_buffers                   32 4k;
    proxy_connect_timeout           30s;
    proxy_read_timeout              300s;
    proxy_send_timeout              180s;

    # Log format
    log_format  main    '$remote_addr - $remote_user [$time_local] $request '
                        '"$status" $body_bytes_sent "$http_referer" '
                        '"$http_user_agent" "$http_x_forwarded_for"';
    log_format  bytes   '$body_bytes_sent';
    log_not_found off;
    access_log off;

    # Mime settings
    include             /etc/nginx/mime.types;
    default_type        application/octet-stream;

    # Compression
    gzip                on;
    gzip_static         on;
    gzip_vary           on;
    gzip_comp_level     6;
    gzip_min_length     1024;
    gzip_buffers        16 8k;
    gzip_http_version   1.1;
    gzip_types          text/plain text/css text/javascript text/js text/xml application/json application/javascript application/x-javascript application/xml application/xml+rss application/x-font-ttf image/svg+xml font/opentype;
    gzip_proxied        any;
    gzip_disable        "MSIE [1-6]\.";

    brotli on;
    brotli_comp_level 6;
    brotli_static on;
    brotli_types application/atom+xml application/javascript application/json application/rss+xml application/vnd.ms-fontobject application/x-font-opentype application/x-font-truetype application/x-font-ttf application/x-javascript application/xhtml+xml application/xml font/eot font/opentype font/otf font/truetype image/svg+xml image/vnd.microsoft.icon image/x-icon image/x-win-bitmap text/css text/javascript text/plain text/xml;

    # Cloudflare https://www.cloudflare.com/ips
    set_real_ip_from 103.21.244.0/22;
    set_real_ip_from 103.22.200.0/22;
set_real_ip_from 103.22.200.0/22;
    set_real_ip_from 103.31.4.0/22;
    set_real_ip_from 104.16.0.0/13;
    set_real_ip_from 104.24.0.0/14;
    set_real_ip_from 108.162.192.0/18;
    set_real_ip_from 131.0.72.0/22;
    set_real_ip_from 141.101.64.0/18;
    set_real_ip_from 162.158.0.0/15;
    set_real_ip_from 172.64.0.0/13;
    set_real_ip_from 173.245.48.0/20;
    set_real_ip_from 188.114.96.0/20;
    set_real_ip_from 190.93.240.0/20;
    set_real_ip_from 197.234.240.0/22;
    set_real_ip_from 198.41.128.0/17;
    #set_real_ip_from  2400:cb00::/32;
    #set_real_ip_from  2405:b500::/32;
    #set_real_ip_from  2606:4700::/32;
    #set_real_ip_from  2803:f800::/32;
    #set_real_ip_from  2c0f:f248::/32;
    #set_real_ip_from  2a06:98c0::/29;
    real_ip_header     CF-Connecting-IP;

    # SSL PCI compliance
    ssl_session_cache   shared:SSL:20m;
    ssl_session_timeout 60m;
    ssl_buffer_size     1400;
    ssl_protocols       TLSv1.2 TLSv1.3;
    ssl_prefer_server_ciphers on;
    ssl_ciphers         "ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA:ECDHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-RSA-AES256-SHA256:DHE-RSA-AES256-SHA:ECDHE-ECDSA-DES-CBC3-SHA:ECDHE-RSA-DES-CBC3-SHA:EDH-RSA-DES-CBC3-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:DES-CBC3-SHA:!DSS";
    ssl_dhparam         /etc/ssl/dhparam.pem;
    ssl_ecdh_curve      secp384r1;
    ssl_session_tickets off;
    resolver 127.0.0.53  valid=300s ipv6=off;
    resolver_timeout    5s;

    # Error pages
    error_page 403 /error/404.html;
    error_page 404 /error/404.html;
    error_page 410 /error/410.html;
    error_page 500 501 502 503 504 505 /error/50x.html;

    # Proxy cache
    proxy_cache_path /var/cache/nginx levels=2 keys_zone=cache:10m inactive=60m max_size=1024m;
    proxy_cache_key "$host$request_uri $cookie_user";
    proxy_temp_path  /var/cache/nginx/temp;
proxy_temp_path  /var/cache/nginx/temp;
    proxy_ignore_headers Expires Cache-Control;
    proxy_cache_use_stale error timeout invalid_header http_502;
    proxy_cache_valid any 1d;

    # FastCGI cache
    fastcgi_cache_path /var/cache/nginx/micro levels=1:2 keys_zone=microcache:10m max_size=1024m inactive=30m;
    fastcgi_cache_key "$scheme$request_method$host$request_uri";
    fastcgi_cache_methods GET HEAD;
    fastcgi_cache_use_stale updating error timeout invalid_header http_500 http_503;
    fastcgi_ignore_headers Cache-Control Expires Set-Cookie;
    add_header X-FastCGI-Cache $upstream_cache_status;

    # Cache bypass
    map $http_cookie $no_cache {
        default 0;
        ~SESS 1;
        ~wordpress_logged_in 1;
    }

    # File cache (static assets)
    open_file_cache          max=10000 inactive=30s;
    open_file_cache_valid    60s;
    open_file_cache_min_uses 2;
    open_file_cache_errors   off;

    # Wildcard include
    include /etc/nginx/conf.d/*.conf;
    include /etc/nginx/conf.d/domains/*.conf;
}

####### the LoadBalancer file

/etc/nginx/conf.d/upstream.nubehits.com.conf

upstream servers_webrtc_http {
        server origen01.nubehits.com:3333 max_conns=200;
#       server edge02.nubehits.com:3334 max_conns=200;
        }

upstream servers_webrtc {
        server origen01.nubehits.com:3334 max_conns=200;
#       server edge02.nubehits.com:3334 max_conns=200;
        }
upstream servers_hls{
        server origen01.nubehits.com:4443 max_conns=200;
#       server edge02.nubehits.com:4443 max_conns=200;
}

As i have a multi domains on same server first the one that is on insecure url that is working http://ws.nubehits.com (demo)

ws.nubehits.com.nginx.conf

server {
    listen      207.244.240.88:80;
    server_name ws.nubehits.com ;

    include /home/nubehits/conf/web/ws.nubehits.com/nginx.forcessl.conf*;
    modsecurity on;
    modsecurity_rules_file /etc/nginx/modsec/main.conf;

    location / {
        include /home/nubehits/conf/web/ws.nubehits.com/nginx.modsec.conf*;
        proxy_pass      http://207.244.240.88:8080;
        location ~* ^.+\.(jpeg|jpg|png|gif|bmp|ico|svg|tif|tiff|css|js|htm|html|ttf|otf|webp|woff|txt|csv|rtf|doc|docx|xls|xlsx|ppt|pptx|odf|odp|ods|odt|pdf|psd|ai|eot|eps|ps|zip|tar|tgz|gz|rar|bz2|7z|aac|m4a|mp3|mp4|ogg|wav|wma|3gp|avi|flv|m4v|mkv|mov|mpeg|mpg|wmv|exe|iso|dmg|swf)$ {
            root           /home/nubehits/web/ws.nubehits.com/public_html;
            access_log     /var/log/apache2/domains/ws.nubehits.com.log combined;
            access_log     /var/log/apache2/domains/ws.nubehits.com.bytes bytes;
            expires        max;
            try_files      $uri @fallback;
        }
    }

    location /error/ {
        alias   /home/nubehits/web/ws.nubehits.com/document_errors/;
    }

    location @fallback {
        proxy_pass      http://207.244.240.88:8080;
    }

    location ~ /\.(?!well-known\/) { 
       deny all; 
       return 404;
    }

    include /home/nubehits/conf/web/ws.nubehits.com/nginx.conf_*;   #<----nginx.conf_OME_nubehits_proxy
}

here the ws.nubehits.com.nginx.conf --> nginx.conf_OME_nubehits_proxy

location /conectar/ {
         # enables WS support
        rewrite ^/conectar(.*)$ $1 break;       
        proxy_pass http://servers_webrtc_http;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
        proxy_set_header Host $host;
        }

Now the problem SSL/TLS issue that only works HLS but not WSS https://nubehits (demo)

server {
    listen      207.244.240.88:443 ssl http2;
    server_name nubehits.com www.nubehits.com;
    ssl_certificate      /home/nubehits/conf/web/nubehits.com/ssl/nubehits.com.pem;
    ssl_certificate_key  /home/nubehits/conf/web/nubehits.com/ssl/nubehits.com.key;
    ssl_stapling on;
    ssl_stapling_verify on;
    error_log  /var/log/apache2/domains/nubehits.com.error.log error;

    modsecurity on;
    modsecurity_rules_file /etc/nginx/modsec/main.conf;

    include /home/nubehits/conf/web/nubehits.com/nginx.hsts.conf*;

    location / {
        include /home/nubehits/conf/web/nubehits.com/nginx.modsec.conf*;

        proxy_pass      https://207.244.240.88:8443;
        location ~* ^.+\.(jpeg|jpg|png|gif|bmp|ico|svg|tif|tiff|css|js|htm|html|ttf|otf|webp|woff|txt|csv|rtf|doc|docx|xls|xlsx|ppt|pptx|odf|odp|ods|odt|pdf|psd|ai|eot|eps|ps|zip|tar|tgz|gz|rar|bz2|7z|aac|m4a|mp3|mp4|ogg|wav|wma|3gp|avi|flv|m4v|mkv|mov|mpeg|mpg|wmv|exe|iso|dmg|swf)$ {
            root           /home/nubehits/web/nubehits.com/public_html;
            access_log     /var/log/apache2/domains/nubehits.com.log combined;
            access_log     /var/log/apache2/domains/nubehits.com.bytes bytes;
            expires        max;
            try_files      $uri @fallback;
        }
    }

    location /error/ {
        alias   /home/nubehits/web/nubehits.com/document_errors/;
    }

    location @fallback {
        proxy_pass      https://207.244.240.88:8443;
    }

    location ~ /\.(?!well-known\/) { 
       deny all; 
       return 404;
    }

    proxy_hide_header Upgrade;

    include /home/nubehits/conf/web/nubehits.com/nginx.ssl.conf_*; # <--- nginx.ssl.conf_OME_NUBEHITS_PROXY
}

here the location implementation nginx.ssl.conf_OME_NUBEHITS_PROXY

location /webrtc/ {
        modsecurity off;
        rewrite ^/webrtc(.*)$ $1 break;
        proxy_pass http://servers_webrtc_http;    #   <---here i made the change that you told me, go to http upstream .... OME 8888 port
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
        proxy_set_header Host $host;
#       proxy_ssl_server_name      on;  
#       proxy_ssl_name $host;
#       proxy_ssl_certificate      /home/nubehits/conf/web/nubehits.com/ssl/nubehits.com.pem;
#       proxy_ssl_certificate_key  /home/nubehits/conf/web/nubehits.com/ssl/nubehits.com.key;

        }
location /hls/ {
        modsecurity off;
        rewrite ^/hls(.*)$ $1 break;
        proxy_pass https://servers_hls;
        proxy_set_header Host $host;
        }

###################################################################

Now the Server origen xml config, the edge has the same configuration just the edge directives diference

Server.xml

<?xml version="1.0" encoding="UTF-8"?>

<Server version="8">
        <Name>AlnuxSoft</Name>
        <!-- Host type (origin/edge) -->
        <Type>origin</Type>
        <!-- Specify IP address to bind (* means all IPs) -->
        <IP>*</IP>
        <PrivacyProtection>false</PrivacyProtection>

        <!-- 
        To get the public IP address(mapped address of stun) of the local server. 
        This is useful when OME cannot obtain a public IP from an interface, such as AWS or docker environment. 
        If this is successful, you can use ${PublicIP} in your settings.
        -->
        <StunServer>stun.l.google.com:19302</StunServer>

        <!-- Settings for the ports to bind -->
        <Bind>
                <!-- Enable this configuration if you want to use API Server -->

                <Managers>
                        <API>
                                <Port>33375</Port>
                                <WorkerCount>1</WorkerCount>
                        </API>
                </Managers>

                <Providers>
                        <!-- Push providers -->
                        <RTMP>
                                <Port>1935</Port>
                                <WorkerCount>1</WorkerCount>
                        </RTMP>
                        <SRT>
                                <Port>9999</Port>
                                <WorkerCount>1</WorkerCount>
                        </SRT>
                </Providers>

                <Publishers>
                        <OVT>
                                <Port>9010</Port>
                                <WorkerCount>1</WorkerCount>
                        </OVT>
                        <HLS>
                                <Port>8888</Port>
                                <!-- If you want to use TLS, specify the TLS port -->
<TLSPort>4443</TLSPort>
                                <WorkerCount>1</WorkerCount>
                        </HLS>
                        <WebRTC>
                                <Signalling>
                                        <Port>3333</Port>
                                        <!-- If you want to use TLS, specify the TLS port -->
                                        <TLSPort>3334</TLSPort>
                                        <WorkerCount>1</WorkerCount>
                                </Signalling>
                                <IceCandidates>
                                        <!-- 
                                                If you want to stream WebRTC over TCP, specify IP:Port for TURN server.
                                                This uses the TURN protocol, which delivers the stream from the built-in TURN server to the player's TURN client over TCP. 
                                                For detailed information, refer https://airensoft.gitbook.io/ovenmediaengine/streaming/webrtc-publishing#webrtc-over-tcp
                                        -->
                                        <TcpRelay>144.126.139.107:3478</TcpRelay>
                                        <!-- TcpForce is an option to force the use of TCP rather than UDP in WebRTC streaming. (You can omit ?transport=tcp accordingly.) If <TcpRelay> is not set, playback may fail. -->
                                        <TcpForce>true</TcpForce>

                                        <IceCandidate>144.126.139.107:10000/udp</IceCandidate>
                                        <TcpRelayWorkerCount>1</TcpRelayWorkerCount>
                                </IceCandidates>
                        </WebRTC>
                </Publishers>
        </Bind>

        <Managers>
                <Host>
                        <Names>
                                <Name>*</Name>
                        </Names>
                        <TLS>
                                <CertPath>/home/nubehitsorigen01/conf/web/origen01.nubehits.com/ssl/origen01.nubehits.com.crt</CertPath>
                                <KeyPath>/home/nubehitsorigen01/conf/web/origen01.nubehits.com/ssl/origen01.nubehits.com.key</KeyPath>
                                <ChainCertPath>/home/nubehitsorigen01/conf/web/origen01.nubehits.com/ssl/origen01.nubehits.com.crt</ChainCertPath>
                        </TLS>
                </Host>
                <API>
                        <AccessToken>acceso-a-api-ovenemdiaengine</AccessToken>
                </API>
        </Managers>

        <VirtualHosts>
                <!-- You can use wildcard like this to include multiple XMLs -->
                <VirtualHost include="VHost*.xml" />
                <VirtualHost>
                        <Name>origen01.nubehits.com</Name>
                        <!--Distribution is a value that can be used when grouping the same vhost distributed across multiple servers. This value is output to the events log, so you can use it to aggregate statistics. -->
                        <Distribution>nubehits.com</Distribution>

                        <!-- Settings for multi ip/domain and TLS -->
                        <Host>
                                <Names>
                                        <!-- Host names
                                                <Name>stream1.airensoft.com</Name>
                                                <Name>stream2.airensoft.com</Name>
                                                <Name>*.sub.airensoft.com</Name>
                                                <Name>192.168.0.1</Name>
                                        -->
                                        <Name>origen01.nubehits.com</Name>
                                        <Name>ws.nubehits.com</Name>
                                        <Name>nubehits.com</Name>

                                </Names>
                                <!--
                                <TLS>
                                        <CertPath>path/to/file.crt</CertPath>
                                        <KeyPath>path/to/file.key</KeyPath>
                                        <ChainCertPath>path/to/file.crt</ChainCertPath>
                                </TLS>
                                -->
                                <TLS>
                                        <CertPath>/home/nubehitsorigen01/conf/web/origen01.nubehits.com/ssl/origen01.nubehits.com.crt</CertPath>
                                        <KeyPath>/home/nubehitsorigen01/conf/web/origen01.nubehits.com/ssl/origen01.nubehits.com.key</KeyPath>
                                        <ChainCertPath>/home/nubehitsorigen01/conf/web/origen01.nubehits.com/ssl/origen01.nubehits.com.crt</ChainCertPath>
                                </TLS>

                        </Host>
                        <!-- Settings for applications -->
                        <Applications>
                                <Application>
                                        <Name>endirecto</Name>
                                        <!-- Application type (live/vod) -->
                                        <Type>live</Type>
                                        <OutputProfiles>
                                                <!-- Enable this configuration if you want to hardware acceleration using GPU -->
                                                <HardwareAcceleration>false</HardwareAcceleration>
                                                <OutputProfile>
                                                        <Name>bypass_stream_1280</Name>
                                                        <OutputStreamName>${OriginStreamName}_1280</OutputStreamName>
                                                        <Encodes>
                                                                <Audio>
                                                                        <Bypass>true</Bypass>
                                                                </Audio>
                                                                <Video>
                                                                        <Bypass>true</Bypass>
                                                                </Video>
                                                                <Audio>
                                                                        <Codec>opus</Codec>
                                                                        <Bitrate>128000</Bitrate>
                                                                        <Samplerate>48000</Samplerate>
                                                                        <Channel>2</Channel>
                                                                </Audio>
                                                        </Encodes>
                                                </OutputProfile>
                                                <OutputProfile>
                                                        <Name>bypass_stream_640</Name>
                                                        <OutputStreamName>${OriginStreamName}_640</OutputStreamName>
                                                        <Encodes>
                                                                <Audio>
                                                                        <Bypass>true</Bypass>
                                                                </Audio>
                                                                <Audio>
                                                                        <Codec>opus</Codec>
                                                                        <Bitrate>128000</Bitrate>
                                                                        <Samplerate>48000</Samplerate>
                                                                        <Channel>2</Channel>
                                                                </Audio>
                                                                <Video>
                                                                        <Codec>h264</Codec>
                                                                        <Bitrate>819200</Bitrate>
                                                                        <Width>640</Width>
                                                                        <Framerate>30.0</Framerate>
                                                                        <Preset>faster</Preset>
                                                                </Video>
                                                        </Encodes>
                                                </OutputProfile>
                                        </OutputProfiles>
                                        <Providers>
                                                <RTMP />
                                                <SRT />
                                        </Providers>
                                        <Publishers>
                                                <AppWorkerCount>1</AppWorkerCount>
                                                <StreamWorkerCount>6</StreamWorkerCount>
                                                <OVT />
                                                <WebRTC>
                                                       <Timeout>30000</Timeout>
                                                        <Rtx>false</Rtx>
                                                        <Ulpfec>false</Ulpfec>
                                                        <JitterBuffer>false</JitterBuffer>
                                                </WebRTC>
                                                <HLS>
                                                        <SegmentDuration>5</SegmentDuration>
                                                        <SegmentCount>3</SegmentCount>
                                                        <CrossDomains>
                                                                <Url>*</Url>
                                                        </CrossDomains>
                                                </HLS>
                                        </Publishers>
                                </Application>
                        </Applications>
                </VirtualHost>
        </VirtualHosts>
</Server>
alnux commented 2 years ago

i have the next results on differents browsers chrome stay on Provisional headers but on firefox return me the 101 Switching Protocols, but firefox does not play too.

here the detail Chrome

Request URL: wss://nubehits.com/demo/endirecto/stream_1280/
Provisional headers are shown

Accept-Encoding: gzip, deflate, br
Accept-Language: es,en-US;q=0.9,en;q=0.8
authorization: basic YWNjZXNvLWEtYXBpLW92ZW5lbWRpYWVuZ2luZQ==
Cache-Control: no-cache
Connection: Upgrade
Cookie: wordpress_test_cookie=WP%20Cookie%20check; wordpress_logged_in_ebcfb6298dcf2004d6c07a9c0709c334=nubehits%7C1640307562%7CAvq3iIjMwOyA5b0jzk9WygR3CHCa0rK4119CjRj9tR2%7C6b392950ee563eba192fb72f9510b2a74c68ea9a03c8588c8ec07fcab7225051; wp-settings-1=libraryContent%3Dbrowse; wp-settings-time-1=1640134763; PHPSESSID=e64cp3ucb29go9kbnuu5qm3cp0
Host: nubehits.com
Origin: https://nubehits.com
Pragma: no-cache
Sec-WebSocket-Extensions: permessage-deflate; client_max_window_bits
Sec-WebSocket-Key: NDqz7RJtwSk4o6o8pLu0ig==
Sec-WebSocket-Version: 13
Upgrade: websocket
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/94.0.4606.85 YaBrowser/21.11.0.2054 Yowser/2.5 Safari/537.36

firefox 

response header ############

HTTP/1.1 101 Switching Protocols   # <----------101 result
Server: nginx
Date: Thu, 23 Dec 2021 14:46:16 GMT
Content-Type: text/html; charset=utf-8
Content-Length: 0
Connection: upgrade
Sec-WebSocket-Accept: hxzkJf11/Ug4ljQM2eSb9KJPluQ=
Strict-Transport-Security: max-age=15768000;

request header ###############

GET /demo/endirecto/stream_1280/ HTTP/1.1
Host: nubehits.com
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.13; rv:95.0) Gecko/20100101 Firefox/95.0
Accept: */*
Accept-Language: es-ES,es;q=0.8,en-US;q=0.5,en;q=0.3
Accept-Encoding: gzip, deflate, br
Sec-WebSocket-Version: 13
Origin: https://nubehits.com
Sec-WebSocket-Extensions: permessage-deflate
Sec-WebSocket-Key: OtHQc+wN+4Bwi7LBOj/Emw==
Connection: keep-alive, Upgrade
Cookie: wordpress_test_cookie=WP%20Cookie%20check; wordpress_logged_in_ebcfb6298dcf2004d6c07a9c0709c334=nubehits%7C1639965508%7COxpuYnzDRX7IcbsWdtZGYB4jEU8gO6LIgbyom1A2Itv%7C4e0c22337755442264ac60a354097122321a4246f093cd3a06420758348128e7; wp-settings-1=libraryContent%3Dbrowse; wp-settings-time-1=1639792710
Sec-Fetch-Dest: websocket
Sec-Fetch-Mode: websocket
Sec-Fetch-Site: same-origin
Pragma: no-cache
Cache-Control: no-cache
Upgrade: websocket
getroot commented 2 years ago

Are you using the latest version of OvenMediaEngine? See if the problem is reproduced with the master branch. Also, check what the nginx log says.

alnux commented 2 years ago

i am using right now OvenMediaEngine v0.12.9 as told before, the nginx log does not show any error or issue, the chrome show just Provisional headers are shown and firefox show response header with 101 status, just i have the OME log of two files

[2021-12-27 10:49:29.278] I [SPRtcSig-T3333:32444] Signalling | rtc_signalling_server.cpp:201  | New client is connected: <ClientSocket: 0x7fa2040010f0, #23, Connected, TCP, Nonblocking, 207.244.240.88:49510>
[2021-12-27 10:49:29.429] I [SPRtcSig-T3333:32444] Signalling | rtc_signalling_server.cpp:339  | Client is disconnected: <WebSocketClient: 0x7fa2040031f0, <ClientSocket: 0x7fa2040010f0, #23, Disconnected, TCP, Nonblocking, 207.244.240.88:49510>> (#edge02.nubehits.com#demo / endirecto, ufrag: local: (N/A), remote: (N/A))

now i will try with the one that you tell me "master branch"

alnux commented 2 years ago

i just install the branch master OME and still the same error trougth Load balancer (please read before messages) and when i try to use directly stream there show me a new error log about certificate on hls and webrtc, i is the way that i have returned to OME 0.12.9 release

[2021-12-27 18:14:01.588] I [SPRtcSig-T3333:21853] Signalling | rtc_signalling_server.cpp:201  | New client is connected: <ClientSocket: 0x7fefcc007
040, #4, Connected, TCP, Nonblocking, 207.244.240.88:53324>
[2021-12-27 18:14:01.750] I [SPRtcSig-T3333:21853] Signalling | rtc_signalling_server.cpp:339  | Client is disconnected: <WebSocketClient: 0x7fefcc0
05df0, <ClientSocket: 0x7fefcc007040, #4, Disconnected, TCP, Nonblocking, 207.244.240.88:53324>> (#edge02.nubehits.com#demo / endirecto, ufrag: loca
l: (N/A), remote: (N/A))
[2021-12-27 18:14:21.151] I [SPRtcSig-T3333:21853] Signalling | rtc_signalling_server.cpp:201  | New client is connected: <ClientSocket: 0x7fefcc007
040, #4, Connected, TCP, Nonblocking, 207.244.240.88:53338>
[2021-12-27 18:14:21.305] I [SPRtcSig-T3333:21853] Signalling | rtc_signalling_server.cpp:339  | Client is disconnected: <WebSocketClient: 0x7fefcc0
05df0, <ClientSocket: 0x7fefcc007040, #4, Disconnected, TCP, Nonblocking, 207.244.240.88:53338>> (#edge02.nubehits.com#demo / endirecto, ufrag: loca
l: (N/A), remote: (N/A))
[2021-12-27 18:14:53.157] E [SPSegPub-T4443:21860] HttpServer | https_server.cpp:83   | Could not handle connection event: there is no certificate
[2021-12-27 18:14:53.297] E [SPSegPub-T4443:21860] HttpServer | https_server.cpp:83   | Could not handle connection event: there is no certificate
[2021-12-27 18:15:22.417] I [SPRtcSig-T3333:21853] Signalling | rtc_signalling_server.cpp:201  | New client is connected: <ClientSocket: 0x7fefcc007
040, #21, Connected, TCP, Nonblocking, 207.244.240.88:53366>
[2021-12-27 18:15:22.570] I [SPRtcSig-T3333:21853] Signalling | rtc_signalling_server.cpp:339  | Client is disconnected: <WebSocketClient: 0x7fefcc0
05df0, <ClientSocket: 0x7fefcc007040, #21, Disconnected, TCP, Nonblocking, 207.244.240.88:53366>> (#edge02.nubehits.com#demo / endirecto, ufrag: loc
al: (N/A), remote: (N/A))
[2021-12-27 18:15:39.018] E [SPRtcSig-T3334:21854] HttpServer | https_server.cpp:83   | Could not handle connection event: there is no certificate
[2021-12-27 18:16:09.324] E [SPRtcSig-T3334:21854] HttpServer | https_server.cpp:83   | Could not handle connection event: there is no certificate
[2021-12-27 18:16:23.067] I [SPRtcSig-T3333:21853] Signalling | rtc_signalling_server.cpp:201  | New client is connected: <ClientSocket: 0x7fefcc007
040, #20, Connected, TCP, Nonblocking, 207.244.240.88:53442>
[2021-12-27 18:16:23.217] I [SPRtcSig-T3333:21853] Signalling | rtc_signalling_server.cpp:339  | Client is disconnected: <WebSocketClient: 0x7fefcc0
05df0, <ClientSocket: 0x7fefcc007040, #20, Disconnected, TCP, Nonblocking, 207.244.240.88:53442>> (#edge02.nubehits.com#demo / endirecto, ufrag: loc
al: (N/A), remote: (N/A))
[2021-12-27 18:16:39.949] E [SPSegPub-T4443:21860] HttpServer | https_server.cpp:83   | Could not handle connection event: there is no certificate
[2021-12-27 18:16:40.089] E [SPSegPub-T4443:21860] HttpServer | https_server.cpp:83   | Could not handle connection event: there is no certificate
[2021-12-27 18:17:20.900] E [SPSegPub-T4443:21860] HttpServer | https_server.cpp:83   | Could not handle connection event: there is no certificate
[2021-12-27 18:17:23.704] I [SPRtcSig-T3333:21853] Signalling | rtc_signalling_server.cpp:201  | New client is connected: <ClientSocket: 0x7fefcc007
040, #22, Connected, TCP, Nonblocking, 207.244.240.88:53486>
[2021-12-27 18:17:23.855] I [SPRtcSig-T3333:21853] Signalling | rtc_signalling_server.cpp:339  | Client is disconnected: <WebSocketClient: 0x7fefcc0
05df0, <ClientSocket: 0x7fefcc007040, #22, Disconnected, TCP, Nonblocking, 207.244.240.88:53486>> (#edge02.nubehits.com#demo / endirecto, ufrag: loc
al: (N/A), remote: (N/A))
[2021-12-27 18:18:25.014] I [SPRtcSig-T3333:21853] Signalling | rtc_signalling_server.cpp:201  | New client is connected: <ClientSocket: 0x7fefcc007
040, #4, Connected, TCP, Nonblocking, 207.244.240.88:53524>
[2021-12-27 18:18:25.165] I [SPRtcSig-T3333:21853] Signalling | rtc_signalling_server.cpp:339  | Client is disconnected: <WebSocketClient: 0x7fefcc005df0, <ClientSocket: 0x7fefcc007040, #4, Disconnected, TCP, Nonblocking, 207.244.240.88:53524>> (#edge02.nubehits.com#demo / endirecto, ufrag: local: (N/A), remote: (N/A))
[2021-12-27 18:18:33.048] E [SPRtcSig-T3334:21854] HttpServer | https_server.cpp:83   | Could not handle connection event: there is no certificate
[2021-12-27 18:19:03.607] E [SPRtcSig-T3334:21854] HttpServer | https_server.cpp:83   | Could not handle connection event: there is no certificate
[2021-12-27 18:19:13.426] E [SPRtcSig-T3334:21854] HttpServer | https_server.cpp:83   | Could not handle connection event: there is no certificate
[2021-12-27 18:19:19.319] E [SPRtcSig-T3334:21854] HttpServer | https_server.cpp:83   | Could not handle connection event: there is no certificate
[2021-12-27 18:19:25.653] I [SPRtcSig-T3333:21853] Signalling | rtc_signalling_server.cpp:201  | New client is connected: <ClientSocket: 0x7fefcc007040, #20, Connected, TCP, Nonblocking, 207.244.240.88:53616>
[2021-12-27 18:19:25.804] I [SPRtcSig-T3333:21853] Signalling | rtc_signalling_server.cpp:339  | Client is disconnected: <WebSocketClient: 0x7fefcc005df0, <ClientSocket: 0x7fefcc007040, #20, Disconnected, TCP, Nonblocking, 207.244.240.88:53616>> (#edge02.nubehits.com#demo / endirecto, ufrag: local: (N/A), remote: (N/A))
[2021-12-27 18:19:29.231] E [SPRtcSig-T3334:21854] HttpServer | https_server.cpp:83   | Could not handle connection event: there is no certificate
[2021-12-27 18:19:46.742] E [SPRtcSig-T3334:21854] HttpServer | https_server.cpp:83   | Could not handle connection event: there is no certificate
[2021-12-27 18:19:53.361] E [SPRtcSig-T3334:21854] HttpServer | https_server.cpp:83   | Could not handle connection event: there is no certificate
[2021-12-27 18:20:23.533] E [SPRtcSig-T3334:21854] HttpServer | https_server.cpp:83   | Could not handle connection event: there is no certificate
[2021-12-27 18:20:26.292] I [SPRtcSig-T3333:21853] Signalling | rtc_signalling_server.cpp:201  | New client is connected: <ClientSocket: 0x7fefcc007040, #20, Connected, TCP, Nonblocking, 207.244.240.88:53698>
[2021-12-27 18:20:26.446] I [SPRtcSig-T3333:21853] Signalling | rtc_signalling_server.cpp:339  | Client is disconnected: <WebSocketClient: 0x7fefcc004ab0, <ClientSocket: 0x7fefcc007040, #20, Disconnected, TCP, Nonblocking, 207.244.240.88:53698>> (#edge02.nubehits.com#demo / endirecto, ufrag: local: (N/A), remote: (N/A))
[2021-12-27 18:20:53.817] E [SPSegPub-T4443:21860] HttpServer | https_server.cpp:83   | Could not handle connection event: there is no certificate
[2021-12-27 18:21:03.836] E [SPSegPub-T4443:21860] HttpServer | https_server.cpp:83   | Could not handle connection event: there is no certificate
[2021-12-27 18:21:13.855] E [SPRtcSig-T3334:21854] HttpServer | https_server.cpp:83   | Could not handle connection event: there is no certificate
[2021-12-27 18:21:27.659] I [SPRtcSig-T3333:21853] Signalling | rtc_signalling_server.cpp:201  | New client is connected: <ClientSocket: 0x7fefcc007040, #4, Connected, TCP, Nonblocking, 207.244.240.88:53730>
[2021-12-27 18:21:27.811] I [SPRtcSig-T3333:21853] Signalling | rtc_signalling_server.cpp:339  | Client is disconnected: <WebSocketClient: 0x7fefcc004ab0, <ClientSocket: 0x7fefcc007040, #4, Disconnected, TCP, Nonblocking, 207.244.240.88:53730>> (#edge02.nubehits.com#demo / endirecto, ufrag: local: (N/A), remote: (N/A))
[2021-12-27 18:21:45.060] E [SPRtcSig-T3334:21854] HttpServer | https_server.cpp:83   | Could not handle connection event: there is no certificate
[2021-12-27 18:22:28.417] I [SPRtcSig-T3333:21853] Signalling | rtc_signalling_server.cpp:201  | New client is connected: <ClientSocket: 0x7fefcc007040, #4, Connected, TCP, Nonblocking, 207.244.240.88:53768>
[2021-12-27 18:22:28.586] I [SPRtcSig-T3333:21853] Signalling | rtc_signalling_server.cpp:339  | Client is disconnected: <WebSocketClient: 0x7fefcc006be0, <ClientSocket: 0x7fefcc007040, #4, Disconnected, TCP, Nonblocking, 207.244.240.88:53768>> (#edge02.nubehits.com#demo / endirecto, ufrag: local: (N/A), remote: (N/A))
[2021-12-27 18:23:29.077] I [SPRtcSig-T3333:21853] Signalling | rtc_signalling_server.cpp:201  | New client is connected: <ClientSocket: 0x7fefcc007040, #4, Connected, TCP, Nonblocking, 207.244.240.88:53800>
[2021-12-27 18:23:29.235] I [SPRtcSig-T3333:21853] Signalling | rtc_signalling_server.cpp:339  | Client is disconnected: <WebSocketClient: 0x7fefcc006c40, <ClientSocket: 0x7fefcc007040, #4, Disconnected, TCP, Nonblocking, 207.244.240.88:53800>> (#edge02.nubehits.com#demo / endirecto, ufrag: local: (N/A), remote: (N/A))
[2021-12-27 18:24:30.428] I [SPRtcSig-T3333:21853] Signalling | rtc_signalling_server.cpp:201  | New client is connected: <ClientSocket: 0x7fefcc007040, #4, Connected, TCP, Nonblocking, 207.244.240.88:53838>
[2021-12-27 18:24:30.578] I [SPRtcSig-T3333:21853] Signalling | rtc_signalling_server.cpp:339  | Client is disconnected: <WebSocketClient: 0x7fefcc006c40, <ClientSocket: 0x7fefcc007040, #4, Disconnected, TCP, Nonblocking, 207.244.240.88:53838>> (#edge02.nubehits.com#demo / endirecto, ufrag: local: (N/A), remote: (N/A))
[2021-12-27 18:25:31.092] I [SPRtcSig-T3333:21853] Signalling | rtc_signalling_server.cpp:201  | New client is connected: <ClientSocket: 0x7fefcc007040, #4, Connected, TCP, Nonblocking, 207.244.240.88:53872>
[2021-12-27 18:25:31.255] I [SPRtcSig-T3333:21853] Signalling | rtc_signalling_server.cpp:339  | Client is disconnected: <WebSocketClient: 0x7fefcc006c40, <ClientSocket: 0x7fefcc007040, #4, Disconnected, TCP, Nonblocking, 207.244.240.88:53872>> (#edge02.nubehits.com#demo / endirecto, ufrag: local: (N/A), remote: (N/A))
getroot commented 2 years ago

Sorry, I didn't see you detailed the OME version.

<TLS>
<CertPath>/home/nubehitsorigen01/conf/web/origen01.nubehits.com/ssl/origen01.nubehits.com.crt</CertPath>
<KeyPath>/home/nubehitsorigen01/conf/web/origen01.nubehits.com/ssl/origen01.nubehits.com.key</KeyPath>                                      
<ChainCertPath>/home/nubehitsorigen01/conf/web/origen01.nubehits.com/ssl/origen01.nubehits.com.crt</ChainCertPath>
</TLS>

The files for CertPath and ChainCertPath are the same in your setup. In OME, CertPath must contain the domain's certificate, and ChainCertPath must contain chain certificates.

This may give erroneous results depending on the HTTP client.

Try setting the certificate file correctly.

getroot commented 2 years ago

I guess the problem you have is of a similar type to the issue below. https://github.com/AirenSoft/OvenMediaEngine/issues/600

alnux commented 2 years ago

hi @getroot really thanks for the help, but the certificate is not the problem becouse im using the same with nginx and in no browser gives me a problem, This problem gave me after installing the master version of OME, but the one that is available as release ( link here) does not have that certificate error using the same certificates. Now mi problem is not similar with #600 becouse him is using proxy not load balancer upstream nginx option. in my case with nginx Load Balancer HLS works without problem but WEBRTC on firefox pass 101 response and on chrome not response is get in twice case the video does not work and on OME log just show two lines that show connect and instantly disconnect, on nginx does not appear anything on log

[2021-12-18 19:09:53.106] I [SPRtcSig-T3334:11525] Signalling | rtc_signalling_server.cpp:201  | New client is connected: <ClientSocket: 0x7f6edc023810, #28, Connected, TCP, Nonblocking, 207.244.240.88:40880>
[2021-12-18 19:09:53.349] I [SPRtcSig-T3334:11525] Signalling | rtc_signalling_server.cpp:339  | Client is disconnected: <WebSocketClient: 0x7f6ef0034550, <ClientSocket: 0x7f6edc023810, #28, Disconnected, TCP, Nonblocking, 207.244.240.88:40880>> (#origen01.nubehits.com#endirecto / stream_1280, ufrag: local: (N/A), remote: (N/A))
getroot commented 2 years ago

Unlike Nginx, OME distinguishes between Cert file and ChainCert file. And depending on the HTTP client, it may behave abnormally (problem that is reproduced only in specific browsers or specific clients, such as issue 600).

Your certificate is of course normal. However, since OME handles ChainCert separately, it is worth trying to put Cert and Chain Cert separately.

And why do you think it is a bug in OME that it works fine when connected directly with a browser, but does not work when connected with nginx?

OME doesn't know if it's a browser or nginx that is trying to connect to with websocket. OME just think of it as the same websocket client. Is the https client specification of browser and nginx different? If different, whose problem is it?

Next year I will see if nginx is sending websocket packets in a different way than browsers with debug logs in OME. (I'm on vacation after finishing work in 2021.) Happy New Year!

alnux commented 2 years ago

thanks @getroot i really really like OME but this issue is going me crazy, have a nice and happy new year too

stale[bot] commented 2 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

tothandras commented 2 years ago

@alnux I'm facing a similar/same issue. Have you figured out what was the cause? I also noticed that the demo player is working fine, but when the UI is on the same domain it fails to load.

alnux commented 2 years ago

Sorry but not