WHIP from OBS v30.1.2 on Windows/Apple platforms fails to establishe DTLS session with "Bad certificate" reply from OBS

paullouisageneau / libdatachannel

C/C++ WebRTC network library featuring Data Channels, Media Transport, and WebSockets

https://libdatachannel.org/

Mozilla Public License 2.0

1.8k stars 362 forks source link

WHIP from OBS v30.1.2 on Windows/Apple platforms fails to establishe DTLS session with "Bad certificate" reply from OBS #1190

Open ianzag opened 5 months ago

ianzag commented 5 months ago

Dear developers,

I have an issue with recently released OBS v30.1.2 and WHIP Streaming. I'm trying to broadcast media stream to WHIP compatible server. Problem is DTLS handshake fails with Windows/Apple OBS official builds but can be successfully established with Linux official build. Not much in OBS logs in both cases, but I got SDP exchange and DTLS network traffic traces. Looks like depending on OBS's platform (and used TLS engine?) it behaves slightly different. Please see attached traces. Will be glad to supply any additional info. Thanks! Bad_DTLS.txt Bad_SDP.txt Good_DTLS.txt Good_SDP.txt

paullouisageneau commented 5 months ago

It looks indeed like a different TLS library behaving differently.

@Sean-Der Is the TLS library in OBS builds for Linux different from the one for Windows and OSX? I guess Windows and OSX rely on Mbed TLS so it would be an issue specific to MbedTLS.

Sean-Der commented 5 months ago

@paullouisageneau exactly!

Sorry I missed this @ianzag is this still a problem? Could I test against a server with the issue?

ianzag commented 5 months ago

@Sean-Der pardon for long response had very busy days. I'll check if it's still applicable tomorrow. On server side there's an ordinary OpenSSL's DTLS state machine so I believe it can be wrapped into dedicated regression test. How FooSSL performs with BarSSL.

Sean-Der commented 1 week ago

@ianzag Is this still a problem? Maybe OBS upgrade fixed?