iOS: websocket client failed due to TLS alert: unknown CA

[chobie]

I was testing on iOS and it seemed that I couldn't connect due to an OpenSSL error when signaling with a websocket client.

here is verbose log. ngrok uses Let's Encrypt R3 certificate.

rtc::impl::WebSocket::WebSocket@41: Creating WebSocket
RTC:rtcCreateWebSocket(String)
rtc.RTCWebSocket:.ctor(String)
VP8Example:Start()

rtc::impl::WebSocket::open@55: Opening WebSocket to URL: wss://af8dd7043ec9.ngrok.app/0aRj
RTC:rtcCreateWebSocket(String)
rtc.RTCWebSocket:.ctor(String)
VP8Example:Start()

rtc::impl::TcpTransport::TcpTransport@33: Initializing TCP transport
RTC:rtcCreateWebSocket(String)
rtc.RTCWebSocket:.ctor(String)
VP8Example:Start()

rtc::impl::WebSocket::setTcpTransport@206: Starting TCP transport
RTC:rtcCreateWebSocket(String)
rtc.RTCWebSocket:.ctor(String)
VP8Example:Start()

rtc::impl::TcpTransport::connect@124: Connecting to af8dd7043ec9.ngrok.app:443
RTC:rtcCreateWebSocket(String)
rtc.RTCWebSocket:.ctor(String)
VP8Example:Start()

rtc::impl::TcpTransport::resolve@138: Resolving af8dd7043ec9.ngrok.app:443
Remote: af8dd7043ec9.ngrok.app:443
VP8Example:Start()

Failed to initialize encoder.
WARNING -> applicationDidReceiveMemoryWarning()
rtc::impl::TcpTransport::createSocket@251: Trying address 2406:da14:540:e900::6e:3:443
rtc::impl::TcpTransport::createSocket@254: Creating TCP socket
rtc::impl::PollService::add@59: Registering socket in poll service, direction=out
rtc::impl::PollService::runLoop@188: Exiting poll
rtc::impl::PollService::runLoop@179: Entering poll, timeout=10000ms
rtc::impl::PollService::runLoop@188: Exiting poll
rtc::impl::PollService::process@141: Poll out event
rtc::impl::TcpTransport::attempt@230: TCP connected
rtc::impl::WebSocket::initTlsTransport@308: Starting TLS transport
rtc::impl::TlsTransport::TlsTransport@562: Initializing TLS transport (OpenSSL)
rtc::impl::TlsTransport::TlsTransport@606: Server Name Indication: af8dd7043ec9.ngrok.app
rtc::impl::VerifiedTlsTransport::VerifiedTlsTransport@21: Setting up TLS certificate verification
rtc::impl::TlsTransport::start@638: Starting TLS transport
rtc::impl::Transport::registerIncoming@27: Registering incoming callback
rtc::impl::TcpTransport::send@89: Send size=439
rtc::impl::PollService::add@59: Registering socket in poll service, direction=in
rtc::impl::PollService::runLoop@179: Entering poll, timeout=10000ms
rtc::impl::PollService::runLoop@188: Exiting poll
rtc::impl::PollService::runLoop@179: Entering poll, timeout=10000ms
rtc::impl::PollService::runLoop@188: Exiting poll
rtc::impl::PollService::process@137: Poll in event
rtc::impl::TcpTransport::incoming@97: Incoming size=1428
rtc::impl::TlsTransport::incoming@692: Incoming size=1428
rtc::impl::PollService::runLoop@179: Entering poll, timeout=10000ms
rtc::impl::PollService::runLoop@188: Exiting poll
rtc::impl::PollService::process@137: Poll in event
rtc::impl::TcpTransport::incoming@97: Incoming size=2862
rtc::impl::TlsTransport::incoming@692: Incoming size=2862
rtc::impl::PollService::runLoop@179: Entering poll, timeout=10000ms
rtc::impl::TlsTransport::InfoCallback@804: TLS alert: unknown CA
rtc::impl::TcpTransport::send@89: Send size=7
rtc::impl::PollService::runLoop@188: Exiting poll
rtc::impl::PollService::process@137: Poll in event
rtc::impl::TcpTransport::incoming@97: Incoming size=24
rtc::impl::TlsTransport::incoming@692: Incoming size=24
rtc::impl::TcpTransport::process@439: TCP disconnected
rtc::impl::PollService::remove@72: Unregistering socket in poll service
rtc::impl::WebSocket::close@127: Closing WebSocket
rtc::impl::TlsTransport::doRecv@781: TLS handshake failed
rtc::impl::WebSocket::closeTransports@463: Closing transports
rtc::impl::TlsTransport::stop@655: Stopping TLS transport
rtc::impl::Transport::unregisterIncoming@34: Unregistering incoming callback
rtc::impl::PollService::runLoop@182: Entering poll
rtc::impl::TlsTransport::stop@655: Stopping TLS transport
rtc::impl::Transport::unregisterIncoming@34: Unregistering incoming callback
rtc::impl::TlsTransport::stop@655: Stopping TLS transport
rtc::impl::Transport::unregisterIncoming@34: Unregistering incoming callback
rtc::impl::PollService::runLoop@188: Exiting poll
rtc::impl::PollService::runLoop@182: Entering poll
rtc::impl::Transport::unregisterIncoming@34: Unregistering incoming callback
rtc::impl::TcpTransport::close@309: Closing TCP socket
rtc::impl::PollService::remove@72: Unregistering socket in poll service
rtc::impl::PollService::runLoop@188: Exiting poll
rtc::impl::PollService::runLoop@182: Entering poll
WebSocket Error: TLS connection failed
rtc.<>c__DisplayClass25_0:<onError>b__0(Object)
UnityEngine.WorkRequest:Invoke()
UnityEngine.UnitySynchronizationContext:Exec()

WebSocket Closed
rtc.<>c__DisplayClass23_0:<onClose>b__0(Object)
UnityEngine.WorkRequest:Invoke()
UnityEngine.UnitySynchronizationContext:Exec()

here is cmake options.

cmake -DUSE_GNUTLS=0 \
    -DUSE_NICE=0 \
    -DCMAKE_BUILD_TYPE=Release \
    -DCMAKE_SYSTEM_NAME=iOS \
    -DOPENSSL_ROOT_DIR=/Users/chobie/github/OpenSSL-for-iPhone \
    -DOPENSSL_INCLUDE_DIR=/Users/chobie/github/OpenSSL-for-iPhone/include \
    -DOPENSSL_CRYPTO_LIBRARY=/Users/chobie/github/OpenSSL-for-iPhone/ \
    -DOPENSSL_USE_STATIC_LIBS=1 \
    -DCMAKE_OSX_DEPLOYMENT_TARGET=17.0 \
    -DCMAKE_OSX_ARCHITECTURES=arm64 \
    -DNO_EXAMPLES=ON \
    -DNO_TESTS=ON \
    -G Xcode \
    -B build/iOS

It's an environment-dependent issue related to the build process. A detailed investigation is needed, but I don't have the time right now, so I'll just create an Issue for the time being.

[paullouisageneau]

The root cause is that OpenSSL is not aware of the system root CAs on the platform, so it can't check the remote certificate. You can of course mitigate this by disabling verification at the price of security.

[chobie]

Thanks. I was able to confirm the operation after checking the connection with disableTlsVerification.

I'm unsure about the best course of action. However, regarding Unity users, they might opt for a different C# WebSocket implementation, so it might not be a big concern.