MacOS M1 Ventura: mbedtls fails certificate verification without -debug flag

[davidbruce]

Original Post on Forum:https://community.haxe.org/t/sys-ssl-socket-broken-in-cpp-build-on-macos-unless-debug-compiler-option-is-included/3838

Overview:

Example code:

class Main {
    static function main() {
        var socket = new sys.ssl.Socket();
        socket.connect(new sys.net.Host("haxe.org"), 443);
        socket.handshake();
        trace("handshake succeeded");
    }
}

Debug build works:

haxe -cp src -main Main -debug -cpp out

Build without debug fails:

haxe -cp src -main Main -cpp out

Error : X509 - Certificate verification failed, e.g. CRL, CA or signature check failed

System Info:

Haxe 4.2.5 hxcpp 4.2.1 macOS Ventura 13.0.1 (M1 Apple Silicon)

Initial Investigation:

Certificate.loadDefaults from the Haxe core-api calls the following and loads all of the same certs with or without the -debug flag:

    var x = NativeSsl.cert_load_defaults();
        if (x != null)
            return new Certificate(x);

https://github.com/HaxeFoundation/haxe/blob/7de58981cc030beb5779a0d89261b2eb09544bb4/std/cpp/_std/sys/ssl/Certificate.hx#L50

I also don't see any debug specific logic in SSL.cpp: https://github.com/HaxeFoundation/hxcpp/blob/25cf4e861f30753aa8d48b8c046ecc4dc85836b2/src/hx/libs/ssl/SSL.cpp#L451

Error is being thrown here after calling NativeSsl.ssl_handshake https://github.com/HaxeFoundation/haxe/blob/7de58981cc030beb5779a0d89261b2eb09544bb4/std/cpp/_std/sys/ssl/Socket.hx#L162

Let me know if I can provide any logs.

[davidbruce]

@hughsando I tested https://github.com/HaxeFoundation/hxcpp/commit/1664b9ec74257c8c9e2a1fcb90c1f984d91f8b0c the simple code on this ticket and found both x86 and arm64( -D HXCPP_ARM64 ) builds are now working properly with or without the -debug flag.

[hughsando]

Great. I think we can call it. updating mbedtls might be the next step.

[tobil4sk]

Looks like this might have been fixed in mbedtls in https://github.com/Mbed-TLS/mbedtls/pull/4941 (backport of https://github.com/Mbed-TLS/mbedtls/pull/4947). After mbedtls is updated in #1026, we can check to make sure it is working properly with the updated version.