MqttClient with TLS and client certificate works on Linux but not on windows using the latest and following samples

[tufberg]

We have a project using the MQTTNet client to connect to a RabbitMQ broker using the MQTT Plugin (v311)

The extension method pasted below works on Linux/WSL and in our docker containers deployed but it does not work for our developers running on their windows machines. The method runs fine but the MQTT client fails to connect when running in windows.

We are using the latest MQTTNet NuGet.

• The clientCert variable below is a self-signed certificate that the RabbitMQ broker have whitelisted in the connected truststore. As I mentioned this works well in WSL/Linux. The certificate is downloaded from an Azure Keyvault and cotnains both private and public key.
• The Broker returns the exception/error "BadUserNaneOrPassword"/Invalid credentials (return code 4) which indicates that the ClientCert is not used properly.
• I've read the samples and tried a variety of ways to make this work, looked at https://github.com/Azure-Samples/MqttApplicationSamples/blob/main/mqttclients/dotnet/MQTTnet.Client.Extensions/WithTlsSettings.cs as it shows that the ca-chain needs to be in place.

    public static MqttClientOptionsBuilder WithSPTServiceAccountSettings(this MqttClientOptionsBuilder builder, string virtualHostName, string clientId, SPTRabbitMQRuntimeEnvironment env, X509Certificate2 clientCert, bool useTls = true, bool allowUntrustedCert = false, int brokerPort = 8883, int keepAlive = 25)
    {
        //timeout
        var host = EnumHelper.GetRabbitMQAddress(env);
        var caCertBytes = clientCert.Export(X509ContentType.Cert);
        var caCert = new X509Certificate2(caCertBytes);

        var tlsParams = new MqttClientTlsOptionsBuilder();
        tlsParams
            .UseTls(useTls)
            .WithSslProtocols(System.Security.Authentication.SslProtocols.Tls12 | System.Security.Authentication.SslProtocols.Tls13)
            .WithClientCertificates(new List<X509Certificate2> {
                     clientCert
                 })
            .WithAllowUntrustedCertificates(true)
            .WithIgnoreCertificateChainErrors(true)
            .WithIgnoreCertificateRevocationErrors(true)
            .WithRevocationMode(X509RevocationMode.NoCheck)
            .WithTargetHost(host)
            .WithCertificateValidationHandler((x) => { return true; })
            .WithTrustChain(new X509Certificate2Collection {
                    caCert
                });

        //Now build the client
        builder
            .WithTcpServer(host, brokerPort)
            .WithClientId(clientId)
            .WithKeepAlivePeriod(TimeSpan.FromSeconds(keepAlive))
            .WithTlsOptions(tlsParams.Build());

        return builder;
    }

[rido-min]

do you have any logs comparing the tests Linux/Windows ?

There is one detail when loading the clientCert

// https://github.com/dotnet/runtime/issues/45680#issuecomment-739912495
return new X509Certificate2(cert.Export(X509ContentType.Pkcs12));

from https://github.com/Azure-Samples/MqttApplicationSamples/blob/1a96ac0889ae7f76c29b42dda6864360a4b05e34/mqttclients/dotnet/MQTTnet.Client.Extensions/X509ClientCertificateLocator.cs#L21-L22

Also, I would not disable all the TLS settings, to see if those also succeed in Linux but not Windows.

[tufberg]

Thank fo the quick reply @rido-min , did some further testing yesterday. What I.ve tried was the following code.

• At first I've tried using .Net 8 but I've also tried .Net 7 and no difference there.
• The line // .WithTrustChain(new X509Certificate2Collection { caCert }); does not compile on WSL but I've tried with it and without it on windows.

    public static MqttClientOptionsBuilder WithSPTServiceAccountSettings(this MqttClientOptionsBuilder builder, string virtualHostName, string clientId, SPTRabbitMQRuntimeEnvironment env, X509Certificate2 clientCert, bool useTls = true, bool allowUntrustedCert = false, int brokerPort = 8883, int keepAlive = 25)
    {
        var host = EnumHelper.GetRabbitMQAddress(env);
        var pkcs12 = new X509Certificate2(clientCert.Export(X509ContentType.Pkcs12));
        var caCert = new X509Certificate2(pkcs12.Export(X509ContentType.Cert));

        var tlsParams = new MqttClientTlsOptionsBuilder();
        tlsParams
            .UseTls(useTls)
            .WithClientCertificates(new List<X509Certificate2> {
                     pkcs12
                 })
            .WithAllowUntrustedCertificates(true)
            .WithRevocationMode(X509RevocationMode.NoCheck)
            .WithTargetHost(host)
            .WithCertificateValidationHandler((x) => { return true; });
        // .WithTrustChain(new X509Certificate2Collection { caCert });

        //Now build the client
        builder
            .WithTcpServer(host, brokerPort)
            .WithClientId(clientId)
            .WithKeepAlivePeriod(TimeSpan.FromSeconds(keepAlive))
            .WithTlsOptions(tlsParams.Build());

        return builder;
    }

Here is the traces I've managed to gather when running the extension and connecting to the broker.

info: Program[0]
      Connecting MQTT Clientwith ClientId:'testclient' to Vhost '/'
dbug: Program[0]
      MQTT Trace: Trying to connect with server 'Unspecified/{redacted}:8883'
dbug: Program[0]
      MQTT Trace: Connection with server established
dbug: Program[0]
      MQTT Trace: TX (24 bytes) >>> Connect: [ClientId=testclient] [Username=] [Password=] [KeepAlivePeriod=25] [CleanSession=True]
dbug: Program[0]
      MQTT Trace: RX (4 bytes) <<< ConnAck: [ReturnCode=ConnectionRefusedBadUsernameOrPassword] [ReasonCode=Success] [IsSessionPresent=False]
warn: Program[0]
      MQTT Warning: Client will now throw an _MqttConnectingFailedException_. This is obsolete and will be removed in the future. Consider setting _ThrowOnNonSuccessfulResponseFromServer=False_ in client options.
fail: Program[0]
      MQTTnet.Adapter.MqttConnectingFailedException: Connecting with MQTT server failed (BadUserNameOrPassword).
         at MQTTnet.Client.MqttClient.Authenticate(IMqttChannelAdapter channelAdapter, MqttClientOptions options, CancellationToken cancellationToken) in /_/Source/MQTTnet/Client/MqttClient.cs:line 479
         at MQTTnet.Client.MqttClient.ConnectInternal(IMqttChannelAdapter channelAdapter, CancellationToken cancellationToken) in /_/Source/MQTTnet/Client/MqttClient.cs:line 528
         at MQTTnet.Client.MqttClient.ConnectAsync(MqttClientOptions options, CancellationToken cancellationToken) in /_/Source/MQTTnet/Client/MqttClient.cs:line 128
dbug: Program[0]
      MQTT Trace: Disconnecting [Timeout=00:01:40]
dbug: Program[0]
      MQTT Trace: Disconnected from adapter.
info: Program[0]
      MQTT Information: Disconnected.

If i look at the logs on the Broker I can see the logs below. An interesting part here is that the broker seems to think the client is trying to authenticate using username/password although no such credentials have been added when creating the client. guest:guest is the default user RabbitMQ falls back to when a client tries to authenticate using username/password.

2024-04-26 06:14:12.427308+00:00 [error] <0.1384153.0> MQTT connection failed: access refused for user 'guest':user 'guest' - invalid credentials

So my bet is that the Clientcertificate is not used properly during the connection procedure? I've tried debugging the MQTTNet library and did some print-outs of the sslStream.IsAuthenticated property in the TcpChannel cllass and is says true.

[tufberg]

The Broker test instance had the possibility to log in by using username/password enabled 🙈 so I disabled that temporarily to see if that made any difference. And behold ....

As suspected the client does not provide any client cert. The lines below are from the Broker

2024-04-26 07:04:29.770786+00:00 [notice] <0.195587.0> TLS server: In state certify at tls_dtls_connection.erl:329 generated SERVER ALERT: Fatal - Handshake Failure
2024-04-26 07:04:29.770786+00:00 [notice] <0.195587.0>  - no_client_certificate_provided

It is the line below that fails in MQTTNet

When inspecting the ssloptions before the client tries to authenticate the ssl stream I can see the client cert is there.

[rido-min]

Hi @tufberg

Before digging into client cert details, I'm surprised of this comment.

The line // .WithTrustChain(new X509Certificate2Collection { caCert }); does not compile on WSL but I've tried with it and without it on windows.

AFAIK WithTrustChain requires dotnet 7or greater, I'd like to understand what's your WSL configuration.

And now, going back to the certificates.

First you need to establish a valid TLS connection validating the ServerCert, then you can try to configure the client certs. I'd say that if the TLS handshake fails the client certs are not sent.

I've seen brokers that have additional requirements when using ClientCerts, such as using the same CN as ClientId, or even keep using UserName/Password.

Can you try to configure mosquitto with client certs and see if it works with MQTTNet ?

[tufberg]

Good point @rido-min ! I was at bit puzzled what was happening when I tried to compile and it did not work ...

But it turns out that I've been working a lot with the library and testing different settings and solutions and forgot that the library that the MQTTClient is built-in to are used by other projects needing the netstandard2.1 so the following code runs on both Windows .net 8 and WSl .net 8 and I can confirm that the row .WithTrustChain() is executed.

 public static MqttClientOptionsBuilder WithSPTServiceAccountSettings(this MqttClientOptionsBuilder builder, string virtualHostName, string clientId, SPTRabbitMQRuntimeEnvironment env, X509Certificate2 clientCert, bool useTls = true, bool allowUntrustedCert = false, int brokerPort = 8883, int keepAlive = 25)
    {
        var host = EnumHelper.GetRabbitMQAddress(env);
        var pkcs12 = new X509Certificate2(clientCert.Export(X509ContentType.Pkcs12));
        var caCert = new X509Certificate2(pkcs12.Export(X509ContentType.Cert));

        var certProvider = new MqttClientSvcAccountCertificatesProvider(pkcs12);

        var tlsParams = new MqttClientTlsOptionsBuilder();
        tlsParams
            .UseTls(useTls)
            .WithClientCertificatesProvider(certProvider)
            .WithAllowUntrustedCertificates(true)
            .WithRevocationMode(X509RevocationMode.NoCheck)
            .WithTargetHost(host)
            .WithCertificateValidationHandler((x) => { return true; });

#if NET7_0_OR_GREATER
        tlsParams.WithTrustChain(
            new X509Certificate2Collection() { caCert }
        );
#endif
        //Now build the client
        builder
            .WithTcpServer(host, brokerPort)
            .WithClientId(clientId)
            .WithKeepAlivePeriod(TimeSpan.FromSeconds(keepAlive))
            .WithTlsOptions(tlsParams.Build());

        return builder;
    }

Regarding the broker and Client cert the broker will extract the CN part of the certificate and use that for authorization. The authentication is done by the broker to see that the client cert exists in the trust store and thereby trusts the client. But since the client is not sending any certs the authentication fails and IF the username/password mechanism is enabled the broker will fall back on that flow.

• I'm using the same client cert in both WSL and Windows. Works on WSL, not on Windows.
• In WSL the ClientId and CN does not match. I've tried a matching CN and ClientId but that had no effect on windows either. RabbitMQ has not requirements regarding CN and ClientId.
• If I supply a username then it fails on both WSL and Windows since the user is not allowed to log in using username/password.

I can try to set up a different broker and see if that changes anything but we cannot change Broker at this point.

Since the sslOptions contains a Client certificate there has to be some other mechanism in the handshake that fails. The broker does not report any CiperSuite issues during the handshake. Only that there is no client cert available.

[rido-min]

I'm guessing that the ClientCerts are sent after the TLS handshake, so you need to make sure the TLS, including trusting the CA is ok before proceeding to auth with ClientCerts.

The most common configuration issue is to trust the chain used to issue the client certs in the server.

What is interesting is that it works in WSL but not Windows. Can you provide more details? Are you using .NET core or .NET Fx in Windows? (asking b/c the netstandard mention). Are you using a custom CA that has been configured in one platform but not the other?

I was not suggesting to change the broker, just to verify that you can auth from both platforms using other broker, In this sample we are collecting some instructions for configuring certs in mosquitto.

[tufberg]

If I enable username/password authentication on the Broker and set up a user that has username/password it works fine on both WSL and Windows. Since username/password works over TLS I make the assumption that the server certificate is trusted by the client.

The server certificate is signed by a public CA but the client certs are self-signed without a CA. And yes that is not the best of ideas. We'll change that in future sprints since we are building an IoT platform and having 10 k clients certs to trust is generally a bad design IMHO.

Ok, testing a different broker to see if there is any differences was a good idea. I've set up a HiveMQ cloud account trial, since I already had a Free one to test client certificate authentication.

Now to the interesting, but very annoying part, it works on HiveMQ 😭. So the crazy Rabbit is doing some weird stuff when the windows client tries to authenticate and not when it is sent from WSL !?!

[tufberg]

Tried to connect to the Broker from windows using the .Net RabbitMQ AMQP client. Since the broker is using the same SSLAuth mechanism for MQTT and AMQP connections.

Turns out that in the case of AMQP it works with the client cert ONLY IF we export the client cert to pkcs12/pfx before using it to create a TLS connection

        var pkcs12 = new X509Certificate2(certificate.Export(X509ContentType.Pkcs12));

        //Create the sslOptions to connect to rabbitmq
        var sslOptions = new SslOption()
        {
            Certs = new X509Certificate2Collection {
                    pkcs12
                },
            Enabled = true,
            ServerName = rabbitAddress
        };

I've tried different Flags when creating the PFX/Pkcs12 cert. E.g.

        var pkcs12 = new X509Certificate2(clientCert.Export(X509ContentType.Pkcs12), (string)null!, X509KeyStorageFlags.PersistKeySet | X509KeyStorageFlags.Exportable | X509KeyStorageFlags.MachineKeySet);

But it does not make a difference. When debugging on WSL there is no Key property on the client cert object. So my current assumption is that there is something strange happening in Windows regarding the private key handling ....

[tufberg]

What is interesting is that it works in WSL but not Windows. Can you provide more details? Are you using .NET core or .NET Fx in Windows? (asking b/c the netstandard mention). Are you using a custom CA that has been configured in one platform but not the other?

We are using .NET version 8.

    <TargetFramework>net8.0</TargetFramework>

[tufberg]

I simply can't let this go 😉

I found a way to make it work!!!. As mentioned above the AMQP client works so I had a look at the RabbitMQ AMQP client for .NET

So therefore I changed line 112 in /Source/MQTTnet/Implementations/MqttTcpChannel.cs to

var sslStream = new SslStream(networkStream, false, InternalUserCertificateValidationCallback, InternalUserLocalCertificateValidationCallback);

and implemented the InternalUserLocalCertificateValidationCallback as

        private X509Certificate InternalUserLocalCertificateValidationCallback(object sender, string targetHost, X509CertificateCollection localCertificates, X509Certificate remoteCertificate, string[] acceptableIssuers)
        {
            if (acceptableIssuers != null && acceptableIssuers.Length > 0 &&
                localCertificates != null && localCertificates.Count > 0)
            {
                foreach (X509Certificate certificate in localCertificates)
                {
                    if (Array.IndexOf(acceptableIssuers, certificate.Issuer) != -1)
                    {
                        return certificate;
                    }
                }
            }
            if (localCertificates != null && localCertificates.Count > 0)
            {
                return localCertificates[0];
            }

            return null;
        }

And voilà .... It works! 🎉

Tested it in WSL and it still works as expected there. :-)

[rido-min]

I'm very surprised it works in Windows but not in WSL under NET8.

if you want to just configure MQTTNet (without modifying MqttTcpChannel.cs) to validate the chain for netstandard you might want to try to configure WithCertificateValidationHandler and X509ChainValidator.cs (note this is not required in dotnet8)

to debug RabbitMQ you can inspect the TLS handlshake with openssl s_cllient -connect your-rabbitmq:8883

[tufberg]

I'm very surprised it works in Windows but not in WSL under NET8.

Maybe I'm a bit unclear or I do not understand :-)

The change I did above to implement InternalUserLocalCertificateValidationCallback is working both on WSL and Windows on .net8 client.

[tufberg]

if you want to just configure MQTTNet (without modifying MqttTcpChannel.cs) to validate the chain for netstandard you might want to try to configure WithCertificateValidationHandler and X509ChainValidator.cs (note this is not required in dotnet8)

I looked at this but I'm having a hard time to see how using WithCertificateHandler and X509ChainValidator would help in our platform. As mentioned above WSL works fine (.net 8) but Windows (.net 8) does not work.

Should I suggest a change to MqttTcpChannel with the following Diff? Would it be possible to get this into any upcoming version of MQTTNet ? 😊

diff.txt

[chkr1011]

@tufberg Please put the expected code in a comment. I will not download any files to my machine.

Or is it the code from https://github.com/dotnet/MQTTnet/issues/1978#issuecomment-2082551516

[chkr1011]

@rido-min Do you think the following code is a good practice which can be added to the library as the default behavior or do you think it has higher potential for security issues?

private X509Certificate InternalUserLocalCertificateValidationCallback(object sender, string targetHost, X509CertificateCollection localCertificates, X509Certificate remoteCertificate, string[] acceptableIssuers)
{
    if (acceptableIssuers != null && acceptableIssuers.Length > 0 &&
        localCertificates != null && localCertificates.Count > 0)
     {
        foreach (X509Certificate certificate in localCertificates)
        {
           if (Array.IndexOf(acceptableIssuers, certificate.Issuer) != -1)
            {
                return certificate;
            }
        }
    }

    if (localCertificates != null && localCertificates.Count > 0)
    {
        return localCertificates[0];
    }

    return null;
}

[rido-min]

I have some doubts about that code blurb.

Proper CA chain validation requires more checks, such as validity dates, OID constraints etc.., than just checking the issuer name.

Before making any decision I'd like to fully understand why it's working in WSL but no in Windows, and then come up with a good test plan.

I would start by defining what's the "reference broker with x509 auth" we want to use to validate. We've been using the existing version with mosquitto/aws/iothub with no issues so far, so if this is something "special" about RabbitMQ they should document the expected behavior.

[chkr1011]

Thanks for sharing your thoughts. Then I will expose the certificate selection handler in the MQTTnet options and do not implement a default handler. MQTTnet will choose the best constructor based on the provided handlers. Then users can execute their own checks and having no custom handler will lead to the same code as when not providing a handler.

https://github.com/dotnet/MQTTnet/pull/1984

@tufberg I added a new handler for certificate selection to the client options. This should allow you to properly select the best matching certificate. Even though it should work with .NET 8.0 exposing the handler anyways should give users more flexibility.

[tufberg]

Awesome @chkr1011 and thanks @rido-min for quick responses and feedback. Greatly appreciated!

When PR #1984 is merged we can implement a handler that takes care of the current problem and add that to the options object. That would help a lot👍 Will keep an eye for upcoming releases.

I have some doubts about that code blurb.

I did too.... 🙈 I borrowed it from RabbitMQ .Net AMQP client code base. Apparently RabbitMQ is doing something special in their SSLImplementation and therefore that code have been added to their .Net AMQP Client. Developers at RabbitMQ have been very helpful on Discord regarding other stuff so maybe a friendly question is in order?

We have looked at several different brokers and the choice landed on RabbitMQ which also supports MQTT v5 since a couple of months back. Have not tried v5 implementation yet though. Adding RabbitMQ to the list of brokers to test sounds like a good plan. MQTTNet works really well and we have used it a lot in other projects, not with RabbitMQ, but with customers using AWS or Mosquitto.

Again, all help here is very much appreciated. Ping me if you need any help.