ModuleClient.OpenAsync throws AuthenticationException

[SLdragon]

• OS, version: Ubuntu 18.04
• Application's .NET Target Framework: dotnet 2.2.202
• Device: Azure VM, VirtualBox VM, Physical Device
• SDK version used: 1.19.0, 1.20.0

Description of the issue:

• On Ubuntu 18.04, Install Docker, VSCode, dotnet SDK, iotedgehubdev, Azure IoT Edge Extension for VSCode
• Follow the steps from tutorial to create a new CSharp Edge Module and try to debug it:
  • Create a new solution template
  • Select C# module
  • Input sample module name, and registry url
  • From command Palette, start simulator for single module
    • Try to start local debug
    • Set a breakpoint in the Init function.
    • When the breakpoint hits, execute the ModuleClient.CreateFromEnvironmentAsync function, successfully
    • Upon return, attempt to execute the OpenAsync method.See the exception.

Other information:

• Only limited machines would throw this error, and we cannot find any clue where is the root cause
• Node JS modules work fine for all the machines, only C# modules would throw this error
• You can refer this issue for more information: https://github.com/Azure/iotedge/issues/1044

Code sample exhibiting the issue:

namespace SampleModule
{
    using System;
    using System.IO;
    using System.Runtime.InteropServices;
    using System.Runtime.Loader;
    using System.Security.Cryptography.X509Certificates;
    using System.Text;
    using System.Threading;
    using System.Threading.Tasks;
    using Microsoft.Azure.Devices.Client;
    using Microsoft.Azure.Devices.Client.Transport.Mqtt;

    class Program
    {
        static int counter;

        static void Main(string[] args)
        {
            Init().Wait();

            // Wait until the app unloads or is cancelled
            var cts = new CancellationTokenSource();
            AssemblyLoadContext.Default.Unloading += (ctx) => cts.Cancel();
            Console.CancelKeyPress += (sender, cpe) => cts.Cancel();
            WhenCancelled(cts.Token).Wait();
        }

        /// <summary>
        /// Handles cleanup operations when app is cancelled or unloads
        /// </summary>
        public static Task WhenCancelled(CancellationToken cancellationToken)
        {
            var tcs = new TaskCompletionSource<bool>();
            cancellationToken.Register(s => ((TaskCompletionSource<bool>)s).SetResult(true), tcs);
            return tcs.Task;
        }

        /// <summary>
        /// Initializes the ModuleClient and sets up the callback to receive
        /// messages containing temperature information
        /// </summary>
        static async Task Init()
        {
            MqttTransportSettings mqttSetting = new MqttTransportSettings(TransportType.Mqtt_Tcp_Only);
            ITransportSettings[] settings = { mqttSetting };

            // Open a connection to the Edge runtime
            ModuleClient ioTHubModuleClient = await ModuleClient.CreateFromEnvironmentAsync(settings);
            await ioTHubModuleClient.OpenAsync();
            Console.WriteLine("IoT Hub module client initialized.");

            // Register callback to be called when a message is received by the module
            await ioTHubModuleClient.SetInputMessageHandlerAsync("input1", PipeMessage, ioTHubModuleClient);
        }

        /// <summary>
        /// This method is called whenever the module is sent a message from the EdgeHub. 
        /// It just pipe the messages without any change.
        /// It prints all the incoming messages.
        /// </summary>
        static async Task<MessageResponse> PipeMessage(Message message, object userContext)
        {
            int counterValue = Interlocked.Increment(ref counter);

            var moduleClient = userContext as ModuleClient;
            if (moduleClient == null)
            {
                throw new InvalidOperationException("UserContext doesn't contain " + "expected values");
            }

            byte[] messageBytes = message.GetBytes();
            string messageString = Encoding.UTF8.GetString(messageBytes);
            Console.WriteLine($"Received message: {counterValue}, Body: [{messageString}]");

            if (!string.IsNullOrEmpty(messageString))
            {
                var pipeMessage = new Message(messageBytes);
                foreach (var prop in message.Properties)
                {
                    pipeMessage.Properties.Add(prop.Key, prop.Value);
                }
                await moduleClient.SendEventAsync("output1", pipeMessage);
                Console.WriteLine("Received message sent");
            }
            return MessageResponse.Completed;
        }
    }
}

Console log of the issue:

Exception has occurred: CLR/System.AggregateException
An unhandled exception of type 'System.AggregateException' occurred in System.Private.CoreLib.dll: 'One or more errors occurred.'
 Inner exceptions found, see $exception in variables window for more details.
 Innermost exception     System.Security.Authentication.AuthenticationException : The remote certificate is invalid according to the validation procedure.
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Net.Security.SslState.StartSendAuthResetSignal(ProtocolToken message, AsyncProtocolRequest asyncRequest, ExceptionDispatchInfo exception)
   at System.Net.Security.SslState.CheckCompletionBeforeNextReceive(ProtocolToken message, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.StartSendBlob(Byte[] incoming, Int32 count, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.ProcessReceivedBlob(Byte[] buffer, Int32 count, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.StartReadFrame(Byte[] buffer, Int32 readBytes, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.PartialFrameCallback(AsyncProtocolRequest asyncRequest)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Net.Security.SslState.ThrowIfExceptional()
   at System.Net.Security.SslState.InternalEndProcessAuthentication(LazyAsyncResult lazyResult)
   at System.Net.Security.SslState.EndProcessAuthentication(IAsyncResult result)
   at System.Net.Security.SslStream.EndAuthenticateAsClient(IAsyncResult asyncResult)
   at System.Net.Security.SslStream.<>c.<AuthenticateAsClientAsync>b__46_2(IAsyncResult iar)
   at System.Threading.Tasks.TaskFactory`1.FromAsyncCoreLogic(IAsyncResult iar, Func`2 endFunction, Action`1 endAction, Task`1 promise, Boolean requiresSynchronization)

[JetstreamRoySprowl]

FWIW I've been working on this as well, and I have the suspicion that the certificate expiry math is missing a constant so that it's expiring in X hours instead of X days or similar.

[drewf7]

Just wanted to leave a comment since I came across this github issue while searching for that particular error.

In our case the problem was that the service being authenticated (iotedge for us) seems to strip underscores out of device hostnames.

So we had a device with underscores in the name, and provisioned a certificate with that hostname. The framework then stripped the underscores causing the hostname to no longer match that which the certificate was provisioned for.

Causing a generic TLS Authentication Error.

Again this might not be the cause in your particular case, but I wanted to leave something here in case others with the same issue stumble on this thread.

[ancaantochi]

@SLdragon,

Are the certificates generated every time there is a new run or how are they generated? It might be similar to an issue that we had with quick start certs and got fixed by adding authorityKeyIdentifier to the certificates extension. You can find here the configuration for openssl that we use: https://github.com/Azure/iotedge/blob/master/tools/CACertificates/openssl_root_ca.cnf

This issue happens when multiple certs are generated for the same device and on Linux the SDK installs the trusted root in the dotnet trust store. Sometimes dotnet can't build the cert chain correctly because it can't find the correct root, if the certs contain authorityKeyIdentifier the chain is build correctly.

[SLdragon]

Thank you @ancaantochi for your information, as @adashen discussed with me yeasterday, she also found if we clear the cert folder (~/.dotnet/corefx/cryptography/x509stores/root) from linux, then reset the cert, and the sample would work.

I am not sure whether it is related to the dotnet runtime or iot c# sdk, so are there some plans to fix this issue from iot c# sdk side to let it choose the right certificate?

[ancaantochi]

@SLdragon , I think the issue is in dotnet core, to be able to correctly construct the chain the certs have to be generated with the authorityKeyIdentifier extension.

Removing the certs from x509stores is a good workaround.

[abhipsaMisra]

@SLdragon Since this looks like a dotnet core limitation on Linux, there isn't much we can do from the SDK side. Please try using the above suggested solution of generating certificates with authorityKeyIdentifier extension, which will assist in the correct certificate chain construction.
Closing this issue; in case you continue hitting an error even with the above solution, please reopen another issue and let us know. Thanks!

[az-iot-builder-01]

@JetstreamRoySprowl, @drewf7, @ancaantochi, @SLdragon, @abhipsaMisra, thank you for your contribution to our open-sourced project! Please help us improve by filling out this 2-minute customer satisfaction survey