HTTPS Requests fail on ASP.NET Core + Service Fabric on Ubuntu

[tastyeggs]

When a ASP.NET Core is deployed on an Ubuntu instance, HTTPS requests start failing with the following exception:

Uncaught exception from the OnConnectionAsync method of an IConnectionAdapter.|System.Security.Authentication.AuthenticationException: A call to SSPI failed, see inner exception. ---> System.UnauthorizedAccessException: Access to the path '/home/sfappsuser/.dotnet/corefx/cryptography/x509stores/ca' is denied. ---> System.IO.IOException: Permission denied
   --- End of inner exception stack trace ---
   at System.IO.UnixFileSystem.CreateDirectory(String fullPath)
   at System.IO.Directory.CreateDirectory(String path)
   at Internal.Cryptography.Pal.DirectoryBasedStoreProvider.Add(ICertificatePal certPal)
   at System.Security.Cryptography.X509Certificates.X509Store.Add(X509Certificate2 certificate)
   at Internal.Cryptography.Pal.ChainPal.SaveIntermediateCertificates(X509ChainElement[] chainElements, HashSet`1 downloaded)
   at Internal.Cryptography.Pal.ChainPal.BuildChain(Boolean useMachineContext, ICertificatePal cert, X509Certificate2Collection extraStore, OidCollection applicationPolicy, OidCollection certificatePolicy, X509RevocationMode revocationMode, X509RevocationFlag revocationFlag, DateTime verificationTime, TimeSpan timeout)
   at System.Security.Cryptography.X509Certificates.X509Chain.Build(X509Certificate2 certificate, Boolean throwOnException)
   at System.Security.Cryptography.X509Certificates.X509Chain.Build(X509Certificate2 certificate)
   at System.Net.Http.TLSCertificateExtensions.BuildNewChain(X509Certificate2 certificate, Boolean includeClientApplicationPolicy)
   at Interop.OpenSsl.AllocateSslContext(SslProtocols protocols, SafeX509Handle certHandle, SafeEvpPKeyHandle certKeyHandle, EncryptionPolicy policy, Boolean isServer, Boolean remoteCertRequired)
   at System.Net.Security.SafeDeleteSslContext..ctor(SafeFreeSslCredentials credential, Boolean isServer, Boolean remoteCertRequired)
   at System.Net.Security.SslStreamPal.HandshakeInternal(SafeFreeCredentials credential, SafeDeleteContext& context, SecurityBuffer inputBuffer, SecurityBuffer outputBuffer, Boolean isServer, Boolean remoteCertRequired)
   --- End of inner exception stack trace ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Net.Security.SslState.StartSendAuthResetSignal(ProtocolToken message, AsyncProtocolRequest asyncRequest, ExceptionDispatchInfo exception)
   at System.Net.Security.SslState.CheckCompletionBeforeNextReceive(ProtocolToken message, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.StartSendBlob(Byte[] incoming, Int32 count, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.ProcessReceivedBlob(Byte[] buffer, Int32 count, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.StartReadFrame(Byte[] buffer, Int32 readBytes, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.PartialFrameCallback(AsyncProtocolRequest asyncRequest)
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Net.Security.SslState.InternalEndProcessAuthentication(LazyAsyncResult lazyResult)
   at System.Net.Security.SslState.EndProcessAuthentication(IAsyncResult result)
   at System.Net.Security.SslStream.EndAuthenticateAsServer(IAsyncResult asyncResult)
   at System.Threading.Tasks.TaskFactory`1.FromAsyncCoreLogic(IAsyncResult iar, Func`2 endFunction, Action`1 endAction, Task`1 promise, Boolean requiresSynchronization)
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.AspNetCore.Server.Kestrel.Https.Internal.HttpsConnectionAdapter.<InnerOnConnectionAsync>d__12.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.FrameConnection.<ApplyConnectionAdaptersAsync>d__50.MoveNext()

This is likely because SF on Ubuntu runs the application as user sfappsuser, and ASP.NET Core is trying to build a certificate cache in the user's home directory, which isn't created.

Creating the above path -- /home/sfappsuser/.dotnet/corefx/cryptography/x509stores/ca makes this work.

This doesn't happen on Windows.

[ashishnegi]

Is there a way to change that directory to point to your local app work directory where your running app has permissions to write ? May be they provide some environment variable for the path.

[tastyeggs]

@ashishnegi, not that I know of.

Things should just work on first run, and ASP.NET Core is seen as a first-class member of Service Fabric, so it should work on first run :)

I think the issue here is:

1. ASP.NET Core by default, looks to locate the cache on the user's home directory, discussion here. This issue isn't going to be resolved, as they believe storing the certificate in the user's home directory is the best option
2. Service Fabric creates a user without a home directory.

So the only way out here is for service fabric to create the sfappuser user, along with the home directory, which is a very easy change to make to the service fabric installer.

[ashishnegi]

/cc @maburlik for question regarding :

So the only way out here is for service fabric to create the sfappuser user, along with the home directory, which is a very easy change to make to the service fabric installer.