"Invalid provider type" error signing Token on one node in web farm but other working fine

[lastbuilders]

Hi All,

I am implementing Identity Server 3 on a Web Farm and am running into a problem with one node whereby the error below is thrown linked to signing the access token. The same configuration steps have been completed on both nodes,

The Servers are both Windows 2012 R2.

The certificate was generated using the following command I found here (https://brockallen.com/2015/06/01/makecert-and-creating-ssl-or-signing-certificates/) makecert -r -pe -n "CN=%1" -b 01/01/2015 -e 01/01/2020 -eku 1.3.6.1.5.5.7.3.3 -sky signature -a sha256 -len 2048 -ss my -sr LocalMachine

To register the certificate I followed the following steps: Open mmc.exe and add the certificates snap-in. Use the local computer store.

1. Expand Personal under Certificates (Local Computer)
2. Right Click Certificates
3. Select All Tasks -> Import...
4. Click Next
5. Browse to the testpfx file
6. Click Next
7. Enter test as the password
8. Click Next
9. Click Next (again)
10. Click Finish

Grant Access to the imported certificate

1. Go to Personal under Certificates (Local Computer) -> Personal -> Certificates
2. Right click the certificate
3. Select All Tasks
4. Click Manage private key on the imported certificate
5. Grant the NetworkService account read access

So far I have checked (and rechecked several times :)) the following between the 2 servers to try and isolate the problem but the settings on the servers are consistent:

1. The ID Server application pool is running under the Network Service account
2. The Certificate is in certificate Manager
3. Permissions on the "\ProgramData\Microsoft\Crypto\RSA\MachineKeys" are set so Network Service has access to the Certificate
4. I tried regenerating the certificate but the same issue occurs

From the error it appears that the ID Server is finding the Certificate but is not able to use it correctly. I am hoping for suggestions a way to resolve this or to troubleshoot it further?

Thanks, Barry

Relevant parts of the log file

2017-01-24 10:09:52.929 -05:00 [Debug] Start client validation
2017-01-24 10:09:52.929 -05:00 [Debug] Start parsing Basic Authentication secret
2017-01-24 10:09:52.929 -05:00 [Debug] Start parsing for secret in post body
2017-01-24 10:09:52.929 -05:00 [Debug] Parser found secret: "PostBodySecretParser"
w3wp.exe Information: 0 : 2017-01-24 10:09:52.929 -05:00 [Information] Secret id found: "d14e42c2-1a97-40a9-9cf1-33a6e867949b"
w3wp.exe Information: 0 : "24-01-2017 10:09:52:9294","Entering","RetrieveClients","RetrieveClients"
w3wp.exe Information: 0 : "24-01-2017 10:09:52:9294","Entering","RegistryFunctions_GetRegXmlValue","ServiceCallHelper.RegistryFunctions_GetRegXmlValue"
w3wp.exe Information: 0 : "24-01-2017 10:09:52:9450","Exiting","RegistryFunctions_GetRegXmlValue","ServiceCallHelper.RegistryFunctions_GetRegXmlValue"
w3wp.exe Information: 0 : "24-01-2017 10:09:52:9450","Entering","RegistryFunctions_GetRegXmlValue","ServiceCallHelper.RegistryFunctions_GetRegXmlValue"
w3wp.exe Information: 0 : "24-01-2017 10:09:52:9450","Exiting","RegistryFunctions_GetRegXmlValue","ServiceCallHelper.RegistryFunctions_GetRegXmlValue"
w3wp.exe Information: 0 : "24-01-20
17 10:09:52:9450","Entering","ExecuteDataSet","DataAccessServiceHelper.ExecuteDataSet"
w3wp.exe Information: 0 : "24-01-2017 10:09:52:9606","Exiting","ExecuteDataSet","DataAccessServiceHelper.ExecuteDataSet"
w3wp.exe Information: 0 : "24-01-2017 10:09:52:9606","Exiting","RetrieveClients","RetrieveClients"
2017-01-24 10:09:52.960 -05:00 [Debug] Secret validator success: "HashedSharedSecretValidator"
w3wp.exe Information: 0 : 2017-01-24 10:09:52.960 -05:00 [Information] Client validation success
w3wp.exe Information: 0 : 2017-01-24 10:09:52.960 -05:00 [Information] Start token request validation
w3wp.exe Information: 0 : 2017-01-24 10:09:52.960 -05:00 [Information] Start client credentials token request validation
w3wp.exe Information: 0 : "24-01-2017 10:09:52:9606","Entering","RetrieveScopes","RetrieveScopes"
w3wp.exe Information: 0 : "24-01-2017 10:09:52:9606","Entering","RegistryFunctions_GetRegXmlValue","ServiceCallHelper.RegistryFunctions_GetRegXmlValue"
w3wp.exe Information: 0 : "24-01-2017 10:09:52:9606","Exiting","RegistryFunctions_GetRegXmlValue","ServiceCallHelper.RegistryFunctions_GetRegXmlValue"
w3wp.exe Information: 0 : "24-01-2017 10:09:52:9606","Entering","RegistryFunctions_GetRegXmlValue","ServiceCallHelper.RegistryFunctions_GetRegXmlValue"
w3wp.exe Information: 0 : "24-01-2017 10:09:52:9762","Exiting","RegistryFunctions_GetRegXmlValue","ServiceCallHelper.RegistryFunctions_GetRegXmlValue"
w3wp.exe Information: 0 : "24-01-2017 10:09:52:9762","Entering","ExecuteDataSet","DataAccessServiceHelper.ExecuteDataSet"
w3wp.exe Information: 0 : "24-01-2017 10:09:52:9762","Exiting","ExecuteDataSet","DataAccessServiceHelper.ExecuteDataSet"
w3wp.exe Information: 0 : "24-01-2017 10:09:52:9762","Exiting","RetrieveScopes","RetrieveScopes"
w3wp.exe Information: 0 : 2017-01-24 10:09:52.976 -05:00 [Information] Client credentials token request validation success
w3wp.exe Information: 0 : 2017-01-24 10:09:52.976 -05:00 [Information] Token request validation success
{
  "ClientId": "d14e42c2-1a97-40a9-9cf1-33a6e867949b",
  "ClientName": "Demo",
  "GrantType": "client_credentials",
  "Scopes": "testwebapi",
  "Raw": {
    "grant_type": "client_credentials",
    "client_id": "d14e42c2-1a97-40a9-9cf1-33a6e867949b",
    "client_secret": "******",
    "scope": "testwebapi"
  }
}
w3wp.exe Information: 0 : 2017-01-24 10:09:52.976 -05:00 [Information] Creating token response
w3wp.exe Information: 0 : 2017-01-24 10:09:52.976 -05:00 [Information] Processing token request
2017-01-24 10:09:52.976 -05:00 [Debug] Creating access token
2017-01-24 10:09:52.976 -05:00 [Debug] Creating JWT access token
w3wp.exe Error: 0 : 2017-01-24 10:09:52.976 -05:00 [Error] Unhandled exception
System.InvalidOperationException: IDX10614: AsymmetricSecurityKey.GetSignatureFormater( 'http://www.w3.org/2001/04/xmldsig-more#rsa-sha256' ) threw an exception.
Key: 'System.IdentityModel.Tokens.X509AsymmetricSecurityKey'
SignatureAlgorithm: 'http://www.w3.org/2001/04/xmldsig-more#rsa-sha256', check to make sure the SignatureAlgorithm is supported.
Exception:'System.Security.Cryptography.CryptographicException: Invalid provider type specified.

   at System.Security.Cryptography.Utils.CreateProvHandle(CspParameters parameters, Boolean randomKeyContainer)
   at System.Security.Cryptography.Utils.GetKeyPairHelper(CspAlgorithmType keyType, CspParameters parameters, Boolean randomKeyContainer, Int32 dwKeySize, SafeProvHandle& safeProvHandle, SafeKeyHandle& safeKeyHandle)
   at System.Security.Cryptography.RSACryptoServiceProvider.GetKeyPair()
   at System.Security.Cryptography.RSACryptoServiceProvider..ctor(Int32 dwKeySize, CspParameters parameters, Boolean useDefaultKeySize)
   at System.Security.Cryptography.X509Certificates.X509Certificate2.get_PrivateKey()
   at System.IdentityModel.Tokens.X509AsymmetricSecurityKey.get_PrivateKey()
   at System.IdentityModel.Tokens.X509AsymmetricSecurityKey.GetSignatureFormatter(String algorithm)
   at System.IdentityModel.Tokens.AsymmetricSignatureProvider..ctor(AsymmetricSecurityKey key, String algorithm, Boolean willCreateSignatures)'.
If you only need to verify signatures the parameter 'willBeUseForSigning' should be false if the private key is not be available. ---> System.Security.Cryptography.CryptographicException: Invalid provider type specified.

   at System.Security.Cryptography.Utils.CreateProvHandle(CspParameters parameters, Boolean randomKeyContainer)
   at System.Security.Cryptography.Utils.GetKeyPairHelper(CspAlgorithmType keyType, CspParameters parameters, Boolean randomKeyContainer, Int32 dwKeySize, SafeProvHandle& safeProvHandle, SafeKeyHandle& safeKeyHandle)
   at System.Security.Cryptography.RSACryptoServiceProvider.GetKeyPair()
   at System.Security.Cryptography.RSACryptoServiceProvider..ctor(Int32 dwKeySize, CspParameters parameters, Boolean useDefaultKeySize)
   at System.Security.Cryptography.X509Certificates.X509Certificate2.get_PrivateKey()
   at System.IdentityModel.Tokens.X509AsymmetricSecurityKey.get_PrivateKey()
   at System.IdentityModel.Tokens.X509AsymmetricSecurityKey.GetSignatureFormatter(String algorithm)
   at System.IdentityModel.Tokens.AsymmetricSignatureProvider..ctor(AsymmetricSecurityKey key, String algorithm, Boolean willCreateSignatures)
   --- End of inner exception stack trace ---
   at System.IdentityModel.Tokens.AsymmetricSignatureProvider..ctor(AsymmetricSecurityKey key, String algorithm, Boolean willCreateSignatures)
   at System.IdentityModel.Tokens.SignatureProviderFactory.CreateProvider(SecurityKey key, String algorithm, Boolean willCreateSignatures)
   at System.IdentityModel.Tokens.SignatureProviderFactory.CreateForSigning(SecurityKey key, String algorithm)
   at System.IdentityModel.Tokens.JwtSecurityTokenHandler.CreateSignature(String inputString, SecurityKey key, String algorithm, SignatureProvider signatureProvider)
   at System.IdentityModel.Tokens.JwtSecurityTokenHandler.WriteToken(SecurityToken token)
   at IdentityServer3.Core.Services.Default.DefaultTokenSigningService.<SignAsync>d__11.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at IdentityServer3.Core.Services.Default.DefaultTokenSigningService.<CreateJsonWebToken>d__8.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at IdentityServer3.Core.Services.Default.DefaultTokenSigningService.<SignTokenAsync>d__0.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at IdentityServer3.Core.Services.Default.DefaultTokenService.<CreateSecurityTokenAsync>d__c.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at IdentityServer3.Core.ResponseHandling.TokenResponseGenerator.<CreateAccessTokenAsync>d__29.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at IdentityServer3.Core.ResponseHandling.TokenResponseGenerator.<ProcessTokenRequestAsync>d__11.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at IdentityServer3.Core.ResponseHandling.TokenResponseGenerator.<ProcessAsync>d__0.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at IdentityServer3.Core.Endpoints.TokenEndpointController.<ProcessAsync>d__7.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at IdentityServer3.Core.Endpoints.TokenEndpointController.<Post>d__0.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Threading.Tasks.System.Web.Http908956.TaskHelpersExtensions.<CastToObject>d__3`1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Web.Http.Controllers.ApiControllerActionInvoker.<InvokeActionAsyncCore>d__0.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Web.Http.Filters.ActionFilterAttribute.<CallOnActionExecutedAsync>d__5.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Web.Http.Filters.ActionFilterAttribute.<CallOnActionExecutedAsync>d__5.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Web.Http.Filters.ActionFilterAttribute.<ExecuteActionFilterAsyncCore>d__0.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Web.Http.Controllers.ActionFilterResult.<ExecuteAsync>d__2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Web.Http.Filters.AuthorizationFilterAttribute.<ExecuteAuthorizationFilterAsyncCore>d__2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Web.Http.Dispatcher.HttpControllerDispatcher.<SendAsync>d__1.MoveNext()

[brockallen]

So it works on one machine, but not another? Sounds like an environmental issue.

[lastbuilders]

Thanks Brock,

I was thinking environmental too. I am wondering is there something else I am missing that could cause this behaviour or are there any troueshooting utilities I could use to investigate it further?

Regards, Barry

[brockallen]

Nothing comes to mind, sorry.

[lastbuilders]

Hi Brock, Thanks for getting back to me. I did some research and found some articles which resolved it by resetting permissions on the MachineKeys folder. https://www.reddit.com/r/sysadmin/comments/339ogk/this_certificate_issue_invalid_provider_type_has/

Barry

[brockallen]

So that was the issue?

[lastbuilders]

Yes, that was it. After resetting the permissions the IDServer signing certificate worked.

[brockallen]

Ok, thanks for the update. Glad it's working now.