Configuring Azure AD Authentication using a Client Secret fails with Unexpected Character Error

Clare-Octopus commented 2 years ago

Team

[X] I've assigned a team label to this issue

Severity

Blocking both On-Prem and Cloud customers who want to use Client Secret for Azure Ad Auth

Version

2022.2.6729, latest Cloud builds

Latest Version

I could reproduce the problem in the latest build

What happened?

Following our documentation on setting up Azure AD with a client secret fails when you sign into Octopus using Microsoft with the below error (line positioning in error may be different):

Reproduction

Follow our documentation to setup Azure Ad Authentication using the Client Secret.
Once configured in the UI sign out and sign in with Microsoft.
Input email and password and try and sign in.
See error message appear.

Error and Stacktrace

{
  "ErrorMessage": "Unexpected character encountered while parsing value: [. Path 'error_codes', line 1, position 500."
}

More Information

Workaround

The workaround we have is to follow this specific section in our Azure AD Authentication documentation to enable implicit grant with the ID Token.

You can then setup Azure Ad Auth the same but not input the Client Secret. This will allow you to use Azure AD Auth.

One thing to note here is that by doing it this way you are using implicit flow to authenticate with Azure Ad, which is less secure. By using the client secret you use PKCE which is more secure. Once we have a fix in place for this you will be able to take advantage of PKCE by using the Client Secret.

You can read more on the differences between PKCE and Implicit Flow here.

SeanStanway-Octopus commented 1 year ago

User coming across this: internal link (User is on latest Cloud instance 2022.4.3913-hotfix.4575)

Clare-Octopus commented 1 year ago

Another user on Cloud (Internal link) - https://octopus.zendesk.com/agent/tickets/100895 User is on 2022.4.5100-hotfix.7233 I will edit this GitHub Issue as its clearly not just On-Prem customers with the issue. Taking out the Client secret and making sure implicit grant with the ID token was setup fixed the issue for this customer.

Clare-Octopus commented 1 year ago

Another customer affected (internal link) - https://octopus.zendesk.com/agent/tickets/101193

Clare-Octopus commented 1 year ago

Another user affected - https://help.octopus.com/t/azure-aad-octopus-instance-connection/28927/3

Clare-Octopus commented 1 year ago

Another user on cloud latest affected - https://octopus.zendesk.com/agent/tickets/119868

Clare-Octopus commented 1 year ago

Good afternoon everyone that has subscribed to this issue. We have been revisiting it in order to try and resolve it but unfortunately we have not actually been able to reproduce it since. I have even re-tested on 2022.2.6729 but cannot seem to replicate the original issue. I am wondering if this was a blip on the Microsoft side.

The only way I can get the Unexpected character encountered error now is to actually put an incorrect value in for the Client Secret, which I am sure I double checked that was correct in my testing and I do not think 5 other separate users also entered an incorrect client secret.

Would anyone be able to test this for us again by creating a client secret in their Azure Application and inputting that into Octopus and see if you get any errors on sign in.

If you could let us know if you are getting the error still and you would not mind us emailing you to share some of your Azure setup to see if I missed anything in my testing.

We would love to hear from any customer experiencing this issue if they are still seeing it after re-entering their client secret. Kind Regards, Clare

Scott-Emberson commented 1 year ago

We have this issue on V2023.1.10046

Clare-Octopus commented 1 year ago

Another customer affected (internal) - https://octopus.zendesk.com/agent/tickets/126932

Clare-Octopus commented 1 year ago

I have been testing this for a few weeks and unfortunately am unable to see a consistent pattern of when or why this happens. It seems to happen mostly on initial Azure AD setup but only sometimes and only on certain Octopus releases (though I am unsure if that is just a co-incidence). I did manage to get the error to occur on 2023.1.10046 but when I then span up a brand new DB and instance of that same release it did not happen on initial setup so this does not seem to be Octopus release specific.

Our engineers have added additional logging to 2023.2.10685 which does give us the below (better) error when this occurs:

{"error":"invalid_client","error_description":"AADSTS7000215: Invalid client secret provided. Ensure the secret being sent in the request is the client secret value, not the client secret ID, for a secret added to app '2a40076f-2f81-4637-a5b8-e0f804dec1c5'.\r\nTrace ID: 6bd0ccfe-eb58-41f8-8585-5dd563f91000\r\nCorrelation ID: 5afa0a4d-308d-4650-b5f4-5c123ac4ceee\r\nTimestamp: 2023-05-19 14:39:15Z","error_codes":[7000215],"timestamp":"2023-05-19 14:39:15Z","trace_id":"6bd0ccfe-eb58-41f8-8585-5dd563f91000","correlation_id":"5afa0a4d-308d-4650-b5f4-5c123ac4ceee","error_uri":"https://login.microsoftonline.com/error?code=7000215"}
2023-05-19 14:39:17.5687   7208     89 ERROR  Unhandled error on request: "Could not authenticate against identity provider. Please check your configuration."

On googling the error code (AADSTS7000215) the suggestions are the user is either putting in the client secret ID not the value (which I know is not the case as I was using the client secret value every time and still seeing the issue). Or the suggestion was to delete your app registration and create a new one.

From all of the testing I have done I did find I could only get this issue to happen once, usually on initial setup of the Azure AD integration with Octopus. Since this error is in google we are thinking its a quirk with Azure not Octopus, since I cant replicate it across releases or even with the same releases but with new blank instances.

I did find that, if I got the error_codes error (and with the new error which is the same issue) if I used a private window or cleared my cookies and cache, I was able to login every time. So this does look like its only on initial connection of Octopus to your Azure App registration, but only sometimes, and its not consistent even on the same blank, new Octopus instance of the same release.

We are sorry we have still not gotten to the bottom of this yet, since we are unable to replicate this consistently we are finding it very hard to narrow down why this is happening.

If any user is affected by this I would recommend trying:

On initial show of that error, try opening a private browser window and logging in again.
If that does not work, delete the app registration and re-create it with a client secret.
If that does not work follow the recommended workaround (Ensure ID Token is selected and remove the Client Secret from Octopus).

If any of those top two workarounds work for you would you be able to pop a note in here to say which one worked as we are also struggling to establish a pattern for the workarounds which allow you to still use the client secret.

Kind Regards, Clare

Clare-Octopus commented 1 year ago

Another user affected by this (internal ticket) - https://octopus.zendesk.com/agent/tickets/153991 Customer is self hosted on 2023.1.12035

paraicoceallaigh commented 10 months ago

Looks like another customer affected: https://octopus.zendesk.com/agent/tickets/164706

jimh-datacom commented 9 months ago

This is what I get when I try in a new private window. { "ErrorMessage": "The type arguments for method 'Nevermore.IWriteQueryExecutor.DeleteAsync<TDocument>(TDocument, Nevermore.DeleteOptions, System.Threading.CancellationToken)' cannot be inferred from the usage. Try specifying the type arguments explicitly." }

OctopusDeploy / Issues