[GHES] GitHubTokenExchangeError: Internal Server Error

[SuibianP]

Type: Bug

Logging in to a self-hosted GitHub Enterprise server from the GitHub sidebar results in "Having trouble logging in? Would you like to try a different way?" and finally "Error signing in to GitHub Enterprise".

Logs

2022-12-09 14:31:11.399 [info] Reading sessions from keychain...
2022-12-09 14:31:11.406 [info] Getting sessions for read:user,repo,user:email,workflow...
2022-12-09 14:31:11.406 [info] Got 0 sessions for read:user,repo,user:email,workflow...
2022-12-09 14:31:11.437 [info] Getting sessions for read:user,repo,user:email...
2022-12-09 14:31:11.437 [info] Got 0 sessions for read:user,repo,user:email...
2022-12-09 14:31:11.454 [info] Getting sessions for read:user,repo,user:email,workflow...
2022-12-09 14:31:11.454 [info] Got 0 sessions for read:user,repo,user:email,workflow...
2022-12-09 14:31:17.161 [info] Logging in for the following scopes: read:user repo user:email workflow
2022-12-09 14:31:17.170 [info] Trying without local server... (read:user repo user:email workflow)
2022-12-09 14:31:27.736 [info] Exchanging code for token...
2022-12-09 14:31:28.467 [error] GitHubTokenExchangeError: Internal Server Error
    at t.GitHubServer.u (/Applications/Visual Studio Code.app/Contents/Resources/app/extensions/github-authentication/dist/extension.js:2:1309015)
    at process.processTicksAndRejections (node:internal/process/task_queues:96:5)
    at async /Applications/Visual Studio Code.app/Contents/Resources/app/extensions/github-authentication/dist/extension.js:2:1304423
2022-12-09 14:31:36.464 [info] Trying with local server... (read:user repo user:email workflow)
2022-12-09 14:31:38.721 [info] Exchanging code for token...
2022-12-09 14:31:39.081 [error] GitHubTokenExchangeError: Internal Server Error
    at t.GitHubServer.u (/Applications/Visual Studio Code.app/Contents/Resources/app/extensions/github-authentication/dist/extension.js:2:1309015)
    at process.processTicksAndRejections (node:internal/process/task_queues:96:5)
    at async /Applications/Visual Studio Code.app/Contents/Resources/app/extensions/github-authentication/dist/extension.js:2:1305597
2022-12-09 14:31:46.451 [info] Trying device code flow... (read:user repo user:email workflow)
2022-12-09 14:31:47.176 [error] Error: Failed to get one-time code: Cookies must be enabled to use GitHub.
    at t.GitHubServer.p (/Applications/Visual Studio Code.app/Contents/Resources/app/extensions/github-authentication/dist/extension.js:2:1305870)
    at process.processTicksAndRejections (node:internal/process/task_queues:96:5)
    at async t.GitHubServer.login (/Applications/Visual Studio Code.app/Contents/Resources/app/extensions/github-authentication/dist/extension.js:2:1303463)
    at async t.GitHubAuthenticationProvider.createSession (/Applications/Visual Studio Code.app/Contents/Resources/app/extensions/github-authentication/dist/extension.js:2:1299643)

Extension version: 0.57.2022120809 VS Code version: Code 1.74.0 (Universal) (5235c6bb189b60b01b1f49062f4ffa42384f8c91, 2022-12-05T16:43:37.5

[akhileshraju]

I see a similar error

Extension version - 0.56.0 VS Code version - 1.74.1 GHES version - 3.4.9

[VSCodeTriageBot]

Thanks for creating this issue! It looks like you may be using an old version of VS Code, the latest stable release is 1.74.1. Please try upgrading to the latest version and checking whether this issue remains.

Happy Coding!

[SuibianP]

Confirmed reproducible on latest version (1.74.2).

[TylerLeonhardt]

Hi @SuibianP @akhileshraju. Couple questions for you two:

1. Is your GitHub Enterprise server accessible from anywhere? In other words, do you need a VPN to resolve your GHES url?
2. Are you using a Remote extension like Remote SSH?
3. When it fails, do you get a notification that says "Do you want to try a different way"? If so, can you click "Yes" and try that different way? If that way also fails, can you try the 3rd way?

Oh I see, @SuibianP tried all 3 auth flows. This is probably coming from your GHES instance:

Failed to get one-time code: Cookies must be enabled to use GitHub

[SuibianP]

@TylerLeonhardt Thank you for your response.

1. Yes, the instance is gated behind corporate VPN, which is global on my computer. The URL is also reachable via curl from the command line so I suppose that is not a problem. GitHub Desktop also logs in just fine.

   EDIT: The issue persists on corporate network (without VPN) as well, so VPN should not be the culprit.

2. No.

As for the cookies issue, I already enabled cookies on my browser. I just tried with another browser and also relaxed CORS requirements, only to encounter the very same error. Also, the third auth flow does not even redirect to my browser at all and just fails out. Could it be referring to the VSCode client not keeping a cookie store?

[pWoLiAn]

1. Yes
2. Yes(I tried with both ssh and local sessions and login failed both times)
3. Yes and all three ways seem to fail

[SuibianP]

@pWoLiAn Maybe can you also share the logs of all three auth flows? That might be helpful for @TylerLeonhardt to debug.

[TylerLeonhardt]

A few months ago we mirrored the GitHub auth experience by lighting up the same experience for GitHub Enterprise Server. The experience is tremendously better then the former PAT auth because it includes 3 additional more ergonomic ways to log in:

1. OAuth with automatic redirect back to the VS Code window that started the flow (only works if you have a vscode:// OS handler setup)
2. OAuth using a local server which displays a “you can close this window” (only works if you have the ability to spin up a local http server)
3. Device Code flow (works pretty universally but isn’t as smooth as the first 2)

but you all have made me realize there is a flaw here. The first 2 methods only work if their GitHub Enterprise Server is available via the open internet. If someone needs a VPN connection to connect to their GHES instance, then 1 and 2 on this list will fail, because they depend on making a request to vscode.dev/codeExchangeProxies/… which will make a request to the configured GHES url in order to get an access token (by using a Secret that vscode.dev has). So the question is…

should we still do number 1 and 2 (or maybe skip 2 if we detect a certain error related to this) since they are easier to use knowing that it might fail for a chunk of folks?

or

should we not do number 1 and 2 and just do number 3 knowing that that will mean everyone will have the same experience?

[TylerLeonhardt]

Additionally, there seems to be a problem with making the request for Device Code Flow for some folks related to cookies. That's unusual to me because:

1. These are POST requests: https://github.com/microsoft/vscode/blob/f233c22e6f780c9a29095eb9ae15e9a6c86bbdd4/extensions/github-authentication/src/githubServer.ts#L302-L312 and https://github.com/microsoft/vscode/blob/f233c22e6f780c9a29095eb9ae15e9a6c86bbdd4/extensions/github-authentication/src/githubServer.ts#L377-L396
2. They're called from Node.js so CORS shouldn't be a problem on the client side

I'm initially suspicious that the URL is malformed in some way. If you are getting is cookie issue, can you double check your github-enterprise.uri setting? It should have the scheme (https://) in the url. It might need to be https and not http but I'm not sure.

[SuibianP]

@TylerLeonhardt Thank you for the investigation that reveals the root cause!

I believe the former approach should be chosen if possible to avoid degrading existing experience for users of Internet accessible GHES servers. Nonetheless, the experience of private GHES server authentication should definitely be smoother, without going through two failure messages first, considering the prevalence of private enterprise servers out there. Preferably it should suppress the error prompt altogether by automatically catching this exception reliably.

On a separate note, why is PAT flow removed instead of being a fourth alternative? PAT seems to work for me on IntelliJ for the same server, though I am not sure if it is something else that worked out.

As for the URL, it looks correct with the https scheme ("github-enterprise.uri": "https://foo.bar.com"). I wonder if the company-specific SSO has anything to do with this?

[TylerLeonhardt]

Maybe try this out. Using like curl or something (in otherwords, not your browser) does this resolve?:

curl https://foo.bar.com/api/v3/meta

obviously use your GHES url.

Your GHES url doesn't have paths in it, does it? Like: https://foo.bar.com/github isn't the home page instead of https://foo.bar.com?

[TylerLeonhardt]

On a separate note, why is PAT flow removed instead of being a fourth alternative?

Yeah I'm considering this for GitHub Enterprise. The downside of the PAT auth is that you need to make sure that the PAT has the correct scopes... and so a lot of people just check all the boxes and it's not really the best practice to give an extension, even if you trust it, more permissions than it's requesting.

[SuibianP]

$ curl https://foo.bar.com/api/v3/meta
{
  "verifiable_password_authentication": true,
  "installed_version": "2.21.23"
}

Yes, the URL does not have paths in it. In other words, a particular git repository named qux under user baz is thus located at https://foo.bar.com/baz/qux.git, for example.

[SuibianP]

On a separate note, why is PAT flow removed instead of being a fourth alternative?

Yeah I'm considering this for GitHub Enterprise. The downside of the PAT auth is that you need to make sure that the PAT has the correct scopes... and so a lot of people just check all the boxes and it's not really the best practice to give an extension, even if you trust it, more permissions than it's requesting.

Adding back the good old PAT flow sounds reasonable because AFAIK it has fewer quirks to go wrong, despite being less user-friendly. It is likely to work even if all other flows fail out for some reason.

Maybe it can be made opt-in and warn the user about the potential security risk and best practices? IntelliJ offers to open the PAT generation page with the minimal required scopes pre-checked, which may also be a way forward to encourage good practices. If possible, a dedicated token with least required privileges can be demanded by rejecting tokens with excessive privileges, so as to thwart the one-omnipotent-token-fits-all anti-pattern.

[TylerLeonhardt]

open the PAT generation page with the minimal required scopes pre-checked

Wow I didn't know this was a thing 😃 it's pretty nice. I can also pre-fill the Note in.

Ok I have now merged in a PR that re-adds PAT for GHES as a fallback.

... now if I could figure out how to pre-fill the expiration to be "never" or at least "90 days".

On the other side, I have sent a PR to add a new endpoint to vscode.dev that will basically test if a GHES is accessible. We will use that route in the client to either do flow 1 & 2..... or not and skip to device code flow.

Device Code flow is still way better than PAT so I'm leaving that in but if for whatever reason that doesn't work, we go to PAT.

[SuibianP]

Thanks a ton for the efforts @TylerLeonhardt!

Back to the device flow issue, I consulted the docs and found that it was added in GHES 3.1 which antedates mine. While all GHES versions in support as of now supports it, it is still explicitly marked as subject to change in the docs, and when the API should change in the future there might be yet another bunch of such issues with confusing error messages. Could there be some kind of guard check implemented to ensure version compatibility before enabling a feature?

By the way, I tried to get Insiders build but the latest version seems to be last week on 6ed4d436a9b6ae131732f4e2a723868231f4473a rather than nightly as described in the docs. Off-topic here but just asking first in case it's something silly of me 😹

[TylerLeonhardt]

@SuibianP it's a holiday pause :) they'll resume next week.

[SuibianP]

@TylerLeonhardt Unfortunately the version of my instance is so old that it does not have the workflow scope or GitHub Action :facepalm:

2023-01-03 21:49:40.153 [error] Error: The provided token does not match the requested scopes: read:user repo user:email workflow
    at t.GitHubServer.q (/Applications/Visual Studio Code - Insiders.app/Contents/Resources/app/extensions/github-authentication/dist/extension.js:2:1307790)
    at process.processTicksAndRejections (node:internal/process/task_queues:96:5)
    at async t.GitHubServer.login (/Applications/Visual Studio Code - Insiders.app/Contents/Resources/app/extensions/github-authentication/dist/extension.js:2:1303595)
    at async t.GitHubAuthenticationProvider.createSession (/Applications/Visual Studio Code - Insiders.app/Contents/Resources/app/extensions/github-authentication/dist/extension.js:2:1299577)

A quick blame landed me on #97396. What is your take on the way going forward? I am not sure if such corner case deserves a special case, especially that such archaic versions have long been EOL and VSCode does not seem particularly strict to never break anything.

That said, is there any workaround for this now as it stands?

EDIT: Same issue as https://github.com/microsoft/vscode-pull-request-github/issues/3828. That issue was closed due to PAT deprecation.

[TylerLeonhardt]

@SuibianP oof... I mean this is getting really tough.

• You can't use flow 1 or 2 because you're behind a VPN
• You can't use device code flow because the version of GHES didn't have it
• You can't use PAT because the version of GHES does not support the workflow scope that GitHub Pull Request is asking for

One of two solutions here:

• GitHub Pull Request could version check and not ask for workflow if the instance is below 3.1
• GitHub Auth could have a setting that doesn't validate that PAT has all the scopes and we just hope for the best

thoughts @alexr00 ?

It's tough to action on this when 2.21 is just so old.. it's been deprecated for over a year now: https://docs.github.com/en/enterprise-server@3.6/admin/all-releases#deprecated-releases

[SuibianP]

@TylerLeonhardt There is indeed some merit in supporting the last release of GHES 2 though -- the upgrade to GHES 3 has different infrastructure requirements and there might still be a number of installations stuck on GHES 2 (just like Python 2.7.18 and its unending phase-out). Maybe failed scope check could be made a warning instead? Also FYI, the GitHub pull request extension does not work even after patching out the offending scope check, due to some usage of new GraphQL fields. I will file another issue in the extension repo later to see if && how that can be sorted out.

More generically, the backward compatibility policy can be better agreed upon and documented so that version support is clear and consistent throughout the project as well as all officially supported extensions.

[alexr00]

@SuibianP at present, Enterprise is only supported in the GitHub Pull Request extension starting with Enterprise version 3.1 (https://github.com/microsoft/vscode-pull-request-github/wiki#2-is-github-enterprise-supported). As you've noticed, it's not just the workflow scope that's the problem. Back in August and September we worked on improving Enterprise support and landed on 3.1 as the oldest version that let us support the most users. I'll open an issue to gauge the need/interest in supporting GHES 2.21, but given the few number of users on it and the difficult for us to test (our test infra only goes back to 3.1) I'm not sure whether supporting 2.21 will work itself up to a high enough priority.

[alexr00]

Issue filed: https://github.com/microsoft/vscode-pull-request-github/issues/4385

[akhileshraju]

I can confirm that with VS Code 1.74.3 the plugin is able to authenticate with a GHES 3.4.12 instance

[TylerLeonhardt]

So this is a weird one to mark as closed because it's "fixed" in the sense that you won't see Internal Server Error anymore because we no longer do the OAuth flow for GitHub Enterprise Servers. We only do Device Code Flow and PAT flow... so there's no problem with VPN'd GHES's.

With that said, @SuibianP, your other issue is that you're on GHES 2... which sadly, isn't supported by us at the moment. I will leave this issue as closed to solely address the VPN'd GHES's issue... but @SuibianP if you would like, can you please open a feature request for supporting GHES 2.x. I can't promise anything as I don't know what the numbers are like for such an old version (and GitHub itself doesn't support anything before 3.1...) but we try to keep one issue per-issue in VS Code.

[TylerLeonhardt]

verification:

• Do a log in flow of GitHub Enterprise
• You should see the device code flow first

[SuibianP]

@TylerLeonhardt https://github.com/microsoft/vscode-pull-request-github/issues/4385 should be serving the exact purpose already I suppose?

[LiveFreeAndRoam]

I stiill see this error using VSC 1.75 and GHE 3.5.6.

From the Output: GitHub Enterprise Authentication:

2023-02-20 12:14:01.734 [info] Reading sessions from keychain...
2023-02-20 12:17:48.864 [info] Reading sessions from keychain...
2023-02-20 12:17:58.815 [info] Getting sessions for read:user,repo,user:email,workflow...
2023-02-20 12:17:58.815 [info] Got 0 sessions for read:user,repo,user:email,workflow...
2023-02-20 12:17:59.379 [info] Getting sessions for read:user,repo,user:email...
2023-02-20 12:17:59.380 [info] Got 0 sessions for read:user,repo,user:email...
2023-02-20 14:24:09.386 [info] Getting sessions for all scopes...
2023-02-20 14:24:09.386 [info] Got 0 sessions for ...
2023-02-20 14:24:24.706 [info] Getting sessions for all scopes...
2023-02-20 14:24:24.706 [info] Got 0 sessions for ...
2023-02-20 14:25:10.235 [info] Getting sessions for read:user,repo,user:email,workflow...
2023-02-20 14:25:10.235 [info] Got 0 sessions for read:user,repo,user:email,workflow...
2023-02-20 14:25:10.790 [info] Getting sessions for read:user,repo,user:email...
2023-02-20 14:25:10.790 [info] Got 0 sessions for read:user,repo,user:email...
2023-02-20 14:25:11.352 [info] Getting sessions for read:user,repo,user:email,workflow...
2023-02-20 14:25:11.352 [info] Got 0 sessions for read:user,repo,user:email,workflow...
2023-02-20 14:25:17.166 [info] Logging in for the following scopes: read:user repo user:email workflow
2023-02-20 14:25:17.172 [info] Trying without local server... (read:user repo user:email workflow)
2023-02-20 14:25:30.082 [info] Exchanging code for token...
2023-02-20 14:26:00.880 [error] GitHubTokenExchangeError: Internal Server Error
    at t.GitHubServer.u (c:\Users\username\AppData\Local\Programs\Microsoft VS Code\resources\app\extensions\github-authentication\dist\extension.js:2:1309597)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at c:\Users\username\AppData\Local\Programs\Microsoft VS Code\resources\app\extensions\github-authentication\dist\extension.js:2:1304449

$ curl https://github.company.com/api/v3/meta
{
  "verifiable_password_authentication": false,
  "installed_version": "3.5.6"
}

[TylerLeonhardt]

It's fixed in VS Code 1.76 which will be released in a week and a half.

To try it early, you can get the insider build: https://code.visualstudio.com/insiders

[SuibianP]

@TylerLeonhardt Thinking from the token issue, I am wondering why is GitHub authentication baked into VSCode itself, instead of decoupled as an extension like the GitLab one? Under the current design, one would have to patch VSCode in addition to the extension to be able to work with old GHE versions.

Also, just a quick passing thought, maybe the error message could also be more descriptive than the generic "Error signing in to GitHub Enterprise"?

[TylerLeonhardt]

I am wondering why is GitHub authentication baked into VSCode itself

Because built-in features, like Settings Sync, need it (well, GitHub auth, not GHES auth) in order to be used.

[TylerLeonhardt]

Also, just a quick passing thought, maybe the error message could also be more descriptive than the generic "Error signing in to GitHub Enterprise"?

Yeah this might be good. Can you get a new issue going for this?

[TylerLeonhardt]

verification:

• Do a log in flow of GitHub Enterprise
• You should see the device code flow first

Can't have the author verify because they are on GHES 2 which doesn't have the device code flow

[SuibianP]

I confirm that it is working with the latest insider build on another GHE instance on a supported version.