[Remote-SSH] Extension Host with Copilot Fails on VS Code 1.95.3 but Works on 1.93.0

[technic960183]

Type: Bug

Description

We are encountering an issue with the GitHub Copilot extension when using Visual Studio Code with Remote-SSH on an HPC login node. The extension works flawlessly with VS Code 1.93.0 (4849ca9bdf9666755eb463db297b69e5385090e3) but fails on VS Code 1.95.3 (f1a4fb101478ce6ec82fe9627c43efbf9e98c813). Interestingly, VS Code 1.95.3 works fine on the computing node of the same cluster, accessed via SSH forwarding from the login node.

To eliminate variables, we have tested the setup thoroughly:

• Two personal computers were used:
  • One running VS Code 1.93.0.
  • The other running VS Code 1.95.3.
• Both with GitHub Copilot 1.245.0 and GitHub Copilot Chat 0.22.4
• Both connect to the same HPC login node and same account using Remote-SSH.
• Before each login, the .vscode-server directory on the remote machine was cleaned up to ensure a fresh environment.
• On the remote machine, GitHub Copilot and GitHub Copilot Chat were the only extensions installed.

Expected Behavior

The GitHub Copilot extension should initialize and function correctly on both VS Code versions.

Actual Behavior

• On VS Code 1.95.3, the Copilot extension causes the Extension Host process to crash only on the login node.
• The issue does not occur on the computing node or when using VS Code 1.93.0 on the login node.
• Disabling the GitHub Copilot extensions eliminates the issue.

Test by VS Code Bisect

Done on our login node with VS Code 1.95.3. Result: Extension Bisect is done and has identified github.copilot as the extension causing the problem.

Before, we don't know that it is possible to disable GitHub Copilot alone but not GitHub Copilot Chat. Later, we found that when VS Code bisect disable GitHub Copilot 1.245.0 alone, GitHub Copilot Chat 0.22.4 works normally on our login node with VS Code 1.95.3.

Logs

Below are logs captured from the Extension Host (remote) process on both the login and computing nodes running VS Code 1.95.3. These logs highlight differences observed during initialization:

Login Node (Fails)

2024-11-21 16:34:43.523 [trace] ExtHostCommands#registerCommand github.copilot.openLogs
2024-11-21 16:34:43.523 [trace] ExtHostCommands#registerCommand github.copilot.signIn
2024-11-21 16:34:43.530 [trace] ExtensionService#_callActivateOptional GitHub.copilot-chat
2024-11-21 16:34:43.555 [trace] extHostWorkspace#findFiles2: fileSearch, extension: GitHub.copilot-chat, entryPoint: findFiles2
2024-11-21 16:34:43.556 [trace] ProxyResolver#tls.connect [{"highWaterMark":16384,"servername":"default.exp-tas.com","session":"null","localAddress":"null","ALPNProtocols":"http/1.1","port":443,"host":"default.exp-tas.com"}]
2024-11-21 16:34:43.562 [debug] ProxyResolver#resolveProxy unconfigured http://169.254.169.254/metadata/instance/compute DIRECT 
2024-11-21 16:34:43.693 [trace] ProxyResolver#tls.connect [443, "default.exp-tas.com", {"servername":"default.exp-tas.com","ALPNProtocols":"h2,http/1.1,http/1.0","signal":"[object AbortSignal]","rejectUnauthorized":true,"ca":"[281 certs]"}]
2024-11-21 16:34:43.723 [trace] ProxyResolver#tls.connect [443, "api.github.com", {"servername":"api.github.com","ALPNProtocols":"h2,http/1.1,http/1.0","signal":"[object AbortSignal]","rejectUnauthorized":true,"ca":"[281 certs]"}]
2024-11-21 16:34:43.760 [debug] ExtHostSearch /work1/user141421/.vscode-server/cli/servers/Stable-f1a4fb101478ce6ec82fe9627c43efbf9e98c813/server/node_modules/@vscode/ripgrep/bin/rg --files --hidden --case-sensitive --no-require-git -g '!**/.git' -g '!**/.svn' -g '!**/.hg' -g '!**/CVS' -g '!**/.DS_Store' -g '!**/Thumbs.db' -g '!**/node_modules' -g '!**/bower_components' -g '!**/*.code-search' --no-ignore-parent --follow --no-config --no-ignore-global
 - cwd: /home/user141421
 - Sibling clauses: {}
2024-11-21 16:34:43.964 [trace] ProxyResolver#tls.connect [443, "api.github.com", {"servername":"api.github.com","ALPNProtocols":"h2,http/1.1,http/1.0","rejectUnauthorized":true,"ca":"[281 certs]"}]
2024-11-21 16:34:43.998 [trace] ExtHostCommands#registerCommand github.copilotChat.signIn
[skip some lines]
2024-11-21 16:34:44.005 [trace] ExtHostCommands#registerCommand github.copilot.buildLocalWorkspaceIndex
2024-11-21 16:34:44.020 [debug] ExtHostSearch Search finished. Stats: {"cmdTime":266,"fileWalkTime":266,"directoriesWalked":0,"filesWalked":0,"cmdResultCount":28852}
2024-11-21 16:34:44.020 [debug] Ext host file search time: 266ms
2024-11-21 16:34:44.174 [trace] ExtHostCommands#registerCommand codereferencing.showOutputPane
2024-11-21 16:34:44.176 [trace] ProxyResolver#tls.connect [443, "copilot-telemet[39 chars]", {"servername":"copilot-telemet[39 chars]","ALPNProtocols":"h2,http/1.1,http/1.0","rejectUnauthorized":true,"ca":"[281 certs]"}]
2024-11-21 16:34:44.215 [trace] ExtHostCommands#executeCommand setContext
2024-11-21 16:34:44.215 [trace] ExtHostCommands#executeCommand _setContext
2024-11-21 16:34:44.215 [trace] ExtHostCommands#registerCommand github.copilot.generate
2024-11-21 16:34:44.215 [trace] ExtHostCommands#registerCommand github.copilot.acceptCursorPanelSolution
2024-11-21 16:34:44.215 [trace] ExtHostCommands#registerCommand github.copilot.previousPanelSolution
2024-11-21 16:34:44.215 [trace] ExtHostCommands#registerCommand github.copilot.nextPanelSolution
2024-11-21 16:34:44.216 [trace] ExtHostCommands#registerCommand _github.copilot.ghostTextPostInsert
[Process Crash]

Computing Node (Works)

2024-11-21 16:30:49.433 [trace] ExtHostCommands#registerCommand github.copilot.openLogs
2024-11-21 16:30:49.433 [trace] ExtHostCommands#registerCommand github.copilot.signIn
[Compare to above, no extra message here.]
2024-11-21 16:30:49.454 [trace] ExtHostCommands#registerCommand github.copilotChat.signIn
[skip the same lines as above]
2024-11-21 16:30:49.459 [trace] ExtHostCommands#registerCommand github.copilot.buildLocalWorkspaceIndex
2024-11-21 16:30:49.664 [trace] ProxyResolver#tls.connect [443, "default.exp-tas.com", {"servername":"default.exp-tas.com","ALPNProtocols":"h2,http/1.1,http/1.0","signal":"[object AbortSignal]","rejectUnauthorized":true,"ca":"[281 certs]"}]
2024-11-21 16:30:49.694 [trace] ProxyResolver#tls.connect [443, "api.github.com", {"servername":"api.github.com","ALPNProtocols":"h2,http/1.1,http/1.0","signal":"[object AbortSignal]","rejectUnauthorized":true,"ca":"[281 certs]"}]
2024-11-21 16:30:49.981 [trace] ProxyResolver#tls.connect [443, "api.github.com", {"servername":"api.github.com","ALPNProtocols":"h2,http/1.1,http/1.0","rejectUnauthorized":true,"ca":"[281 certs]"}]
2024-11-21 16:30:50.116 [trace] ExtHostCommands#registerCommand codereferencing.showOutputPane
2024-11-21 16:30:50.118 [trace] ProxyResolver#tls.connect [443, "copilot-telemet[39 chars]", {"servername":"copilot-telemet[39 chars]","ALPNProtocols":"h2,http/1.1,http/1.0","rejectUnauthorized":true,"ca":"[281 certs]"}]
2024-11-21 16:30:50.156 [trace] ExtHostCommands#executeCommand setContext
2024-11-21 16:30:50.156 [trace] ExtHostCommands#executeCommand _setContext
2024-11-21 16:30:50.156 [trace] ExtHostCommands#registerCommand github.copilot.generate
2024-11-21 16:30:50.156 [trace] ExtHostCommands#registerCommand github.copilot.acceptCursorPanelSolution
2024-11-21 16:30:50.156 [trace] ExtHostCommands#registerCommand github.copilot.previousPanelSolution
2024-11-21 16:30:50.156 [trace] ExtHostCommands#registerCommand github.copilot.nextPanelSolution
2024-11-21 16:30:50.156 [trace] ExtHostCommands#registerCommand _github.copilot.ghostTextPostInsert
2024-11-21 16:30:50.176 [debug] ProxyResolver#loadSystemCertificates count 137
2024-11-21 16:30:50.191 [debug] ProxyResolver#loadSystemCertificates count filtered 134
2024-11-21 16:30:50.191 [debug] ProxyResolver#resolveProxy unconfigured https://mobile.events.data.microsoft.com/OneCollector/1.0?cors=true&content-type=application/x-json-stream DIRECT 
2024-11-21 16:30:50.192 [trace] ProxyResolver#tls.connect [{"protocol":"https:","hostname":"mobile.events.d[32 chars]","port":443,"path":"null","method":"POST","headers":"[object Object]","agent":"[object Object]","_defaultAgent":"[object Object]","host":"mobile.events.d[32 chars]","lookupProxyAuthorization":"[Function: bound dz]","noDelay":true,"servername":"mobile.events.d[32 chars]","secureEndpoint":true,"_vscodeAdditionalCaCerts":"[134 certs]","keepAlive":true,"scheduling":"lifo","timeout":5000,"_agentKey":"mobile.events.d[57 chars]","encoding":"null","keepAliveInitialDelay":1000}]
[Process keep running without crashing]
2024-11-21 16:30:50.176 [debug] ProxyResolver#loadSystemCertificates count 137
2024-11-21 16:30:50.191 [debug] ProxyResolver#loadSystemCertificates count filtered 134
2024-11-21 16:30:50.191 [debug] ProxyResolver#resolveProxy unconfigured https://mobile.events.data.microsoft.com/OneCollector/1.0?cors=true&content-type=application/x-json-stream DIRECT 
2024-11-21 16:30:50.192 [trace] ProxyResolver#tls.connect [{"protocol":"https:","hostname":"mobile.events.d[32 chars]","port":443,"path":"null","method":"POST","headers":"[object Object]","agent":"[object Object]","_defaultAgent":"[object Object]","host":"mobile.events.d[32 chars]","lookupProxyAuthorization":"[Function: bound dz]","noDelay":true,"servername":"mobile.events.d[32 chars]","secureEndpoint":true,"_vscodeAdditionalCaCerts":"[134 certs]","keepAlive":true,"scheduling":"lifo","timeout":5000,"_agentKey":"mobile.events.d[57 chars]","encoding":"null","keepAliveInitialDelay":1000}]

Additional Notes

• Network configuration remains the same across tests on both the login node and the computing node. Firewall are disable for all nodes.
• The system administrator has confirmed that no OS-level rules (e.g., process termination) are in effect on the login node.
• All of the hardware been exactly the same for the login node and the computing nodes in our cluster.

Key questions

• Why it works on VS Code 1.93.0 but not on VS Code 1.95.3 ?
• Why VS Code 1.95.3 works on our computing node but not on the login node?

VS Code version: Code 1.95.3 (f1a4fb101478ce6ec82fe9627c43efbf9e98c813, 2024-11-13T14:50:04.152Z) OS version: Windows_NT x64 10.0.19045 Modes: Remote OS version: Linux x64 5.15.0-78-generic

System Info

|Item|Value| |---|---| |CPUs|Intel(R) Core(TM) i7-7700HQ CPU @ 2.80GHz (8 x 2808)| |GPU Status|2d_canvas: enabled
canvas_oop_rasterization: enabled_on
direct_rendering_display_compositor: disabled_off_ok
gpu_compositing: enabled
multiple_raster_threads: enabled_on
opengl: enabled_on
rasterization: enabled
raw_draw: disabled_off_ok
skia_graphite: disabled_off
video_decode: enabled
video_encode: enabled
vulkan: disabled_off
webgl: enabled
webgl2: enabled
webgpu: enabled
webnn: disabled_off| |Load (avg)|undefined| |Memory (System)|7.89GB (1.49GB free)| |Process Argv|--log info --crash-reporter-id edf07c68-6f96-49c4-a777-c6a188cd4542| |Screen Reader|no| |VM|0%| |Item|Value| |---|---| |Remote|SSH: Spock| |OS|Linux x64 5.15.0-78-generic| |CPUs|AMD Ryzen Threadripper PRO 5975WX 32-Cores (64 x 1793)| |Memory (System)|251.70GB (244.89GB free)| |VM|0%|

Extensions (7)

Extension|Author (truncated)|Version ---|---|--- copilot|Git|1.245.0 copilot-chat|Git|0.22.4 jupyter-keymap|ms-|1.1.2 remote-ssh|ms-|0.115.1 remote-ssh-edit|ms-|0.87.0 remote-explorer|ms-|0.4.3 vscode-speech|ms-|0.12.1 (1 theme extensions excluded)

A/B Experiments

``` vsliv368cf:30146710 vspor879:30202332 vspor708:30202333 vspor363:30204092 vscod805cf:30301675 binariesv615:30325510 vsaa593:30376534 py29gd2263:31024239 c4g48928:30535728 azure-dev_surveyone:30548225 962ge761:30959799 pythonnoceb:30805159 asynctok:30898717 pythonmypyd1:30879173 2e7ec940:31000449 pythontbext0:30879054 cppperfnew:31000557 dsvsc020:30976470 pythonait:31006305 dsvsc021:30996838 g316j359:31013175 dvdeprecation:31068756 dwnewjupytercf:31046870 nativerepl2:31139839 pythonrstrctxt:31112756 cf971741:31144450 iacca1:31171482 notype1cf:31157160 5fd0e150:31155592 dwcopilot:31170013 stablechunks:31184530 ```

[lramos15]

Does Copilot Chat work?

Please try using VS Code bisect to help us narrow down the version which broke this

[technic960183]

Thank you for your response.

The result of VS Code bisect confirm that the issue is caused by the github.copilot extension (Version 1.245.0).

When VS Code bisect disable GitHub Copilot alone, GitHub Copilot Chat works normally.

Additionally, we have verified that both GitHub Copilot (1.245.0) and GitHub Copilot Chat (0.22.4) extensions are using the same version on VS Code 1.93.0 and VS Code 1.95.3 during our previous tests. I've updated these information to the issue description.

Please let us know if further information is needed.