VS Code executable not found after downloading it

DonJayamanne commented 1 year ago

Does this issue occur when all extensions are disabled?: Yes/No

VS Code Version: Insiders
OS Version: Windows

CI logs https://github.com/microsoft/vscode-jupyter/actions/runs/6940223424/job/18878834977 I am downloading VS Code using the package @vscode/test-electron

I have noticed that the windows tests can fail sometimes due to some file not found error (see below). Re-running the CI seems to fix it (hence the reason I think its flaky)

Please let me know if additional information/logs are required.

Here are the logs from the Jupyter CI

Run GabrielBB/xvfb-action@v1.4
C:\Windows\system32\cmd.exe /D /S /C "C:\hostedtoolcache\windows\node\18.15.0\x64\npm.cmd run testSmokeLogged"

> jupyter@2023.11.100 testSmokeLogged
> cross-env VSC_JUPYTER_FORCE_LOGGING=true node --no-force-async-hooks-checks ./out/test/testBootstrap.node.js ./out/test/smokeTest.node.js

Starting socket server for tests.
Creating test server
Listening to test server
Test server running
Test server listening on port 53222
Spawning C:\hostedtoolcache\windows\node\18.15.0\x64\node.exe : ./out/test/smokeTest.node.js
Python used in tests is c:\hostedtoolcache\windows\python\3.10.11\x64\python.exe
Start Test Runner
Launch tests in test runner
****************************************************************************************************
Start Standard tests
Downloading VS Code insiders from https://update.code.visualstudio.com/latest/win32-x64-archive/insider
Downloading VS Code (131091780B)
Downloading VS Code [=-----------------------------] 6%Downloading VS Code [=====-------------------------] 17%Downloading VS Code [=========---------------------] 33%Downloading VS Code [==============----------------] 48%Downloading VS Code [===================-----------] 64%Downloading VS Code [====================----------] 69%Downloading VS Code [=====================---------] 71%Downloading VS Code [======================--------] 74%Downloading VS Code [=======================-------] 77%Downloading VS Code [=======================-------] 80%Downloading VS Code [=========================-----] 85%Downloading VS Code [==========================----] 90%Downloading VS Code [============================--] 94%Downloaded VS Code into D:\a\vscode-jupyter\vscode-jupyter\.vscode-test\vscode-win32-x64-archive-insiders
Downloaded VS Code into D:\a\vscode-jupyter\vscode-jupyter\.vscode-test\vscode-win32-x64-archive-insiders
Installing Python Extension ms-python.python to C:\Users\RUNNER~1\AppData\Local\Temp\vscode_jupyter_exts
Installing Pylance Extension to C:\Users\RUNNER~1\AppData\Local\Temp\vscode_jupyter_exts
Installing Renderer Extension to C:\Users\RUNNER~1\AppData\Local\Temp\vscode_jupyter_exts
Test error: Error: spawn D:\a\vscode-jupyter\vscode-jupyter\.vscode-test\vscode-win32-x64-archive-insiders\Code - Insiders.exe ENOENT
Exit code:   -4058
End Standard tests (with errors) Failed
****************************************************************************************************
Start Standard tests
End Standard tests (with errors) Error: ENOENT: no such file or directory, open 'D:\a\vscode-jupyter\vscode-jupyter\.vscode-test\vscode-win32-x64-archive-insiders\resources\app\product.json'
    at Object.openSync (node:fs:601:3)
    at Object.readFileSync (node:fs:469:[35](https://github.com/microsoft/vscode-jupyter/actions/runs/6940223424/job/18878834977#step:13:36))
    at Object.insidersDownloadDirMetadata (D:\a\vscode-jupyter\vscode-jupyter\node_modules\@vscode\test-electron\out\util.js:114:41)
    at download (D:\a\vscode-jupyter\vscode-jupyter\node_modules\@vscode\test-electron\out\download.js:281:72)
    at downloadAndUnzipVSCode (D:\a\vscode-jupyter\vscode-jupyter\node_modules\@vscode\test-electron\out\download.js:[36](https://github.com/microsoft/vscode-jupyter/actions/runs/6940223424/job/18878834977#step:13:37)1:18)
    at start (D:\a\vscode-jupyter\vscode-jupyter\out\test\standardTest.node.js:144:83)
    at D:\a\vscode-jupyter\vscode-jupyter\out\test\standardTest.node.js:177:16
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5) {
  errno: -[40](https://github.com/microsoft/vscode-jupyter/actions/runs/6940223424/job/18878834977#step:13:41)[58](https://github.com/microsoft/vscode-jupyter/actions/runs/6940223424/job/18878834977#step:13:59),
  syscall: 'open',
  code: 'ENOENT',
  path: 'D:\\a\\vscode-jupyter\\vscode-jupyter\\.vscode-test\\vscode-win32-x[64](https://github.com/microsoft/vscode-jupyter/actions/runs/6940223424/job/18878834977#step:13:65)-archive-insiders\\resources\\app\\product.json'
}
Tests Exited with code 1
Error in running Smoke Tests Failed with code 1.
Exiting with test failures
Killing VSC
Error: The process 'C:\hostedtoolcache\windows\node\18.15.0\x64\npm.cmd' failed with exit code 1

DonJayamanne commented 1 year ago

The second run with unrelated changes resulted in a clean run without any issues https://github.com/microsoft/vscode-jupyter/actions/runs/6940290925/job/18879022414?pr=14747

Here are two runs on on the same commit in CI:

First run that failed https://github.com/microsoft/vscode-jupyter/actions/runs/6940223424
Second run that worked https://github.com/microsoft/vscode-jupyter/actions/runs/6940331546

DonJayamanne commented 1 year ago

Hmm, ran into this exact same issue on MacOS on the CI server This time with the latest version of @vscode/test-electron

https://github.com/microsoft/vscode-jupyter/actions/runs/6951074099/job/18912450458

Still trying to figure out whats going on here.

DonJayamanne commented 1 year ago

Looks like solmething is very wrong with the way files get extracted https://github.com/microsoft/vscode-jupyter/actions/runs/6951254070/job/18913278166?pr=14758

Downloaded VS Code into D:\a\vscode-jupyter\vscode-jupyter\.vscode-test\vscode-win32-x64-archive-insiders
VS Code executable D:\a\vscode-jupyter\vscode-jupyter\.vscode-test\vscode-win32-x64-archive-insiders\Code - Insiders.exe exists = false
Files in folder D:\a\vscode-jupyter\vscode-jupyter\.vscode-test\vscode-win32-x64-archive-insiders => af.pak
am.pak
app
ar.pak
bg.pak
bn.pak
ca.pak
cs-cz
cs.pak
da.pak
de-de
de.pak
el.pak
en-GB.pak
en-us
en-US.pak

The files are in the wrong place. The files *.pak should be in D:\a\vscode-jupyter\vscode-jupyter\.vscode-test\vscode-win32-x64-archive-insiders\locales and not Files in folder D:\a\vscode-jupyter\vscode-jupyter\.vscode-test\vscode-win32-x64-archive-insiders

DonJayamanne commented 1 year ago

Found that in all of the cases downloading never goes to 100%. I.e. download stops before getting to 100%, thus the stream is no complete, hence extraction of the file fails.

connor4312 commented 1 year ago

Because this is happening across different environments in your CI, and seemingly(?) only in your CI as we've not had other reports of the issue, I suspect something might be up with the archives that get downloaded. This is supported by

seeing the issue on multiple platforms, which use different unzip implementations (on macOS we use the command-line unzip utility, on windows we use the jszip module.)
I'm assuming seeing this issue come up all of a sudden without package changes, whereas we have changed CDNs

Currently we don't do any validation on downloaded archives, so I'll add some.

DonJayamanne commented 1 year ago

Because this is happening across different environments in your CI, and seemingly(?) only in your CI as we've not had other

Logan and Steven came across this in their pipeline too. @lramos15 /cc

connor4312 commented 1 year ago

I believe still it's a good theory with the two bullet points I mentioned.

I made a fix to try to solve that, please validate it once published. Note that hash validation is only available when you ask for a specific commit in the version you pass (unless Joao tweaks the update server before I publish in the morning) so if you do still see the issue, please try passing a specific commit and seeing if that avoids the issue.

DonJayamanne commented 1 year ago

Still running into this on CI, (with the latest npm package) This time on Mac CI https://github.com/microsoft/vscode-jupyter/actions/runs/6962842187/job/18947373636

Start Standard tests
Downloading VS Code insiders from https://update.code.visualstudio.com/latest/darwin/insider
Downloading VS Code (130406165B)
Downloading VS Code [------------------------------] 3%Downloading VS Code [=-----------------------------] 3%Downloading VS Code [=-----------------------------] 4%Downloading VS Code [=-----------------------------] 6%Downloading VS Code [==----------------------------] 9%Downloading VS Code [===---------------------------] 13%Downloading VS Code [====--------------------------] 15%Downloading VS Code [=====-------------------------] 17%Downloading VS Code [=====-------------------------] 17%Downloading VS Code [=====-------------------------] 18%Downloading VS Code [======------------------------] 20%Downloading VS Code [======------------------------] 21%Downloading VS Code [======------------------------] 22%Downloading VS Code [======------------------------] 23%Downloading VS Code [=======-----------------------] 25%Downloading VS Code [=======-----------------------] 27%Downloading VS Code [========----------------------] 28%Downloading VS Code [========----------------------] 29%Downloading VS Code [========----------------------] 30%Downloading VS Code [=========---------------------] 31%Downloading VS Code [=========---------------------] 33%Downloading VS Code [==========--------------------] 35%Downloading VS Code [==========--------------------] 35%Downloading VS Code [==========--------------------] 36%Downloading VS Code [===========-------------------] 39%Downloading VS Code [===========-------------------] 39%Downloading VS Code [============------------------] 41%Downloading VS Code [=============-----------------] 44%Downloading VS Code [=============-----------------] 44%Downloading VS Code [==============----------------] 48%Downloading VS Code [===============---------------] [51](https://github.com/microsoft/vscode-jupyter/actions/runs/6962842187/job/18947373636#step:13:52)%Downloading VS Code [===============---------------] [52](https://github.com/microsoft/vscode-jupyter/actions/runs/6962842187/job/18947373636#step:13:53)%Downloading VS Code [================--------------] 54%Downloading VS Code [=================-------------] 59%Downloading VS Code [===================-----------] 64%Downloading VS Code [====================----------] 68%Downloading VS Code [====================----------] 68%Downloading VS Code [=====================---------] 71%Downloading VS Code [=====================---------] 73%Downloading VS Code [======================--------] 74%Downloading VS Code [=======================-------] 78%Downloading VS Code [========================------] 82%Downloading VS Code [========================------] 82%Downloading VS Code [=========================-----] 84%Downloading VS Code [=========================-----] 87%Downloading VS Code [==========================----] 87%Downloading VS Code [==========================----] 90%Downloading VS Code [===========================---] 93%Downloading VS Code [============================--] 96%Downloading VS Code [=============================-] 100%Downloaded VS Code into /Users/runner/work/vscode-jupyter/vscode-jupyter/.vscode-test/vscode-darwin-insiders
Downloaded VS Code into /Users/runner/work/vscode-jupyter/vscode-jupyter/.vscode-test/vscode-darwin-insiders
Installing Python Extension ms-python.python to /var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/vscode_jupyter_exts
Installing Pylance Extension to /var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/vscode_jupyter_exts
Installing Renderer Extension to /var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/vscode_jupyter_exts
Test error: Error: spawn /Users/runner/work/vscode-jupyter/vscode-jupyter/.vscode-test/vscode-darwin-insiders/Visual Studio Code - Insiders.app/Contents/MacOS/Electron ENOENT
Exit code:   -2
End Standard tests (with errors) Failed
****************************************************************************************************
Start Standard tests
End Standard tests (with errors) Error: ENOENT: no such file or directory, open '/Users/runner/work/vscode-jupyter/vscode-jupyter/.vscode-test/vscode-darwin-insiders/Visual Studio Code - Insiders.app/Contents/Resources/app/product.json'
    at Object.openSync (node:fs:[60](https://github.com/microsoft/vscode-jupyter/actions/runs/6962842187/job/18947373636#step:13:61)1:3)
    at Object.readFileSync (node:fs:469:35)

DonJayamanne commented 1 year ago

Will try a specific commit today

alexr00 commented 1 year ago

I'm also hitting this. Trying with the latest npm package now...

alexr00 commented 1 year ago

No luck. I'm able repro locally too. I can see that D:\repos\Microsoft\vscode-pull-request-github\.vscode-test\vscode-win32-x64-archive-insiders\Code - Insiders.exe simply doesn't exist.

alexr00 commented 1 year ago

Rerunning CI has not fixed it for me. Maybe interesting: so far, this has only occurred in my public PR CI and locally, not in the internal CI that I have for releasing builds.

Edit: actually after 5 reruns it's succeeded.

connor4312 commented 1 year ago

@alexr00 can you confirm it happens when you request the version via commit hash so that it did the SHA verification after download?

alexr00 commented 1 year ago

Yes, I set the version here to the latest insiders commit. I only tried this locally though, I didn't try this in the pipeline.

connor4312 commented 1 year ago

Thanks, will try to repro and investigate more after the holiday.

joaomoreno commented 1 year ago

This is too coincidental with happening at the same time we're rolling out the new CDN to insider builds. Investigating...

BTW, update service now has the X-SHA256 changes: https://dev.azure.com/monacotools/Monaco/_build/results?buildId=246022&view=results

joaomoreno commented 1 year ago

Tests were only running against stable. I've added tests for insiders, which are now failing:

joaomoreno commented 1 year ago

Found the issue. It's indeed related to the new CDN.

We're relying on the Content-Type header to detect whether the download is a ZIP or a TGZ: https://github.com/microsoft/vscode-test/blob/c1a49663d17d86c76cc74c1a7ac4d955d1906d73/lib/download.ts#L223

The new CDN unfortunately always sends application/octet-stream:

➜  ~ http -F -h https://update.code.visualstudio.com/latest/darwin-arm64/insider
HTTP/1.1 200 OK
Accept-Ranges: bytes
Age: 12449
ApiVersion: Distribute 1.1
Cache-Control: public, max-age=86400
Content-Disposition: attachment; filename=VSCode-darwin-arm64.zip; filename*=UTF-8''VSCode-darwin-arm64.zip
Content-Length: 123910672
Content-Type: application/octet-stream
Date: Fri, 24 Nov 2023 10:43:49 GMT
Etag: "0xA44299607ACF10F270F36D79ADF383369949B4C0BE16DEAFEC16D8071E72B32A"
Last-Modified: Fri, 24 Nov 2023 06:29:55 GMT
Server: ECAcc (mil/6BBD)
X-Cache: HIT

➜  ~ http -F -h https://update.code.visualstudio.com/latest/darwin-arm64/stable
HTTP/1.1 200 OK
Accept-Ranges: bytes
Access-Control-Allow-Origin: *
Access-Control-Expose-Headers: x-ms-request-id,x-ms-version,x-ms-lease-status,x-ms-lease-state,x-ms-blob-type
Age: 1268675
Cache-Control: max-age=31536000, public
Content-Disposition: attachment; filename="VSCode-darwin-arm64.zip"
Content-Length: 125047073
Content-MD5: g75j7rfjdHvKVTNfWLbMQg==
Content-Type: application/zip
Date: Fri, 24 Nov 2023 10:43:54 GMT
Etag: "0x8DBE11687A6A0BA"
Last-Modified: Thu, 09 Nov 2023 11:25:08 GMT
Server: ECAcc (mil/6C9A)
X-Cache: HIT
x-ms-blob-type: BlockBlob
x-ms-lease-state: available
x-ms-lease-status: unlocked
x-ms-request-id: e891b733-401e-0035-0d39-13490d000000
x-ms-version: 2013-08-15

This explains why Windows and macOS are broken, while Linux isn't: the ZIP archives are incorrectly "detected" as TGZ.

joaomoreno commented 1 year ago

Fixed in 2.3.8, pipeline running

joaomoreno commented 1 year ago

🚀 https://www.npmjs.com/package/@vscode/test-electron/v/2.3.8

connor4312 commented 1 year ago

Thank you for looking into and fixing this! I'm surprised this didn't surface in a more explicit way. I owe you a testing/debug bug fix of your choice 😁

joaomoreno commented 1 year ago

Yeah me too! It turns out tar -xzf also enjoys munching on zip files and just does weird things with them. 😄

alexr00 commented 1 year ago

I get a different error now: https://dev.azure.com/vscode/vscode-pull-request-github/_build/results?buildId=109640&view=logs&j=bf770c6b-39e5-569e-4808-1859a08d14ab&t=ede73121-db1b-5427-c6bc-86115846f909

Still coming from test-electron though.

connor4312 commented 1 year ago

Please ensure you're using a version of Node >=16 where the AbortController is available.

alexr00 commented 12 months ago

That was it, thanks!

microsoft / vscode-test

VS Code executable not found after downloading it #246