search

quick-lint / quick-lint-js

quick-lint-js finds bugs in JavaScript programs
https://quick-lint-js.com
GNU General Public License v3.0
1.52k stars 192 forks source link

VS Code test flaky failure #1055

Open strager opened 1 year ago

strager commented 1 year ago

macOS AArch64 https://github.com/quick-lint/quick-lint-js/actions/runs/5828965577/job/15807702364

Running rename open .js file to .py ...
OK      rename open .js file to .py
Running rename open .py file to .js ...
AssertionError [ERR_ASSERTION]: Expected values to be strictly equal:

false !== true

    at /Users/github-builder/github-actions-runner/_layout/_work/quick-lint-js/quick-lint-js/plugin/vscode/test/vscode-tests.js:1278:12
FAIL    rename open .py file to .js
    at pollAsync (/Users/github-builder/github-actions-runner/_layout/_work/quick-lint-js/quick-lint-js/plugin/vscode/test/vscode-tests.js:1567:9)
    at async waitUntilAnyDiagnosticsAsync (/Users/github-builder/github-actions-runner/_layout/_work/quick-lint-js/quick-lint-js/plugin/vscode/test/vscode-tests.js:1276:3)
    at async rename open .py file to .js (/Users/github-builder/github-actions-runner/_layout/_work/quick-lint-js/quick-lint-js/plugin/vscode/test/vscode-tests.js:155:5)
    at async runTestAsync (/Users/github-builder/github-actions-runner/_layout/_work/quick-lint-js/quick-lint-js/plugin/vscode/test/test-support.js:51:5)
    at async runTestsAsync (/Users/github-builder/github-actions-runner/_layout/_work/quick-lint-js/quick-lint-js/plugin/vscode/test/test-support.js:35:7)
    at async testMainAsync (/Users/github-builder/github-actions-runner/_layout/_work/quick-lint-js/quick-lint-js/plugin/vscode/test/test-support.js:22:5)
    at async runAsync (/Users/github-builder/github-actions-runner/_layout/_work/quick-lint-js/quick-lint-js/plugin/vscode/test/vscode-tests.js:1595:3)
    at async Object.runCatchingErrorsAsync [as run] (/Users/github-builder/github-actions-runner/_layout/_work/quick-lint-js/quick-lint-js/plugin/vscode/test/vscode-tests.js:1602:12)
Error: test failed
    at /Users/github-builder/github-actions-runner/_layout/_work/quick-lint-js/quick-lint-js/plugin/vscode/test/vscode-tests.js:1596:11
    at testMainAsync (/Users/github-builder/github-actions-runner/_layout/_work/quick-lint-js/quick-lint-js/plugin/vscode/test/test-support.js:27:5)
    at processTicksAndRejections (internal/process/task_queues.js:97:5)
    at async runAsync (/Users/github-builder/github-actions-runner/_layout/_work/quick-lint-js/quick-lint-js/plugin/vscode/test/vscode-tests.js:1595:3)
    at async Object.runCatchingErrorsAsync [as run] (/Users/github-builder/github-actions-runner/_layout/_work/quick-lint-js/quick-lint-js/plugin/vscode/test/vscode-tests.js:1602:12)
No authentication provider 'microsoft' is currently registered.
Settings Sync: Accounts are not initialized
2023-08-10 21:44:43.295 Code Helper (Renderer)[79383:49471702] CoreText note: Client requested name ".NewYork-Regular", it will get Times-Roman rather than the intended font. All system UI font access should be through proper APIs such as CTFontCreateUIFontForLanguage() or +[NSFont systemFontOfSize:].
Exit code:   1
undefined
error Command failed with exit code 1.
strager commented 1 year ago

Another failure of the same test soon after (suspicious): https://github.com/quick-lint/quick-lint-js/actions/runs/5829905074/job/15810535318

strager commented 1 year ago

https://github.com/quick-lint/quick-lint-js/actions/runs/5844970826/job/15848765850

strager commented 1 year ago

https://github.com/quick-lint/quick-lint-js/actions/runs/5889667835/job/15973875729

strager commented 1 year ago

https://github.com/quick-lint/quick-lint-js/actions/runs/5923102122/job/16058538086

strager commented 1 year ago

I ran the test suite on my M1 laptop for an hour and a half and it didn't hit this flaky failure. (It did hit another flaky failure, though: Running open existing non-JS file ... Error: Could NOT open editor for "file:///private/var/folders/fw/nlljnqw16lx56p_8s4jf15j00000gn/T/quick-lint-js-vscode-test-92ca1H/hello.txt" because another editor opened in the meantime.)

strager commented 1 year ago

Unusual logs in this failure: https://github.com/quick-lint/quick-lint-js/actions/runs/5945589414/job/16125151877

Running open existing non-JS file ...
OK      open existing non-JS file
Running rename open .js file to .py ...
Unable to read file '/private/var/folders/24/722hs_6j7bx_w0t84_8ywyzh0000gn/T/quick-lint-js-vscode-test-PJu7au/hello.py' (Error: File not modified since): Error: Unable to read file '/private/var/folders/24/722hs_6j7bx_w0t84_8ywyzh0000gn/T/quick-lint-js-vscode-test-PJu7au/hello.py' (Error: File not modified since)
    at o.doReadFileStream (file:///Users/github-builder/github-actions-runner/_layout/_work/quick-lint-js/quick-lint-js/plugin/vscode/.vscode-test/vscode-darwin-arm64-1.55.2/Visual Studio Code.app/Contents/Resources/app/out/vs/workbench/workbench.desktop.main.js:626:48867)
OK      rename open .js file to .py
    at async E.doRead (file:///Users/github-builder/github-actions-runner/_layout/_work/quick-lint-js/quick-lint-js/plugin/vscode/.vscode-test/vscode-darwin-arm64-1.55.2/Visual Studio Code.app/Contents/Resources/app/out/vs/workbench/workbench.desktop.main.js:2151:80864)
Running rename open .py file to .js ...
    at async E.readStream (file:///Users/github-builder/github-actions-runner/_layout/_work/quick-lint-js/quick-lint-js/plugin/vscode/.vscode-test/vscode-darwin-arm64-1.55.2/Visual Studio Code.app/Contents/Resources/app/out/vs/workbench/workbench.desktop.main.js:2151:80510)
    at async kr.loadFromFile (file:///Users/github-builder/github-actions-runner/_layout/_work/quick-lint-js/quick-lint-js/plugin/vscode/.vscode-test/vscode-darwin-arm64-1.55.2/Visual Studio Code.app/Contents/Resources/app/out/vs/workbench/workbench.desktop.main.js:2149:49103)
AssertionError [ERR_ASSERTION]: Expected values to be strictly equal:

false !== true

    at /Users/github-builder/github-actions-runner/_layout/_work/quick-lint-js/quick-lint-js/plugin/vscode/test/vscode-tests.js:1278:12
    at pollAsync (/Users/github-builder/github-actions-runner/_layout/_work/quick-lint-js/quick-lint-js/plugin/vscode/test/vscode-tests.js:1567:9)
    at async waitUntilAnyDiagnosticsAsync (/Users/github-builder/github-actions-runner/_layout/_work/quick-lint-js/quick-lint-js/plugin/vscode/test/vscode-tests.js:1276:3)
    at async rename open .py file to .js (/Users/github-builder/github-actions-runner/_layout/_work/quick-lint-js/quick-lint-js/plugin/vscode/test/vscode-tests.js:155:5)
FAIL    rename open .py file to .js
    at async runTestAsync (/Users/github-builder/github-actions-runner/_layout/_work/quick-lint-js/quick-lint-js/plugin/vscode/test/test-support.js:51:5)
    at async runTestsAsync (/Users/github-builder/github-actions-runner/_layout/_work/quick-lint-js/quick-lint-js/plugin/vscode/test/test-support.js:35:7)
    at async testMainAsync (/Users/github-builder/github-actions-runner/_layout/_work/quick-lint-js/quick-lint-js/plugin/vscode/test/test-support.js:22:5)
    at async runAsync (/Users/github-builder/github-actions-runner/_layout/_work/quick-lint-js/quick-lint-js/plugin/vscode/test/vscode-tests.js:1595:3)
    at async Object.runCatchingErrorsAsync [as run] (/Users/github-builder/github-actions-runner/_layout/_work/quick-lint-js/quick-lint-js/plugin/vscode/test/vscode-tests.js:1602:12)
Error: test failed
    at /Users/github-builder/github-actions-runner/_layout/_work/quick-lint-js/quick-lint-js/plugin/vscode/test/vscode-tests.js:1596:11
    at testMainAsync (/Users/github-builder/github-actions-runner/_layout/_work/quick-lint-js/quick-lint-js/plugin/vscode/test/test-support.js:27:5)
    at processTicksAndRejections (internal/process/task_queues.js:97:5)
    at async runAsync (/Users/github-builder/github-actions-runner/_layout/_work/quick-lint-js/quick-lint-js/plugin/vscode/test/vscode-tests.js:1595:3)
    at async Object.runCatchingErrorsAsync [as run] (/Users/github-builder/github-actions-runner/_layout/_work/quick-lint-js/quick-lint-js/plugin/vscode/test/vscode-tests.js:1602:12)
No authentication provider 'microsoft' is currently registered.
Settings Sync: Accounts are not initialized
Exit code:   1
strager commented 1 year ago

Unusual logs again: https://github.com/quick-lint/quick-lint-js/actions/runs/5954183289/job/16150616742

The unusual logs correlate with commit 246d60ad02ab5d2bf7d93a10ec4cb36b87e2efc8 which upgraded the VS Code we test against.

strager commented 1 year ago

Git commit 550b51d67e2fca7834f29de191d695e2dd50c3fa added --disable-gpu to the VS Code flags. I think this will help with macOS AArch64 flakiness. It doesn't seem to help with Windows flakiness though: https://github.com/quick-lint/quick-lint-js/actions/runs/6129946246/job/16638617081