A couple of issues have been noticed in the Editor Tests.
1) As mentioned in https://github.com/pulsar-edit/pulsar/pull/954#issuecomment-1997177492, there are some silent fails possible, where an individual test will fail, the retry will happen but with an empty set of tests to run, for some reason, which "succeeds" with no failures and 100% specs skipped, which exits "cleanly", so CI thinks the run was good and "passes".
We should Ensure the re-run actually runs the intended tests... rather than having an empty set of tests to run and skipping 100% of tests, thereby false-passing the CI run. Or at least have the script error out if an error was detected and the set of tests we are about to re-run is malformed as empty.
2) A false pass in this test?
AtomEnvironment > saving and loading > it serializes the project state with all the options supplied in saveState [1924:0328/191144.391928:INFO:CONSOLE(0)] "Uncaught (in promise) Error: ENOENT: no such file or directory, lstat '/private/var/folders/rd/9ccg_bdj4v949y62zjrcv9100000gn/T/dir2-2024228-1927-c8ixq1.py8ie'", source: file:///Users/user/pulsar/static/index.html (0)
[1924:0328/191144.392056:INFO:CONSOLE(0)] "Uncaught (in promise) Error: ENOENT: no such file or directory, lstat '/private/var/folders/rd/9ccg_bdj4v949y62zjrcv9100000gn/T/dir2-2024228-1927-c8ixq1.py8ie'", source: file:///Users/user/pulsar/static/index.html (0)
[1924:0328/191144.392144:INFO:CONSOLE(0)] "Uncaught (in promise) Error: ENOENT: no such file or directory, lstat '/private/var/folders/rd/9ccg_bdj4v949y62zjrcv9100000gn/T/dir1-2024228-1927-a6vtc6.egd1f'", source: file:///Users/user/pulsar/static/index.html (0)
[1924:0328/191144.392224:INFO:CONSOLE(0)] "Uncaught (in promise) Error: ENOENT: no such file or directory, lstat '/private/var/folders/rd/9ccg_bdj4v949y62zjrcv9100000gn/T/dir1-2024228-1927-a6vtc6.egd1f'", source: file:///Users/user/pulsar/static/index.html (0)
[pass]
(This looks more like it should be returning a "fail" result rather than a "pass" result, IMO? (Presumably there is some test assertion that coincidentally passes if nothing happens, and despite the run erroring out, the assertion passes?) Would be great to get this test properly running and true-passing, not false passing, if it is indeed having problems as it appears to be at a glance.)
3) More? Worth doing a review of how well the editor tests are working.
A couple of issues have been noticed in the Editor Tests.
1) As mentioned in https://github.com/pulsar-edit/pulsar/pull/954#issuecomment-1997177492, there are some silent fails possible, where an individual test will fail, the retry will happen but with an empty set of tests to run, for some reason, which "succeeds" with no failures and 100% specs skipped, which exits "cleanly", so CI thinks the run was good and "passes".
We should Ensure the re-run actually runs the intended tests... rather than having an empty set of tests to run and skipping 100% of tests, thereby false-passing the CI run. Or at least have the script error out if an error was detected and the set of tests we are about to re-run is malformed as empty.
2) A false pass in this test?
(This looks more like it should be returning a "fail" result rather than a "pass" result, IMO? (Presumably there is some test assertion that coincidentally passes if nothing happens, and despite the run erroring out, the assertion passes?) Would be great to get this test properly running and true-passing, not false passing, if it is indeed having problems as it appears to be at a glance.)
3) More? Worth doing a review of how well the editor tests are working.