Ideas for CI-only tests

[savetheclocktower]

Certain tests we could run would require installing actual software into specific locations, thus are unfeasible to run on contributors' machines. If someone could figure out how to set up a CI server with the necessary environment, I could write the tests.

Testing nodeBin behavior

This would involve installing various versions of Node into known locations.

• Install three different versions of Node:

  • One bleeding-edge
  • One LTS
  • One explicitly too old to support module syntax (maybe?)
  • Maybe use NVM for this? Not sure how well it plays with CI environments

• Set nodeBin to the LTS version via package settings

• Open a project with ESLint v8 or greater

  • lint a file; expect results
  • run the debug action; get output with a Node version number that matches what we expect

• Create a .linter-eslint file with { "nodeBin": "[path to bleeding-edge node]" }

  • save the .linter-eslint file; wait a few seconds
  • lint a file; expect results
  • run the debug action; get output with a Node version number that matches what we expect

• Modify the .linter-eslint file to read { "nodeBin": "[path to too-old node]" }

  • save the .linter-eslint file; wait a few seconds
  • run the debug action; get output with a Node version number that matches what we expect
  • lint a file; expect… not sure? failure of some kind

• Delete our .linter-eslint file

  • expect behavior to match what we saw above with LTS node

• Change the node path in package settings via atom.config.set to bleeding-edge

  • expect behavior to match what we saw above with bleeding-edge node

• Change the node path in package settings to something nonexistent

  • expect a warning message

Testing behavior with various ESLint versions

We read ESLint's package.json to figure out what version we're dealing with. So to write specs against this behavior, we'd have to have at least three different dummy projects, each of which installs a different version of ESLint:

• One v8 or greater (latest stable version)
• One v7 (to test overlap behavior)
• One <v7 (to test eslint-too-old behavior)

Things to test:

• Open the v8-or-greater version with linter-eslint package installed, but linter-eslint-node disabled

  • Expect a notification that prompts us to upgrade

• Open the v8-or-greater version with linter-eslint package enabled and linter-eslint-node enabled

  • Ensure no notification appears that prompts us to upgrade
  • Ensure the “Debug” command correctly identifies linter-eslint-node as handling the linting for this project

• Open the v7 project with linter-eslint disabled

  • Ensure linter-eslint-node lints as expected

• Open the v7 project with linter-eslint enabled

  • Ensure the “Debug” command correctly identifies linter-eslint as handling the linting for this project
  • Ensure linter-eslint-node doesn't try to lint; explicitly calling provideLinter().lint should produce zero results, even on a file with a syntax error

• Open the <v7 project with linter-eslint disabled

  • Expect a notification that prompts us to install linter-eslint
  • Repeat steps, but with relevant setting unchecked; expect no notification
  • Ensure linter-eslint-node doesn't try to lint; explicitly calling provideLinter().lint should produce zero results, even on a file with a syntax error

• Open the <v7 project with linter-eslint enabled

  • Expect no notification
  • Ensure linter-eslint-node doesn't try to lint; explicitly calling provideLinter().lint should produce zero results, even on a file with a syntax error

[UziTech]

We can setup GitHub Action workflows for the different setups. If we have some tests we only want to run in CI we can use process.env.CI.

something like:

if (process.env.CI) {
  describe('CI tests' () => {
      ...
    }
  );
}