talonvoice / talon

Issue Tracker for the main Talon app
85 stars 0 forks source link

Add a way for user repositories to run CI tests inside talon #375

Open pokey opened 2 years ago

pokey commented 2 years ago

We just need to be able to do a pnpm install, and then run a node script on a machine that has a Cursorless commit checked out and has Talon running. See https://github.com/talonvoice/talon/issues/375#issuecomment-1667621659 for an overview of how our spoken form tests work

Not sure if this was what you already had in mind, but I wonder if using your servers as a GitHub self-hosted runner would be a way to do this

pokey commented 2 years ago

Fwiw here is our CI setup to run cursorless and VSCode:

pokey commented 11 months ago

Just a heads up that we're all set now on our side; we have >1k spoken form tests that can run without requiring VSCode, so whenever you get CI set up, we should be good to go.

The way it works is that we have a Mocha test runner using node.js that does the following:

  1. Enables a mode that turns off every command except Cursorless commands (see node.js side; Talon side), and overrides RPC to just store command payloads rather than sending them to VSCode.
  2. For each spoken form to test, sends it via the REPL to a Talon action that just mimics it, then prints the JSON payload of any commands that were run as a result of the spoken form
  3. Checks that the JSON matches the expected payload, or prints it out if it wasn't JSON-parseable, as we assume it was actually REPL printing an error message

As of today, this test runner does require community to be installed, but mocking that would be trivial, as we just require a couple basic actions (see https://github.com/cursorless-dev/cursorless/issues/1739)

Today we run this script locally, but it should be fairly straightforward for us to do this in CI if it ran on a box where we can do a pnpm install and have Talon running