passtool: implement the `gen-checks` command

[zerbina]

Summary

• add the gen-checks passtool command, for generating a syntax check module derived from a language grammar
• add the generate command to koch
• integrate the syntax checks into the test runners and REPL
• fix some grammar issues

Details

Passtool

• add commands to passtool. verify only performs the grammar verification; gen-checks generates the syntax check modules
• the code generator is kept simple in order to allow for easier adjustments later on

Check Integration

• integrate the syntax checks into all pass test runners. Both the test input and immediate output are checked
• add source language syntax checks to the REPL

Koch

• add the generate command, for generating the language-derived modules
• automatically run passtool gen-checks prior to building programs that require the generated modules

CI

• run the language checks before building the executables and running the tests
• update the passtool invocations

Fixes

• fix multiple grammar issues with various languages, where the grammar didn't reflect reality
• fix some pass3 tests violating the L3 grammar

[zerbina]

I'm not sure whether committing the generated modules is a good idea. On the upside:

• it's clear to reviewers when and how the generated code is affected by some
• it's possible to just clone the repository and compile the code without an intermediate step or executable

The downside are that:

• it introduces some noise to the PR changesets (at least in my opinion)
• it's a source of friction when creating PRs (the generated files need to be up-to-date, otherwise CI fails)

I'm currently leaning towards not committing the files, but this would mean that one has to first run nim r koch.nim generate after cloning the repository in order to be able to compile various executables. Thoughts?

[saem]

I'm currently leaning towards not committing the files, but this would mean that one has to first run nim r koch.nim generate after cloning the repository in order to be able to compile various executables. Thoughts?

If we can have a single bit of metadata in a file that koch can check on each run, then it could check the metadata and automatically run generate.

[zerbina]

If we can have a single bit of metadata in a file that koch can check on each run, then it could check the metadata and automatically run generate.

That would work, though it would still require going through koch. Maybe that's not too much of a problem -- NimSkull effectively requires one to do the same, after all.

[zerbina]

I've excluded the generated files from being committed and extended koch to regenerate the generated modules when needed.

The up-to-date check is as naive as it can get, with only the existence of the directory being taken into account, though I can foresee this becoming annoying very quickly, since it's easy to forgot to manually run koch generate after rebasing some branch onto main (after an update). The command is quick to run, so it'll hopefully not be too much of an issue.

While a proper task graph system with a input/artifact tracking would work and scale a lot better, it's also a whole lot more complex and I don't think it to be necessary at the moment.