Update for async API calling pattern (for debug-time use of linter failures)

[cabiad]

For large codebases, it would be good if lint results could be communicated to devs before CI / pre-commit time. We were wondering if we could integrate with our web back end when it's in debug mode to show devs ASAP if there are problems with what they're doing.

[cabiad]

@lime-green investigated pretty thoroughly but we haven't found a great way for the tool to operate asynchronously. We may come back to this another day.

[lime-green]

It was hard to integrate with our UWSGI django app, notably because multiple processes would run and this required locking (I used django cache but this only works if the underlying cache implementation is atomic) or to start it independently of the UWSGI processes. It also either requires a file watcher (but then codewatch restarts on every little change) or someway to restart whenever the dev server restarts.

After discussion with @francoiscampbell, I propose the following:

• Have this async service run inside a docker container
• Have it watch an arbitrary directory (passed as an argument to the container)
• On file change, run codewatch only if it's not already running
• Have an optional throttle argument so that it only runs if it hasn't been run in the last N seconds
• expose a simple HTTP API which returns the failures of the last run, or a 200 status code if none

[noahnu]

Alternative solution:

Explicitly define and support a monotonic assertion type.

A monotonic assertion would have the properties:

• All dependant visitor functions must be monotonic visitors.
  • Monotonic Visitor: visitor in which all operations to the stats dictionary follows a monotonic function. That is, if the effects of the visitor function on the stats dictionary is quantified, the results in the dictionary clearly follow a single explicit ordering. An example would be a visitor function which only ever does stats.increment and never does a stats.decrement. Or vice versa.
• If the assertion "passes" (doesn't throw AssertError) for N executions of the dependant visitor functions, then it must pass for < N executions of the visitor functions.

Now given monotonic assertions, provide CLI "watch" mode (or IDE hook) such that: When a file is saved, check that it matches file/dir filters. Now, for every monotonic assertion, alternate between a visit to each dependant visitor and an execution of the parent assertion. When the assertion fails, report an "error" and stop traversing the AST.

What does this solve?

We no longer need to traverse the entire AST of the codebase before determining if an assertion fails. In CI you'll probably still want the complete stats, but for something like an IDE / watch mode integration, you're probably only concerned with the files you're actively editing (or diff between current code and HEAD of upstream).

Caveats

This would require a way to know if an assertion is monotonic. We could provide a way to explicitly label the function and explicitly note the dependant visitor functions.

Need to support way to alternate between visitors and assertions efficiently.

Comments

Most of our assertions in use are already monotonic according to the above definition.