Engine overhaul (WIP)

[cassaundra]

Engine overhaul

This is a large engine overhaul that is depended upon by a number of features, in particular lazy task evaluation, but also a number of others (detailed below) which are interdependent on each other in various ways. Many of those features have been made trivial to implement as part of this overhaul (and useful as a proof of concept), so a few of them have been included here as well.

The features directly implemented by this overhaul are:

• Add signatures to def-task blocks (#40)
• Lazily evaluate task declaration blocks (#37)

Normally, this would be broken down into smaller PRs and/or commits, but as the priority right now is primarily to push out a working 0.1.0, this is not too much of a concern.

Changes

In support of the features implemented above, this PR also contains the following changes:

• [x] Split tasks into TaskStubs (the result of a def-task declaration) and TaskCalls (the result of a call of a given task) and propagate these changes to the rest of the engine
  • Task stubs contain only the name and signature of the task (as well as any corresponding BlockIds, internally)
  • Task calls are instantiated from direct command-line invocation (e.g. quake build --release or quake build -- --release depending on the final syntax) and depends calls inside task declaration block (e.g. depends build --release)
• [x] Move the heavy lifting of def-task into the early parsing phase
  • Populates task stubs before any evaluation as occurred
  • Updates the signatures of task blocks to match the signature argument
  • Due to a nushell parsing quirk, we seem to end up needing to reparse the signature anyway, so this is the best design for this approach
• [x] Lazily evaluate task declaration blocks
  • See EngineState::populate_metadata_for_call
• [x] Simplify task eval scopes with stored
• [x] Validate Metadata::insert_task_call_metadata against task stub signatures
  • This should already effectively be caught on actual invocations, but this would raise errors earlier
• [x] Implement a conversion from quake to parse errors (IntoParseError, like IntoShellError)
  • Unfortunately this is a pretty lossy conversion, might be worth revisiting to solve this issue more generally
• [x] Reorganize modules to clean up structure as a result of this refactor
  • Moves most nushell internals into nu submodules
• [x] Standardize exit codes
  • 100: load failure
  • 101: task decl failure
  • 102: task run failure
• [ ] Convert command-line arguments into nu_protocol::Arguments (deferred)

This overhaul also includes a number of improvements to documentation, both to the existing code and to what has been introduced.

To-do

Other than the above, some items to complete internal to this refactor:

• [x] Re-implement subtask
  • An easy hack might be to give subtasks signatures with their argument/value pairs as named arguments with defaults
• [x] Re-implement depends
• [x] Re-implement sources and produces
• [ ] Remove #![allow(dead_code)]
• [x] Switch away from temporary expose-parser branch of nushell (which just exports all parser internals for convenience)
• [x] Split squashed commits where possible/convenient (not blocking)

Not listed above are a number of generally inconsequential fixes made along the way.

Subsequent/deferred work

(some of these should have issues)

• Put def-task, and separately the commands used inside of it (e.g. depends), into their own overlays to ensure at parse-time they are used only in the appropriate contexts
• Implement file-watching mode (#14) with these changes in consideration.
• Make internal state a RwLock instead of Mutex
• Ensure no mutable variables are captured by task blocks
• Associate tasks with nushell VarIds (see #48)

Comment

Note: I've squashed the intermediate commits made in this process since this overhaul has undergone several iterations already. An extra TODO has been added to split out some of these commits in the interest of maintaining an easier to read git log.

[cassaundra]

Added #48 as potential future work

[cassaundra]

Fixed AST traversal. Need to update nushell branch then ready to merge