jonsequitur commented 2 weeks ago

Magic command API redesign

Background and motivations

In the early days of .NET Interactive, providing magic command features as seen in Jupyter required a parser for a magic command grammar that would be independent of the various languages (C#, F#, PowerShell) that .NET Interactive supports. A POSIX command line-style grammar was a fairly clear choice, and System.CommandLine was robust and ready to use, so rather than build something custom, we took a dependency on System.CommandLine. This API has served .NET Interactive well for several years now. But as System.Commandline's design is being reset for inclusion as a core .NET library, some of the features that .NET Interactive uses are likely to be removed. At the same time, .NET Interactive's input system is undergoing a long-planned set of improvements whose ergonomics can be much better if we support features that are well outside of what a command line parser would include.

The largest impacts of these changes will be on people who've written extensions to .NET Interactive that customize magic commands. We are attempting to minimize impacts on notebook users by maintining backwards compatibility to the greatest extent possible, but it's important to recognize that some breaking changes are unavoidable unless we want to bring the entirety of the System.CommandLine codebase into .NET Interactive, which has a high cost for maintainability and concept complexity.

So what's changing?

Options and arguments are now just parameters

The terms option and argument were chosen based on common Gnu/POSIX naming conventions for the command line. Separating the magic command API from the command line use case is an opportunity to choose the more commonly-recognized term parameter. Both named parameters (i.e. options) and unnamed parameters (i.e. arguments) will now use the same type, KernelDirectiveParameter.

What's being removed?

The following features of System.CommandLine are not currently planned to be reimplemented by the new magic command parser.

A magic command can no longer have multiple unnamed parameters (i.e. arguments).
Optionally, a single unnamed parameter can be allowed if the magic command is configured to allow it. It is not required of API users to configure their magic commands to have a name-optional parameter (KernelDirectiveParameter.AllowImplicitName).

The following two lines are equivalent, assuming --name is the parameter whose name is optional:
```
#!fruit --name apple 
#!fruit apple
```
Similarly, the following are equivalent:
```
#!fruit --color red --name apple
#!fruit --color red apple
```
Relative ordering between parameters is not significant, so the following are equivalent:
```
#!fruit --color red apple 
#!fruit apple --color red
```
Inputs (e.g. @input and @password) are now only valid for a parameter values. They can no longer be used to supply subcommands or parameter names.

Assuming a #!fruit magic command with two options, --color and --name, neither of which allow an implicit parameter name, the following would be invalid:
```
#!fruit @input:"Which parameter are you setting?" red
```
The allowed parameter name prefix / (used for Windows-style options) will no longer be supported. The only allowed prefixes for parameter names will be - and --.
System.CommandLine allows three ways to separate an option name from its argument: --option argument, --option=argument, and --option:argument. Only the first (space-separated) syntax will now be supported.
POSIX-style option bundling (e.g. git clean -fdx which is equivalent to git clean -f -d -x) will no longer be supported.
The POSIX-style -- delimiter, used as an escape in POSIX command lines, will no longer be supported.
Minimum arities are no longer supported. Parameters can be required but a minimum arity can't be specified. Only maximum arities (now specified using KernelDirectiveParameter.MaxOccurrences) are supported.

What's being added

Inline JSON can now be used to specify parameter values. (Multi-line magic commands are still not allowed, so JSON must be all on one line.)
```
#!fruit --name apple { "color": "red", "varieties": ["Macintosh", "Honeycrisp"] }
```
Variable sharing and input tokens are supported as before, but are not allowed within inlined JSON.

Inline JSON can also be used to configure expressions such as @input and @password.

#!connect mssql --data-source @input:{ "prompt": "Please choose a database", "type":"list", "recall": true, "provider": "AzureAD", "clientId": "abc", "tenantId": "def", "subscriptionId": "xyz" }

Type hints can be specified directly on a KernelDirectiveParameter. (Previsouly, they were inferred from the generic parameter of the target Argument<T> or Option<T>.)
More granular diagnostic squiggles are now provided. The System.CommandLine parser is unaware of character positions in the original text. This isn't knowable in a command line because the arguments to be parsed are sent to a .NET application entry point already broken into an array. This limitation need not apply in a notebook.
Richer completion support is now possible (e.g. within an argument, or within JSON).

Handling magic command invocation

As of System.CommandLine beta 4 (which is the most recent version that .NET Interactive depends on), the code that handles the execution of a magic command is specified using the Command.Handler property, which mainly does two things:

Calls a user-specified delegate passing strongly-typed parameters.
Parses those strongly-typed parameters from string input.

The Command.Handler API has been a major pain point in System.CommandLine and has seen a number of breaking changes over the last few years. The magic command API redesign is an opportunity to replace a couple of pieces with existing, stable APIs.

Since there's already an API for kernels to handle KernelCommand-derived commands, magic commands will now use the same approach. On invocation, magic commands will be parsed into KernelCommand implementations and passed to Kernel.SendAsync. For custom magics, these will typically be commands defined by the magic command author. All of the existing command handling APIs, including middleware, will work with these commands just like they do with existing commands such as SubmitCode.

This has the additional effect of allowing magic command behaviors to be invoked directly by sending the corresponding KernelCommand, bypassing the magic command syntax. For example, #r nuget is now parsed into a new AddPackage command, making the following two examples functionally equivalent:
```
#r "nuget: Plotly.NET, 5.0.0"
```
```
var command = new AddPackage(
    packageName: "Plotly.NET",
    packageVersion: "5.0.0");
await kernel.SendAsync(command);
```
You can think of a magic command as a gesture for the notebook user while the KernelCommand is its underlying API, which might be more ergonomic for programmatic usage.
Rather than using a custom binding and serialization implementation to create KernelCommand instances from parsed magic commands, .NET Interactive will now use JSON serialization. The new parser has the ability to serialize parsed magic commands into JSON. Magic commands are then deserialized using System.Text.Json. Any custom deserialization you might need can be configured using standard System.Text.Json attributes on your custom command type.

With all of that in mind, here's a simple example of the new API:

var fruitDirective = new KernelActionDirective("#!fruit");

kernel.AddDirective<ExampleCommand>(
    fruitDirective,
    async (ExampleCommand command, KernelInvocationContext context) =>
    {
        // magic command logic goes here
    });

marckruzik commented 2 weeks ago

I'm glad to see this being worked on! Bravo! 😃

jonsequitur commented 2 weeks ago

It's been on the back burner for a minute and is a very complex change. I'll be updating the details above as more of the design details are worked out. Please give us any thoughts or questions that occur to you.

IntegerMan commented 1 week ago

This is great to know about in this level of detail. This makes a lot of sense and I love the clarity on implicit parameters as well. I think this change makes the overall system more approachable with consistent terminology, in addition to being a necessary change due to the commandline dependency.

dotnet / interactive