Upcoming changes

[KathleenDollard]

Upcoming changes

A short history

Nearly five years ago, @JonSequitur and I believed that .NET needed a proper POSIX-based command line (CLI) parser. We set out to build one based on the parser that was in the version of the .NET SDK current at that time. Along the way we have had support and contributions from so many people - with a special shout out to @Keboo, @PatrikSvensson and @HowardvanRooijen. It has mostly been a side project, which has made progress slow.

Along the way the bar - the expectations - for our initial vision has been raised many times. Early on, we realized the non-parser problems were bigger than the parser itself. We also discovered that no matter how complicated you think proper POSIX parsing is, you underestimate the complexity. We realized that programmers vary in their need for simplicity and in their aversion to magic. And we also learned that there is a huge variety in CLI designs. Performance was called out. The ability for you to trim your app became essential. The modernization of dotnet new required a dynamic approach. Help was not adequate for the .NET CLI. API Review raised significant issues. Usability studies went badly. The ability to leverage source generation with the current pipeline was called into question.

These issues are valid and handling them was more than we could handle alone, and happily we have not needed to. @AdamSitnik massively improved perf and @JonSequitur allowed trimming (Jon has also done the majority of the design and coding). @Vlada-Shubina made important changes while implementing System.CommandLine for dotnet new including updates to help. @MarcPopMSFT, @SfOslund, and @Baronfel updated the .NET CLI to use System.CommandLine making additional improvements along the way.

Moving forward, we have begun a new partnership with the .NET Libraries team. You'll see involvement by @AdamSitnik, @Jozkee, @JeffHandley, and @Terrajobst.

Upcoming changes

The work that remains to be done is important and large. We had hoped to GA within the .NET 7 time-frame but GA will be delayed.

The .NET Libraries team partnership brings some amazing new folks to the team and allows us to address problems that we could not previously. The first step will be assessing the current design and implementation. As we complete that, we'll share designs and planning.

We regret this delay, but it puts us on track for a better, stronger, smaller, faster, simpler .NET command line parser!

What does this mean for you?

We do not yet know what changes will impact you, but I'll share my best current guesses.

These things will stay the same

We do not expect changes to parsing behavior. We will work hard not to break your users who we know may use your CLI via scripts. We have extensive tests to protect parser behavior and will communicate in release notes if there are any changes in parsing behavior.

We expect to continue to use System.CommandLine in important internal efforts including the .NET CLI, ML.NET and many internal tools.

We expect SetHandler to remain available, probably as a separate package as simplifying approaches are moved to Roslyn source generation. We will also work with the F# community to ensure we remain friendly to DSLs where source generation is not available.

We remain committed to open source and working in the open.

These things will change

If you use System.CommandLine as a parser today, we anticipate that you will see future changes to package dependencies, namespaces and some naming. There is likely to be changes to supporting aspects, especially middleware pipeline customization and localization where we have an opportunity to improve the design.

Once we release as part of the System namespace, we will consider the API locked and will work to avoid further breaking changes. Since we will live with that API, we need to take the time now to get it right.

We need your help

If you are customizing the middleware pipeline we need to hear from you!. We plan to replace the pipeline with a faster and simpler approach. We are making guesses about how people are using the pipeline and we may miss your scenario. By customizing the pipeline, we mean - if you are adding middleware via one of the AddMiddleware methods, replacing any of the default Use... methods, or removing anything from the pipeline. If you are doing these things please tell us in an issue or discussion. Hearing from you is critical to not breaking you.

If there is an issue in the System.CommandLine repo that you care about please ping it, perhaps with an updated perspective.

If something about System.CommandLine annoys you, let us know. If it's specific, create an issue, and if you think it's too vague for an issue, start a discussion.

If you are a CLI parser author or might consider using a System parser as a foundation for a tool, please reach out. We want to support System.CommandLine being the foundation for other parsers, particularly to provide consistent behavior for complex POSIX issues.

If you forked System.CommandLine, we would really like to know why. Maybe we can make changes so you do not need to maintain your fork.

If you picked a different parser, we would love to know why, obviously.

Closing

We come back to our mission: Create a great and consistent command line experience for end users while requiring as little effort as possible from CLI authors using the library. The partnership with the .NET Libraries team let us maintain and enhance our experience for end users, while simplifying and improving the experience of CLI authors.

[KalleOlaviNiemitalo]

Are future versions going to require .NET SDK and .NET Runtime, or will they still support older Visual Studio tooling and .NET Framework? For example, if you are planning to make C# source generators mandatory, then I have to start migrating out.

[jonsequitur]

We plan to continue to support netstandard2.0. Source generator support will be at another layer.

[KathleenDollard]

@KalleOlaviNiemitalo Thanks for the feedback. It confirms our expectation that we need to ensure things keep working for those scenarios - and particularly that the non-generator scenario needs to work well.

[KalleOlaviNiemitalo]

Once we release as part of the System namespace,

Are you going to name the types something like System.CommandLineOption, rather than System.CommandLine.Option? I hope it won't be just System.Option.

[jonsequitur]

A name change is likely here as the single-word type name is more prone to collide with type names in other namespaces.

[KathleenDollard]

@KalleOlaviNiemitalo

We are likely to retain the namespace System.CommandLine. "Release as part of the System namespace" referred to the parent namespace and the high bar we have for anything under that umbrella.

Also, we will probably disambiguate the name due to API Review feedback. Especially Option has other meanings in both .NET and the broader tech space. I currently favor CliOption, CliArgument and CliCommand, etc.

[JobaDiniz]

@KathleenDollard One aspect that touches middleware and global options that I find hard: is the verbosity option. Usually you want to provide -v as a global option for any command. And you would wire up this option in a middleware, but how to access the -v value from there?

internal class RpaCommand : RootCommand
    {
        //static to be accessed from the middleware
        internal static readonly Option<Verbosity> VerbosityOption = CreateVerbosityOption();

        public RpaCommand() : base("Provides features to manage RPA through the command line")
        {
            AddGlobalOption(VerbosityOption);

Middleware extension method

internal static CommandLineBuilder RegisterLoggerFactory(this CommandLineBuilder builder)
        {
            return builder.AddMiddleware(async (context, next) =>
            {
               //get the value for Verbosity
                var verbosity = context.ParseResult.GetValueForOption(RpaCommand.VerbosityOption);
               //set the logging based on the verbosity
                var loggerFactory = LoggerFactory.Create(builder =>
                {
                    builder.AddOpenTelemetry(o => o
                        .SetIncludeFormattedMessage(true) //TODO: for some unknown reason, the 'Message' of the LogRecord is empty and not showing in the Jaeger UI. (https://github.com/open-telemetry/opentelemetry-dotnet/discussions/3696)
                        .ConfigureResource(r => r.AddService(RpaCommand.ServiceName, serviceVersion: RpaCommand.AssemblyVersion))
                        .AttachLogsToActivityEvent())
                    .SetMinimumLevel(verbosity.ToLogLevel());
                });
                context.BindingContext.AddService(s => loggerFactory);

                await next(context);
            }, MiddlewareOrder.Configuration);
        }

---

Another aspect the bothers me is the how cumbersome is to tell parser to parse custom value object classes/structs: The Verbosity struct for instance, look

private static Option<Verbosity> CreateVerbosityOption()
        {
            var verbosity = new Option<Verbosity>("--verbosity", ParseVerbosity, isDefault: true, description: "Specifies how much output is sent to the console.")
               .FromAmong(Verbosity.Quiet, Verbosity.Minimal, Verbosity.Normal, Verbosity.Detailed, Verbosity.Diagnostic);
            verbosity.Arity = ArgumentArity.ZeroOrOne;
            verbosity.AddAlias("-v");
            return verbosity;

            static Verbosity ParseVerbosity(ArgumentResult result)
            {
                if (result.Tokens.Count == 1)
                    return new Verbosity(result.Tokens[0].Value);

                if ((result.Parent as OptionResult)?.Token?.Value == "-v")
                    return Verbosity.Diagnostic;

                return Verbosity.Normal;
            }
        }

If I need to reuse this parser, I would need to expose the ParseVerbosity somewhere. Maybe if I had a place to wire up parsers based on type it would be good.

---

What about when you want to tell the user the next command he might be interested to use? That is, you want to print in the console something like: use 'cli env add [alias]' to add more environments. How to accomplish such a thing without hardcoding the command names? I need to create a public static string CommandName = ".." in every command and create the string myself in the code - not ideal.

Console.Writeline($"Use '{RpaCommand.CommandName} {EnvironmentCommand.CommandName} {EnvironmentCommand.AddCommand.CommandName} [alias]' to configure more environments.");

---

How to get the command that was issued by the user? For example, I'm tracking command usage using the Activity:

 internal static CommandLineBuilder TrackUsage(this CommandLineBuilder builder)
        {
            return builder.AddMiddleware(async (context, next) =>
            {
               //there is another previous Middleware that injects the ActivitySource...
                var activitySource = context.BindingContext.GetRequiredService<ActivitySource>();
                using var activity = activitySource.StartActivity(context.ParseResult.CommandResult.GetIssuedCommand());
                foreach (var symbolResult in context.ParseResult.CommandResult.Children)
                    activity?.AddTag(symbolResult.Symbol.Name, string.Join(' ', symbolResult.Tokens));

                try
                {
                    await next(context);
                }
                catch (Exception ex)
                {
                    activity?.RecordException(ex);
                    activity?.SetStatus(ActivityStatusCode.Error, ex.Message);
                    throw;
                }
            }, MiddlewareOrder.Default);
        }

        internal static string GetIssuedCommand(this CommandResult result)
        {
            ArgumentNullException.ThrowIfNull(result, nameof(result));

            var stack = new Stack<string>();
            var command = result.Command;
            var commandResult = result;
            do
            {
                stack.Push(command.Name);
                commandResult = commandResult?.Parent as CommandResult;
                command = commandResult?.Command;
            } while (command != null);

            return string.Join(' ', stack);
        }

See the GetIssuedCommand method? It was my way to get the command. Using Environment.CommandLineArguments is not ideal because I'm not interested in all arguments, just the command names (not options or arguments).

[jonsequitur]

See the GetIssuedCommand method? It was my way to get the command.

@JobaDiniz You should be able to get this information using ParseResult.CommandResult.Command.Name. To see the actual tokens that were specified, try ParseResult.Command.CommandResult.Tokens.

[JobaDiniz]

ParseResult.CommandResult.Command.Name

This is just 1 command. Usually, I have sub commands, like my-cli env add --param 1. "env" is a command, and "add" is subcommand of "env". I want to track env add and not only add, because add can also be a subcommand of another command.

[KalleOlaviNiemitalo]

Will there be a 2.0.0-beta5 release with the changes from the API review https://github.com/dotnet/command-line-api/issues/1891 so that people can try it out and comment before you freeze the API?

Daily builds were requested in https://github.com/dotnet/command-line-api/issues/1809 but don't seem to have been implemented.

[KalleOlaviNiemitalo]

@JobaDiniz, you could also use string.Join(" ", context.ParseResult.Tokens.Where(token => token.Type == TokenType.Command)), but if your commands have aliases, then this will return the user-specified names rather than the canonical names.

[tmds]

@jonsequitur @KathleenDollard is the plan still to do a 2.0 GA based on the current API on short term with some specific issues still fixed (which?)?

[lonix1]

As mentioned in #1828:

"System.CommandLine" is not just a collection of utility methods, it has become a huge library with MANY! pages of documentation, and seems to be useful to many devs both inside and outside Microsoft.

Please consider giving it a name. We could refer to the "web stuff" in .NET, but we don't because it's so massive and deserved a name, viz. "ASP.NET", similarly, "EF" rather than "the dotnet ORM", etc.

Naming it would give it more prominence, attract more users, more plugin libraries, and also be something people could put on their resumes: instead of "I have skills in the dotnet web stuff", I can say "I have skills in ASP.NET".

[jonsequitur]

@tmds We're putting off GA until after we've settled #1891 and any additional issues that come out of that work. I'll add the issues within #1891 (since it's a meta-issue) into the 2.0 GA milestone so it more accurately reflects our progress, which will be faster now that we have more people working on it.

@KalleOlaviNiemitalo We're planning the next preview release to be a roll-up of quite a few changes and we want to get as many of the breaking changes as we can into as few releases as possible. Your feedback is absolutely critical. We won't freeze the API until people have had time to try it out and tell us how we're doing.

[jonsequitur]

Please consider giving it a name. We could refer to the "web stuff" in .NET, but we don't because it's so massive and deserved a name, viz. "ASP.NET", similarly, "EF" rather than "the dotnet ORM", etc.

@lonix1 I love this idea.

[eajhnsn1]

@KathleenDollard, similar to @JobaDiniz above, I'm using middleware to look for a --debug option to configure a minimum logging level the application should output. This is using InvocationContext.ParseResult.GetValueForOption().

[KalleOlaviNiemitalo]

Do you intend to merge this repository to https://github.com/dotnet/runtime/? I'd prefer keeping this separate, so that I can more easily manage GitHub notifications and grep the sources.

[KathleenDollard]

@lonix1 Do you have any thoughts on what kind of a name?

[lonix1]

@KathleenDollard @jonsequitur No idea, but any would do. You guys have been doing the hard work for all these years, so I think you have the right to choose something cool. If you leave it to corporate you'll end up with another bland name like "EF" :-).

[sashagit]

The Spectre project has great features and feels more convenient to use.

[KalleOlaviNiemitalo]

For me, the primary attraction of this library has been the shell completion integration. The line-wrapping help is also somewhat nice but not very much better than a fixed-80-column string resource, and doesn't yet work right with fullwidth characters (https://github.com/dotnet/command-line-api/issues/1973). Response files are not useful for my applications. Subcommands would be easy enough to implement without this library.

My interest in custom directives stems from the possibility of implementing better shell completion integration (CompletionItem, long-running process for faster responses) that way before it is supported by the library itself.

I keep being frustrated by how this parses invalid command lines. Particularly --option=argument, where the option was not designed to take an argument (https://github.com/dotnet/command-line-api/issues/2010), is parsed the same as --option argument against existing practice in getopt_long and argp_parse. Likewise it is impossible to define a long option with an optional argument and use that unambiguously as it grabs the next token even if not connected with an equals sign. And there seems to be no intention to change this, instead I was told that even --option=@path would start being expanded as a response file reference.

Recent changes focus on performance, .NET SDK needs, and .NET BCL style rules rather than fixing those issues.

I'm now thinking about emigrating my applications from System.CommandLine so that I can have traditional parsing of long options and a single-file executable on .NET Framework. I would still like to keep them dotnet-suggest compatible though. To that end, I'm very interested in how you are going to change the application registration and the [suggest] directive protocol. AFAICT those should be implementable in unmanaged applications as well, rather than depend on .NET.

[jonsequitur]

Particularly --option=argument, where the option was not designed to take an argument (https://github.com/dotnet/command-line-api/issues/2010), is parsed the same as --option argument against existing practice in getopt_long and argp_parse.

This looks like a bug and we should fix it.

Likewise it is impossible to define a long option with an optional argument and use that unambiguously as it grabs the next token even if not connected with an equals sign.

Was there an issue opened for this one? It looks like this would present some ambiguity in parsing when the next token collides with an identifier, requiring the user to know how to escape it.

And there seems to be no intention to change this, instead I was told that even --option=@path would start being expanded as a response file reference.

This isn't a fixed design decision. The relevant discussion (#1625) isn't settled and your suggestion to keep = as an effective escape mechanism for response files is reasonable. It also happens to be the current behavior, though I believe it was unintentional.

[KalleOlaviNiemitalo]

Was there an issue opened for this one?

Now filed as https://github.com/dotnet/command-line-api/issues/2072.

[IanKemp]

There's a whole bunch of unfinished stuff in the GA release that looks very much like navel-gazing. Please remember that perfect is the enemy of good enough.

[KalleOlaviNiemitalo]

Huh, apparently there has been a release to nuget.org after 2.0.0-beta4.22272.1: dotnet-suggest 1.1.415701 was built from commit 259d24fa2ed4d9114bc560b68c1bb319f4d6dd39. The corresponding version of the System.CommandLine package was however not released.

The previous dotnet-suggest 1.1.327201 was from commit 209b724a3c843253d3071e8348c353b297b0b8b5, which matches tag 2.0.0-beta4.22272.1.

[iBicha]

Is there an ETA on when GA (or any version really) will be out? The last published version is from over a year ago. I would advocate GA should take longer, and have more incremental versions coming up, as opposed to have the releases come to a halt. Thank you!

[CEbbinghaus]

It would be great if more package versions could be released to nuget. The last published version was early last year and a lot has changed since. Having nightly / beta versions for people to test/use is going to gain more immediate feedback. I know there is a private feed that I dug out because I wanted a new feature but I see no reason why these cannot be published to nuget with a -nightly tag