Embedded quotes / json data as argument issue with examples

[slee-accesso]

Using C# .NET 8 on Windows 11 in PowerShell (5.1) I have this simple console program:

using System.CommandLine;
namespace DemoIssue;
internal static class Program
{
    private static async Task<int> Main(string[] args)
    {
        var rootCommand = new RootCommand {Description = "This application is to demonstrate a parsing issue "};
        var postDataOption = new Option<string>("-data", "PostData");
        rootCommand.AddOption(postDataOption);
        rootCommand.SetHandler(postData => { Console.WriteLine($"PostData: {postData}"); }, postDataOption);
        return await rootCommand.InvokeAsync(args);
    }
}

By varying the placement of spaces in the json string, I can make it fail or succeed. If any embedded spaces are present, then at least one colon outside embedded quotes must have a space next to it (before or after). I also tried experiments with multiple colons. As long as one outside embedded quotes had a space, the rest seemed ok.

>.\DemoIssue.exe -data '{\"myKey\":\"myValue\"}'
PostData: {"myKey":"myValue"}
>.\DemoIssue.exe -data '{\"myKey\": \"myValue\"}'
PostData: {"myKey": "myValue"}
>.\DemoIssue.exe -data '{\"myKey\": \"my Value\"}'
PostData: {"myKey": "my Value"}
>.\DemoIssue.exe -data '{\"my Key\": \"my Value\"}'
PostData: {"my Key": "my Value"}

>.\DemoIssue.exe [parse] -data '{\"my Key\": \"my Value\"}'
[ DemoIssue [ -data <{"my Key": "my Value"}> ] ]

These all fail:

>.\DemoIssue.exe -data '{\"myKey\":\"my Value\"}'
Unrecognized command or argument 'Value"}'.
>.\DemoIssue.exe -data '{\"my Key\":\"myValue\"}'
Unrecognized command or argument 'Key":"myValue"}'.
>.\DemoIssue.exe -data '{\"my Key\":\"my Value\"}'
Unrecognized command or argument 'Key":"my'.
Unrecognized command or argument 'Value"}'.

>.\DemoIssue.exe [parse] -data '{\"my Key\":\"my Value\"}'
![ DemoIssue [ -data <{"my> ] ]   ???--> Key":"my Value"}
>.\DemoIssue.exe [parse] -data '{\"myKey\":\"my Value\"}'
![ DemoIssue [ -data <{"myKey":"my> ] ]   ???--> Value"}
>.\DemoIssue.exe [parse] -data '{\"my Key\":\"myValue\"}'
![ DemoIssue [ -data <{"my> ] ]   ???--> Key":"myValue"}

Experiments with multiple colons. These work:

>.\DemoIssue.exe -data '{\"myKey\":\"my Value\",\"myKey2\": \"my:Value2\"}'
>.\DemoIssue.exe -data '{\"myKey\" :\"my Value\",\"myKey2\":\"my:Value2\"}'

These fail:

>.\DemoIssue.exe -data '{\"myKey\":\"my Value\",\"myKey2\":\"my:Value2\"}'
>.\DemoIssue.exe -data '{\"myKey\":\"my Value\",\"myKey2\":\"my :Value2\"}'
>.\DemoIssue.exe [parse] -data '{\"myKey\":\"my Value\",\"myKey2\":\"my :Value2\"}'
![ DemoIssue [ -data <{"myKey":"my> ] ]   ???--> Value","myKey2":"my :Value2"}

Is this going to be a case of "Me: Hey doc - it hurts when I do this ... " "Doc: Then stop doing that!"?

[elgonzo]

You should have checked the contents of the args array you pass to the library. Because, if the content in the args array is not "right", then there is nothing System.CommandLine can do about it.

Basically, in your situation it's the typical Powershell command-line argument quoting f**kery, mkay...

For example, the command-line:

.\DemoIssue.exe -data '{\"myKey\":  \"myValue\"}'

will result in PS using the following effective command-line to execute the app: .\DemoIssue.exe -data "{\"myKey\": \"myValue\"}". Note the double-quotes surrounding the json argument. Due to the surrounding double-quotes, the command-line will be correctly parsed into an args array with two arg strings: -data and {"myKey": "myValue"}.

However, with for example the command-line:

.\DemoIssue.exe -data '{\"myKey\":\"my Value\"}'

Powershell in its incomprehensible wisdom decides now not to use surrounding double-quotes, with the effective command-line now being .\DemoIssue.exe -data {\"myKey\":\"my Value\"}. The args array will therefore now have three arg strings: -data, {"myKey":"my and Value"} Kaboom. It's the power of Powershell, baby.

As you might have noticed, i am not a big fan of Powershell. I have given up long ago on PS precisely due to its byzantine command-line escaping/quoting/transformation rules, which work in mysterious ways and imho are far worse then ye'olde cmd.exe. Hence i am unfortunately also not able to advice on how to correctly escape/quote the command-line arguments in PS, because PS makes me angry. Although, you might try using double double-quotes "" instead of escaped double-quotes \" and see what it gets you... ;-)

(Disclaimer: I am not associated with the System.CommandLine project nor its authors/maintainers.)

[slee-accesso]

Well, @elgonzo - if nothing else, you are hilarious : ) However I'm pretty sure System.CommandLine is not parsing correctly but PowerShell is. To test this, I made a PowerShell function that takes two string arguments so I could verify the parsing. I made two parameters to prove it took the first parameter (the json) as a single string, and still could see the second parameter. All of the test cases above parsed properly both as named and positional parameters. So - PowerShell can pass the string to PowerShell just fine - therefore, I think it is System.CommandLine's parser.

PowerShell test function:

function Reflect-Text ([Parameter()][String] $stringyString, [Parameter()][String] $secondString) {
    Write-Host "Reflection stringyString: $stringyString";
    Write-Host "Reflection secondString: $secondString";
}

Two sample calls/results - the first fails with System.CommandLine, the second passes but both work fine in PowerShell:

> Reflect-Text '{\"myKey\":\"my Value\",\"myKey2\":\"my:Value2\"}' 'Hello!'
Reflection stringyString: {\"myKey\":\"my Value\",\"myKey2\":\"my:Value2\"}
Reflection secondString: Hello!

> Reflect-Text '{\"myKey\":\"my Value\",\"myKey2\": \"my:Value2\"}' 'Hello again...'
Reflection stringyString: {\"myKey\":\"my Value\",\"myKey2\": \"my:Value2\"}
Reflection secondString: Hello again...

[slee-accesso]

Although - @elgonzo - you are correct that the args array is not what I expect. I added output of the args array. Now I get:

> .\DemoIssue.exe -data '{\"myKey\":\"my Value\",\"myKey2\":\"my:Value2\"}'
Arg 1: -data
Arg 2: {"myKey":"my
Arg 3: Value","myKey2":"my:Value2"}
Unrecognized command or argument 'Value","myKey2":"my:Value2"}'.

So - does this remove System.PowerShell from the equation? It is so weird that one space adjacent to any colon seems to be the key. Here are some larger examples:

Succeeds with space after first colon:

> .\DemoIssue.exe -data '{\"Op_Code\": \"ADMIN \",\"ContactId\":2000000,\"OperatorId\":1},{\"Op_Code\":\"ANNE  \",\"ContactId\":35000000,\"OperatorId\":8},{\"Op_Code\":\"AXESS \",\"ContactId\":39000000,\"OperatorId\":12},{\"Op_Code\":\"BOB   \",\"ContactId\":21000000,\"OperatorId\":4},{\"Op_Code\":\"EXPREV\",\"ContactId\":38000000,\"OperatorId\":11}'
Arg 1: -data
Arg 2: {"Op_Code": "ADMIN ","ContactId":2000000,"OperatorId":1},{"Op_Code":"ANNE  ","ContactId":35000000,"OperatorId":8},{"Op_Code":"AXESS ","ContactId":39000000,"OperatorId":12},{"Op_Code":"BOB   ","ContactId":21000000,"OperatorId":4},{"Op_Code":"EXPREV","ContactId":38000000,"OperatorId":11}
PostData: {"Op_Code": "ADMIN ","ContactId":2000000,"OperatorId":1},{"Op_Code":"ANNE  ","ContactId":35000000,"OperatorId":8},{"Op_Code":"AXESS ","ContactId":39000000,"OperatorId":12},{"Op_Code":"BOB   ","ContactId":21000000,"OperatorId":4},{"Op_Code":"EXPREV","ContactId":38000000,"OperatorId":11}

Fails after removal:

> .\DemoIssue.exe -data '{\"Op_Code\":\"ADMIN \",\"ContactId\":2000000,\"OperatorId\":1},{\"Op_Code\":\"ANNE  \",\"ContactId\":35000000,\"OperatorId\":8},{\"Op_Code\":\"AXESS \",\"ContactId\":39000000,\"OperatorId\":12},{\"Op_Code\":\"BOB   \",\"ContactId\":21000000,\"OperatorId\":4},{\"Op_Code\":\"EXPREV\",\"ContactId\":38000000,\"OperatorId\":11}'
Arg 1: -data
Arg 2: {"Op_Code":"ADMIN
Arg 3: ","ContactId":2000000,"OperatorId":1},{"Op_Code":"ANNE
Arg 4: ","ContactId":35000000,"OperatorId":8},{"Op_Code":"AXESS
Arg 5: ","ContactId":39000000,"OperatorId":12},{"Op_Code":"BOB
Arg 6: ","ContactId":21000000,"OperatorId":4},{"Op_Code":"EXPREV","ContactId":38000000,"OperatorId":11}
Unrecognized command or argument '","ContactId":2000000,"OperatorId":1},{"Op_Code":"ANNE'.
Unrecognized command or argument '","ContactId":35000000,"OperatorId":8},{"Op_Code":"AXESS'.
Unrecognized command or argument '","ContactId":39000000,"OperatorId":12},{"Op_Code":"BOB'.
Unrecognized command or argument '","ContactId":21000000,"OperatorId":4},{"Op_Code":"EXPREV","ContactId":38000000,"OperatorId":11}'.

[elgonzo]

I don't know what you are getting at. Your application is not a PS function, and unlike a PS function it's not executed within the PS environment. Your test with the PS test function would only meaningful to your problem if you were to turn your program also into a PS function instead of it being an executable.

For executable, PS assembles a command-line and passes it to the process of the exeutable launched. It's not a matter of belief, everything i said in my previous post is testable and verifiable. Just look in Task Manager what kind of command-line PS assembles when executing your program. And look at the args array passed to the Main method of your program, instead of chasing false equivalencies with PS functions... [EDIT: Saw your last comment, you got it... ;-)]

It is so weird that one space adjacent to any colon seems to be the key.

It's Powershell... i wasn't joking when i said i don't like Powershell. It's behavior is sometimes really mindboggling. For what it's worth, i don't believe it's the colon, but to me it rather seems (i guess) Powershell falsely assuming the escaped double-quotes being argument delimiters and thus only using surrounding double-quotes when it detects a white-space outside of a pair of double-quotes while completely ignoring that the double-quotes are escaped/prefixed with a backslash. That's my speculation of what PS is doing, but as with speculations, it might not turn out to be true...

[slee-accesso]

They seem to have fixed the parsing sometime, apparently. PowerShell 7.4.3 does not screw it up. Thanks for steering me right, @elgonzo .

[slee-accesso]

Closing - as this isn't System.CommandLine : )

[elgonzo]

They seem to have fixed the parsing sometime, apparently. PowerShell 7.4.3 does not screw it up.

Oh, they did? Well, guess i have try it out, it seems ;-)