Restructure result conventions (again)

[shadowspawn]

A new attempt to improve the results (#12 #22 #23 #38 #55) prompted by proposed changes to input configuration (#63), and with better understanding of some of original intent (https://github.com/pkgjs/parseargs/issues/38#issuecomment-1017327974).

For context, the configuration syntax proposed in #63 is:

parseArgs({
  args: process.argv.slice(2),
  options: {
    foo: {
      short: 'f',
      type: 'string', // defaults to 'boolean'
      multiple: true,
      // default: 'bar' - Future iteration
    }
  },
})

Result Properties

I propose result has properties for:

• optionFound: object with properties and true value for all parsed options found in args.
  • e.g. optionFound: { debug: true, name: true, mult: true }
• values: object with properties and values for the parsed options
  • e.g. values: { debug: true, name: 'John', mult: ['a', 'b'] }
• positionals: array of positionals from args
  • e.g. positionals: ['alpha']

This is a rename of flags to optionFound. The intent of flags name was the options without values, but the proposed meaning is all options found in args as per original proposal.

vs status quo: storing all the options found in optionFound matches the current implementation, but not the current README examples which are only showing true for boolean options. See also following section on values.

The optionFound field is initially redundant and can be calculated from values. However, it offers a semantic affordance now, and is not redundant if we add support for defaults (or setting known boolean flags to false, say) . Both Command and Yargs did not start out with a concept of whether the option was found in the input args, and that led to requests later as not possible to tell from a value whether was an explicit default (say for a string) or an implicit default (say for a boolean) or for from the parsed arguments.

Values

I propose values property holds for an option:

• string for option with a value in args (normal case for option with type:'string')
• true for option used as boolean flag (normal case for option with type:'boolean')
• array if multiples:true, with true and/or string elements

The change is storing true as the value for a found boolean option. I think this is the natural value, and is what other parsing libraries do such as Command and Yargs.

vs status quo: the current README examples are omitting boolean options entirely. The current implementation is storing undefined rather than true.

[bcoe]

This design seems quite reasonable to me, with a nit that I don't love the variable name optionFound. To match the other values returned, seems like a picking something plural would makes sense, some suggestions:

• optionsSet.
• optionsProvided.
• passedOptions

[aaronccasanova]

+1 for passedOptions

Agreed, this design seems reasonable to me as well. However, while were discussing the results object, I'd like to challenge the upcoming relevance of the name values. If #63 lands and withValues is removed, is values still the best name to convey it is the parsed args based on the options config? I quite like the intuitiveness of the meow api where the flags configuration is also the name of the parsed flags result.

What are folks thoughts around this convention?

const { options } = parseArgs({
  options: {
    foo: {
      type: 'string'
    },
  },
})

[shadowspawn]

I like passedOptions too on its own and not too bad beside options:

const { options, positionals, passedOptions } = parseArgs({ strict: false });

A wild challenge would be to drop passedOptions for now. A quick sanity check, is anyone actually wanting to use passedOptions in code or examples? I'm not looking for a compelling use case, I'm just checking if there is any interest in the extra result, or everyone just ignoring it because it does not affect how you envisage library being used!

[shadowspawn]

One complication with using options as property in both directions is it requires extra code to make this work:

const { options, positionals, passedOptions } = parseArgs({ args, options });

(I don't see any issues against Meow though, so perhaps does not come up much in practice.)

[ljharb]

I've definitely run into that kind of issue before, but i think "options" is a pretty confusing name to pass as a property here.

[shadowspawn]

Nit. I have gone off passedOptions as a result property name since we pass "options" into the configuration. In fact, the unit tests use passedOptions as the variable name! https://github.com/pkgjs/parseargs/blob/b4120957d90e809ee8b607b06e747d3e6a6b213e/test/index.js#L31

I wasn't enthusiastic about my original suggestion of optionFound either. My current plan is I'll have a go at refactor with foundOptions which at least has plural on end.

[aaronccasanova]

One complication with using options as property in both directions is it requires extra code to make this work

Can you point out the extra code? I'm struggling to see complications in the example.

I have gone off passedOptions as a result property name since we pass "options" into the configuration.

passedOptions still makes sense and reads well to me with that framing. Authors pass options into parseArgs and retrieved the (parsed) passedOptions on the other side.

In fact, the unit tests use passedOptions as the variable name!

I'm not sure this should influence API naming conventions. We could simply rename the test vars from passedOptions to optionConfigs

[shadowspawn]

Can you point out the extra code? I'm struggling to see complications in the example.

I wasn't very clear, or accurate... What I was thinking was if there was already a variable called options in play then need to rename when destructuring, like from:

const { options, positionals, passedOptions } = parseArgs({ args, options });

to

const { options: resultOptions, positionals, passedOptions } = parseArgs({ args, options });

[shadowspawn]

passedOptions still makes sense and reads well to me with that framing. Authors pass options into parseArgs and retrieved the (parsed) passedOptions on the other side.

1) Morally I feel the foundOptions are determined by the input args, and not the input options.

Although might be a good time to check expectations! In the future, if we added defaults to the input options, I was assuming a default would not cause the option to appear in foundOptions but would cause the option to appear in values.

2) A different name I thought of and you hint at is parsedOptions. How does that scan?

[ljharb]

If we ever add defaults, it absolutely must be possible to determine if a defaulted value was explicitly passed or not; to do this in yargs I have to default things to, eg, Object('foo'), and then see if the value is the string foo or the cached Object foo, which is a horrific workaround.

"parsed" is better than "passed", because "passed" implies it's unchanged from what the user passed - but it is changed, because it's parsed.

[shadowspawn]

For interest, Commander only added support for determining source of option value in v8.3.0 in late 2021 after about a decade without a mechanism. e.g. 'default' vs 'CLI'

• https://github.com/tj/commander.js/blob/master/CHANGELOG.md#830-2021-10-22

[bcoe]

@shadowspawn @ljharb what if rather than an object that contains all objects that were explicitly set, we have the opposite defaulted which contains all the options that were set to their default value?

I think this could serve a similar purpose we're looking to solve, but to me makes the return values less confusing. We could then perhaps run with:

const { values, positionals, defaulted } = parseArgs({ args, options });

I could also be pushed towards:

const { options, positionals, defaulted } = parseArgs({ args, options });

But I don't love the overloaded term options.

[aaronccasanova]

How do we feel about just having one object for parsed options containing the value and usage metadata? I think all the discussion points in issue #43 and PR #63 for input options apply to the outputs e.g. More natural to have all the information in one place and lays the foundation for future iterations:

// node parsedOptions.js --foo bar

const optionConfigs = {
  foo: { type: 'string' },
  baz: { type: 'string', default: 'hello' },
}
const { options, positionals } = parseArgs({ options: optionConfigs })
/*
options === {
  foo: {
    value: 'bar',
    defaulted: false,
    parsed: true,
  },
  baz: {
    value: 'hello',
    defaulted: true,
    parsed: false,
  },
}
*/

[ljharb]

I like https://github.com/pkgjs/parseargs/issues/70#issuecomment-1069278041, assuming positionals was an array of objects matching the schema of the options values.

(altho when would parsed !== !defaulted?)

[aaronccasanova]

assuming positionals was an array of objects matching the schema of the options values.

I could definitely see this being helpful. I think it addresses the same concerns and establishes a pattern for future iterations.

(altho when would parsed !== !defaulted?)

Good point 👍 I didn't fully reason about that and was focused on covering the edge cases discussed so far (e.g. identifying when options are passed and/or defaulted). So we can probably nix the parsed key.

[aaronccasanova]

Here's a scrappy example of the ParsedOptions type with some examples of future iterations: ​

type ParsedOptions = {
    [option: string]: {
      /** Value corresponding to the `optionConfig.type`; Set to `optionConfig.default` if option was not passed */
      value: string | boolean
      /** True if the parsed option's value used the `optionConfig.default` */
      defaulted: boolean;
​
      // Future iterations...      
​
      /**
       * Could be helpful to understand how the parser inferred `strict:false` options: e.g.
       * node inferred.js --foo // parsedOptions === { foo: { type: 'boolean', value: true }}
       */
      type: 'string' | 'boolean'
      /** Number of times the option was defined */
      count: number
    }
}

As an example of how this pattern can be leveraged, a parsedOption.count field could address @ljharb's callout in https://github.com/pkgjs/parseargs/pull/80#discussion_r825355444 about optionConfig.type === 'boolean' and optionConfig.multiple === true having a somewhat obscure result (e.g. an array of true values [true, true, etc.]). Instead the result could simply be:

// node verbose.js -vvv
const { options } = parseArgs()
/*
options === {
  v: {
    type: 'boolean', // Helpful to see what the parser inferred
    value: true, // Returns a single option for type:'boolean'
    count: 3, // Metadata includes the frequency of the option
  }
}
*/

[shadowspawn]

@shadowspawn @ljharb what if rather than an object that contains all objects that were explicitly set, we have the opposite defaulted which contains all the options that were set to their default value?

Actually, I would rather delete it. (Which I'll suggest in next comment!) The sense of "defaulted" is referencing a feature we don't even have yet.

[shadowspawn]

Short version: I propose we drop optionFound. (The property previously known as args and flags, and also suggested as passedOptions, parsedOptions, and most recently inverted defaulted...)

Long version

I already speculated about dropping it earlier in this issue, and got no defenders (https://github.com/pkgjs/parseargs/issues/70#issuecomment-1053697997). I think it is time to get serious about deleting it! I have been asking what it is for since Nov last year and got no compelling answers, and we are still trying to give it a sensible name, with no compelling use case. 😱

Why have the field at all?

Suggested: differentiate between the case where an option was passed no argument, vs., an option not being set at all (https://github.com/pkgjs/parseargs/issues/13#issuecomment-968344407) Counter: we can tell the difference without it. An option that appears in the input args will have an entry in values, an option not set at all will not appear in values.

Suggested: simplify testing for existence of option in args (https://github.com/pkgjs/parseargs/issues/24#issuecomment-1013945423). e.g. const fooExists = !! optionFound.foo Counter: with values as described here: const fooExists = values.foo !== undefined (or != null if you prefer!) Slightly longer to be fair, but you are going to be working with values anyway so not a new concept. Other packages use single property for the parsed options (https://github.com/pkgjs/parseargs/issues/12#issuecomment-968677456).

Suggested: tell where value came from in future (e.g. if add defaults) Counter: do what makes sense in future

[shadowspawn]

How do we feel about just having one object for parsed options containing the value and usage metadata?

My initial reaction is no, I think combining the values and usage metadata is overkill and complicating the primary target audience. i.e.

Thus, process.parseArgs() is not necessarily intended for library authors; it is intended for developers of simple CLI tools, ad-hoc scripts, deployed Node.js applications, and learning materials.

In the future, if there is demand for this as a base for building richer parsers and custom behaviour, I think there could be a metadata containing property. (Which could perhaps include the value as well so was self-contained.)

[aaronccasanova]

I think combining the values and usage metadata is overkill

This feels similar to the initial reactions when I proposed the restructured option configs API. I'm trying to imagine what parseArgs will look like one year from now and make sure we're considering what API will support new features and enhancements. From what I can see, we are going down the same path as the original option configs API where information for a single option is scattered across the primary config object.

Here is an example of what I'm trying to avoid after say one year of features and enhancements: Please don't take my example features as literal suggestions, but rather to illustrate general enhancements to parseArgs.

// Where we're headed...
const {
  foundOptions: { foo: true },
  values: { foo: true },
  defaulted: { foo: false },
  count: { foo: 2 },
  types: { foo: 'boolean' },
  etc...
} = parseArgs()

// Proposal...
const {
  options: {
    foo: {
      type: 'boolean',
      value: true,
      defaulted: false,
      count: 2,
      etc...
    },
  },
} = parseArgs()

As far as the proposal being complicated for the target audience, I'm not sure I follow. I believe my suggestion improves the robustness of the API and is actually more intuitive for the target audience.

[ljharb]

Certainly if we have no "defaults" at all, then it's easy to see when an option isn't provided.

However, it seems pretty important for almost every CLI workflow i know to have defaults, and for some, it's critical to be able to know whether they're defaulted or not.

[shadowspawn]

Here is an example of what I'm trying to avoid after say one year of features and enhancements

Fair concern. I don't see organic growth of user-level properties as a likely outcome. Other libraries have had a decade of features and enhancements, and have a single layer of object values available. See https://github.com/pkgjs/parseargs/issues/12#issuecomment-968677456

If we add more information for people building richer parsers, I do prefer the idea of putting that richer metadata into one property and not scattering across multiple top level properties.

Other metadata mentioned in https://github.com/pkgjs/parseargs/issues/52#issuecomment-1030740881 https://github.com/pkgjs/parseargs/issues/52#issuecomment-1030828425 #84

[shadowspawn]

(More metadata in: #84)

I am thinking a property like parseDetails which has data of use for people building on top of the library, but is usually ignored by direct users of the library. As with Aarons example, these properties are not suggestions just illustrations.

e.g.

const {
  values: { foo: true },
  positionals: ['index.js'],
  parseDetails: {
    options: {
      foo: { argIndex: 1, argIndices: [1], choosyArgumentIgnored: true, inconsistentUsage: true },
    },
    optionDelimiter: { argIndex: 2 },
    positionals: [
       { argIndex: 3 }
    ]
    }
  }
} = parseArgs({ strict: false })

[aaronccasanova]

Other libraries have had a decade of features and enhancements, and have a single layer of object values available.

100%, Totally valid argument. However, I still (personally) feel the combination of option value and metadata is the more intuitive/future-facing API. The closest example I can think of is RegExp match objects.

Notice: Each match object contains the full match, number and named capture group values, the start index, indices of capture groups, etc.

Now imagine if instead String.prototype.matchAll() returned the matches in one object and all the metadata in a separate one. Wouldn't be as intuitive imo.

I may be reaching a bit with the above demo, but am attempting to provide a real-world example of a parsed input that includes values and metadata in the same result to demonstrate the intuitiveness of the API.

[bakkot]

@aaronccasanova:

I think the overwhelmingly common case is going to be just getting the value without any metadata, possibly with destructuring, as in

let { values } = parseArgs(opts);
let { foo, bar = defaultBar } = values;

And that gets much more annoying if metadata is mixed with values. I think it's important to optimize for the common case.

Note that even in the matchAll example, there are separate groups and indices properties, and the groups property is an object which directly associates values with named groups; the metadata for named groups, i.e. the indices, are in a separate .indices.groups field. We had the destructuring example in mind when designing named capture groups for regexes, IIRC.

[shadowspawn]

🔨 Moving towards MVP Milestone 1 (#87)

See #83 !

[shadowspawn]

(Exploring interest, not pushing for a change!)

I do like options as the result property with flags gone:

const { options, positionals } = parseArgs({ strict: false });

I am not keen on using same property on input and output despite the symmetry (https://github.com/pkgjs/parseargs/issues/70#issuecomment-1053625924), and feel there is additional confusion on the input usage with common usage of "options" for an option bag.

Perhaps the input property could be renamed? Inspired in part by strict:false usage:

const { options, positionals } = parseArgs({ 
   strict: false, 
   knownOptions: { host: { type: 'string' }}
});

[aaronccasanova]

Perhaps the input property could be renamed?

No objections from me. I was thinking the same thing after the initial concerns around using options for both the input and output. I'm a fan of knownOptions, optionConfigs, optionsConfig, etc. Kind of liking optionsConfig 🤓 (naming convention reminds me of the package.json publishConfig key).

const { options, positionals } = parseArgs({ 
   strict: false, 
   optionsConfig: { host: { type: 'string' }}
});

[shadowspawn]

Landed #83. Closing this, the culmination for five months of issues. 😅