Dropping Vorpal and Migrating to Shargs

[AndreMaz]

@Yord yesterday I've started to play with shargs-repl. I've managed to implement some basic commands (e.g., cls, exit, etc.) but I need some guidance on the call command. If possible, I want args to have the same structure as vorpal's args

Here's how my implementation looks like:

const subCommandOpt = subcommand([
    stringPos('actionName', { desc: "Action name (e.g., greeter.hello)", descArg: 'actionName', required: true} ),
    stringPos('jsonParams', { desc: `JSON Parameters (e.g. '{"a": 5}' )`, descArg: 'jsonParams'} ),
    stringPos('meta', { desc: "Metadata to pass to the service action. Must start with '#' (e.g., --#auth 123)", descArg: 'meta'} ),
    string("load", ["--load"], { desc: "Load params from file.", descArg: 'filename' }),
    string("stream", ["--stream"], { desc: "Save response to file.", descArg: 'filename' }),
    string("save", ["--save"], { desc: "Save response to file.", descArg: 'filename' }),
]);

and this is vopral implementation:

vorpal
    .removeIfExist("call")
    .command("call <actionName> [jsonParams] [meta]", "Call an action")
    .autocomplete({
        data() {
            return _.uniq(_.compact(broker.registry.getActionList({}).map(item => item && item.action ? item.action.name: null)));
        }
    })
    .option("--load [filename]", "Load params from file")
    .option("--stream [filename]", "Send a file as stream")
    .option("--save [filename]", "Save response to file")
    .allowUnknownOptions()
    .action((args, done) => call(broker, args, done));

and below are some examples of how vorpal and shargs parse the input data:

---

1. Call with --load option.

Command

call greeter.data --load ./package.json --save p.json

vorpal args

{
  options: { load: './package.json', save: 'p.json' },
  actionName: 'greeter.data',
  rawCommand: 'call greeter.data --load ./package.json --save p.json'
}

shargs args

{
  _: [],
  actionName: 'greeter.data',
  load: './package.json',
  save: 'p.json'
}

---

2. call with JSON string

Command

call greeter.data '{"a": 5, "b": "Bob", "c": true, "d": false, "e": { "f": "hello" } }'

vorpal args

{
  options: {},
  actionName: 'greeter.data',
  jsonParams: '{"a": 5, "b": "Bob", "c": true, "d": false, "e": { "f": "hello" } }',
  rawCommand: `call greeter.data '{"a": 5, "b": "Bob", "c": true, "d": false, "e": { "f": "hello" } }'`
}

shargs args

{
  _: [],
  actionName: 'greeter.data',
  jsonParams: '{"a": 5, "b": "Bob", "c": true, "d": false, "e": { "f": "hello" } }'
}

---

3. Call with random params

Command

call greeter.data --a 5 --b Bob --c --no-d --e.f "hello"

vorpal args

{
  options: { a: 5, b: 'Bob', c: true, d: false, e: { f: 'hello' } },
  actionName: 'greeter.data',
  rawCommand: 'call greeter.data --a 5 --b Bob --c --no-d --e.f "hello"'
}

shargs args

{
  _: [ '--b', 'Bob', '--c', '--no-d', '--e.f', 'hello' ],
  actionName: 'greeter.data',
  jsonParams: '--a',
  meta: '5'
}

With a little work 1. and 2. can be transformed into the desired shape but the 3. is completely wrong. What's is the correct way of parsing unknown params? Vorpal has a function called .allowUnknownOptions() that parses unknown options that start with -- (e.g. --something 1) into

{
  options: { something: 1 },
  actionName: 'greeter.data',
  rawCommand: 'call greeter.data --something 1'
}

How can I achieve the same behavior with shargs?

[Yord]

@AndreMaz Off the top of my head:

• The options group can be realized using a custom field.
• The rawCommand can be realized by wrapping the parser(Sync) function.
• Interpreting unknown options is added to the parser by the bestGuessOpts or bestGuessArgs stages.

I will be able to look into it in detail later today and add some code that aligns things!

[AndreMaz]

Sweet :smile: Thanks for your help. You can clone my repo and run node dev/new-repl.js to test things.

In dev/new-repl.js you can switch between vorpal repl and shargs.

const broker = new ServiceBroker({
    replLocation: '../../../index-shargs',
    // replLocation: '../../../index-vorpal',
    logLevel: 'debug'
})

Vorpal's commands are located in commands/vorpal and their shargs equivalents are in commands/shargs

Once the call command is implemented it will be easy to port the remaining ones.

[Yord]

Nice, got it running without problems ;).

I am working on it now and will send another update in a few hours!

[Yord]

Ok, so I am halfway there:

---

1. Call with --load option.

Command

call greeter.data --load ./package.json --save p.json

vorpal args

{
  options: { load: './package.json', save: 'p.json' },
  actionName: 'greeter.data',
  rawCommand: 'call greeter.data --load ./package.json --save p.json'
}

shargs args

{
  _: [],
  actionName: 'greeter.data',
  rawCommand: 'call greeter.data --load ./package.json --save p.json',
  options: { _: [], load: './package.json', save: 'p.json' }
}

---

2. call with JSON string

Command

call greeter.data '{"a": 5, "b": "Bob", "c": true, "d": false, "e": { "f": "hello" } }'

vorpal args

{
  options: {},
  actionName: 'greeter.data',
  jsonParams: '{"a": 5, "b": "Bob", "c": true, "d": false, "e": { "f": "hello" } }',
  rawCommand: `call greeter.data '{"a": 5, "b": "Bob", "c": true, "d": false, "e": { "f": "hello" } }'`
}

shargs args

{
  _: [],
  actionName: 'greeter.data',
  jsonParams: '{"a": 5, "b": "Bob", "c": true, "d": false, "e": { "f": "hello" } }',
  rawCommand: `call greeter.data '{"a": 5, "b": "Bob", "c": true, "d": false, "e": { "f": "hello" } }'`,
  options: { _: [] }
}

---

I have had no time yet to work out 3, but I have an idea how to make it work.

You can find the changes in two commits in my fork:

• 3bb73b8
• 8760d2a

I have some more time tomorrow evening to work on it ;).

[AndreMaz]

Awesome :+1: you rock

[Yord]

Ok, I was able to get all three examples working:

---

1. Call with --load option.

Command

call greeter.data --load ./package.json --save p.json

vorpal args

{
  options: { load: './package.json', save: 'p.json' },
  actionName: 'greeter.data',
  rawCommand: 'call greeter.data --load ./package.json --save p.json'
}

shargs args

{
  actionName: 'greeter.data',
  rawCommand: 'call greeter.data --load ./package.json --save p.json',
  options: { load: './package.json', save: 'p.json' }
}

---

2. call with JSON string

Command

call greeter.data '{"a": 5, "b": "Bob", "c": true, "d": false, "e": { "f": "hello" } }'

vorpal args

{
  options: {},
  actionName: 'greeter.data',
  jsonParams: '{"a": 5, "b": "Bob", "c": true, "d": false, "e": { "f": "hello" } }',
  rawCommand: `call greeter.data '{"a": 5, "b": "Bob", "c": true, "d": false, "e": { "f": "hello" } }'`
}

shargs args

{
  actionName: 'greeter.data',
  jsonParams: '{"a": 5, "b": "Bob", "c": true, "d": false, "e": { "f": "hello" } }',
  rawCommand: `call greeter.data '{"a": 5, "b": "Bob", "c": true, "d": false, "e": { "f": "hello" } }'`,
  options: {}
}

---

3. Call with random params

Command

call greeter.data --a 5 --b Bob --c --no-d --e.f "hello"

vorpal args

{
  options: { a: 5, b: 'Bob', c: true, d: false, e: { f: 'hello' } },
  actionName: 'greeter.data',
  rawCommand: 'call greeter.data --a 5 --b Bob --c --no-d --e.f "hello"'
}

shargs args

{
  actionName: 'greeter.data',
  rawCommand: 'call greeter.data --a 5 --b Bob --c --no-d --e.f "hello"',
  options: { a: '5', b: 'Bob', c: true, d: false, e: { f: 'hello' } }
}

---

I have added three new commits to my fork:

• be43f0b:\ This commit adds "bestGuess" functionality to enable custom options. I have reused code that I have written (and tested!) for bestGuessOpts previously. So I think it is pretty stable. However, it would benefit from some tests, later. When the time comes, I can support here, just message me ;).
• d8b567f:\ This commit does two things: First, it adds the flagsAsBools stage to the parser that transforms all flags to booleans. Second, it adds the nestValue args stage to the parser that transforms keys of the form {"a.b": 42} to {"a": {"b": 42}} using Lodash's _.set function (code taken from the "Can I use keys like 'a.b' FAQ").
• 9ed9d14:\ This commit removes the rest keys _: [] from shargs args results.

For the examples, vorpal and shargs should now yield the same results. Of course, there may be edge cases that are not yet tried. Also, the shargs code should definitely be tested with unit tests at some point. I would be willing to support here, when the time comes ;).

[AndreMaz]

Thank you for your work 👍 I'm going to try to port the remaining commands just to see if everything works. But I agree, some unit tests will be necessary to iron out any bugs.

[icebob]

@AndreMaz @Yord please test the arrays as well:

mol $ call greeter.hello --a 5 --a 6
{
  options: { a: [ 5, 6 ] },
  actionName: 'greeter.hello',
  rawCommand: 'call greeter.hello --a 5 --a 6'
}

[AndreMaz]

@icebob nice catch. At this moment shargs can't handle this case

mol~$ call greeter.hello --a 5 --a 6

shargs

{
  actionName: 'greeter.hello',
  rawCommand: 'call greeter.hello --a 5 --a 6',
  options: { a: '5' }
}

@Yord regarding autocomplete, is there a way of adding a custom function to do autocomplete over the required param? To give you and idea, this is how vorpal autocomplete looks:

.autocomplete({
    data() {
                // Returns an array of strings
        return _.uniq(_.compact(broker.registry.getActionList({}).map(item => item && item.action ? item.action.name: null)));
    }
})

First Tab does command autocomplete and the second Tab does actionName autocomplete. actionName is the required param.

and this is shargs autocomplete

[Yord]

@AndreMaz @Yord please test the arrays as well:

mol $ call greeter.hello --a 5 --a 6
{
  options: { a: [ 5, 6 ] },
  actionName: 'greeter.hello',
  rawCommand: 'call greeter.hello --a 5 --a 6'
}

@icebob That one is an easy fix, I have added arrayOnRepeat from shargs-parser in the following commit to make it work:

• ee942cc

One follow-up question, though: Adding arrayOnRepeat makes this behavior (repeated options are arrays) the default. Is that the intended semantics?

[Yord]

@AndreMaz I have an idea of how an autocomplete could be added without much overhead. One question, though:

Should the list returned by autocomplete be the full list of options?

Aka no other than the values in the autocomplete list are valid.

[AndreMaz]

One follow-up question, though: Adding arrayOnRepeat makes this behavior (repeated options are arrays) the default. Is that the intended semantics?

Yes, it solves the issue.

Should the list returned by autocomplete be the full list of options? Aka no other than the values in the autocomplete list are valid.

I think that showing only the values in the autocomplete list should work just fine. It avoids writing the complete actionName every time that we want to call a service action. If the user wants to find out the remaining options he will just ask for help call

Another question, currently when a command is called without the required param it shows the following info:

Can we do the same thing with shargs?

[Yord]

I think that showing only the values in the autocomplete list should work just fine. It avoids writing the complete actionName every time that we want to call a service action. If the user wants to find out the remaining options he will just ask for help call

I connected the broker action list in commit 83d0bb4. The actions are now included in the autocompletion list:

I have identified a minor bug in shargs-repl in the process that prevents completing positional arguments. I am in the process of fixing it. If it is fixed, a partially written action name should be completed like all other options as well.

Currently, the default completer behavior is to show all positional arguments and option args, in the same order as they are defined in the subcommand. Overriding the completer is not possible in the current version of shargs-repl, but I will provide the possibility to inject a new function in the next version (that is a good idea anyway).

[Yord]

Another question, currently when a command is called without the required param it shows the following info:

Can we do the same thing with shargs?

First regarding required fields. I have added the requiredOpts parser stage in 84d5ad9 and now errors are thrown if a required field is missing:

mol~$ call
{ rawCommand: 'call', options: {} }
RequiredOptionMissing: An option that is marked as required has not been provided.

Second regarding the help message. This is possible. However, I realize now I have messed up the action function signature:

action: (args) => call(broker, args)

action: (args, errs) => call(broker, args, errs)

The first signature is currently used, the second signature (including errors) would be much better. If we had the second signature this would be straight forward. I will fix it in shargs-repl and publish a new version tomorrow.

[Yord]

I have updated the action signature in shargs-repl version 0.3.0 and made some commits to my fork:

• 1ea1f97:\ I have bumped shargs-repl's version to 0.3.0.
• fe5f8b1:\ I have added error messages and usage help to the call command.
• d2bf877:\ I have adjusted the error message for missing required arguments to match Vorpal's.

[icebob]

@Yord awesome job! By the way, how do you make these gifs?

[Yord]

@icebob Uh that's terminalizer. I love it, it's super awesome! ;)

[AndreMaz]

Awesome work @Yord One question, can actions be async? Vorpal action signature has a done callback

.action((args, done) => call(broker, args, done));

to handle this.

And another question, can shargs-repl do autocomplete? Right now shargs shows the list of available options but doesn't do the actual autocomplete.

To see what I mean, run vorpal repl and type ca and then press Tab. It will add ll (and create call). Then if you type g and press Tab. It will add reeter. (and create call greeter.). If you press Tab again it will show the list of actions that start with greeter.: greeter.data greeter.hello greeter.welcome)

[Yord]

And another question, can shargs-repl do autocomplete? Right now shargs shows the list of available options but doesn't do the actual autocomplete.

To see what I mean, run vorpal repl and type ca and then press Tab. It will add ll (and create call). Then if you type g and press Tab. It will add reeter. (and create call greeter.). If you press Tab again it will show the list of actions that start with greeter.: greeter.data greeter.hello greeter.welcome)

Yes, shargs-repl can autocomplete. However, it is currently broken when positional arguments are involved... but I am working on it. It is a small fix.

You can see it in action if you type cal and then tab. Here no positional arguments are involved and it still works.

[Yord]

One question, can actions be async? Vorpal action signature has a done callback

Currently actions are synchronous. However, there is no good reason for them not to be asynchronous. I will have a look.

[AndreMaz]

Yes, shargs-repl can autocomplete. However, it is currently broken when positional arguments are involved... but I am working on it. It is a small fix.

Awesome 👍

Currently actions are synchronous. However, there is no good reason for them not to be asynchronous. I will have a look.

Thanks, this is the last thing that we need to implement call command

[icebob]

Moreover, would be awesome, if it can be intelligent like in other libs. So if done parameter is not in the function signature but it returns a Promise, it handles the promise. :)

[AndreMaz]

@Yord I took a quick look at shargs-repl source code:

  return (cmd, context, filename, callback) => {
    const { errs, args } = parse(cmd)

    Object.entries(args).forEach(([key, value]) => {
      const cmd = commands.opts.find(_ => _.args.includes(key)) || { action: _ => undefined }
      const action = cmd.action || (_ => undefined)

      action(value, errs)
    })

    callback(null, undefined);
  }

I'm assuming that action(value, errs) calls the shargs command, right? If so, then shouldn't this

Promise.resolve(action(value, errs))
        .then(res => callback(null, undefined))
        .catch(err => callback(null, err))

do the trick?

[Yord]

I think we need to throw some Promise.all in the mix, like:

Promise.all(
    Object.entries(args).map(([key, value]) => {
        const cmd = commands.opts.find(_ => _.args.includes(key)) || { action: _ => undefined }
        const action = cmd.action || (_ => undefined)

        return action(value, errs)
    })
)
.then(() => callback(null, undefined))
.catch(err => callback(null, err))

PS: Sorry for being a little bit unresponsive. Life has thrown things at me that I need to resolve :). But I will be back in business early next week. And in the meantime I have some minutes to respond of course ;).

[icebob]

@Yord no problem, take your time! And thanks for all your work on this project 👍

[Yord]

Ok guys, I am back and reporting progress on asynchronous actions.

I have done the following things:

1. I have updated shargs-repl to 0.4.0:\ In this release I have introduced a new asynchronous execution mode alongside synchronous execution.
2. I have updated my molecular-repl branch as follows:
   1. f9976eb:\ molecular-repl now uses shargs-repl 0.4.0, still with synchronous actions.
   2. 7c2df7a:\ The synchronous replSync is replaced by an asynchronous repl. The call action is still synchronous to demonstrate that the asynchronous repl supports value-returning, as well as Promise-returning functions.
   3. 84de1a3:\ I replace the synchronous call action with a Promise-based asynchronous action. The action contains a setTimeout of 1 second, to demonstrate asynchronicity.
   4. 779127e:\ I remove setTimeout from the code.

I suggest you checkout 84de1a3 (the commit with setTimeout) to test async actions.

[Yord]

Next up is including positional arguments in autocomplete. :)

[AndreMaz]

@Yord great work :+1: Thank you for your help. Async works very nice

Two more questions:

1. Isn't the bestGuess in (variadicPos('customOptions', {bestGuess: true}),) supposed to cast the params? Currently all params are strings.

   mol~$ call greeter.data --a 1 --b true -c a
   {
   actionName: 'greeter.data',
   rawCommand: 'call greeter.data --a 1 --b true -c a',
   options: { a: '1', b: 'true', c: 'a' }
   }

2. I've noticed that help (not the call --help) command isn't working. Do you know what's happening?

[Yord]

1. Isn't the bestGuess in (variadicPos('customOptions', {bestGuess: true}),) supposed to cast the params? Currently all params are strings.

Actually it was supposed to only output strings. But casting those strings is straight forward:

• 063e364:\ I have added the bestGuessCast args stage to the parser. And strings that could be cast to numbers or booleans are converted now.

[Yord]

2. I've noticed that help (not the call --help) command isn't working. Do you know what's happening?

The issue was a change in API that was not applied to help:

The action is now responsible for printing the results. In the case of help, it was still returning a string. I have fixed the issue in f965228.

I assume the same goes for the other commands: Instead of returning a string, said string must be printed.

Edited to add: The other commands are fine. They already print their results instead of returning them!

[AndreMaz]

@icebob can you do a quick test of new REPL? You can install it with npm i -s https://github.com/AndreMaz/moleculer-repl#shargs

I think that the only thing that's missing is the autocomplete for positional arguments.

[Yord]

I think that the only thing that's missing is the autocomplete for positional arguments.

I will be working on this today btw. and will report on my progress soon.

[icebob]

@AndreMaz great. I will check it next days. @Yord thanks in advance!

[Yord]

Ok, so the autocomplete should now work as expected. I have changed the following things:

• shargs-parser 0.6.1:\ I have changed restrictToOnly's behavior: It does not remove incomplete values and only reports errors.
• shargs-repl 0.5.0:\ This release brings autocomplete for positional arguments.
• shargs-repl 0.6.0:\ This release fixes a small bug with command escapes (--).
• fafc0d0:\ This new commit to my molecular-repl fork bumps shargs-parser, shargs-repl, and shargs dependency versions.

[AndreMaz]

@Yord tested over here and autocomplete is working nice :+1:

[icebob]

I've checked and I found these issues:

1. = between key & value is not working:

2. Another that if I write a wrong command it says nothing.

3. Using shorthand meta keys not working (But I think this issue is in moleculer-repl and not in shargs) In this case b: "Bob" should be in ctx.meta. Example

4. dcall is not working:

5. Strange message if I use not existing action or event name

[Yord]

@icebob nice findings :). I look at the issues one by one:

1. 3706b53:\ I have imported the equalsSignAsSpace stage and added it to the shargs parser. Options are now split at an equals sign.

That means, however, that no '=' can be used in option values any longer (like --name "E=mc2"). The reason is that shargs does not restrict options to start with '--'. If this restriction is, well, too strict, I can write a more fitting stage.

[Yord]

2. I have fixed this by adding a default action to shargs-repl that is called if no command is found:

• shargs-repl 0.6.1 and shargs-repl 0.6.2:\ Add the default action field to repl options.
• 8124dbf, d3b6cc1, 633fd6b, 7e180ed, and fce8add:
  • Bump shargs-repl to 0.6.2.
  • Add a default action that prints an error message if a command was not found: Your command was not found!

• a7b6a1d:\ I thought it would be nice to use shargs' suggestOpts stage to suggest existing commands for mistyped commands:

  mol~$ carl greeter.welcome --name=Icebob
  Your command was not found! Did you mean:
  call, clear, cls

  But if you don't need suggestions, it is easy to remove ;).

[Yord]

5. I have improved the error message in b037258:

mol~$ call some.thing
  The 'actionName' field cannot be 'some.thing'. Choose one of the following:
  $node.list, $node.services, $node.actions, $node.events, $node.health, $node.options, $node.metrics, greeter.hello, greeter.welcome, greeter.data

  Usage: call [options] (<actionName>) [<jsonParams>] [<meta>]
  ...

mol~$ emit "user.created"

  The 'eventName' field cannot be 'user.created'. Choose one of the following:
  hello.event, *

  Usage: emit [options] (<eventName>)
  ...

[icebob]

Great, thanks!

[AndreMaz]

Thanks for the quick fixes @Yord :wink:

3. Using shorthand meta keys not working (But I think this issue is in moleculer-repl and not in shargs)

I'll have a look at it.

4. dcall is not working:

Ups, I forgot to implement this one.

5. Strange message if I use not existing action or event name

@icebob vorpal's autocomplete acts as a suggestion, i.e., you are still able to make a call some.thing even if the same.thing is not a valid action. Shags, on the other hand, is strict. It only allows to call actions that are present in the broker.registry. Which approach do you prefer? The strict one (shargs) or the "relaxed" one (vorpal)?

[icebob]

5. Good question. I can imagine situation when you call something which does not exist. E.g. in case of using inter-namespace middleware. In this case, you call greeter.hello@ns-venus action which doesn't exist, but the middleware will change it.

[AndreMaz]

Thanks for the input :+1: @Yord is it possible to make autocomplete more "relaxed"?

[Yord]

@AndreMaz yes, that is possible. I suggest just dropping the error message. This way we get the "pros" (autocomplete) without the "cons" (error message if an invalid value is used).

[Yord]

Note: I have relaxed autocomplete in two different ways. First I chose a path that turned out to be suboptimal, later. But at that point, it was already pushed to my fork, so I decided to not rewrite history, but leave it be:

1. e98990a:\ I factored out a rephraseErrMessages stage from adjustErrMessages.
2. 889aed5:\ I stopped rephrasing the ValueRestrictionsViolated error that occurs if a value that is not in only is used.
3. 12ad83e:\ I introduced a filterErrMessages step that removes errors and put ValueRestrictionsViolated to its blacklist.

At this point, I rerolled all my changes and went another, much more straight forward way:

1. b72ac16:\ I removed restrictToOnly from the shargs parser

Edited to add: This change retains the autocompletion functionality, but does not report errors if values are used that are not specified in only.

[Yord]

I have added more functionality and fixed some bugs in shargs-repl. The fixes should not affect molecular-repl, but I would just in case bump the shargs-repl version to 0.7.0.

• shargs-repl 0.6.3:\ Added autocompletion for options with only fields.
• shargs-repl 0.6.4:\ Report an error if a command is not found.
• shargs-repl 0.7.0:\ Fix a case where reporting an error if a command was missing did not work if the rest field _ was removed from args.
• 03a36e4:\ Bump shargs-repl version to 0.7.0.

[AndreMaz]

Thanks :+1: I'm working on moleculer-repl right now and so far didn't encounter any errors. But yeah, I'm definitely going to update

[icebob]

@AndreMaz I've checked again the new repl.

1. The prefix is mol$~$ It's by design? In vorpal it's mol $
2. Executing actions, I've got this:

Could you check it?

[Yord]

@icebob I can help with the first issue:

A custom prompt can be set in repl's options as follows:

const options = {prompt: 'mol $ ', ...}

repl(lexer, parser, command, options)

[AndreMaz]

@icebob thanks for testing and @Yord thanks for the fix. The actions command should be working now.

@Yord there's an issue with the autocomplete. It doesn't like white spaces and gives wrong suggestions