Modernize cli scripts in Arcs

[lindner]

Arcs has a number of node shell scripts. Each one is bespoke and adding a new script or options is not well defined. Here are some issues with the current state

• Scripts are using node require instead of modules
• Lots of manual processing of argv
• No help output
• Difficult to discover which commands are available.
• Inconsistent help output.
• Scripts may or may not work on windows.

Here is a non-exhaustive list of scripts in the repo:

• tools/sigh.js
• dataflow/cli/flowcheck.js
• tools/spotify-importer.js
• various node deploy.sh/serve.sh shell scripts

A desirable end state would satisfy the following requirements

• A single arcs command that you can easily execute.
• Git style subcommands that extend the core.
• A consistent method of writing and linking cli code into a common whole.
• An Arcs developer can quickly and easily add a new command line option or sub command without much effort using consistent idioms (es6 modules, typescript, etc.)
• A particle/sdk developer can quickly download the arcs environment, including the command line tool and have easy to understand help options and ability to execute.
• The tool works consistently across platforms.

I evaluated available options and found that there are three main tools that are in active use in the Node ecosystem:

• yargs 29m weekly,
• commander 23m weekly, no deps
• minimist 25m weekly, abandoned.
• Write our own.

The following factors were weighed:

• The sigh.js tool in Arcs does use minimist for command line flag parsing only.
• All modules are available in internal repos.
• minimist has not been updated in 4 years.
• yargs has more feature but requires a number of transitive dependencies

To that end I propose that we adopt commander as our default CLI solution for running code in node environments.

A prototype exists here:

https://github.com/lindner/arcs/commit/714a11a15b8a1e96c3bc61350d788b642da4176f

After build and npm link you will have an arcs command in your path. There is a main arcs executable and each subcommand is installed as arcs-subcommand

Running arcs --help shows

Usage: arcs [options] [command]

Arcs System

Options:
  -V, --version  output the version number
  -h, --help     output usage information

Commands:
  init           Initialize Arcs
  status         Show the arcs status
  export         Export Repo Contents
  ingest         Ingest Data into Arcs
  flowcheck      Analyze and Diagnose Dataflow
  manifest       Handle Manifest files
  help [cmd]     display help for [cmd]

Here's an example of src/cli/arcs-manifest.ts

#!/usr/bin/env -S node --require esm --require source-map-support/register
// ...
commander.command('parse <filename>')
    .description('Parse a Manifest')
    .action(async (filename) => {
         // ....
    })
    .parse(process.argv);

[lindner]

Would appreciate comments on this proposal. Assigning to owner of infra @raulverag .

[shaper]

Thanks for exploring this! It sounds interesting and I can see it being useful during exploration of data import and making it easier for folks to learn about how to install, update, and work with the Arcs system. I defer to @raulverag but wanted to briefly note some initial thoughts:

• I like the DSL-like way in which the commands are defined, it would make it much more pleasant to add new scripts or extend existing ones.
• Probably obvious but I suspect we may not want to move all of the scripts under an overarching arcs prefix. Some of these tools are quite disparate -- early steps in what will eventually probably be build-time-only dataflow analysis, a utility to test parsing a manifest, a single demo's one-off utility script to hack data import, and so forth.
• It's less clear to me how this fits with sigh specifically and our build system in general, using an overarching arcs command to kick off the equivalent of a make or build step feels strange.
• The unix pattern of --help and individual scripts feels poignant to move away from vs having to read through a hierarchy of command info with arcs help command which I assume this would lead to, but maybe I'm dating myself.
• Having to run npm link is a bummer, we've had discussion on limiting use of/dependence on npm or even moving away from using it in time.

[lindner]

@shaper thanks for the feedback.

I think we can address your concerns:

• sigh remains as-is. We just convert the internal structure to es6 and use commander.js
• As for npm link -- we can still create shell script wrappers in common places as need be. It's just a convenience of putting arcs in the global node path.
• Or, since the js files have a shebang and can be executable you can just run them in the build directory on Mac/Linux.
• One-off scripts and things like that could be included in an arcs-dev command which can be run as arcs devor arcs-dev This is what Perkeep uses.

Also note that I took some inspiration from the js-ipfs CLI. It's well thought out and works nicely.

[lindner]

Also of note is that if we implement a CLI system we can drop usage of npm start

[shans]

This is an interesting direction to think about going in; I don't see a strong value proposition in doing it now. Specifically, this doesn't seem to contribute to our current goals. While I agree with @shaper that this would make an exploration of data import easier, I can't see how that contributes towards the current goal of a data import MMVP.

It isn't clear that doing this would actually have much impact on onboarding users in other ways; particularly, it sounds like sigh would remain basically the same as it currently is from a usage perspective.

When we do have spare capacity to move on something like this (or if you decide you'd like to do it as a 20%-style thing) I think you will need to ensure that we don't at any stage support multiple different command-line arg parsers, to make sure we don't take a step backwards in this regard.

Some specific questions/observations:

• are you going to retire the rest of the npm commands at the same time? Re-regularizing our build processes would be an onboarding improvement.
• I agree with Walter that our dataflow analysis tool shouldn't be moved to an arcs rooted command
• why do we need a utility for parsing a manifest?
• why should deployment / serving commands be somewhere separate from the build commands in sigh?

[raulverag]

Another reason I think we should hold off on this is that we will relatively soon need to start distinguishing clearly between tools and docs for people like ourselves who are working on the Arcs system, and tools and docs for developers of Particles and Recipes. Piotr even mentioned an Arcs Developer Kit, which I think makes a lot of sense. We will need this for onboarding 3P developers, and it will be important to get it right early, as a clunky early ADK can be a disaster for adoption. So while I agree that our current set of tools is a bit messy, it doesn't cause very much friction, so I would suggest we clean it up as part of a broader design that includes the tools to be included in an ADK.

Also, am I the owner of infra? I don't mind, but I don't particularly recall signing up for this.

[lindner]

Okay, will defer this for now. (And @raulverag with your sigh and build expertise it seemed that you were owning infra, sorry!)

That said, I did recently need to fix the spotify importer so I converted it to a command line tool using some (but not all) of the above concepts:

https://github.com/PolymerLabs/arcs/pull/2870

One nice part is that it doesn't require sigh to set up the environment (at least on Unix/Mac)

I do hope that this exploration is useful and that we learn something from the effort.