search

arcanis / clipanion

Type-safe CLI library / framework with no runtime dependencies
https://mael.dev/clipanion/
1.1k stars 61 forks source link

Implements tokenization #146

Closed arcanis closed 1 year ago

arcanis commented 1 year ago

This PR makes Clipanion expose a tokens field to each command instance. This field contains a list of all tokens that Clipanion identified in the original command line. I plan to use this feature on the Yarn documentation, to automatically display the options' descriptions when hovering them inside the examples.

I removed the suggestion engine since we never really used it and it's on its way out with @paul-soporan's work in #89.

paul-soporan commented 1 year ago

What impact does this have on performance, since tokenization will run on all command invocations?

arcanis commented 1 year ago

What impact does this have on performance, since tokenization will run on all command invocations?

I didn't check; a little hard to say, generally there are a couple of places that perform work in a non-optimized ways (like by traversing an array with includes), and this PR addresses some of them, so I'd expect perfs to be better here but not because of the tokenization (which I expect to be a little slower, but negligible all things considered).

I considered to make the tokenizer an option of the CLI object, but it didn't feel a good approach to treat it as an entirely different "build target", with a different state machine. Can be reconsidered if it becomes a problem 🤔