Colorize the generated help [feature request]

[kaushalmodi]

Hello,

I have developed an internal application that has about a dozen subcommands. So running foo help generates a wall of help for all the subcommands.

I have opened this issue to discuss what would be the best way to colorize the output so that each subcommand help is easy to read.

Would it make sense to colorize the $command and $options differently in the usage string?

Or could you add a user-facing hook that can be used to colorize different parts of the usage differently?

[c-blake]

Well, if you want to colorize the various "sections" of the help wholesale you can do that already with the usage template strings...the things where the defaults are defined in cligen/helpTmpl.nim.

[kaushalmodi]

hmm, I was thinking of using writeStyled and related procs from terminal.nim and wasn't sure how to put those proc calls in the usage templates.

I think what you are suggesting is to wrap hardcoded ANSI color codes around $command, etc... let me try that.

[c-blake]

Yeah. That is, of course, compile-time, much like writeStyled. In my own little end-CL-user-driven color scenarios, I switch off at run-time on an $LC_THEME environment variable. I actually run terminals with both light & dark backgrounds and there are precious few color schemes which work well on either. Sometimes just going "bold and not bold" or tossing an "inverse" in there is enough to really break things up visually. Another wrinkle is making sure things like "auto-completion inferred from help strings" still works with escape codes embedded in there.

[c-blake]

The "LC" in that originally came from my "lc" program, but it also works well because older sshd's will propagate any LC* env vars from your local shell to a remote shell. More recently, they just have a specific list of LC_FOO vars instead of taking LC_ANYTHING. :-( You can always "overload" something automagically propagated of course, and in the olden days people would do this with $TERM. (Setting some follow on text like xterm;follow-on-text' and then splitting it off in shell init files.)

[c-blake]

Oh, and if you didn't know about it, cligen/humanUt.nim has a variety of tools for run-time ANSI escape generation. And no, I do not worry about terminal portability. There are only very few oddball terminals that would ever use something different for bold/inverse/normal (\e[1m, \e[7m, and \e[0m). So, that might be another reason to stick to that subset. Italic and blinking also work well, but those are less well supported by terminals in the wild.

Anyway, I am also interested in this feature, but the "various backgrounds and terminals" issues sort of indicate a level of indirection..like meta-colors, but then some CL-user rather than CL-author way to bind them, but then use directly by the cligen engine. That's all kind of a new direction for cligen proper. I'm certainly willing to discuss it, but it may be trickier than feels worthwhile. I did try to make the multi-commands with no help, just the raw command, dump a nice summary table. But, yeah, if you ask for all the help you get a wall of text and even minor embellishments really aid visual digestion of walls of text.

[c-blake]

If we were to do something in cligen proper, the easiest thing might be to stake out an actual $CLIGEN or $CLIGEN_* namespace for users to control colorization schemes & themes.

[kaushalmodi]

I tried something real quick.. it kind of worked.. only for -h output, but not for help output.

Here's an incomplete snippet:

  const
    topLvlUse = clUseMultiPerlish &
      "\n\nURI\n  " & uri &
      "\n\nAUTHOR\n  " & nimbleData.fromNimble("author") &
      "\n\nVERSION\n  " & version
    colorUsage = "\e[31m$command\e[0m $args\n${doc}Options(opt-arg sep :|=|spc):\n$options"

  dispatchMulti(
    ["multi", usage = topLvlUse],
    [edit, usage = colorUsage, short = {"filetype": 't'}],
    [opened],
    ..

• When running foo edit -h, it shows the "edit" subcommand in red color.
• But when running foo help, it literally prints the variable name "colorUsage" instead of the edit subcommand help:

This is a multiple-dispatch command.  Top-level --help/--help-syntax
is also available.  Usage is like:
    p4x {SUBCMD} [subcommand-opts & args]
where subcommand syntaxes are as follows:

  colorUsage

  opened [optional-params] [patterns: string...]

Also, it looks like I will need to add usage = colorUsage for each subcommand?

[kaushalmodi]

Oh, and if you didn't know about it, cligen/humanUt.nim has a variety of tools for run-time ANSI escape generation.

I did not know of that.. I will see how I can use it in this case.

[c-blake]

The usage templates for the multi-command and the leaf dispatch commands use different variable sets - clUseMulti* vs clUse*. I bet that is your problem (See also https://github.com/c-blake/cligen/issues/129). Also, writeStyled hard-codes for ANSI color escapes as well.

[kaushalmodi]

I am using $command (and not subcommand) in the variable passed to the subcommand edit's usage parameter. So the foo edit -h outputs fine.

When foo help is run though, it looks like it is literally printing the passed variable colorUsage instead of evaluating it. I do not see any compile or run time errors.

[c-blake]

If you come up with something you think works well with normal/bold/inverse, I am happy to include new variants in cligen/helpTmpl.nim. I'd hold off on exact colors like red and especially not-bright or light blue. And if you want to use those new leaf templates globally you can just change the clCfg convenience global variable.

[kaushalmodi]

If you come up with something you think works well with normal/bold/inverse, I am happy to include new variants in cligen/helpTmpl.nim.

Sure thing! I am using red so it's easy to test as I try this out :)

I am also not a fan of bright colors.. just colorful enough to show distinction.

[kaushalmodi]

And if you want to use those new leaf templates globally you can just change the clCfg convenience global variable.

ClCfg type does not have "usage": http://c-blake.github.io/cligen/cligen.html#ClCfg

[c-blake]

Oh..Oops. I guess I misremembered that in my hasty reply. Sorry. I guess you have to repeat it. That swapping in of the variable name seems like it might be related to this very recent change taking non-string-literals. Are you using cligen#head or a release?

The problem I usually have is that dark blue makes a nice background color on dark|light BG terminals, and also a fine almost bold-y color on light BG terminals but an almost unreadable foreground color on black background terminals. But, of course, that is just my setup and eyeballs and some big fraction of men are red-green color blind (like a few % or something).

[kaushalmodi]

Are you using cligen#head or a release?

Release.. 0.9.42

The problem I usually have is that dark blue makes a nice background color on dark|light BG terminals, and also a fine almost bold-y color on light BG terminals but an almost unreadable foreground color on black background terminals.

Yep, I've dealt with that.. my solution is to change my blue (.Xdefaults) :)

! 4,12 - blue and bold
XTerm.VT100.color4: cornflowerblue

[kaushalmodi]

Sorry. I guess you have to repeat it.

That's alright. That's not a gating problem.. The problem is that the passed usage parameter value is not evaluated in the multi help.

[c-blake]

You should try cligen#head. I did some work ending January 19th finishing off (I thought..maybe for real) https://github.com/c-blake/cligen/issues/110 that probably at the very least alters the manifestation of your issue.

And the problem with your cornflowerblue override is that it then no longer makes a great background color for light foreground text. Anyway, the obvious ultimate answer is end-user controls.

[c-blake]

(and ideally end-user controls that are easy to propagate across ssh's).

[kaushalmodi]

You should try cligen#head.

I just did, but still the same problem is literally printing the variable name..

I will work on creating a minimum reproducible example.

[c-blake]

Ok. That will be helpful in tracking this down.

[c-blake]

Being able to use compile-time values instead of literals is a very recent feature. It may also matter if you use const string variables vs. not.

[kaushalmodi]

Here's a minimal example:

## cligen reproducible example

## edit
proc edit(patterns: seq[string]; openInEditor = false; filetype = "") =
  ## Open / Check out one or more files for editing.
  discard

## opened
proc opened(patterns: seq[string]) =
  ## List opened files.
  discard

## main
when isMainModule:
  import cligen

  const
    colorUsage = "\e[31m$command\e[0m $args\n${doc}Options(opt-arg sep :|=|spc):\n$options"

  dispatchMulti(
    ["multi", usage = clUseMultiPerlish],
    [edit, usage = colorUsage, short = {"filetype": 't'}],
    [opened]
    )

[c-blake]

Ok. I will take a look, but perhaps in a little while after I've had a cup of tea. It does work in a single-dispatch setting as in test/PassValues.nim..Just not dispatchMulti. Probably dispatchMulti parameter handling just needs some more macro/compile-time love.

[c-blake]

We should probably make a test/MultiPassValues.nim, too.

[kaushalmodi]

I will take a look, but perhaps in a little while after I've had a cup of tea.

No rush. Thanks for looking at it.

[kaushalmodi]

Beautiful fix :D

I always look for such fixes in my projects.

Thanks! I confirm that this issue is resolved.

[c-blake]

Well, I think at least the one thing you wanted to do as a const global works now. It turned out to be a pretty easy fix. The new test/PassValuesMulti.nim exercises the feature. Some other params are probably still broken but may not matter as much. Let me know if it still doesn't work for you somehow.

Note that unfortunately escape sequences like \e only work in regular Nim literals, not triple-quoted string literals (as per the manual). With text attribute escape sequences that puts more pressure for regular literals in this context. I actually kind of like that new template better. We have --help-syntax now to explain things like :/space/=/etc.

[kaushalmodi]

We have --help-syntax now to explain things like :/space/=/etc.

👍

[c-blake]

These github issue threads are almost like instant messaging for you & me. ;-)

[kaushalmodi]

I ended up with this:

I am ignoring terminal users with white background.. I know

[c-blake]

That looks pretty good to me (for a dark background, anyway. Yellow is usually not so bad on a light background). I liked the "inverse text" for that command row as a way to almost be a "bar across the screen" separating the sections. Depending on the terminal, the text attributes might "chase the newline" so if you didn't turn them off until the next line it would literally extend across the whole terminal.

Incidentally, had I not been able to figure out that easy fix, I think you could also have done something like create a cligen/ subdirectory in your project (or anywhere else ahead of my cligen/ in your nim path), done a cp cbVsn/cligen/helpTmpl.nim myPkg/cligen/, then hacked away at clUse. That is the main reason I put all those help templates in their own file and just do an include.

[c-blake]

Anyway, I am happy to include an inverse-bold (maybe italic/underscore) value in helpTmpl.nim, but I think I would shy away from specific colors until some of those thornier color-indirection/deployment to varying environments at run-time questions get ironed out.

On my own terminal I actually get colors on bold/italic/underline "for free" as a redundant color switch on top of the font rendering switches. I map bold default FG color to a bright orange on a dark background terminal and a dark orange on a white background terminal (and similarly for italic & underline but mapped to different colors). This kind of re-interpretation of font rendering attributes to colors dates way back. I believe I was doing it with italic on the Linux virtual console back in the mid-90s. That kind of color-mapping can often be a useful compromise because people can pick the mapping with the backgrounds.

Anyway, orange and yellow are kinda similar but different enough to never be confused with each other at a glance. Had the default set of 8 colors included orange but not yellow, I would surely have added in the yellow instead and then some of my experiments would have looked almost exactly like yours. :-)

[kaushalmodi]

Anyway, I am happy to include an inverse-bold (maybe italic/underscore) value in helpTmpl.nim

I am wondering if an example of this should go into the docstring of dispatchGen..

but I think I would shy away from specific colors until some of those thornier color-indirection/deployment to varying environments at run-time questions get ironed out.

Makes sense.

I map bold default FG color to a bright orange on a dark background terminal and a dark orange on a white background terminal (and similarly for italic & underline but mapped to different colors).

That's a neat idea.

My yellow is a custom color as well:

! 3,11 - yellow and bold
XTerm.VT100.color3: rgb:f3/dc/55
XTerm.VT100.color11: rgb:f3/dc/55

[c-blake]

Well, test/PassValuesMulti.nim is an example. I think it's mostly just a relaxation of literalness constraints that most would "expect to work". So, I'm not sure it needs much more elaboration. Multiple people filed issues expecting it to "just work" and now it does.

(And, yeah, all my colors are specific RGBs and so on and I even used to hack the palette on the Linux Console with \e]PnRRGGBB sequences..I even did color palette hacking on my Vic20 in 1982..Why some might well have forecast back then that "this guy might do his own color-ls/ps stuff someday...". If you don't know about lc and procs you might check out some screenshots. In like 50 lines of Nim that cligen/humanUt.nim is more general than ANSI colors - it also does the full 24-bit color that xterm/st support and all that as well as the older 6x6x6 color cube and grey scale stuff. Examples of using that are mostly in the lc configs.)

[kaushalmodi]

"this guy might do his own color-ls/ps stuff someday"

Cool! I am nowhere close to that.. There's a lot of learn in this terminal coloring area.

If you don't know about lc and procs

It's on my list to check out lc; I've already starred it. I need to look at https://github.com/c-blake/procs through.

[c-blake]

lc is actually the 3rd incarnation of my own color-ls. The first was in Python back around 2000 which was about half defined in environment variables that got evald. That was way too slow. So, I re-wrote it in C around 2006. After about 13 years of using that, I had several ideas percolating around in my head about how to change it, and more ideas appeared as I coded them up. So, the new Nim version is really the 3rd iteration "done right this time" with several pretty novel & noteworthy ideas (colors-or-not, actually). No one thing is all that "slam-dunk", but taken as a whole I think it's pretty spiffy. It's definitely spruced up my own command-line life. Can't do much without listing your files. And the Nim version is about as fast as the C ever was, and about as elegant and user-defined as the Python ever was. /sales-pitch

[c-blake]

(I should maybe say the performance gap with the old C version is about as much as the average speed-up from --gc:arc I have been seeing. So, there is indeed much hope for performance parity, but presently there is at least one pretty bad bug in arc mode that I just worked around in procs and either the same or another causes trouble in lc, too. I haven't gotten them to small enough reproductions to report yet. I kind of hope they fix these things before 1.2.)

[c-blake]

Good morning, Kaushal. First no rush on this as I probably won't code up most of it until after this weekend. So, if you give me any feedback before Monday I can probably incorporate it.

Before I punch 1.0, I thought I would revisit this feature request. What I am thinking for the CL user UI is just a config file located in the file ${CLIGEN:-${XDG_CONFIG_HOME:-$HOME/.config}/cligen}. If stat(2) says that "file" is actually a directory, look for said-path/"config".

In said config file, the CL user can say things like:

[include__LC_THEME]  # grab file $LC_THEME defining "hotN"
OptKeys: bold hot1
ValType: inverse hot2 onHot3
subCmdName: italic yellow

That $LC_THEME can then be "darkBG" or "lightBG" or whatever and propagated around by ssh/fork/etc. People with only one kind of terminal color theme can just pick one and not include. Others might even have three or four. This kind of setup works well for my lc and procs programs. That [include__VAR] syntax is something I hacked up for those programs which can have other "ini file sections"..So it kind of fit cleanly there, but for just this $CLIGEN file we could have it be written just include: $LC_THEME.

There could be a some var cgConfigDfl = "" a CL author can re-assign to something between import cligen and dispatch to provide defaults for an uncolor-sophisticated userbase, though we might advise folks to keep such defaults pretty toned down.

With such a helpAttr: Table[string,string] built from the config file, all the various places where we stitch together help output can grow little helpAttr.getOrDefault("key", "")s. These would be kind of "structural" in the same way that composing the output is like my boxed examples above.

This would all be bypassed if a variable called NO_COLOR is present in the environment. Well, we may still want to bypass only the text attribute setting but still parse the config file anyway to allow that same config framework to let CL users adjust other formatting fields of cligen.nim:ClCfg such as (hTabCols, hTabRowSep, hTabColGap) and maybe even CL syntax things like reqSep and sepChars. There could someday be a "strict" mode to turn off unique prefix matching on various sets of tokens. So, then at the top of some script someone could export CLIGEN=~/.config/cligen-strict (or something) to force using fully spelled out tokens in such contexts. Anyway, point is there are various things besides colors that might vary more at run-time by user than at compile-time by CL author.

I think with this set up a user (like me, actually) that deals with dozens of cligen commands as well as multiple terminal themes can get everything they want by just editing a file or two.

Also, I can confirm that at least the _gnu_generic autocompletion code in Zsh-5.8 is very confused by escape sequences in the help table. Put codes in the 1st option column and it sees no options..color codes later cause a bunch of fields to be highly corrupted, etc. It should be pretty easy to wedge a NO_COLOR=1 in the foo --help invocation there, though.

[kaushalmodi]

Good morning, Kaushal. First no rush on this as I probably won't code up most of it until after this weekend. So, if you give me any feedback before Monday I can probably incorporate it.

Hello. It's already Monday!

Before I punch 1.0, I thought I would revisit this feature request. What I am thinking for the CL user UI is just a config file located in the file ${CLIGEN:-${XDG_CONFIG_HOME:-$HOME/.config}/cligen}. If stat(2) says that "file" is actually a directory, look for said-path/"config".

+1

In said config file, the CL user can say things like:

[include__LC_THEME]  # grab file $LC_THEME defining "hotN"
OptKeys: bold hot1
ValType: inverse hot2 onHot3
subCmdName: italic yellow

It's probably too late to suggest this, but I'll still put it out there.. have you considered using TOML? (Hint: It's awesome!)

What are "hot1", "hot2", ..? Have you used them as place holders for "red", "green", etc. colors?

That $LC_THEME can then be "darkBG" or "lightBG" or whatever and propagated around by ssh/fork/etc. People with only one kind of terminal color theme can just pick one and not include. Others might even have three or four. This kind of setup works well for my lc and procs programs.

Works for me.

That [include__VAR] syntax is something I hacked up for those programs which can have other "ini file sections"..So it kind of fit cleanly there, but for just this $CLIGEN file we could have it be written just include: $LC_THEME.

There could be a some var cgConfigDfl = "" a CL author can re-assign to something between import cligen and dispatch to provide defaults for an uncolor-sophisticated userbase, though we might advise folks to keep such defaults pretty toned down.

What will that var hold? As it's a string, it looks like it's the path to the above cligen config file? If this variable is not set, I am assuming it will be picked up from XDG_CONFIG_HOME as you mentioned above?

Nitpick: Can you name that cgConfigDflt ("Dflt" does a better job at sounding like "default" than "Dfl", for me).

With such a helpAttr: Table[string,string] built from the config file, all the various places where we stitch together help output can grow little helpAttr.getOrDefault("key", "")s. These would be kind of "structural" in the same way that composing the output is like my boxed examples above.

Is helpAttr a Table parsed from the cligen config file? If so, should it be called cgConfig? I am a bit confused here. Also, would this be exposed to the user?

This would all be bypassed if a variable called NO_COLOR is present in the environment.

Sure. Though I won't be a good tester for that :)

Anyway, point is there are various things besides colors that might vary more at run-time by user than at compile-time by CL author.

I agree.

[c-blake]

In terms of TOML..I don't like imposing external dependencies. cligen is already one. The stdlib has parsecfg. So, that's what I use. It's admittedly a little glitchy, but one-stop shopping is nice, too.

"fhot1" is foreground heat-level 1. I do 0-5 as purple->red as a "spectrum order" kind of sequence. I find that easy to remember. I also have a fhot-+ that relates to the background being day mode or night mode. It's all defined in my lc and procs config files.

I think your other comments turned out to be about ideas that didn't wind up being how I went. I'm about to commit a big change in the next half hour. It would be helpful if you read the RELEASE_NOTES.md and let me know if you have any questions. If you use Bash not Zsh it would be helpful to figure out where the NO_COLOR=1 setting should be to advise people. I figured it out for Zsh.

[kaushalmodi]

If you use Bash not Zsh

I use (have to! Please don't judge .. work env) 🥁 tcsh

---

But yes, I'll try it out.. (setenv NO_COLOR 1 && ./bin_using_cligen)

[c-blake]

Oh. Well, I don't know if tcsh even has a system to go from --help output to an autocompletion table. Zsh and Bash do (as mentioned in this package's README.me). Zsh was definitely impacted by color in the help table, but I found where to patch that. I guess I have to look into Bash myself. Anyway, the big change was just pushed. Give it a spin and let me know what you think.

[c-blake]

I included an example Nim-parsecfg config-file parser in the package and a sample config file in the RELEASE_NOTES.md, but I mention you by name in the notes in case you want to re-write cligen/clCfgInit.nim to use a 3rd party TOML parser. It would probably take you all of 30-60 min to code that up. Then I could also just distribute it and CL authors could use a define switch to select it. (Sort of like clUseMultiPerlish).

As it is, you could do all of that without my cooperation just via some directory/file ahead of cligen in Nim's path that is called cligen/clCfgInit.nim. Still, I'm happy to add it to the cligen distro as long as it is "opt-in" to the dependency..much as the new color stuff is CL user opt-in. Anyway, just something to think about if Nim's accepted parsecfg syntax hurts your eyeballs too much.

[kaushalmodi]

I mention you by name in the notes case you want to re-write cligen/clCfgInit.nim to use a 3rd party TOML parser. It would probably take you all of 30-60 min to code that up.

Yes, I will come up with an alternative clCfgInit.nim using parsetoml, but I will not be able to get it today.

Then I could also just distribute it and CL authors could use a define switch to select it. (Sort of like clUseMultiPerlish).

Awesome, may be -d:cgConfigToml :)

if Nim's accepted parsecfg syntax hurts your eyeballs too much.

The init syntax looks almost exact to TOML except when it comes to arrays, tables, etc. It's more about using a config that is well spec'd, a sane (subjective) syntax and also widely adopted.

[c-blake]

No rush on an alternative clCfgInit.nim. Probably just before we push 0.9.46 on the world. Happy to call it -d:cgConfigToml. But if you have major objections to other aspects of this new setup, this week is definitely the time.

[c-blake]

Also, we may want to use it for both an alternative in clCfgInit.nim and in cligen/cfUt.nim (either as separate procs or separate files). The latter is only used by the include cligen/mergeCfgEnv mechanism, but any CL author wanting your parser for $CLIGEN probably wants it for that, too.

[kaushalmodi]

I included an example Nim-parsecfg config-file parser in the package and a sample config file in the RELEASE_NOTES.md

I just rebuilt cligen from HEAD and that example config works well.

I have 3 comments:

• Can you document all the config file keywords?
• Also, some existing keywords, I believe, can be more verbose for readability: e.g. dflval -> defaultValue, descrip -> description.
• The color* can be maps (can we have those in current config format?). In TOML , I would do:

  [colors]
    command = "bold"
    doc = "italic"
    args = "underline"

  Similar comment for hTab*.

[c-blake]

Well, yeah. At present there is an awful large lack of documentation, but the sample config has all of them that exist. :-) I'm not sure where to document them. Maybe a new file hooked off of the README.md. Let's get the interface settled first. :-)

I'm fine renaming most of the CL user-visible names to be more distinct from the programmer-centric naming inside the package. It's actually pretty easy to just do of name1,name2,... to have several aliases. The one exception are the names of the included columns (values for what is now called hTabCols). I'd have to do another parseEnum for that. Not impossible, but more work. And I can't really change the enum idents without it being a won't-compile-breaking change for anyone using them.

parsecfg can do [colors] like you say, but I already use colors as a directive to establish color aliases. I'd like to keep it that way so the whole file can just be the same for any-cligen, lc, procs, etc. So, that might be confusing anyway. I don't really expect more than those three elements ever. So, I'd kind of lean toward keeping those "color"-prefixed rather than introducing config file "sections"/"maps".

[kaushalmodi]

Here's a quickly drafted TOML config (~/.config/cligen/config.toml):

themes = ["theme1", "theme2"]

# Below settings will override the values set by the above themes.
[textAttr]
  switches = ["RED", "bold"]
  valueType = ["CYAN"]
  defaultValue = ["italic", "GREEN"]
  description = ["PURPLE"]
  command = ["bold"]
  doc = ["italic", "magenta"]
  args = ["underline"]

[hTab]
  colGap = 1
  minLast = 12
  cols = ["clOptKeys", "clDflVal", "clDescrip"]

# Be very careful with the next two "CL syntax modifiers" as changing
# them can easily break config files or script-usages of programs.
#reqSep = true
sepChars = ["=", ":"]

useHdr = "%(underline)Usage:\n  "
use = "$command $args\n${doc}options:\n$options"

useMulti = """${doc}Usage:
  $command {SUBCMD}  [sub-command options & parameters]
where {SUBCMD} is one of:
$subcmds
$command {-h|--help} or with no args at all prints this message.
$command --help-syntax gives general cligen syntax help.
Run "$command {help SUBCMD|SUBCMD --help}" to see help for just SUBCMD.
Run "$command help" to get *comprehensive* help.${ifVersion}"""

[kaushalmodi]

.. and yes, TOML values have types.. so we can quickly detect if user screws up a type for a value.

E.g.

• reqSeq is a boolean, so only true or false would make sense there.
• parameters in [ .. ] are arrays. So even if there's only 1 element, it has to be an array.
• Strings have to be strings, and ints have to ints, and so on