Allow parsing cligen config from a TOML file

[kaushalmodi]

To enable reading from a TOML config file, the user needs to:

1. Install the parsetoml package using nimble install parsetoml
2. Compile the project using cligen with -d:clCfgToml

Default location for the TOML config file: ${XDG_CONFIG_HOME}/cligen/config.toml

[kaushalmodi]

Here are some TOML examples files that are tested to work together:

~/.config/cligen/darkBG.toml

[global]
  [global.colors]
    #A dark bkgd (black BG, really) color theme with heatmap doing purple..red.
    fhotm = "black"        #default fg => useless
    fhot0 = "PURPLE"
    fhot1 = "BLUE"
    fhot2 = "CYAN"
    fhot3 = "GREEN"
    fhot4 = "YELLOW"
    fhot5 = "RED"
    fhotp = "WHITE"

    FHOTm = "BLACK"
    FHOT0 = "purple"
    FHOT1 = "blue"
    FHOT2 = "cyan"
    FHOT3 = "green"
    FHOT4 = "yellow"
    FHOT5 = "red"
    FHOTp = "white"        #default bg => usless

    bhotm = "on_black"     #default fg => useless
    bhot0 = "on_PURPLE"
    bhot1 = "on_BLUE"
    bhot2 = "on_CYAN"
    bhot3 = "on_GREEN"
    bhot4 = "on_YELLOW"
    bhot5 = "on_RED"
    bhotp = "on_WHITE"

    BHOTm = "on_BLACK"
    BHOT0 = "on_purple"
    BHOT1 = "on_blue"
    BHOT2 = "on_cyan"
    BHOT3 = "on_green"
    BHOT4 = "on_yellow"
    BHOT5 = "on_red"
    BHOTp = "on_white"     #default bg => usless

~/.config/cligen/config.toml

includes = ["darkBG.toml"]

 # Below settings will override the values set by the above themes.
[color]
  switches = ["fhot5", "bhotp"]
  valueType = ["FHOT4"]
  defaultValue = ["fhot3"]
  description = ["FHOT2"]
  command = ["fhot1"]
  doc = ["FHOT0"]
  args = ["fhotm"]

[layout]
  # rowSep = "\n"
  colGap = 1
  minLast = 12
  cols = ["clOptKeys", "clDflVal", "clDescrip"]

[syntax]
  # Be very careful with the next two "CL syntax modifiers" as changing
  # them can easily break config files or script-usages of programs.
  reqSep = false
  sepChars = ["=", ":"]

[templates]
  useHdr = "%(underline)Usage:\n  "
  usage = "$command $args\n${doc}options:\n$options"
  usageMulti = """${doc}Usage:
  $command {SUBCMD}  [sub-command options & parameters]
where {SUBCMD} is one of:
$subcmds
$command {-h|--help} or with no args at all prints this message.
$command --help-syntax gives general cligen syntax help.
Run "$command {help SUBCMD|SUBCMD --help}" to see help for just SUBCMD.
Run "$command help" to get *comprehensive* help.${ifVersion}"""

Let me know what's the best way to add this to the RELEASE_NOTES / documentation. Also how would we add a test for this?

[c-blake]

Right. Well, I would say there is no need to have a giant 4-way spectrum alias system in an example file and those theme1/theme2 don't do anything. That's "hot" stuff is kind of "my personal thing" (though everyone knows "rainbow order"). In lc/procs I have a configs directory with cb0 as an example setup. It makes a lot of sense when "hotness" is "amount of data" or "amount of time"..maybe not for discrete categories like "amount of "command-ness". They're just aliases I know I have that work with both backgrounds.

We should probably do a config or two in the distro here, too. Happy to accept some "km0" one from you. I'm only so motivated to create an entire alternate framework. ;-)

Testing will be tricky. I don't have any behavioral tests, really. I just test all this stuff on command lines/in personal configs and then call it ready. That doesn't trap regressions if someone changes something and people have pushed back on that, but eh. In 5 years working on this that's actually never happened to my knowledge and I've used commands using this daily since then. Mostly what happens is compile failures and test.sh catches that. So, I'm ok with you just being the tester for TOML things and telling me if something ever breaks. You can just put -d:cgCfgToml in your config.nims or whatever.

We may want to document some of this color syntax on the new wiki you prodded me to make. I feel like a couple examples and the wiki is probably an ok first cut at documentation. See what kind of questions come up that it doesn't answer.

All that sound reasonable?

[c-blake]

One other thought was doing that when defined(clCfgToml): switch in cligen.nim. Then neither file needs indenting. And I suppose, in theory, other users/projects could locally override either/both of the impls.

[kaushalmodi]

One other thought was doing that when defined(clCfgToml): switch in cligen.nim.

I thought about that, but then also wanted to have common handing of the CLIGEN env var etc. Is it OK to move this piece to cligen.nim?

var cfNm = getEnv("CLIGEN", os.getConfigDir() / "cligen" / cgConfigFileBaseName)
if cfNm.existsFile: clCfg.apply(move(cfNm), existsEnv("NO_COLOR"))
elif cfNm.splitPath.head == "config" and (cfNm / cgConfigFileBaseName).existsFile:
  clCfg.apply(cfNm / cgConfigFileBaseName, existsEnv("NO_COLOR"))
# Any given end CL user likely wants just one global system of color aliases.
# Default to leaving initial ones defined, but clear if an env.var says to.
if existsEnv("CLIGEN_COLORS_CLEAR"): textAttrAliasClear()

or.. should I create a new file clCfgDflt to hold the default parsing code? and use clCfgInit.nim as a wrapper?

[kaushalmodi]

Happy to accept some "km0" one from you. I'm only so motivated to create an entire alternate framework.

OK, can I add the examples in this thread to cligen/examples/config/{config,darkBG}.toml?

[c-blake]

Yeah. Moving that would be ok. I can do all that tomorrow. Or you can do it and I'll just merge it. I would make a new directory called "configs" at the top level.

[kaushalmodi]

hmm, did you merge this by mistake.. I thought I would first move the default code and then do this PR..

OK then, I will send PR for example TOML configs in the next PR.

[c-blake]

Nah. I'm just merging as you make changes. I'm easy. I'd even give you write access to the repo if you wanted.

[kaushalmodi]

Cool! Thanks, I don't need the write access yet. Thanks for the quick merges.

[c-blake]

Ok. I may clean up some things tomorrow morning like making parsecfg and toml modes more symmetric. But it's kinda late for me to keep working tonight. Good night!

[c-blake]

So, I added a file configs/config with some more elaborate commented explanations, not entirely duplicative of the RELASE_NOTES.md. I tried to be "complete but brief". (Well, I guess I didn't explain all the keyword aliases. I'm not sure anyone will care beside you and you know from the code, and I'm really not sure anyone who can't read those short parser procs will care.)

I don't know if we want the TOML example to be as descriptive. Maybe not. Maybe just showing the altenate "config syntax" is enough..Being descriptive in two different ways might lower cognitive load on would be readers.

[kaushalmodi]

I didn't explain all the keyword aliases.

Makes sense.

Maybe just showing the altenate "config syntax" is enough..Being descriptive in two different ways might lower cognitive load on would be readers.

Sure. The descriptions in the config.toml are a plain copy of what you had in the config in the RELEASE_NOTES.md. You can remove the descriptions from there, and add a note to refer to the configs/config for descriptions of keywords.

Additionally, the wiki would be a better place for keywords, default values and descriptions, as we can have a nicely formatted table there.

[kaushalmodi]

Additionally, the wiki would be a better place for keywords, default values and descriptions, as we can have a nicely formatted table there.

I started documenting attribute keywords in https://github.com/c-blake/cligen/wiki/Text-Attributes-supported-in-Config-files

Now from a separate page for config files, we can link this text attributes page to guide the user to how colors and styling keywords are used in the [color] section

[c-blake]

Sounds good. I'm not like in love with my color/attr specification syntax but it was really easy to code up in cligen/humanUt when I was working on lc and it's more complete than most things in this problem space. Why, I don't even know any terminals that support two rates of blinking. :-)

[kaushalmodi]

Why, I don't even know any terminals that support two rates of blinking

I never use the blink attribute (I am scarred by the websites in 90's which blinked text in webpages). So I was surprised you gave extra special treatment to the blink attr :D

[c-blake]

I always liked blink for broken symbolic links. Many terminals also do not support "faint"/"dim" text either which would also make sense for orphaned links.

There are a few other dark horse codes on that Wikipedia page for SGR that I skipped like Fraktur, double underlined, framed, encircled, etc. Someday I may write my own terminal emulator in Nim and complete those features.

[c-blake]

Hey, @kaushalmodi - you should verify that my changes to clCfgToml.nim Re: [render] work when you get a chance. (Also, you will probably like the new, better wordwrap and rST/md style rendering if you haven't yet tried it.)

[kaushalmodi]

@c-blake Hello, yes, certainly. I will get to it early next week.

[c-blake]

I should mention that, yes, I did test it, but I do not routinely use the TOML fork. So, the test was one-off/not thorough and I may have missed something.

Also, just to cross link on the github, I was following up on my comments in https://github.com/c-blake/cligen/issues/145 - more about general help formatting than inferring the main help[param] question. At the time, I should have realized dual API-CLI demands just interpreting the rST similarly to how nim doc does it -- only we drop anything but text style rendering/SGR sequences since text tables, bullet lists, URIs and other markup are designed to "look nice" in ASCII. Actually, even the single-triple * and backslash are designed that way, too, but using them a lot uses up scarce terminal columns especially in the help[param] section.

My current rST/md style parser is a very hacky regex based thing. I tried to use the packages/docutils/[rst, rstast], but honestly it was not really designed to "parse partially" just the text style stuff and pass through the tables, lists, URIs, etc. So, I would have had to write a ton of output rendering for all that stuff. In some sense the Nim stdlib stuff was "too complete" to be nice to use, with some irony. I am pretty sure markdown at least was designed to be parsable by regex, though. Most stuff I have tried works about as well there as in nim doc or github boxes/etc.

[c-blake]

Also, totally off topic, but I got another 1.1-1.5x speed boost (depending on directory branching ratios) out of forPath on Linux with a custom kernel module that adds a generic batch system call. I also added a couple utilities (rr and dirt) that take advantage of the exposed recursive hierarchy in forPath that would be unwieldy-to-impossible with regular iterator-based methods. With that module, it probably is the fastest ftw in The West, at least on Unix when file Stat metadata is needed, but that module work remains preliminary. I mention it since you just seemed interested before.

[c-blake]

Oh, and if you happen to want to combine your new formatting config testing with sys_batch file tree walk acceleration, I just added a new examples/du.nim that is quite a bit faster than GNU du with my https://github.com/c-blake/batch activated. { There are likely more sensitive examples than ftw for my batch, but getdents is already a batch interface. So, the follow on to do a batch fstatat is very "natural". } That's a bit more effort, but you had seemed interested.

Oh, and also my du.nim is about 10x fewer source LOC than the GNU coreutils du.c, though probably more than half of that gain just comes from the cligen approach. :-) It saddens me more and more to see all this boilerplate CL stuff everywhere now -- without even colorized help, spell check, the long-to-short-continuum of unique prefix matching, CL end user tweaks, etc. Just all this bloat with even less functionality. Yuck.

[c-blake]

Hey, dude. Not trying to be a pest, but I'm probably a week or two away from stamping a 1.1 version. It would be good to check your TOML works before then. I'm personally liking my new rST -> ANSI SGR help message formatting with fancier word wrap quite a bit. :-)

[kaushalmodi]

Hello, sorry for the wait. Last week was too busy and then I was away in the mountains, hiking. I'll be able to get to my development computer tonight US eastern time (after 4 days). I'll ping you.

[c-blake]

Ah. No worries. Just making sure this doesn't get forgotten. Anytime this week is A.Ok. I enjoy hiking myself.

[kaushalmodi]

I'm probably a week or two away from stamping a 1.1 version. It would be good to check your TOML works before then.

It works beautifully! ❤️

I am aware that italics show as inverse on my terminal (xterm). I had spent a couple of hours to get italics to work for me, but I couldn't do that in a way that tmux could also work in parallel. So I am just giving up on italics for now.

[c-blake]

Great to hear. Thanks for checking!

And yeah...SGR is more limited than HTML as a render target mostly due to variation in terminal capabilities..another reason I didn't want a "full parse" and having to worry about like line-drawing characters/unicode support/whatnot. Even just inverse and underline help a lot with documentation, though. Unix man pages often only use 3 styles, and it is fine to map the 5 (single..triple star, single..double backtick) styles to different colors like white or something, too..whatever a user likes, not only "font embellishments". I don't usually colorize either the doc section or the help[param] section, myself. I think Nim may recommend only 2 or 3 embellishments in proc docs. Anyway, I'm liking the nicer help dumps than most man pages quite a bit, and it also fits perfectly with "dual API/CLI" mentality of cligen.

[kaushalmodi]

I'm liking the nicer help dumps than most man pages quite a bit, and it also fits perfectly with "dual API/CLI" mentality of cligen.

+1

[c-blake]

BTW, though it was kind of staring me in the face conceptually since 2015 when I first did this package, our misunderstanding about you forgetting help[param] is mostly to credit for my sitting down and actually doing it. Sometimes a misunderstanding can bear unforeseen fruit. :-)