brimdata / super

A novel data lake based on super-structured data
https://zed.brimdata.io/
BSD 3-Clause "New" or "Revised" License
1.39k stars 64 forks source link

refactor zio reader/writer options and flags and improve CLI tools re-use of flags #1250

Closed mccanne closed 4 years ago

philrz commented 4 years ago

Verified in zq commit ca0ef53.

The linked PR #1249 is primarily described as cleaning up technical debt, but it also speaks to some user-facing differences. A specific tour of what's changed in the help text for each tool in commit ca0ef53 compared with the commit 059c003 that came right before it:

$ git checkout ca0ef53 && make install
$ for tool in microindex pcap zapi zar zq zqd; do   $tool help 2> ${tool}.ca0ef53; done
$ git checkout 059c003 && make install
$ for tool in microindex pcap zapi zar zq zqd; do   $tool help 2> ${tool}.059c003; done
$ for tool in microindex pcap zapi zar zq zqd; do echo "$tool:"; diff ${tool}.059c003 ${tool}.ca0ef53; echo; done
microindex:
8c8,11
<     no flags for this command
---
>     -cpuprofile write cpu profile to given file name
>     -memprofile write memory profile to given file name
>     -sortmem maximum memory used by sort, in MiB (default "128")
>     -version print version and exit (default "false")

pcap:
8c8,11
<     no flags for this command
---
>     -cpuprofile write cpu profile to given file name
>     -memprofile write memory profile to given file name
>     -sortmem maximum memory used by sort, in MiB (default "128")
>     -version print version and exit (default "false")

zapi:
7a8
>     -cpuprofile write cpu profile to given file name
9a11
>     -memprofile write memory profile to given file name
11a14,15
>     -sortmem maximum memory used by sort, in MiB (default "128")
>     -version print version and exit (default "false")

zar:
8c8,11
<     no flags for this command
---
>     -cpuprofile write cpu profile to given file name
>     -memprofile write memory profile to given file name
>     -sortmem maximum memory used by sort, in MiB (default "128")
>     -version print version and exit (default "false")
27c30
<     chunk. For  filesystem archives, the associated files are stored in a
---
>     chunk. For filesystem archives, the associated files are stored in a

zq:
24a25
>     -validate validate the input format when reading ZNG streams (default "true")
26d26
<     -zngcheck check input records when reading ZNG streams (default "true")

zqd:
7a8,10
>     -cpuprofile write cpu profile to given file name
>     -memprofile write memory profile to given file name
>     -sortmem maximum memory used by sort, in MiB (default "128")

In addition to the profiling options being available in all tools as the PR said to expect, closest to my heart is the -version option arriving in all the tools. Here running against current master:

$ for tool in microindex pcap zapi zar zq zqd; do echo "$tool:"; $tool -version; echo; done
microindex:
Version: v0.21.0-13-g2742f18

pcap:
Version: v0.21.0-13-g2742f18

zapi:
Version: v0.21.0-13-g2742f18

zar:
Version: v0.21.0-13-g2742f18

zq:
Version: v0.21.0-13-g2742f18

zqd:
Version: v0.21.0-13-g2742f18

I was somewhat unsure about -sortmem being in so many tools I don't think of as doing sorting, and indeed, @mccanne confirmed this was unintentional, so #1279 has been opened to circle back and clean that up in a refactor.

Thanks @mccanne!