Data pipes

[annacrombie]

This pr totally refactors the input system. It simplifies the actual reading of numbers and makes data processing more flexible.

Pipelines

With this pr, you specify a pipeline for each input. Currently, this is with the option '-p'.

example:

the equivalent of seq 1 100 | plot -a 5 would become seq 1 100 | plot -p avg:5

or you could get a simple moving average with this:

seq 1 100 | plot -p sma:5

furthermore, these data processors or data procs, can be composed

seq 1 100 | plot -p "avg:5|sma:2"

They can also be set per input. Using -p before any input sets the default pipeline which is copied to each input.

@kuro > ~/src/c/plot build/plot -p avg:40 -i gas_prices -i gas_prices -p sma:9
      12.29 ┤  ╭╮
      11.82 ┤  ││
      11.36 ┤  ││
      10.90 ┤  ││
      10.44 ┤ ╭╯│
       9.98 ┤ │ │
       9.52 ┤ │ │
       9.06 ┤╭╯ │
       8.60 ┤│  │
       8.14 ┼───╮╮
       7.68 ┼╯  ╰╮
       7.22 ┤    ╰╮
       6.76 ┤    ╰╰╮
       6.30 ┤     ││
       5.84 ┤     ╰╰╮    ╭╮
       5.38 ┤      ││    ││
       4.92 ┤      │╰╮   ││                       ╭─╮
       4.46 ┤      ╰╮╰─╮╭╯╰╭──╮╭╮╭─╮              │ ╰─╮
       3.99 ┤       │╭╮╰───╯ ╰╰───╮╰╮        ╭─╭───────╮╮                        ╭╮
       3.53 ┤       ╰╯│╭╯     ╰╯  ╰──╮     ╭╭──╯──╯    ╰─╮           ╭╮          ││
       3.07 ┤         ╰╯             ╰──────╯            ╰─╮       ╭╮╭──────────────╮
       2.61 ┤                         ╰╮╭╯                ╰╰─────────╯       ╰╯     ╰───╮
       2.15 ┤                          ╰╯                     ╰─╮╭╯                  ╰╯ ╰╮
       1.69 ┤                                                   ╰╯                       ╰───

data source original: https://datahub.io/core/natural-gas

The pipelines for the above look like

in1 -> avg:40 -> output
in1 -> avg:40 -> sma:9 -> output

implemented data procs:

• avg:n - averages every n inputs together into a single data point
• sma:n - a simple moving average (boxcar filter) for n inputs
• cma - a cumulative moving average
• roc:t - determines rate of change with the basic formula (val - old_val) / t

Other changes

• -a has been deprecated, but still works as a shorthand for -p avg:n
• you can set flags for the pipeline by putting them after colon at the end of the input filename.
  • e.g. -i data.txt:r will add the flag rewind to the pipeline for data.txt
  • currently the only flags available are r (rewind) and n (nonblock)
  • rewind causes the file to be rewound once an eof is encountered, allowing you to do e.g. plot -f -i /sys/class/net/wlan0/statistics/rx_packets:r and get a stream of numbers
  • nonblock opens the file in nonblocking mode
  • this syntax kinda sucks but I do have a branch where I am playing around with an improvement of the options syntax.
• debug logging via the macro L
• general improvements
  • input files are opened in nonblocking mode so that -f behavior is more controllable this is now an input flag
  • data sources are synchronized so that they always line up in the output
  • escape sequences in animate.c (used to be follow.c) have been clarified with an extremely thin wrapper

TODO

• [x] add docs for new features
• [x] fix busy loop caused by running with no args
• [x] implement validator for rate of change
• [x] tidy up internal names (e.g. ctx_validator)

[annacrombie]

@AdrianDC @holmanb would you like to take a look at this before I merge it?

[holmanb]

I've only read the description so far, but it sounds promising. I'll take a looks at the implementation and make comments if I have any.