Better developer experience CLI params and exhaust behavior

[jawache]

After working with the new version of IE I have some suggestions for improving the developer experience with a different CLI args as well as other approaches I would take.

ie 
--manifest: <your manifest file>
--exhaust: yaml (default) or csv or a configured custom exhaust plugin name
--outfile: the name of the file to output to (full path, the user should expect this to be the name of the file they open without having to remember to append something to it)
--filter: only output this set of comma separated parameters (useful for CSV exporter and perhaps useful for Yaml exporter also)
--stdout: if this flag is present always also prints the YAML to STDOUT.
--verbose: raises the log level, prints a lot more detail.

Exhaust Configuration

• It doesn't feel right to me that exhaust configuration is embedded in the manifest file, it feels right for the manifest file to be just focussed on the measurement, esp. if we are telling people it's an "executable audit".
• Multiple people might want to develop on the same manifest file but one wants to look at YAML and the other with CSV, what get's checked in? Feels like something we should have external to the manifest file itself.
• How you output, publish the data feels secondary and should be a combination of CLI flags and/or perhaps environment variables, something external to the manifest file itself.
• Some exhaust plugins are going to require things like credentials (publish to prometheus etc...) so environment variables seem the right way to go from the outset.
• The current exhaust configuration in the manifest is called outputs which is used in the manifest file to represent the computed outputs in the nodes, so a little confusion there also.
• Instead of in the manifest file we can just specify outputs in the CLI command with the --exhaust option.

Supporting native CSV export

• Certain exhaust plugins, e.g. CSV need configuration that should be passed on the command line, for that use case it needs to be passed a set of fields and only export those fields, see #411
• I think the best way for now is to pass in the args (perhaps all of them) to every exhaust function and allow it to do what it wants.
• For the CSV exhaust passing in filter and outfile will be enough for it to have the configuration it needs.
• This is the exception to the use case of using environment variables, CSV exporting is going to be a very very common use case during development of a manifest file so I feel having an exception here might be worthwhile.

Be consistent with YAML as the manifest file format

• If we are printing a manifest file it should only be printed in YAML.
• Both if you are logging and if you are outputting to STDOUT.
• Displaying JSON in the log when the user is editing a YAML forces them to do some mental translation to match what they are seeing in the log to what they are typing in the YAML manifest. If we are logging their manifest file, it should match their input format which is YAML.

Regarding STDOUT

• Bring back printing the output to STDOUT!
• STDOUT is useful to quickly see the output of a computation in development and means you can use IF in a linux command pipeline (ie | some other command)
• If nothing is specified for the exhaust OR the user has specifically requested it with stdout, output to STDOUT.
• But same as above, the STDOUT output should always be YAML. You should be able to take the output and pipe it back into IE and have it execute it again.

Logging

• Logging the manifest file seems like a choice regarding the log level and verbosity flag than anything to do with exhausting an output.
• Logging is something you might do to trace execution but it's never going to be a normal way for someone to extract the output of a computation.
• It shouldn't be the default and feels more appropriate to be behind a verbosity flag or log level = debug or something like that.

Perhaps there is no need for multiple simultaneous exhausts?

• I'm not 100% sure about this one I can see some pros and cons but I can't think of many use cases where there is a need to output multiple exhausts at the same time, e.g. YAML and CSV.
• During development, I personally would only want to output YAML, or a CSV or something else custom.
• The way I'm using IE to develop complex manifest files is that I want to execute a bunch of times to see YAML, then a few times with CSV to check something complex is working and summing up as expected, then back to exhausting YAML. That feels like the approach I would take to developing a manifest file and that's much easier to do if I can just quickly edit the --exhaust option than edit the manifest file itself.
• I think if you are outputting CSV and also wanted to see the YAML you can add the --stdout flag and have the YAML printed to the console or redirected to a file.
• If they are exhausting to something custom like prometheus and also want to see the YAML file as well locally, same thing I would add the --stdout flag to have the YAML printed to console or i'd redirect to a file.
• Supporting multiple simultaneous exhaust means we are appending a .yaml or .csv to a file name and treating output as a folder, it took me quite a while to figure this out, if it confused me it will confuse many others! If we choose to keep this approach we should rename output as folder to add clarity.
• In production, I can't imagine anyone would configure multiple exhausts that would seem like a waste. I'd imagine you would pick one exhaust type and and then re-run the IF on a cron and processing the output exhaust, you wouldn't waste resources exhausting something you're not going to read.

[pazbardanl]

@jawache @jmcook1186 @narekhovhannisyan The exhaust related directives above make sense to me, these are my inputs:

1. If we take CSV as our main use case / example, this is how a command line will look like:

   ie 
   --manifest: my-manifest.yaml
   --exhaust: csv
   --outfile: my-exhausted-file.csv // this file is the being created by the csv export exhaust plugin which was specified in previous line
   --filter: energy, carbon
   --stdout
   --verbose

   question regarding filter - can you think of other meta-data such as timestamp that are always filtered-in?

2. It might seem minor but it effects DX: if we are dynamically loading exhaust plugins (like regular ones), declaring the plugin in the --exhaust option must have the actual name of the plugin, same as we declare regular plugins:

   plugins:
   'time-sync':
     method: TimeSync // <==

   ... because this is the name that's being used for dynamically loading the plugin (@narekhovhannisyan please correct me here if this is inaccurate). Otherwise we'll need to maintain a mapping of alias (csv) --> plugin_name (ExportCsv) which is hard to maintain. So the command line should actually be:

   ie 
   --manifest: my-manifest.yaml
   --exhaust: ExportCsv // ...which is the actual name of the plugin
   ...

3. Notice we have an issue this sprint for refactoring the exhaust infra to dynamically load the exhaust plugins: https://github.com/Green-Software-Foundation/if/issues/492. I pretty sure these 2 could very well be worked on in parallel by 2 different ppl, but in this case some syncing is needed to prevent conflicts.

[pazbardanl]

hi @jmcook1186 seems this issue is for sprint 8 (ends 15/3) but with the upcoming freeze i dont think we will make it.

[zanete]

as per @jmcook1186 comment in slack: this is backlogged, and maybe actually could be closed out and restarted with a fresh issue later. The backstory was that we wanted to move the export of results from IF to a set of plugins rather than being core features, but we've had some doubts about whether this is really the way we want to go, and bumped the decision making until after the "trust" epic.

[jmcook1186]

Closing as obseleted by new epics