Common: Introduce support for storing data in variables

[rsenden]

The fcli output framework allows for outputting data in arbitrary formats to files, however it may be useful to (temporarily) store the original JSON response (or particular properties) from the remote system for later use, while at the same time having normal (human-readble) fcli output.

There are two main use cases that warrant this enhancement:

• A user may want to perform multiple (client-side) queries, or output the data in multiple formats as supported by fcli; for large data sets it may be more performant to temporarily store the JSON response rather than querying the remote system multiple times.
• Allow data to be passed between commands, without having the user bother about retrieving id's and passing them on to subsequent commands. For example, users may want to invoke a scan start command, and subsequently run some other commands like wait or download on that scan, without having to manually get the scan id from the scan start command and passing it on to the other commands.

This enhancement would need to cover the following:

• Introduce a new helper class in the common util package for storing and retrieving variable contents. Optionally, this helper class could provide cleanup on write functionality, for example automatically cleaning up the oldest variable(s) if the write operation would result in more than x variables being stored.
• Introduce a new command-line option in OutputMixin that triggers storage of the response under a user-specified variable name. OutputMixin should first process standard input and record transformers, and write each record to a file named after the variable name in the FCLI home folder. We need to decide whether we want to support writing the full object only, or support writing a subset of properties, i.e. --store-as <var-name> or --store <var-name>[:<properties>]
• All commands should probably properly set the OutputConfig.singular() property to have OutputMixin write either a single object or array to the variable file, or maybe instead of the singular() property we should have marker interfaces ISingleRecordProducer and IMultiRecordProducer, or add an isSingular() method to IHttpRequestSupplier and IJsonNodeSupplier interfaces (through a new superinterface).
• Optionally, introduce a new none output format, in case users just want to store the response data as a variable, without displaying any output
• Introduce new commands (in the config module?) for managing these variables:
  • delete: Delete one or more variables
  • list: List all available variables
  • clear: Delete all variables
  • get: Get the contents of a variable, providing the -q option to allow for client-side filtering, and the -o option to output the data in any of the supported formats
• Introduce generic functionality for resolving variables on the command line, either by preprocessing main args before invoking picocli, or by using picocli-specific option/parameter value processing (if available; picocli has parameter preprocessing functionality for individual options/paramaters, but maybe not for global value processing?). Variable references should use markers that are unlikely to conflict with existing property expressions (like in -o expr={id}) or shell expressions (like in -o 'expr=echo {envVar:-DefaultValue}: {id}'), so probably use something like {var:<name>.<propertyPath/jsonPath>}
• Introduce generic functionality for allowing commands to explicitly store data under a predefined variable name, like currentDastScan/lastDastScan; in addition to any user-provided variable name; we could introduce a new interface IOutputToVariable containing a single method that returns the variable name (and properties to be stored, if we decide to support storing a subset of properties), in the same format as the --store option
• Have each applicable command implement the IOutputToVariable interface, like scan start and <entity> create commands
• Implement generic functionality for clearing predefined variables if a command fails, to avoid users referencing a previous variable value after a failing command
• Have each applicable resolver treat - as a reference to the applicable property of the applicable predefined variable name; commands that take a DAST scan id would for example look up the lastDastScan variable and use its id property when a user specified - as the scan id

Instead of having the IOutputToVariable interface, we can also look into generalizing this functionality to write a predefined variable for every entity whenever a command that returns a single entity is invoked. For example, sc-dast scan get, sc-dast scan create, sc-dast scan pause, ... would all automatically set the current<entity> property (with <entity> being DastScan in this case), allowing users to use - to reference the 'current' scan id as used for the last (single-entity) sc-dast scan command.

This would allow for this functionality to be provided in a much more consistent way, without having to decide whether or not a particular command should implement the IOutputToVariable interface. The main difficulty is defining/retrieving the <entity> to be used for the variable name; you'd want to have a generic lookup mechanism for all commands related to a particular entity.

[rsenden]

This feature has been implemented