Generic max-results and order-by

[MikeTheSnowman]

With fcli SSC's list commands, if you try to limit the number of results being output by using the RESP APIs limit parameter, it won't work for two reasons:

• If any -q options are specified that result in client-side filtering, you may get less results than specified (if some records on the first page don't match) or no results at all (if none of the records on the first page match), even if there are results on next pages that do match the query.
• If we add generic paging support for SSC (similar to FoD and SC DAST), then specifying a limit like this will still result in all records being loaded, they will just be loaded in smaller chunks

Instead, we'd need to add a generic --max-results option, either in OutputMixin (so the option becomes available on all commands) or as a separate Mixin (so we can make this option available only on list commands for example). Alternatively, we could integrate this into the -q option, like -q maxResults=100, not sure whether that's a good idea though.

OutputMixin would then need to keep track of the number of filtered records actually written, and stop making requests for getting a next page once we've written the maximum number of results. Generic functionality in the SSC module would need to update the request to set the appropriate page size; if --max-results is not specified, this functionality adds limit=-1 to the request, or adds limit=<max-results> if --max-results is specified.

As for the --order-by option:

• We'd need to decide whether we want to support ordering on all (list) commands for all possible properties (possibly requiring client-side ordering), or only support this for endpoints that provide server-side ordering, possibly supporting only a limited set of properties
• As this is generic functionality, we should have a generic Mixin (probably in fcli-common) that defines standard ordering options, and then have generic functionality in each product module that retrieves the ordering options from this mixin (if defined on a particular command) and generates the appropriate request parameters. Apart from --order-by, this mixin should also provide an option to specify 'ascending' or descending', rather than requiring the user to understand the server-side orderBy syntax (like using + or - to specify ascending or descending).

[rsenden]

Order-by functionality is already being discussed in #86

[rsenden]

Having an option to limit the number of results without the ability to provide a starting position probably doesn't make much sense. Potentially we could have a generic --records <start>-<end> output option:

• Applied after any filters have been applied, i.e. --query is evaluated before --records
• Maybe should have a default value, like --records 0-5000, with an option to disable this default, like with --records all
• If the maximum record number has been reached, stop loading additional pages, and maybe output a warning stating that there may be more results available.

[rsenden]

Although the comment above could potentially work for returning the first x records, it won't be performant when processing later pages, for example --records 100000-100005 as we'd still need to load the first 100000 or more records from the source system to be able to determine which ones are removed by client-side queries.

The only way to implement this in a performant way is to have a product-specific --records option, for example defined in a Mixin, and using this Mixin in all commands for which the primary endpoint supports paging. This means that paging will take place before applying client-side queries, so --records 0-1000 could potentially output 0 records if none of them match the given query.

As such, it may make sense to also have a generic --max-results option on all list commands (defined as a mixin referenced from either OutputHelperMixins::List or OutputWriterWithQueryFactoryMixin) that is applied after any client-side queries. This would allow users to simply see the first x matching records, without having to re-run the same command with different --records values to find the first x matching records. Obviously, once max results has been reached, no more pages should be loaded from the source system.