Run workflow and composite commands

[rsenden]

This issue holds ideas and proposals for running workflows and composite commands.

• Investigate possibility to run multiple subcommands in one go:
  • For example: fcli login ssc --url ... get applications -o applications.json get versions -o versions.json logout ssc
  • See picocli subcommandsRepeatable = true configuration option
• Allow shell-based chaining of commands by providing suitable output and functionality for parsing input; some examples:
  • FCLI_FMT=json fcli create ssc version app/version … | fcli … --someOption=${{ stdin.json.project.id }}
  • for v in $(fcli get versions); do fcli download ssc artifact --from ${v}

Idea for workflow syntax:

# Define environment variables for use during workflow run, inheriting from parent environment
env:    
  name: value

# Should we have a separate element for logging in? This would open transient session and would auto-logout
# after workflow completion 
login:   
  ssc:
    url: ...

steps:
  - id: <id used for referencing output of this step in other steps>
    name: Name to display when running this step
    if: ${{ steps.previousStepId.output.json.criticalIssueCount > 0 }}
    forEach: ${{ steps.previousStepId.output.json.data[].id }}
    fcli: <fcli command to run>
    composite:
      async: true|false # Run each step in a separate thread, for example allowing to spawn both sast and dast scans
      maxThreads: 5   # Maximum number of threads to spawn
      joinThreads: true|false # If async==true, wait until all steps defined below have completed
      steps:
        - id, name, fcli, ...

Regarding the fcli: entry:

• Can use expressions like ${{ env.xyz }}, ${{ steps.abc.output.plain }}, ${{ steps.abc.output.json.someProperty}}, ...
• Can use run cmd subcommand to run arbitrary commands (run cmd should be disabled by default for security reasons, can be enabled through fcli config run-cmd enabled true|false or something similar
• Can use run workflow to run other workflows

[rsenden]

Note that the 'mentioned' above is incorrect; #205 doesn't have anything to do this with this issue, but used #1 to refer to a number list item.

[rsenden]

Potentially we could use an approach like the following for integrating workflows into fcli:

• Have an fcli run <workflow> [workflow-options] command for easy invocation, even though this doesn't really correspond to current fcli structure
• <workflow> would either be:
  • A built-in workflow from fcli resources folder
  • A custom workflow added through an fcli config workflow add/import command
  • Potentially, the name of a local file or even remote URL containing a workflow (but non-preconfigured workflows may be more difficult to handle when generating usage information; see below)
• Workflows are (for example) in yaml format, defining properties like listed below for defining workflow characteristics

metadata:
  usage.header: 'My workflow'
  usage.description: 'Some description (maybe accept array of multiple description lines)'
  # usage.descriptionKey: ... (for built-in workflows, referencing resource bundle properties)
  options:
    - names:
      - long: myOption
      - short: o
    - description/descriptionKey: ...
    - required: true
    - interactive: true
    - ... (Other properties defined by picocli OptionSpec.Builder and ArgSpec.Builder

workflow:
  ...

Upon startup, fcli would load all predefined/pre-configured workflows, and/or the workflow file listed on the command line (if we can easily obtain this before picocli is actually being invoked), and use the yaml properties listed above to programmatically register each workflow as a sub-command on the fcli run command (see https://picocli.info/picocli-programmatic-api.html).

Now:

• If the user runs fcli run (without workflow) or fcli run --help, picocli would simply list all known workflows as sub-commands, including workflow description (from usage.header property).
• If the user runs fcli run fod-sast-scan -h, picocli would display the the fod-sast-scan workflow usage header, synopsis, description, options, ..., all based on the information defined in the workflow definition

If we can easily identify which workflow is being requested, the process above can probably be optimized to only load that particular workflow, i.e. if fcli run fod-sast-scan ... is being invoked, we would only need to load the fod-sast-scan workflow. We'd only need to load all workflows if fcli run is invoked without workflow (or with unknown workflow), such that the usage information can display all known workflows as subcommands.

[rsenden]

Alternatively, based on the same principles as above, we could have commands like the below for managing workflows and running workflows:

fcli util workflow list
fcli util workflow generate-sample
fcli util workflow run
...

But, in addition, allow for registering workflows as sub-commands of existing fcli commands, i.e., registering the fod-sast-scan.yml workflow as fcli fod-sast-scan command, or as fcli fod sast-scan command:

fcli util workflow as-cmd --workflow fod-sast-scan.yml --cmd 'fod sast-scan'

Advantages:

• This allows workflows to be even more tightly integrated in the existing fcli command structure
• Upon startup, we know exactly which workflows have been registered as fcli commands, so we only need to load data from those workflows
• We don't need an fcli run <workflow> command that doesn't follow fcli conventions, while still allowing any workflow to be run through fcli util workflow run

Disadvantages:

• Users may be wondering why another user has different fcli commands, so we should probably clearly indicate in usage help that these are custom workflows

[rsenden]

One of the main difficulties of such workflows would be support for arbitrary 'shell' commands and/or custom logic. One of the main features of (for example) GitLab or GitHub workflows is the ability to run arbitrary shell commands, which includes both running external commands, and utilizing shell features like conditionals, loops, environment variables, ...

We could have fcli provide similar functionality, with yaml syntax for specifying which shell to use and the actual shell commands. This would mean that these fcli workflows may only run on platforms that have the specified shell(s) available, which could be a limiting factor as to workflow reusability.

Alternatively, we could use some shell/scripting languages that we can natively support in fcli, like JShell, JavaScript, ... This would require more research though, for example whether this is viable (compatible with native images, acceptable fcli executable size, ...) and how to handle interaction with external commands (for example, running ScanCentral Client or FortifyVulnerabilityExporter from such a workflow).

[kadraman]

I like the idea of command recording/playback - I think most O/Ss have some of ability to do this: script on Linux/UNIX and Start-Transcript on PowerShell. However, It would be good if we could do something more generically though. I still struggle with remembering the correct options and also querying the data (especially as you need different quotes on different OS/s) so it would be good to have something like .fcli file with commands in that we could load and run and wasn’t O/S specific.

So yes, I really think it useful if we could save, load and run workflows/snippets/playbooks/examples etc and then we could also deliver a library of example sets of commands? As in the issue Ruud mentions that we should really have some form of scripting support in these workflows (unless we provide the ability to store them for different platforms/shells?).