Add generic tasks to list flows or namespaces and to download/upload flows (ideally also across tenants)

[anna-geller]

Problem

Some users want to extract some information from the kestra instance without interacting directly with the API, e.g. to:

• fetch all flows
• fetch all namespaces
• download all flows, and all versions of specific flows.

Proposed Solution

We may provide dedicated core tasks for that:

• ListFlows
• ListNamespaces
• DownloadFlows

Proposed syntax

ListNamespaces:

id: export_flows
namespace: dev

tasks:
  - id: list
    type: io.kestra.plugin.core.namespace.ListNamespaces 
    childNamespaces: true # true by default — whether to include flows from child namespaces
    # outputs a list of strings

ListFlows:

id: list_flows
namespace: dev

tasks:
  - id: list_flows
    type: io.kestra.plugin.core.namespace.ListFlows 
    namespace: dev # only fetched from the current namespace by default
    childNamespaces: true # true by default — whether to include flows from child namespaces
    # outputs a list of strings

DownloadFlows:

id: export_flows
namespace: dev

tasks:
  - id: export
    type: io.kestra.plugin.core.namespace.DownloadFlows 
    namespace: dev # only fetched from the current namespace by default
    childNamespaces: true # true by default — whether to include flows from child namespaces
    pastRevisions: false # false by default — whether to include past revisions of each flow
    # outputs ION with columns: namespace, flowId, revision, content. 1 row per flow

Workaround until then

id: export_flows
namespace: dev

tasks:
  - id: fetch_namespaces
    type: io.kestra.plugin.core.http.Request
    uri: http://host.docker.internal:28080/api/v1/flows/distinct-namespaces

  - id: for_each
    type: io.kestra.plugin.core.flow.ForEach
    values: "{{ outputs.fetch_namespaces.body }}"
    tasks:
      - id: get_zipfile
        type: io.kestra.plugin.core.http.Download
        uri: http://host.docker.internal:28080/api/v1/flows/export/by-query?namespace={{taskrun.value}}
        method: GET
        contentType: "application/json"

      - id: unzip
        type: io.kestra.plugin.compress.ArchiveDecompress
        algorithm: ZIP
        from: "{{ outputs.get_zipfile.get(taskrun.value).uri }}"

[anna-geller]

FYI moved to 0.20 as there is only 1 week left before the release