getporter / porter

Porter enables you to package your application artifact, client tools, configuration and deployment logic together as an installer that you can distribute, and install with a single command.
https://porter.sh
Apache License 2.0
1.23k stars 208 forks source link

Add param/cred source for data that porter can resolve internally #2226

Open carolynvs opened 2 years ago

carolynvs commented 2 years ago

When we are running a dependency workflow, we need a way to pass the result of a bundle run to a bundle that runs later. Parameter and credential sets should support a new source porter that is resolved by porter, not a plugin.

name: internal-ps-for-dep-workflow
parameters:
  - name: connStr
     source:
       porter: plan.tasks.mysql.outputs.connStr

I haven't settled on the exact format but the idea is that porter would be aware of the current workflow execution plan that the parameter/credential set is being used within, and can resolve outputs from the run that occurred within that execution plan. So in the above reference plan.tasks.mysql.outputs.connStr, Porter would intercept the source and resolve it without going through the plugin protocol. Porter can use its internal context to know which execution plan it is currently executing, then lookup the named task "mysql" within the plan, figure out the run and result id from that task (which must have already completed successfully) and then use those ids to lookup the output value.

This should only be internal functionality in Porter but not to end users at this time. So someone couldn't make a ps/cs that used the porter source for now.

ℹ️ Read PEP003 - Advanced Dependencies for context about how dependencies should work, design details, and notes about desired behavior.

carolynvs commented 1 year ago

I think we can just support retrieving outputs from another job in the workflow (or another installation later when we support user-defined workflows). We can avoid implementing looking up another job's param/credential by resolving the original source of a value and referencing it directly from the dependent installation. That way when we resolve params/creds for a dependent installation, we aren't having to look up the dependency graph to find the value, the original source has already been identified.

This let's us work with a dependent installation either through a parent installation in the dependency graph or individually. When upgrading individually, it has a reference to the original parameter/credential set used when it was installed. When it's upgraded as part of the larger dependency graph, we will repeat the process of identifying the original source and copying it down onto the dependent installation to ensure that values are passed down the graph not just for install but also for subsequent actions against the graph.

carolynvs commented 1 year ago

Instead of using porter's template syntax for the value of the source mapping,lets go with something more structured so that we don't run into trouble data packing/unpacking.

Instead we can store a json document that specifies the type of data to find and return. We should not directly feed the source mapping into a query against the database.

Installation Output

Retrieve the last output generated for the installation by name

{
    "resource": "Output",
    "filter": {
        "installationID": "abc123",
        "name": "password"
    }
}

Run Output

Retrieve the last output generated from the run by name

{
    "resource": "Output",
    "filter": {
        "runID": "abc123",
        "name": "password"
    }
}