Support for structured config

shailsirohi commented 2 years ago

Hello!

Vote on this issue by adding a 👍 reaction
If you want to implement this feature, comment to let us know (we'll work with you on design, scheduling, etc.)

Issue details

Pulumi supports structured configuration which allows using config.GetObject. This is really a great feature to represent arrays as yaml array and represent the entire config as go struct. Currently Stack object config field is of type map[string]string. Having support for yaml arrays and object will be helpful.

The value would assumedly come from any of the supported sources, mainly:

a literal YAML-encoded (or JSON-encoded) string
a Secret with a key containing a YAML-encoded string

Ideally this feature would be orthogonal to whether or not the value is Pulumi secret.

There may be need for a new configuration block to allow for non-secret structured configuration. Note that secretsRef defines Pulumi secrets from a secret or from a literal value, while non-secrets are simply typed as map[string]string.

Note that the Automation API recently added support for structured configuration: https://github.com/pulumi/pulumi/pull/12265

Examples

Some examples for discussion purposes.

These examples assume the following program:

interface Data {
    active: boolean;
    nums: number[];
}

let config = new pulumi.Config();
let data = config.requireObject<Data>("mydata");
console.log(`Active: ${data.active}`);

And intend to generate the following stack configuration:

# Generated: Pulumi.dev.config
config:
  proj:mydata:
    active: true
    nums:
    - 1
    - 2
    - 3

Using a YAML value from a Kubernetes Secret as a Pulumi secret:

apiVersion: v1
kind: Secret
metadata:
  name: example-secret
type: Opaque
stringData:
  data.yaml: |
    active: true
    nums:
    - 10
    - 20
    - 30
---
apiVersion: pulumi.com/v1
kind: Stack
metadata:
  name: example-1
spec:
  stack: example/proj/dev
  config:
    aws:region: us-east-2
  secretsRef:
    mydata:
      type: secret
      secret:
        name: example-secret
        key: data.yaml
        type: object
  ...

Using an inline YAML value as a Pulumi secret:

apiVersion: pulumi.com/v1
kind: Stack
metadata:
  name: example-1
spec:
  stack: example/proj/dev
  config:
    aws:region: us-east-2
  secretsRef:
    mydata:
      type: literal
      literal:
        jsonValue:
          active: true
          nums:
          - 10
          - 20
          - 30
  ...

Using inline YAML as a non-secret (via a hypothetical config2 block):

apiVersion: pulumi.com/v1
kind: Stack
metadata:
  name: example-1
spec:
  stack: example/proj/dev
  config:
    aws:region: us-east-2
  config2:
    mydata:
      jsonValue:
        active: true
        nums:
        - 10
        - 20
        - 30
  ...

Open Questions

Config values are currently typed as string as opposed to apiextensionsv1.JSON. If we were to change the API to use JSON, then the values would be interpreted as JSON values (e.g. true would become a bool). Is that an acceptable (breaking) change? Or should a jsonValue field be introduced? Or should literals be string-encoded? Note that Program.spec.configuration.default is a JSON value (not encoded).
Should the API types use json.RawMessage rather than apiextensionsv1.JSON? See Grafana and Tanzu for examples.

matthewriedel-flux commented 2 years ago

From what I can tell, this is really critical, as it blocks the ability to configure the AWS Provider to assume a role for deployment execution, since ProviderAssumeRole is an object. This limits use of the operator to IAM users with access keys created and appropriate roles attached.

My desired use case is to use a Service-Linked Role for the operator service account so the operator can assume roles, then define the role to use in the stack's config under aws:assumeRole provider option.

ljtfreitas commented 1 year ago

Looking the CRD spec, this option:

Using a YAML value from a Kubernetes Secret as a Pulumi secret:

apiVersion: v1
kind: Secret
metadata:
  name: example-secret
type: Opaque
stringData:
  data.yaml: |
    active: true
    nums:
    - 10
    - 20
    - 30
---
apiVersion: pulumi.com/v1
kind: Stack
metadata:
  name: example-1
spec:
  stack: example/proj/dev
  config:
    aws:region: us-east-2
  secretsRef:
    mydata:
      type: secret
      secret:
        name: example-secret
        key: data.yaml
        type: object
  ...

would work with the current implementation? (or now does it just support simple,non-structured values?)

About the jsonValue field, personally, it would sound strange to me because I would expect that a complex yaml object would be passed forward as an entire object anyway with no need to put an extra field saying that. but in order to doesn't break compatible looks acceptable. Maybe a value key, as in the structured project-level config could be an option?

Also, here we are thinking that would be great if we could inject values from a ConfigMap. Maybe something like that?

apiVersion: pulumi.com/v1
kind: Stack
metadata:
  name: example-1
spec:
  stack: example/proj/dev
  config:
    aws:region: us-east-2
  configsRef:
    mydata:
      name: my-config-map
      key: data.yaml
      type: object

geowalrus4gh commented 7 months ago

This is a key functionality for me to elect Pulumi for new-gen platform engineering. I would need to configure the stack CR config parameters for my entire stack. I need to convert all my YAML style configurations to map[string][string] if i use pulumi at this moment. waiting eagerly for https://github.com/pulumi/pulumi-kubernetes-operator/pull/575

ljtfreitas commented 7 months ago

@geowalrus4gh please I would love to hear your thoughts about the PR and my suggested implementation. It's a key feature for us as well 😅

blampe commented 6 months ago

Quick note - while reviewing #575 I confirmed that structured config is somewhat supported although it's not obvious and may not be a viable workaround for all scenarios (particularly secrets).

Config values retrieved with GetObject will deserialize themselves from JSON if they happen to be simple strings.

Stack CR:

config:
  project-name:object: '{"nums": [1, 2, 3]}'

index.ts

const cfg = new pulumi.Config();
export const obj = cfg.requireObject("object") as { nums: number[] };

console.log("object looks like:", obj);
console.log(`type is ${typeof obj}`);
console.log(`nums are ${obj.nums}`);

Output:

    object config looks like: { nums: [ 1, 2, 3 ] }
    type is object
    nums are 1,2,3

I'm still working through the PR but wanted to share in case it might be helpful to anyone currently blocked by this.

pulumi / pulumi-kubernetes-operator