Support template-based variable for secure and reuse specifications

Objective

To hide sensitive values from all users by using templates with variable support in specifications, allowing for reusable and secure configurations.

Current System State

Currently, specifications are statically compiled, which means all values must be determined at compile time. This results in sensitive information, such as database credentials, being directly included in the specification and exposed to all users. Additionally, the static nature of the current system makes it difficult to reuse specifications across multiple environments.

Proposed Changes

- kind: block
  specs:
    - kind: rdb
      driver: postgresql
      source: 'postgresql://{{ username }}:{{ password }}@{{ host }}:{{ port }}/{{ database }}'
  links:
    init:
      - name: postgresql
        port: in

- kind: snippet
  name: postgresql
  language: json
  code: >
    {
      "username": "myuser",
      "password": "mypassword",
      "host": "localhost",
      "port": 5432,
      "database": "mydatabase"
    }

Using runtime variables as first-class objects is not ideal due to the varied security requirements and formats of these variables. Instead, we can designate an initialization port init to initialize the specification. Using the load hook, a request is sent to the init port when the symbol is loaded, creating a packet based on the symbol's specification. The response is used as environment variables during the initialization of the specification.

These environment variables, along with the target sub-specification, are rendered using a template engine to produce a complete specification, which is then compiled to the sub-node. This approach allows for flexible control over variables and lets users define the template format, enhancing extensibility. Additionally, the init process can define compilation dependencies across workflows.

Some specification parameters need to be obtained at runtime rather than compile time. If the specification itself does not support payload-based expressions, these variable specifications must be supported not only at the initialization stage but also during the runtime when packets are transmitted.

- kind: block
  specs:
    - kind: snippet
      language: cel
      code: self.data
    - kind: write
      filename: '{{ filename }}'

It may not be the most elegant solution.

this method requires an additional specification to wrap the variable specification. Since other nodes might reference the port during compilation, intermediate ports need to expose the internal node's ports externally, which can lead to performance overhead.

siyul-park / uniflow