How should we handle secrets in flow.yaml or its referenced components

[dbkegley]

Certain fields in our flows will require setting sensitive values like AWS secret keys for AWS credential controllers or ListS3 processors. How can we handle these sensitive properties so that they are not stored in plain text and are not propagated elsewhere in a way that is not transparent to the user

[dbkegley]

As a temporary workaround for now, I have implemented an env_var custom jinja filter which can be used like this:

defaults:
  aws_access_key: ''
  aws_secret_key: ''

process_group:
- name: list-s3
  type: processor
  config:
    package_id: org.apache.nifi.processors.aws.s3.ListS3
    properties:
      Access Key: "{{ aws_access_key | env_var('AWS_ACCESS_KEY') }}"
      Secret Key: "{{ aws_secret_key | env_var('AWS_SECRET_KEY') }}"

aws_secret_key is given a default value of empty string, if the environment variable AWS_SECRET_KEY is defined in the environment of the process running flowlib --flow-yaml ... then the environment variable value is used, otherwise the default is used.

[done] TODO: It would be useful to allow the env_var filter to be used in the vars field of a process_group element

- name: write-to-s3
  type: process_group
  component_ref: common/s3_write_with_retry.yaml
  vars:
    aws_access_key: "{{ '' | env_var('AWS_ACCESS_KEY') }}"
    aws_secret_key: "{{ '' | env_var('AWS_SECRET_KEY') }}"

[dbkegley]

See parameter contexts: https://cwiki.apache.org/confluence/display/NIFI/Better+Support+for+Parameterizing+Flows

[dbkegley]

I think the env lookup is sufficient for this until parameter contexts are introduced in 1.10 We will have more work to do once that's released so I'm closing this for now