TrivadisPF / platys-modern-data-platform

Support for generating modern platforms dynamically with services such as Kafka, Spark, Streamsets, HDFS, ....
Apache License 2.0
72 stars 15 forks source link

Add support for externalizing Password into environment variables #920

Open gschmutz opened 1 month ago

gschmutz commented 1 month ago

Currently passwords are configured in the config.yml file and by that are visible inside the docker-compose.yml file. This is fine for most PoC environments, but we should also support an alternative where these passwords can be defined as environment variables.

gschmutz commented 1 month ago

We can use the pwgen utility to generate passwords:

Usage: pwgen [ OPTIONS ] [ pw_length ] [ num_pw ]

Options supported by pwgen:
  -c or --capitalize
    Include at least one capital letter in the password
  -A or --no-capitalize
    Don't include capital letters in the password
  -n or --numerals
    Include at least one number in the password
  -0 or --no-numerals
    Don't include numbers in the password
  -y or --symbols
    Include at least one special symbol in the password
  -r <chars> or --remove-chars=<chars>
    Remove characters from the set of characters to generate passwords
  -s or --secure
    Generate completely random passwords
  -B or --ambiguous
    Don't include ambiguous characters in the password
  -h or --help
    Print a help message
  -H or --sha1=path/to/file[#seed]
    Use sha1 hash of given file as a (not so) random generator
  -C
    Print the generated passwords in columns
  -1
    Don't print the generated passwords in columns
  -v or --no-vowels
    Do not use any vowels so as to avoid accidental nasty words
gschmutz commented 1 month ago

Initial jinja template to extract all the password env variables from the labels com.platys.password.envvars:

{% for service in services.items() | sort() %}
 {%- if service[1].init is not defined %}
{%- if service[1].labels and service[1].labels['com.platys.password.envvars'] is defined -%} {% for var in service[1].labels['com.platys.password.envvars'].split(',') %}
 {{-var}} = abc123!
{% endfor %}
 {%- endif -%}  
{%- endif -%}  
{% endfor %}
gschmutz commented 1 month ago

The goal should be to offer a feature to remove all passwords from the docker-compose.yml file. One option could be to add the passwords as environment variables with the user deciding how and where to store it.

Instead of using directly the property for a password

AWS_SECRET_ACCESS_KEY: {{MINIO_secret_key}}

we use the environment

AWS_SECRET_ACCESS_KEY: ${PLATYS_AWS_SECRET_ACCESS_KEY:-{{MINIO_secret_key}}}

And add a label with all the password env variables used in the service

      com.platys.password.envvars: "PLATYS_AWS_SECRET_ACCESS_KEY,PLATYS_XXXX"