fidelity / spock

spock is a framework that helps manage complex parameter configurations during research and development of Python applications
https://fidelity.github.io/spock/
Apache License 2.0
121 stars 13 forks source link

Environment Variable Resolver #254

Closed ncilfone closed 2 years ago

ncilfone commented 2 years ago

What does this PR do?

This adds a resolver for environment variables thus allowing the definition of simple spock parameters with the following syntax: ${spock.env:value, default} -- this will read the value from an env variable and fall back on the default if given. Currently supports only simple python types: float, int, string, bool (there are no future plans to support complex types as value and type resolution would require significant effort)

Additionally, this PR also implements the ability to annotate these resolvers. Currently implemented are inject and crypto.

Inject

This will 'inject' the original env variable definition back into the saved spock state when writing to file. For instance, if a parameter is defined as with the inject annotation (by adding .inject to the env annotation):

class Example:
    one: Optional[int] = "${spock.env.inject:DUMMY}"

DUMMY will be read from the environment variable and set to its actual value within the Spockspace, however when written to file instead of the read value being written, the original syntax will be written instead (thus still referencing the env variable and not fixing the value):

Example:
  one: ${spock.env.inject:DUMMY}

Crypto

Sometimes these env variables or other given variables within a spock config might be sensitive information (i.e. a lot of cloud infra uses env variables that might contain passwords, internal DNS domains, etc.). Therefore, the crypto annotation (by adding .crypto to a resolver) provides a simple way to hide these sensitive variables while still maintaining the written/loadable state of the spock config by using the cryptography package (via a salt and key) to 'encrypt' these sensitive values and prevent them from being stored in plaintext.

For instance, here you can add the .crypto annotation to the spock.env notation to indicate that this variable should be 'encrypted' when writing to file.

@spock
class Example:
    two: Optional[str] = "${spock.env.crypto:DUMMY,Foo}"

Producing...

Example:
  two: "${spock.crypto:gAAAAABiesGutuzXLyE-jICdI3gFANcWydM2MhK9-WmB73wi4daP38gu1jmlCv-FjiP54Mv9cqT5YQdMn4qFrRg32-wzeZmOYAAm5uKmBXHWx6pxa70BO8c=}"

This will also dump a *.spock.cfg.salt.yaml *.spock.cfg.key.yaml to file with the same UUID which can be used to 'decrypt' these values within python code.

The SpockBuilder class now takes key and salt arguments which can be paths to the salt and key yaml files, direct values of the key (ByteString) and salt (string), or env resolvers to the salt and key (e.g. '${spock.env:SALT}') which will automatically 'decrypt' the values when building the Spockspace

Partial implementation of discussion in #243.

Bonus addition, saving the spock config with the extra_info flag now dumps all the currently installed packages within a commented block at the end of the file.

Checklist

coveralls commented 2 years ago

Pull Request Test Coverage Report for Build 2340445829

Warning: This coverage report may be inaccurate.

This pull request's base commit is no longer the HEAD commit of its target branch. This means it includes changes from outside the original pull request, including, potentially, unrelated coverage changes.

Details


Changes Missing Coverage Covered Lines Changed/Added Lines %
spock/backend/resolvers.py 101 102 99.02%
spock/backend/field_handlers.py 68 70 97.14%
spock/builder.py 46 48 95.83%
spock/handlers.py 45 48 93.75%
<!-- Total: 337 345 97.68% -->
Totals Coverage Status
Change from base Build 2240354546: 0.4%
Covered Lines: 2240
Relevant Lines: 2331

💛 - Coveralls