Interpolate Amazon SSM parameters into configuration file

ckan / ideas

[DEPRECATED] Use the main CKAN repo Discussions instead:

https://github.com/ckan/ckan/discussions

40 stars 2 forks source link

Interpolate Amazon SSM parameters into configuration file #225

Closed ThrawnCA closed 4 years ago

ThrawnCA commented 5 years ago

We run a CKAN stack on Amazon, and it would be nice if we could interpolate AWS Systems Manager Parameter Store values into the config file; we could get all our passwords off the disk, rotate them easily, etc.

Would this be best achieved by making a custom WSGI file, which would use the 'boto' library to load an SSM tree and pass that to the app? Or perhaps a CKAN extension that would intercept all config values at startup and replace placeholders at that point? Ideally we would want to interpolate after all config plugins have added their values, but before the values are consumed, which makes me think an extension is the best choice, but I can't help noticing that the config file already has interpolation available, and it seems a shame to reinvent that wheel.

metaodi commented 5 years ago

Maybe ckanext-envvars is already doing what you need?

If not, you could create a plugin that implements the IConfigurer interface and replace all values loaded via boto. Just make sure this plugin is the last one loaded (i.e. the last entry in ckan.plugins) so you get all config values.

ThrawnCA commented 5 years ago

Hmm. Thanks for pointing out envvars. We could probably do something with it, but for our use case I'm not sure it has any real advantage over just importing boto into the WSGI file and using paste interpolation.

One useful point that was made on the envvars repo, though: can an extension update the SQLAlchemy URL early enough, before it's used? If not, then a custom WSGI file may be the only feasible approach.

ThrawnCA commented 4 years ago

Draft implementation is up at https://github.com/qld-gov-au/ckanext-ssm-config/tree/develop and it appears to basically work.

More refinement is needed eg:

Make the region configurable
Provide an option to configure an AWS key instead of relying on IAM roles
Handle whitespace inside placeholders
Optionally restrict the list of parameters that can be interpolated
Retrieve parameters in batches instead of individually, for better performance

ThrawnCA commented 4 years ago

ckanext-ssm-config has had a 0.0.1 release. Extra features 1, 2, and 5 from the above comment have been implemented.

I'll close this issue.