Separate pack config from code

[michaelansel]

Scenario:

• You have an awesome pack that needs an API key

Options:

• You commit the API key in config.yaml in your pack repo (a fork if you aren't the maintainer)
  • Problem: You now need to keep up with the fork and continually merge upstream into your fork.
  • Problem: You are committing secrets into a system not designed for secrets (git ain't encrypted).
• The pack maintainer kindly wrote all the special code necessary to read from the datastore.
  • Problem: Writing "special code" over and over for every single pack is pretty lame.
  • Problem: Not every pack maintainer is going to do it.
  • Problem: Not every pack maintainer is going to do it the same.
• The pack maintainer added a config action that self-modifies the config (a la #2196)
  • Problem: No reproducibility of your st2 deployment
  • Problem: Upgrading a pack destroys your config

Pack configuration regularly contains deployment-specific values and secrets, and as a standard pattern code and configuration should be able to be maintained separately (if you desire it). Having something like the above linked AWS example (read from the datastore and fallback to the on-disk, in-pack config file) would make this a lot better. Naming should be simple and predictable, but we also shouldn't prevent shared configs (e.g. common credentials shared by multiple packs).

Ideally, there would also be support for an on-disk config file since the primary purpose of the KV store (and everything in Mongo) is transient data (or at least that is my understanding); on-disk config also allows for near-infinite extensibility to work with external secrets management systems (Conjur, Vault, etc.).

cc #1832 #2196

[manasdk]

Few points -

1. I believe that vault or some other secrets management will be a natural pattern to evolve with credentials and using in workflows.
2. StackStorm might even look at making a the KV store pluggable with something secure. Ideally, its some opensource tool that manages secrets as its core-competency. It would be my personal preference to not push StackStorm into secrets management but who knows how that comes about.
3. I have been thinking of a notion of partial packs. Basically, a way to define a pack fragment that can supplement the original pack from github. This way there can be a install specific config overlays coming from a pack fragment.

Most importantly thanks for the thoughtful issue and certainly this is one of our burning concerns :)

[pixelrebel]

For those who prefer to use config.yaml, these values can't be referenced inside a rule, simple yaml action, or action chain.

For example, I have a slack channel that will be the hub for the pack’s communication. So it's a common parameter I’m setting for many actions.

If this were set in the datastore, you could access this with channel: "{{ system.mychannel }}". But there is no equivalent for file-based config variables.

It would be nice to access this variable in a similar way, perhaps: channel: "{{ mypack.mychannel }}"

[michaelansel]

The evolving picture in my brain is something like hiera - a config store that handles multiple backends and knows how to stack/prioritize duplicates across backends. That way, you could set defaults or "I'm just writing a simple pack. Shut up and let me hard code stuff." using config.yaml, use the web UI or API for customizing externally developed packs, and use an external config store (either consolidated yaml or secret store) for more advanced/prescriptive deployments.

[pixelrebel]

If we're working to simplify config access, the best solution from an end-user perspective would be:

channel: "{{ mypack.mychannel }}"

The engine should be smart enough to check the datastore and config.yaml and serve up the proper variable based on order of precedence.

Same goes for a script's config access:

self.mychannel = config['mychannel']

This should also serve up the proper variable based on a standard order of precedence.

[michaelansel]

I have been thinking of a notion of partial packs.

That's a very interesting idea. That could be a way to support injecting configs, but it feels kind of funky. Very much a container-style model, where you always wrap around someone else's image to build your own. Not necessarily wrong, but I don't see that as being the best way... :question: :question:

[manasdk]

I honestly think it is going to end up being varying degrees of all of the above. Really, at this point all of us are coming to this problem with different uses for config.

partial pack - I see cases where appending to a pack extends beyond config. The reason I find this idea useful is for source controlling the config as well. Pack is the way of delivering content to StackStorm and in some cases config itself can be content.

hiera like - This is actually another way to look at the problem. It is particularly useful since there are override semantics naturally built into the system.

Both the ideas can live together since one is about source and the other is about precedence.

At some point though there will be a very clear demarcation between types of config e.g. -

1. secrets - please do not save them in a config file. Use a secure store of some sort.
2. connection info - Seems like config is a perfectly reasonable place.
3. operation parameters - Better if these live super close to the code.

There are certain more categories and different are suited to come from different sources. Once we define clear rules of precedence and setup guidelines on what should live where I think it will be simpler to come up with the needed mechanism.

[Kami]

Alright everyone.

Feature, which among other things also solves this problem (separates pack configs from code) is now finally in a usable enough state so it can be tested end to end by the users (this feature will released in next stable release - StackStorm v1.5).

Initial documentation for new pack configuration feature is available at https://docs.stackstorm.com/latest/pack_configs.html (your feedback on what can be improved, what is not clear enough, etc. is of course very welcome).

In case you are interested, here is also a video which showcases and explains the feature a bit - https://www.youtube.com/watch?v=_4xn5b8xVlo (list of commands I used is here - https://gist.github.com/Kami/072eb1a42a2d57e45aabeae1ab94d0ce).

Keep in mind that this video was produced quickly just for internal consumption so it's kinda long, hasn't been edited and there are some internal puns (pardon the pun!).

Some time next week or so you can also expect a blog post on this new feature. This blog post will basically provide some background and context, explain how this new feature works (with examples) and provide information on how to migrate from existing configuration to a new one (hint - it's very easy, move existing config to the right place and run st2-register-content --register-configs).

In short - if that's something which has been bothering you or something you are interested in, please go over the docs, play and experiment with the feature a bit and let us know how it goes. Feedback is very important since it will allow us to improve this feature, docs, etc. and make it better and more user-friendly.

If the documentation is too dry and / or the video is too long and you prefer something in a more tutorial like form, please hang tight and wait for the blog post.

[arm4b]

This cool :tada: feature was released with 1.5.1 and described in: https://stackstorm.com/2016/06/30/improved-pack-configuration-user-scoped-datastore-items-secure-secrets-store/