How would you suggest setting secrets?

[jacobthemyth]

Hi, this looks like an interesting project and our team is investigating whether it's a good fit for our workflow. We're wondering how you would suggest giving developers access to change secrets though, since this tool seems to only read from the secret stores. When using it, do you give developers direct access to the secret store to update values? Thanks!

[jondot]

Great question @systemist Currently we are doing two things for security reasons:

1. We don't have an opinion on how to authenticate -- meaning, Teller will not ask you for credentials for your secret stores, but it will basically "assume connected" and federate the authentication to how it currently works on your machine. A good example would be anything based on AWS, where almost every AWS SDK "knows" where your authentication details are and knows how to connect to it. When Teller uses this SDK, it will just ask it to connect the best way it knows.
2. We're restricting to read only, and so -- it would mean lesser privileges, which is naturally healthy. We are assuming the way secrets enter the secret store is: automated/by secops/by a devops person in the org with the specific tooling of the store. However we've gotten a few request to "push" values back to the stores from Teller itself -- so we're exploring a push interface.

For the push interface -- I'm happy to get more details especially for your use case so we can build it right 😄

[rcII]

Thanks @jondot. I'll step in here and add my two cents. The basic use case would be the following:

• Ability to set a secret on multiple providers at once
• Detect "drift" between providers
• Ability to sync secrets between providers when drift occurs, sort of like resolving a merge conflict

[jacobthemyth]

And for what it’s worth, our team (including @rcII) would be happy to work on an implementation but wanted to see if you had any existing designs so we didn’t just create a fork that’s only a good fit for us. Our current workflow is to give all devs access to set configs (including secrets) on Heroku but only have DevOps/SecOps have access to AWS secrets and Vault, but want to unify a single developer friendly workflow as Raney mentioned.

[jondot]

Yea I think actually when you draw out the use case this way it's really exciting! since we automate the source, we don't get drifts much and we were looking on rotation as a use case (update in all providers at once because a key has been exposed/stale).

I love that you call it "drift", makes complete sense.

In terms of implementation -- yea happy to accept any suggestion/PR you might have. Basically since we're dealing with what is essentially a K/V store, it's just to map back a value to the key, and push via the relevant provider API (all APIs should support this). Because authority and authentication is federated, we don't care about roles (either user has permissions or they hasn't, but its not for us to decide).

In terms of user experience, that's where it would be more interesting. How would you suggest interfacing this? For example:

setting secrets

teller put [provider] [KEY] [value]
(remember, path is in the configuration file so there's a small win here to save typing)

or maybe you do want to provide a specific path:

teller put [provider,provider2] [path] [value]

and then maybe we can map a complete file? (but it's kind of dangerous to encourage people to keep files with secrets on disk)

teller put [provider,provider2] [env_file]

and then, maybe put on all providers this way?

teller put-all [key] value

drift detection

teller drift provider1 provider2 provider3

Or maybe check a key drift?

teller drift K1 provider1 provider2

sync

Assumption: there's always one source and many sinks and we want to "sync from source to sink":

teller sync [source provider] [dest provider1, dest provider2] --exclude [KEY]

And just for usefulness include an --exclude/--include logic for the sync, optional.

What do you think? can you draw out your perfect interface in your mind?

[jondot]

One more idea after we've discussed it internally, is to encode all this in the configuration itself. So basically we want to model:

[ ] handle/id (a secret has a unique handle, although it may appear in multiple stores), the handle is used to do matching between keys/secrets in multiple stores. so basically this means: add one more property to configuration [ ] again in the configuration, mark up relevant secrets as origin, meaning this way we can automatically infer where to grab the source data and sync it into any other provider that's has a key/secret that isn't marked as origin

So maybe something like this:

hashicorp_vault:
  env:
    SMTP_PASS:
      path: secret/data/demo/wordpress
      field: smtp
      origin: true                          # this means the Vault value is always the source
      handle: demo-smtp-pass # this will help "link" both secrets regardless of path, key structure

other_store:
  env:
    SMTP_PASS_FOO:
      path: smtp_pass_foo
      handle: demo-smtp-pass

other_store_not_in_the_sync_game:
  env:
    SMTP_PASS_FOO:
      path: smtp_pass_foo

And now we can apply a fix, because from configuration, teller should know everything it needs to

teller drift --fix

same for this:

teller sync

[rcII]

Thanks for the brainstorming and potential implementation criteria @jondot! The one potentially complicated use case that may become more common than not is when organizations separate out secret and non-secret values into Vault (secret) and Consul (non-secret). In this case, a user would need to be able to set a source for two discrete sets of KVs, one secret and one non-secret. However, maybe we don't need to disambiguate between secret and non-secret since the providers handle the encryption portion of the story. I understand that teller was built around secrets and not non-secret configs, so maybe this is a use case that teller shouldn't support by nature. Also, this particular use case is most likely specific to Vault/Consul being used in conjunction. The most useful pieces of your descriptions above would be the following:

1. Set a source and/or sources in a config file. (i.e. Vault and Consul as sources of truth)
2. Have a CLI interface that can put a specific secret (i.e.teller put [provider] [KEY] [value]) or KV config to a provider or multiple providers (put-all)
3. Drift detection and remediation (teller drift provider1 provider2 provider3 and teller sync [source provider] [dest provider1, dest provider2]

Since most app configs contain secret and non-secret values, it would be a nice-to-have to control them all under one tool, but we could easily chunk them into a single source. This may make the user experience a bit rough however, as everything would be redacted in the teller output.

[rcII]

Or maybe one source per path...

[jondot]

Got it. Sounds good, in terms of direction -- yea, I feel that Teller controls "values" and "secrets" are just one specific case that has a couple interesting use cases (redaction, scanning) -- by all means it was built to have this dual roles (you can mark KV as "non sensitive" actually).

So here's my initial sketch, which I'm going to try and play with. While sketching it out, we have another use case of "sync two providers of the same kind, like consul <> consul or vault <> vault",

So the general direction had to be something like this (to be the ultimate and most flexible):

providers:
  dotenv:
    env_sync:
      path: ~/my-dot-env.env
      handle: my-dot-env
      source: true
    env:
      FOO_BAR:
        path: ~/my-dot-env.env
        redact_with: "**FOOBAR**" # optional
        source: true
        handle: foo-bar

providers_backup:
  dotenv:
    config: { ..}
    env_sync:
      path: ~/env_backup.env
      handle: my-dot-env
    env:
      FOO_BAR:
        path: ~/env_backup_2.env
        redact_with: "**FOOBAR**" # optional
        handle: foo-bar

And now you can pit a provider list against a provider list, and you're free to "mark out" the lists, you can have:

1. source and backup
2. various environments (dev, stg, prod)

And you can "link" KVs with a handle and mark one as source while the other, if not marked as source, would be the destination.

And in this way you can create a "web" of connections between various providers in different groups, if KVs were not linked, they won't be actually playing in the sync/drift game.

In terms of interface, it would become easy:

teller sync providers providers_backup
teller drift providers_dev providers_stg

And the "regular" use cases remain:

teller run --providers providers_dev

In terms of "put", I'm thinking two directions:

1. Just keep the teller put <...> interface for verbatim input
2. And/or have actually a source provider (ENV file) where you drop stuff, and, with the new capability just teller sync source_providers providers_production

So in practice this story starts with some elbow grease to build the handle/link infra, and then add some write ability to providers which I'm going to start. After this we could still shape up the interface.

[jondot]

First results in:

And then:

Details

• Looks like no need for handle, we're going to use the key itself and source: true is enough to signal a source<>target pair
• teller.Drift() will work on its internally collected entries and return a DriftEntry{ Source, Target } which can later be used for sync easily
• Added a configuration option of kind: <provider type> for each provider now, minimal change to allow for multiple providers in the same providers list. We now have Provider which is the type and ProviderName which is a user-given name for the provider (in the example: dotfiles and dotfiles2)

Teller works on a single providers list which outputs a single, long, entry list from all providers. This entry list can be use for run, show, .. and now drift.

Future work

I already see a need to separate environments, which this new change supports by using multi-file drift check, for example:

teller drift config1.yml config2.yml

And we can check between two Teller instances:

teller.DriftWith(teller2)

Next steps I'll polish out the drift feature, which is usable already, and then it's all about the write features:

• put
• sync

[rcII]

Thanks for this @jondot. This initial spike on the drift feature looks promising! Please let us know if you want help on the put/sync features. My Go is rusty, but I could probably jump in given some guidelines! As soon as this is ready, we will try it out in our staging environment.

[jondot]

No worries, I think what I'll start doing is the skeleton of the 'write' workflow (data model and refactoring) and then update back here

[jondot]

Hi all, happy to introduce the write/put interface, you can jump to the README updates here to see the addition from a user's point of view.

https://github.com/SpectralOps/teller/blob/ee791c2eeea68c9af5bb0c2b5a857023eaac46f5/README.md#bike-multi-write-rotation--sync

In terms of implementation, everything is in place to easily implement a drift fixing feature -- just waiting for some comments about the put interface if there are any

[rcII]

Thanks @jondot, we will do some preliminary testing with Heroku, Vault, and Consul. This looks great!

[rcII]

I was able to set a secret in a single provider (vault)

➜  teller git:(put-interface) teller put TESTING_TELLER=true --path app/data/config --providers hashicorp_vault -c  ~/.teller.yml
Put TESTING_TELLER (app/data/config) in hashicorp_vault: OK.

But unable to put in multiple providers

➜  teller git:(put-interface) teller put TESTING_TELLER_1=true --path app/data/config --providers hashicorp_vault,heroku -c  ~/.teller.yml
Put TESTING_TELLER_1 (app/data/config) in hashicorp_vault: OK.
Error: cannot put TESTING_TELLER_1 in provider heroku: Patch "https://api.heroku.com/apps/app/data/config/config-vars": The requested API endpoint was not found. Are you using the right HTTP verb (i.e. `GET` vs. `POST`), and did you specify your intended version with the `Accept` header?

I suspect this is because the path semantics are different between the two providers? Vault follows a true "path" format, while Heroku's "path" is actually the name of the app that is being configured. I may be doing something completely wrong as well.

.teller.yml :

project: newproject

# Set this if you want to carry over parent process' environment variables
# carry_env: true 

#
# Variables
#
# Feel free to add options here to be used as a variable throughout
# paths.
#
opts:
  region: env:AWS_REGION    # you can get env variables with the 'env:' prefix
  stage: development

#
# Providers
#
providers:
  # requires an API key in: HEROKU_API_KEY (you can fetch yours from ~/.netrc)
  heroku:
  # sync a complete environment
    env_sync:
      path: $HEROKU_APP
      source: s1

  # # pick and choose variables
  # env:
  #   JVM_OPTS:
  #      path: drakula-demo

  # configure only from environment
  # https://github.com/hashicorp/vault/blob/api/v1.0.4/api/client.go#L28
  # this vars should not go through to the executing cmd
  hashicorp_vault:
    env_sync:
      path: app/data/config
      sink: s1

(NOTE: $HEROKU_APP is a placeholder in this snippet for the actual name of the app)

[jondot]

Yes you are correct, there are two ways to put

1. You "work through" the key-value path configuration in your teller.yml
2. You supply a path in a "direct mode" where you know there's no configuration defined (kind of like raw setting the value)

And yes in case (2) it makes less sense to set multiple providers, the way I think might be interesting to solve it is perhaps one of two (would be happy to get feedback here):

1. Restrict the put --path case to just one provider
2. Accept a full /path/to/FINAL_KEY=value in the case of put --path and then --path just becomes a bool indicating that the mapping includes a full path

What do you think?

Actually, scratch that, (2) is invalid because of 1-to-many semantics (leaving it here just for academic purposes). Sounds like we're left with just (1), which is restricting put --path to one provider at a time. Make sense?

[rcII]

If I am understanding correctly, a user can put to multiple providers at once, but their .teller.yml must be configured? Example real-life use case: A developer needs to set a configuration/secret on Heroku and Vault simultaneously to avoid environment disparity. Can this be accomplished via CLI flags, or must the teller.yml be configured correctly?

[jondot]

Correct. Think of it like the config file containing the coordinates of every secret or value and how it physically maps into the provider. If this information does not exist there eventually we have to somehow provide it (it may include the path but optionally additional bits of information that is provider specific such as encryption in the case of AWS based stores)

[rcII]

Thanks @jondot. From initial testing, the put functionality seems to be working as expected. Thanks for implementing this. Once the drift-fixing feature is place, we would like to start testing within our org at a wider scale. Would there be a way to run the drift-fix as daemon somewhere? Example use case would be:

• Developer makes a change to Heroku config
• drift-fix detects the change and syncs the sink provider(s) We could run this in a container to monitor changes.

[jondot]

Great @rcII, yes I believe that's easily done. We can add a --watch flag that's commonly done in these kind of tools. I would go ahead and add in some kind of standard logging/eventing subsystem so you could get notified of these, I'll look around to see what isn't so heavy to use, and is pluggable (maybe enough to pick a good logging library that can have some plugins to connect to).

[voodooGQ]

Hey @jondot. Just wanted to let you know we believe we now have a path to success with your additions! We're pretty sure we can get around needing the daemon functionality for the time being but would still be a neat addition to the library. Great work and thank you for all your help with this!

[kaplanelad]

I'm closing this issue. You are more than welcome to open new issues with new thoughts for future discussions :).