STDIN Shell Provider

[brandonwestcott]

Just came across this tool, super great. Been in devops for a few years and this definitely solves a need. Love the support for multiple providers within a single invocation.

I'm personally interested in gopass (#11) for some local only api keys. While I fully appreciate first class integration with providers, I'm wondering if #3 #10 #11 (and other future requests) could all be achieved if there was a generic shell provider. Using that provider would just invoke that shell script with a known STDIN api, ex [command] [path] [options].

[lreuven]

@brandonwestcott Hi 👋 Apologizes for the late replay, Can you share a bit more about this ER? not sure how teller will fits into this use case.

[jondot]

I think I got it, the original request is about making a generic adapter for any shell executable -- it's in my backlog

[brandonwestcott]

@jondot yes exactly. Quick escape hatch for devs to use other tools that aren't yet supported providers.

[jondot]

First design:

  cli:
    env_sync:
      path: '' # fmt k=v, ignore empty or no '='. also first '=' counts. take a splitter as optional config
      meta: env
    env:
      FOO_BAR:
        path: foo/bar/baz
        meta: echo ${{path|key}}  # fmt v, also take first line

1. Interchange format was hard. Eventually, go with the environment format (k=v) (and not impose structuring JSONs etc). Specific key just expects value on STDOUT
2. Concluded that there must be an introduction of a new 'meta' key. For CLI provider this will be the command. a meta key is also a gotemplate, and we will provide it with the entry itself for substitution.
3. Cmd execution is simple exec

@kaplanelad WDYT? got some feedback? specifically on the interchange format or new meta key

[jondot]

Example case: gopass

---

sync:

$ gopass list -f
foo

🚨 not good, shows only key list

some CLI golf we get at:

$ gopass list -f | xargs gopass show -o
1234

🚨 not there yet, shows a list of values but we lose the key

---

get:

$ gopass show -o foo
1234

✅ works according to assumption

---

General challenges:

• Unattended, requires interactive user input of GPG password to unlock the key every X time.

[jondot]

Final design:

• full sync get: end CLI should produce k=v mapping, if not existing, the end user should create a shell script/wrapper script to generate it
• get: easy to do, as described above
• add a meta key to allow for flexible CLI running

Configurable:

• input format (maybe we can allow more than k=v ?), or regex input format

Tradeoffs:

• CLI provider cannot know about the specific CLI that's running
• Also, cannot know about the format that its about to accept, other than one specific format
• Everything that doesn't mold into that, the end user should create a shell script / wrapper to run instead

[jondot]

For now finalizing the design but parking implementation. This is good in theory but then to provide value end users would need to do too much work (write wrapper scripts, deal with standardizing a certain CLI) and it's better that we build "tailored" providers per tool.

[brandonwestcott]

@jondot thanks for putting in some mind work toward this.

I definitely agree first class providers are a MUCH better user experience and should be the default answer for tools with any meaningful usage.

I will also note I see value in this feature for complex use cases, such as using aws-vault where you are populating secrets from two different roles. In this example, you can't just prefix teller run with aws-vault, because you need the AWS credentials at the secret / provider level. Having a shell wrapper would unblock that use case, albeit not as elegantly as a first class provider.

[jondot]

@brandonwestcott thanks -- feedback like yours really helps prioritizing, actually, we're doing a public steering every second week, hopefully soon we can open it to the public. I can share that we really went back and forth with this one. What can help - is that maybe we can ask you a few question about this, specifically:

1. We've identified that some CLI tools need human interaction (enter your master password for example), and that's tricky to automate
2. Some tools won't really "deliver" the settings in a straightforward manner (gopass as an example), and will require a user to do some shell scripts on their side to hand Teller a format that's compatible (say a .env format, lines with KEY=VAL). We were wondering if end users would actually invest the time in doing this
3. Happy to get a specific use case you'd have, say aws-vault you mention. Can you maybe give some more detail on the blockers you found?

[brandonwestcott]

Thanks for the insight @jondot. I will say, I see this as a "power user" type feature, and deserves to be prioritized lower than core features / additional providers. That being said, I would love to have it 😄

1) I think the majority of the cases I'm seeing would include human interaction (master password / Touch ID). I understand programs that expect stdin input for that would be tricky, but others (such as aws-vault) will prompt outside of terminal 2) I would agree that most users wouldn't invest time, however my experience would say existing secret environments aren't always in a clean standardized format that would be covered by the existing providers. In a larger org, I could see a devops / secops team investing in these types of scripts for their unique situations. 3) aws-vault stores aws api keys in keychain and then requests session token from STS and injects that into the env. On my current infrastructure, we have multiple AWS accounts, which means invoking aws-vault multiple times for each account (aws api creds are distributed via sub account IAM roles rather than root account IAM roles). This results in having multiple AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY / AWS_SESSION_TOKEN that need to be injected depending on which secret is being accessed. I'm already writing scripts to manage that process, so writing that in the convention of a standardized tool like Teller seems great.

Also from my perspective, in these advanced cases I think the main usage is reading of secrets, not writing / updating. As it relates to 2 above, I personally would not go through the effort of making a custom script support the write ops. I realize having some providers in "read-only" mode is a a bit odd though.

All in all, I think for new applications / environments this isn't valuable - one should just conform to existing teller providers. For existing environments, I foresee people either having to adjust their secret storage strategy to match providers supported in teller, or they skip on teller for those projects because there isn't a custom escape hatch like this.

Thanks again for all the great work on this project, I can see easily see teller becoming the "terraform of secrets" !

[npalladium]

Love the tool! This solves a pain point that I've been patching with shell scripts. Kudos and thanks for the amazing work.

To add one more data point, I would definitely use this feature and build scripts around it if needed (maybe using expect).