Proposal: Programmatic "secrets" API

We need a way to encrypt secret strings securely in various environments.

Previously, the only way to encrypt secrets on fly was to use the fly secrets CLI and "inject" the secret names in your config. We stored each value encrypted via KMS and distributed everywhere.

Shortcomings

[Technical] Very slow: decrypting these values takes minimum 50ms (up to 1s!) via the AWS API. Tempting to cache the plaintext results, making this less secure.
[UX] Awkward workflow: we need the secrets before we can deploy an app. Adding secrets also count as a release. Therefore the first release has no code.
[UX] Error prone: users sometimes forget they need to update secrets since adding secrets is disconnected from producing code.
[UX] Awkward API: how we merge the secrets into the config is not very intuitive and requires a specific format.

Possible solution: Asymmetric public key cryptography

I'm a big fan of how Travis CI does it. I think it's the right tradeoff between convenience and security.

Using public key encryption allows putting encrypted strings directly in code while keeping the values secret.

Workflow example

CLI: fly encrypt [-f <filename>] [text] returns an encoded string (could be base64, but I took a liking to base58 recently)
Copy/paste this string anywhere in code, usually assigning it to a variable. Could be in a .json file of some kind that's imported.
Code: fly.decrypt(str) returns the plaintext value

In production, this means we generate a private & public key for their app (like travis does), encrypt it using AWS KMS (or similar) and store it for distribution. Our API can return the public key and the binary can encrypt the value (or we can encrypt from the API.) If the project is not hosted with fly.io, this can be setup as an environment variable (we need a trait for how these can be retrieved) or stored in a manner similar to ours. Hell, it could just use KMS directly with their own AWS access and secret keys if they don't mind the slow path and putting encrypted ciphertext blobs in their code.

Locally, this would be a different key living close to a $HOME directory or to the project. Maybe it's something developers within the same company can share so they benefit from the same values everywhere. Encrypted values can be checked in source control.

It should be possible to cycle the key in the case of a breach. This would be mitigated by encrypting private keys with AWS KMS and never distributing private keys. Not sure if private keys should be distributed to allow for the same public keys in every environment. Sounds a convenience with a security cost. If they can be cycled easily, that might be "ok".

I think that it might work well to use "fake modules". I.E.

import { someSecret } from "@some-secret-namespace/some-secret-store";

connectToSomething({ username: "user", password: someSecret });

This should be pretty easy to implement just add some additional logic into module resolution, and generate a modules from secrets at module resolution.

This solution makes a lot of sense since secrets are really just dependencies that require special treatment. Since module dependencies can be determined and resolved before execution, This would allow you to do things like:

Check if the secret dependencies can be resolved in a environment before even trying to run code or deploy code to production.
Pull secrets from deployment machine during the deployment process and check the required ones exist before deploying.
Pull secrets from a cloud environment to a dev environment for testing.
Resolve and decrypt secrets before runtime instead of waiting during runtime.

superfly / fly.rs