Managing secret keys

[mnapoli]

The goal of this task is to make it easy to configure applications with secret keys. That may be through documentation or through tooling.

See https://serverless.com/blog/serverless-secrets-api-keys/ because that will help.

[bubba-h57]

We wrote a composer package https://github.com/stechstudio/laravel-env-security to manage secrets in Laravel. It would be easy enough to port that over to some bref specific tooling.

[sandrokeil]

I don't like env files but it could be reasonable for some projects. I use and prefer a kind of Managing Secrets for Larger Projects and Teams. Is it possible to store a config cache file at runtime in Lambda and it's populated for the next request? This has the advantage to only load "once" the secret keys.

With this, you can use the get_secret function in your PHP config file and if config cache file is available no AWS SSM API call is needed. See zend-config-aggregator for an example.

return [
   'my_secret' => get_secret('MY_SECRET_AWS_KEY'),
]

For development it is useful to use getenv in the get_secret() method to allow also to inject env variables. If no env variable is available a AWS SSM API call is executed. With this approach we have two possibilities combined.

[mnapoli]

I was more thinking about documenting how to use AWS secret manager (as well as another similar tool I forgot the name). These are the recommended way to store and access secrets in Lambda.

[bubba-h57]

I see. That turned out to be expensive and not cost effective for us.

It cost $0.025 per 1,000 API calls to SSM. It cost 1 API call to decrypt to one parameter. We average about 6 secrets that need encryption per Lambda Function. We average 100,000 Lambda executions a day.

Worst case scenario, that is 600,000 SSM API calls per day. 600 * $0.025 = an average cost of $15/day or $450/mnth

We were not seeing the worst case scenario, we did some caching, and tried to work around with it. But ultimately, it cost us far too much because our workload is not consistent throughout the day, and then I run the various bits of work in async parallel across multiple differing Lambda functions. AWS has expanded our concurrent jobs to 20K, and most of the work we do is in spikes of 10-15K jobs in parallel for 30 minutes or so and it can be an hour before we spike again. That means the majority are cold starts.

So we went with AWS KMS instead and found it much more cost efficient.

For one thing, it only costs $0.03 per 10,000 requests.

As we see it, the simplest way that KMS is used to better secure secrets in Lambda projects:

1. A customer master key (CMK) is generated in KMS, with access only to production lambda functions (an IAM role) and the administrator account
2. All static secrets are stored in an encrypted .env file.
3. These .env files containing encrypted secrets (e.g. DB password, mail credentials) as well as any unencrypted non-secrets (e.g. DB user, DB host, etc) can then even be committed to your favorite version control system, if desired.
4. Lambda bootstrap function is modified so that it uses the KMS decrypt function (part of AWS SDK) on the encrypted .env file and then exports the values to environment variables, ensuring unencrypted secrets are never stored anywhere but the memory.
5. Because the bootstrap is doing the decryption only on cold starts, the values are effectively cached in memory, only costing 1 API call per cold start for all environment variables.

It is important to us that the secrets are never stored on disk, anywhere, at any time, unencrypted. It is also important to us to minimize the cost in API calls as well as the performance impact of decryption.

Thus, we came up with a way to manage encrypted .env files for this purpose.

I supose for smaller Lambda projects that only make a few 1,000 API calls a month, SSM could be cost effective. That is, simple enough to implement that it is worth the few pennies to use.

However, would never use a solution that writes the secrets to disk, even in the lambda environment, because it is critical security flaw that would result in Financial, Governmental, and Healthcare (and anyone else that requires rigorous security practices) institutions could never leverage it.

Addmittedly, there are always multiple ways to solve this sort of problem, and if we can come up with a more cost effective method, without sacrificing security, I will implement it as fast as possible. If for not other reason than it would have a positive impact on Signature Tech Studio's bottom line. :-)

[mnapoli]

Thank you that is super useful! That shows that we definitely need a guide for all of that 😄

[sandrokeil]

To reduce SSM costs we can query 10 secrets at once via GetParametersByPath. Your security concerns are reasonable.

Here is an example to store the variables in the env vars so you can use getenv() in you config files or elsewhere.

$ssmPath = '[Your environment SSM path e.g. /production/awesome-app/]';

$ssm = new \Aws\Ssm\SsmClient([
    'region' => getenv('AWS_REGION'),
    'version' => getenv('SSM_VERSION'),
]);

$options = [
    'MaxResults' => 10,
    'Path' => $ssmPath,
    'Recursive' => true,
    'WithDecryption' => true,
];

do {
    $result = $ssm->getParametersByPath($options);

    $options['NextToken'] = $result->get('NextToken');

    foreach ($result->get('Parameters') as $parameter) {
        putenv(substr($parameter['Name'], strrpos($parameter['Name'], '/') + 1) . '=' . $parameter['Value']);
    }
} while ($result->get('NextToken'));

[bubba-h57]

GetParametersByPath ... that does put the costs back on par! As long as we have ten or less secret variable to manage, the performance would be on par as well (1 API call). I dig it.

[mnapoli]

Some notes:

• AWS SSM Parameter Store is a sub-service of "SSM" and it allows to store values

  Cost: free

• AWS Secrets Manager is basically the same thing with more features:

  • audit log of changes
  • auto-renewal of secrets
  • finer access control

  Cost: $0.4/month per secret + $0.05 per 10,000 API calls.

The fixed cost of Secrets Manager is too bad because it was a very good solution. For bigger projects it might be interesting.

Here is a solution that doesn't require the Lambda to fetch the secrets on every execution, or even on every cold start. This is a way to import the value in template.yaml:

            Environment:
                Variables:
                    FOO: '{{resolve:ssm:MY_PARAMETER:1}}'

Note that 1 is the version number.

The downside with SSM is that we must create one parameter per environment variable.

With Secrets Manager we can create 1 secret and store multiple values inside via JSON. Then we can reference them using the : as a separator for a "JSON query":

            Environment:
                Variables:
                    FOO: '{{resolve:secretsmanager:MY_PARAMETER:FOO:BAR}}'

No need to set a version as well.

---

Note as well that with this technique environment variables are fixed on deploy. They can't be changed dynamically: we must redeploy the application. (note that this is my interpretation and I haven't tested this further)

[mnapoli]

I am trying to define a parameter in template.yaml and reference it in the environment variables, but that doesn't work. For reference I have asked on StackOverflow: https://stackoverflow.com/questions/55286991/how-to-define-and-use-at-the-same-time-a-ssm-parameter-in-cloudformation

[mnapoli]

I have opened #277 to solve this issue.

In the future we may want to document how to fetch secrets from PHP. That would allow to update configuration values without having to redeploy the application. One step at a time though!