Handling secrets

[synth]

The best way to handle secrets in the AWS world, to my knowledge, is using Systems Manager > Parameter Store.

When getting started, connecting to PS was pretty much my first task as it will link me into the rest of our AWS world basically: api endpoints, tokens, database endpoints/credentials, configuration settings, etc.

It would be great if there was first class support for accessing parameters in Parameter Store. From the AWS side, I figured out how to add ParameterStore following this article: https://aws.amazon.com/blogs/compute/sharing-secrets-with-aws-lambda-using-aws-systems-manager-parameter-store/. It wasn't too hard, but also is fairly boilerplate and so Imagine it can be baked into the cookie cutter project fairly easily.

In general, I imagine cookiecutter-ruby asking us for what modules to include - ParameterStore being one of them. It then puts in the requisite Cfn into the template and also gives you a ruby starter class to query PS for variables.

[synth]

Looks like this is handled by Lamby! https://github.com/customink/lamby/blob/master/lib/lamby/ssm_parameter_store.rb

• perhaps abstract this out so it's available by itself? In any case, I can probably adapt the code :)

[zhuqing662k]

I agree using SSM Parameter Store is a good way to handle secrets. However, there is a cost associated with accessing it. And in a lesser sense it may take a bit time each time we access the Parameter Store. I know the service is lightening fast. But still it is a web service request.

On the other hand, I hope we can put the secrets in CI/CD protected variables and during building/deployment we substitute the place-holders with the real secret values. I thought it earlier when I used this project's sister project for python. But I did not investigate further. So this is just a thought. I am not sure if this is feasible. Just my 2 cents.

[synth]

Fyi parameter store is free as long as you are using standard parameters and need a throughput of 40 queries/s or less.

In any case, you make good points. I think a hybrid approach of being able to use parameter store to set the env at build/deploy time could also be good.

[zhuqing662k]

@synth Thank you so much for the info! Yes, I looked it up - using the SSM Parameter Store in standard tier is indeed free. Googling reveals a similar service - "AWS Secrets Manager". That is not free. But I don't think I need the advanced features there. While reading your comments again, I found the link to an AWS blog. I will bookmark it and read it again. I think I will modify my project to use the SSM Parameter Store. Trying to figuring out how to use github secrets to replace the place-holders in code might be a bit more involved. Kudos!

[synth]

Below is our template for anyone interested (it took me a little while to figure it out correctly). Two notes:

• Replace "jobs-manager" with whatever your function is called.
• Update the tail end of the ssm:getParameter Resource to match any naming convention you use in ParameterStore.
• Update the resource on kms:Decrypt. We aren't using KMS keys uniformly across environments, so we are granting * access until we fix it.

  JobsManagerLambda:
    Type: AWS::Serverless::Function
    Properties:
      CodeUri: .
      Handler: lib/jobs_manager.handler
      Runtime: ruby2.7
      Timeout: 60
      MemorySize: 512
      FunctionName: !Sub jobs-manager-${StageEnv}
      Environment:
        Variables:
          STAGE_ENV: !Ref StageEnv
          RUBYOPT: '-W0'
      AssumeRolePolicyDocument:
        Version: '2012-10-17'
        Statement:
          -
            Effect: Allow
            Principal:
              Service:
                - 'lambda.amazonaws.com'
            Action:
              - 'sts:AssumeRole'
      Policies:
        -
          Version: '2012-10-17'
          Statement:
            -
              Effect: Allow
              Action:
                - 'ssm:GetParameter*'
              Resource: !Sub 'arn:aws:ssm:${AWS::Region}:${AWS::AccountId}:parameter/${StageEnv}/*'
            -
              Effect: Allow
              Action:
                - 'kms:Decrypt'
              Resource: '*' # FIXME: need to limit once there is a convention with KMS keys across the environments

Then our Ruby handler code looks like this:

require 'json'
require 'dotenv'

require_relative './jobs_manager/env'
require_relative './jobs_manager/ssm_parameter_store' # copied from https://github.com/customink/lamby/blob/master/lib/lamby/ssm_parameter_store.rb

def handler(event:, context:)

  path = "/recognize/#{ENV['STAGE_ENV']}/"
  envs = JobsManager::SsmParameterStore.new path
  envs.get!

  # sample code to output what name and env look like (without values)
  envs.params.each do |param|
    puts param.name, param.env
  end

  puts event
  { statusCode: 200,
    headers: [{'Content-Type' => 'application/json'}],
    body: JSON.dump(event) }
end

[metaskills]

Thanks Qing! So a few thoughts. I'll try to be brief on each.

• There are lots of ways to do this, I think I've come full circle back to SSM Parameter Store with Dotenv.
• For a while (current) I was recommending encrypted secrets. That is what the guides speak to now.
• Here is a link to the old guides that mention our SSM helper. https://github.com/customink/lamby_site/blob/c5b661d9ffe8c4c0f3fed4317a7f58c47b2894de/app/views/docs/environment_and_configuration.erbmd The important parts from that are:

Simple rake take to take all ENVs in a path and write out a Dotenv file. I've sued this method recently in the Environment section of build file (https://github.com/customink/lamby-cookiecutter/blob/master/%7B%7Bcookiecutter.project_name%7D%7D/bin/build-rails#L17) vs that CLI usage.

./bin/rake -rlamby lamby:ssm:dotenv \
  LAMBY_SSM_PARAMS_PATH="/config/${RAILS_ENV}/myapp/env" \

As you pointed out, there is a need for a Policies. I like how yours also has KMS encrypted. Here is what I had in the old guides that will come back. I'll pull in your KMS one too when I do.

Policies:
  - Version: "2012-10-17"
    Statement:
      - Effect: Allow
        Action:
          - ssm:GetParameter
          - ssm:GetParameters
          - ssm:GetParametersByPath
          - ssm:GetParameterHistory
        Resource:
          - !Sub arn:aws:ssm:${AWS::Region}:${AWS::AccountId}:parameter/config/myapp/*

REMINDER: SAM has policy templates (syntactic sugar) for common things that make this easier. https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/serverless-policy-templates.html So an alternative would be this:

Policies:
  - SSMParameterReadPolicy: { ParameterName: '/config/myapp/*' }
  - KMSDecryptPolicy: { KeyId: '...' }

IMPORTANT! If you do not use Dotenv to write a .env.production file during your build phase with the rake task... DO NOT do SSM calls in your handler. This function is called for each request. Instead put envs = JobsManager::SsmParameterStore.new path out of the handler. But please do check out Dotevn and the old guides https://github.com/customink/lamby_site/blob/c5b661d9ffe8c4c0f3fed4317a7f58c47b2894de/app/views/docs/environment_and_configuration.erbmd

[synth]

IMPORTANT! If you do not use Dotenv to write a .env.production file during your build phase with the rake task... DO NOT do SSM calls in your handler.

Good point. This is a quick fix 👍

envs = nil
def handler(event:, context:)

  envs ||= begin
    path = "/recognize/#{ENV['STAGE_ENV']}/"
    ps = JobsManager::SsmParameterStore.new path
    ps = envs.get!.to_env
  end
 # ... the rest
end

That said, I think you are right that using dotenv is still the best approach

[metaskills]

Maybe this at the top of your app.rb file.

require 'lamby/ssm_parameter_store'
Lamby::SsmParameterStore.new("/recognize/#{ENV['STAGE_ENV']}").to_env

Then everything will be ENV as expected.

[synth]

better, thanks! UPDATE: I think it needs the .get!, so Lamby::SsmParameterStore.new("/recognize/#{ENV['STAGE_ENV']}").get!.to_env but yea :) UPDATE2: For some reason needs the trailing /, otherwise I get a strange policy exception, ha.

[metaskills]

CORRECT! And thank you! I knew I left this code in for a reason. I'll resolve this issue when I restore all the documentation.