Error: dispatch failure

[hsearcy]

Hey all,

I am using this via a rails app I am deploying with Lamby. I had it working with the following in my template.yaml

Globals:
  Function:
    Architectures:
      - x86_64
    AutoPublishAlias: live
    DeploymentPreference:
      Type: AllAtOnce
    Environment:
      Variables:
        RAILS_ENV: !Ref RailsEnv
        SECRET_KEY_BASE: x-crypteia-ssm:/my_app/SECRET_KEY_BASE
        DATABASE_URL: x-crypteia-ssm:/my_app/DB_URL

I have a policy attached to my lambda role:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "Statement1",
            "Effect": "Allow",
            "Action": [
                "ssm:Get*",
                "ssm:Describe*"
            ],
            "Resource": [
                "arn:aws:ssm:*:MY_ACCOUNT_ID:parameter/my_app/*"
            ]
        }
    ]
}

This is the error I get after deploying my app:

{"All":"all","ErrorMessage":"Error calling ssm:GetParameter. Environment variable: SECRET_KEY_BASE Path: x-crypteia-ssm:/my_app/SECRET_KEY_BASE Error: dispatch failure","_aws":{"CloudWatchMetrics":[{"Dimensions":[["All","ssm"]],"Metrics":[{"Name":"error","Unit":"Count"}],"Namespace":"Crypteia"}],"Timestamp":1719280028337},"error":1,"ssm":"ssm"}

I get one for each Crypteia env var. I was getting these, but then I modified my policy to use arn:aws:ssm:* instead of arn:aws:ssm:us-east-1 and that seemed to work. My app was working, I could access it via my domain and interact with the database an all! Then I tried to add a couple more secrets in Parameter Store and in my template.yaml, and it broke again!

I reverted the changes to my template.yaml, but am once again getting dispatch failure, even though I reverted to the previously working state. I tried setting CRYPTEIA_DEBUG: true as an env var as well, but it didn't seem to give me any additional information.

I feel like I'm a crazy person 😅 but I'm really hoping that I am just missing something obvious.

Edit: I should mention, I am using the default VPC security group for my lambda, and all of the default VPC subnets.

[jeremiahlukus]

Here is how my lambda looks like

Parameters:
  RailsEnv:
    Type: String
    Default: staging

Globals:
    Architectures:
      - arm64
    AutoPublishAlias: live
    DeploymentPreference:
      Type: AllAtOnce
    Environment:
      Variables:
        DATABASE_URL: !Sub "x-crypteia-ssm:/my_app/${RailsEnv}/DATABASE_URL"
        DATABASE: !Sub "x-crypteia-ssm:/my_app/${RailsEnv}/DATABASE"
        RAILS_ENV: !Ref RailsEnv

  RailsLambda:
    Type: AWS::Serverless::Function
    Description: !Sub "Rails application Lambda function for ${RailsEnv} environment"
    Metadata:
      DockerContext: .
      Dockerfile: Dockerfile
      DockerTag: web
    Properties:
      FunctionName: !Sub "${RailsEnv}-rails" 
      Policies:
        - Statement:
            - Effect: Allow
              Action: ["ssm:Get*", "ssm:Describe*"]
              Resource:
                - !Sub arn:aws:ssm:*:${AWS::AccountId}:parameter/my_app/*

And in my Dockerfile im using v2

COPY --from=ghcr.io/rails-lambda/crypteia-extension-debian:2.0.0 /opt /opt
ENTRYPOINT [ "/usr/local/bundle/bin/aws_lambda_ric" ]
ENV LD_PRELOAD=/opt/lib/libcrypteia.so

Can you try using this type of config?

Also verify the ssm param is in the correct region.

[hsearcy]

Update, I got it working! I'm not sure why it was temporarily working in between, but the fix was to create a VPC Endpoint for SSM in AWS. I had not done so before. I troubleshot by installing the aws-sdk-ssm gem into my lambda project and just manually setting the env vars in the AWS console. Then I opened an irb console on the lambda via lambda console, created an SSM client, and tried to get the parameters from there. This resulted in:

#<Seahorse::Client::NetworkingError:Failed to open TCP connection to ssm.us-east-1.amazonaws.com:443 (getaddrinfo: No address associated with hostname)>

This lead me down the path of creating a VPC endpoint pointed to SSM

• Go to the VPC Console: Open the Amazon VPC console in your AWS Management Console.
• Navigate to Endpoints and click "Create Endpoint."
• Select the service name for Systems Manager (com.amazonaws.us-east-1.ssm)
• Select my VPC

I did that, then I made sure my vpc security group was attached so that traffic from the lambda would be allowed.

What I failed to do was to also attach it to my subnets. After attaching it to my subnets and waiting a minute, I was able to do ssm_client.get_parameter(...) in my lambda console and it succeeded! I have since deployed with my config from my original post and it is working 👍

[metaskills]

Interesting, did you deploy lambda to a vpc with no nat gateway or something? Networking is fun.

[hsearcy]

Indeed, I do not have a NAT Gateway, and I knew I was missing something 😆 I saw that I had an Internet Gateway and thought "oh, that's the thing I needed already." Different gateway 🤦