troubles with aws sam cli

[cdepillabout]

I've been trying to use aws-lambda-haskell-runtime with the AWS SAM CLI. I've run into multiple problems. I'll describe them in this issue. I'm new to AWS Lambda and AWS SAM, so you should take this issue with a grain of salt.

Resources

I learned about SAM from the AWS SAM Developer Guide. In particular,

• Tutorial: Deploying a Hello World application
• Building custom runtimes
• Testing and Debugging

Setup

I've setup an example Haskell application using aws-lambda-haskell-runtime. I've used sam to generate a template.yaml file, and then edited to match my Haskell application.

Here's my sam version:

$ sam --version
SAM CLI, version 1.37.0

I believe I generated my template.yaml file with a command like sam init, but I forget the specifics. You can find an example in the above resources section.

My template.yaml looks like the following:

AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Description: |
  haskell-app

  SAM App for custom runtime with haskell

# More info about Globals: https://github.com/awslabs/serverless-application-model/blob/master/docs/globals.rst
Globals:
  Function:
    Timeout: 3

Resources:
  HeyWorldFunction:
    Type: AWS::Serverless::Function # More info about Function Resource: https://github.com/awslabs/serverless-application-model/blob/master/versions/2016-10-31.md#awsserverlessfunction
    Properties:
      Description: Says hello world
      CodeUri: .
      Handler: handler
      Runtime: provided.al2
      Events:
        HeyWorld:
          Type: Api # More info about API Event Source: https://github.com/awslabs/serverless-application-model/blob/master/versions/2016-10-31.md#api
          Properties:
            Path: /hey
            Method: post
    Metadata:
      BuildMethod: makefile

Outputs:
  # ServerlessRestApi is an implicit API created out of Events key under Serverless::Function
  # Find out more about other implicit resources you can reference within SAM
  # https://github.com/awslabs/serverless-application-model/blob/master/docs/internals/generated_resources.rst#api
  HeyWorldApi:
    Description: "API Gateway endpoint URL for Prod stage for Hey World function"
    Value: !Sub "https://${ServerlessRestApi}.execute-api.${AWS::Region}.amazonaws.com/Prod/hey/"
  HeyWorldFunction:
    Description: "Hey World Lambda Function ARN"
    Value: !GetAtt HeyWorldFunction.Arn
  HeyWorldFunctionIamRole:
    Description: "Implicit IAM Role created for Hey World function"
    Value: !GetAtt HeyWorldFunctionRole.Arn

The one thing to note here is that the value for Resources.HeyWorldFunction.Properties.Handler is handler. I use this function in my Haskell application.

I have a Makefile for building this application that is used by sam build:

clean:
        rm -rf .aws-sam/build*

# Build application.
build:
        sam build
.PHONY: build

# Run application in Docker.
#
# XXX: hot loading doesn't work correctly if the .aws-sam/build directory exists
# for some reason.
#
# https://github.com/aws/aws-sam-cli/issues/1921
# https://github.com/aws/aws-sam-cli/issues/921
#
# Note that this works for interpreted languages (like bash) since they
# don't need a build step, but won't work for compiled languages.
#
# For compiled languages, you have to run `sam build` before `sam local
# start-api`. While `sam local start-api` is running, you have to run `sam
# build` for any changes to be reflected.
start-local:
        sam local start-api
.PHONY: start-local

invoke-local:
        sam local invoke
.PHONY: invoke-local

# This command can be used for the initial deploy.  This is just for
# documentation purposes.  I don't expect to use this.
guided-deploy: build
        sam deploy --guided
.PHONY: guided-deploy

# Re-deploy application.
#
# Note that the application has to be built
deploy: build
        sam deploy
.PHONY: deploy

# Completely delete the whole application cloudformation stack.
#
# Note that once you delete the whole application cloudformation stack,
# then you much do `make guided-deploy` to redeploy it.
destroy:
        aws cloudformation delete-stack --stack-name bash-app # --region region
.PHONY: destroy

# Validate template.yaml
validate:
        sam validate
.PHONY: validate

#############################################
## Targets used internally by `sam build`. ##
#############################################

build-HeyWorldFunction:
        cabal build
        cp ./dist-newstyle/build/x86_64-linux/ghc-9.0.2/bootstrap-0.1.0.0/x/bootstrap/build/bootstrap/bootstrap $(ARTIFACTS_DIR)/

.PHONY: build-HeyWorldFunction

The only important target here is build-HeyWorldFunction (but it is not directly related to this issue).

Here is the .cabal file for my application.

bootstrap.cabal:

cabal-version:      2.4
name:               bootstrap
version:            0.1.0.0

executable bootstrap
    main-is:          Main.hs
    build-depends:    base == 4.15.1.0
                    , aeson
                    , amazonka-dynamodb
                    , aws-lambda-haskell-runtime
                    , bytestring
                    , base64-bytestring
                    , http-conduit
                    , text
    default-extensions:  DataKinds
                       , DefaultSignatures
                       , DeriveAnyClass
                       , DeriveFoldable
                       , DeriveFunctor
                       , DeriveGeneric
                       , DerivingStrategies
                       , EmptyCase
                       , ExistentialQuantification
                       , FlexibleContexts
                       , FlexibleInstances
                       , GADTs
                       , GeneralizedNewtypeDeriving
                       , InstanceSigs
                       , KindSignatures
                       , LambdaCase
                       , MultiParamTypeClasses
                       , NamedFieldPuns
                       , OverloadedLabels
                       , OverloadedLists
                       , OverloadedStrings
                       , PatternSynonyms
                       , PolyKinds
                       , RankNTypes
                       , RecordWildCards
                       , ScopedTypeVariables
                       , StandaloneDeriving
                       , TypeApplications
                       , TypeFamilies
                       , TypeOperators
    other-extensions:    TemplateHaskell
                       , QuasiQuotes
                       , UndecidableInstances
    hs-source-dirs:   app
    default-language: Haskell2010

Here's my app/Main.hs file:

module Main where

import Aws.Lambda (ApiGatewayRequest, ApiGatewayResponse, Context, addAPIGatewayHandler, addStandaloneLambdaHandler, defaultDispatcherOptions, mkApiGatewayResponse, runLambdaHaskellRuntime)
import Data.Aeson (FromJSON (parseJSON), ToJSON, Value, genericParseJSON, defaultOptions, eitherDecode)
import Data.Aeson.Types (Parser, parseEither)
import qualified Data.ByteString.Base64 as B64
import Data.Text (Text)
import GHC.Generics (Generic)
import System.IO (hFlush, stdout, stderr)
import Data.Text.Encoding (encodeUtf8)
import Data.ByteString.Lazy (fromStrict)
import Debug.Trace (traceM)
import System.IO.Unsafe (unsafePerformIO)

data Person = Person
  { name :: String
  , age  :: Int
  }
  deriving (Generic, Show, ToJSON)

instance FromJSON Person where
  parseJSON :: Value -> Parser Person
  parseJSON v = do
    -- v might be a base64-encoded JSON String, or a normal JSON value of
    -- Person.
    case parseEither (genericParseJSON defaultOptions) v of
      -- v was a JSON-encoded Person, just return it.
      Right person -> do
        pure person
      -- v may be a base64-encoded JSON String.
      Left jsonErr -> do
        -- parse it as a JSON String
        b64str :: Text <- parseJSON v
        case B64.decode (encodeUtf8 b64str) of
          -- failed to decode the JSON String as base64
          Left b64Err -> do
            fail "Trying to parse Person, but input value was not a JSON-encoded Person, nor a base64-encoded String"
          Right rawPerson -> do
            -- try to decode it as JSON
            case eitherDecode (fromStrict rawPerson) of
              Left innerJsonErr -> do
                fail innerJsonErr
              Right person -> do
                pure person

examplePerson :: Person
examplePerson = Person "hellohello" 33

handler2 :: ApiGatewayRequest Person -> Context () -> IO (Either (ApiGatewayResponse String) (ApiGatewayResponse Person))
handler2 _ _context = do
  let resp = mkApiGatewayResponse 200 [] examplePerson
  pure (Right resp)

main :: IO ()
main = do
  runLambdaHaskellRuntime
    defaultDispatcherOptions
    (pure ())
    id
    (addAPIGatewayHandler "handler" handler2)

This application can be built by running sam build. You can find the built application in the current directory at .aws-sam/build/HeyWorldFunction/bootstrap.

Problem 1: sam local generate-event leaves out some fields that aws-lambda-haskell-runtime is expecting

(I've sent a PR for this at https://github.com/theam/aws-lambda-haskell-runtime/pull/119.)

AWS SAM gives you a way to define example events in files, and then run your function passing it an event from a file. AWS SAM provides the command sam local generate-event to generate event files. This is described in the document Invoking functions locally.

One problem is that the events that sam local generate-event generates don't have all the fields that ApiGatewayRequest is expecting.

For instance, here is an example of generating an event, and then the resulting event file:

$ sam local generate-event apigateway aws-proxy --path "hey"  > events/event2.json

This generates an API Gateway Lambda proxy event. Here is the resulting event file:

{
  "body": "{\"name\": \"helll\", \"age\": 39}",
  "resource": "/{proxy+}",
  "path": "/hey",
  "httpMethod": "POST",
  "isBase64Encoded": false,
  "queryStringParameters": {
    "foo": "bar"
  },
  "multiValueQueryStringParameters": {
    "foo": [
      "bar"
    ]
  },
  "pathParameters": {
    "proxy": "/hey"
  },
  "stageVariables": {
    "baz": "qux"
  },
  "headers": {
    "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8",
    "Accept-Encoding": "gzip, deflate, sdch",
    "Accept-Language": "en-US,en;q=0.8",
    "Cache-Control": "max-age=0",
    "CloudFront-Forwarded-Proto": "https",
    "CloudFront-Is-Desktop-Viewer": "true",
    "CloudFront-Is-Mobile-Viewer": "false",
    "CloudFront-Is-SmartTV-Viewer": "false",
    "CloudFront-Is-Tablet-Viewer": "false",
    "CloudFront-Viewer-Country": "US",
    "Host": "1234567890.execute-api.us-east-1.amazonaws.com",
    "Upgrade-Insecure-Requests": "1",
    "User-Agent": "Custom User Agent String",
    "Via": "1.1 08f323deadbeefa7af34d5feb414ce27.cloudfront.net (CloudFront)",
    "X-Amz-Cf-Id": "cDehVQoZnx43VYQb9j2-nvCh-9z396Uhbp027Y2JvkCPNLmGJHqlaA==",
    "X-Forwarded-For": "127.0.0.1, 127.0.0.2",
    "X-Forwarded-Port": "443",
    "X-Forwarded-Proto": "https"
  },
  "multiValueHeaders": {
    "Accept": [
      "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8"
    ],
    "Accept-Encoding": [
      "gzip, deflate, sdch"
    ],
    "Accept-Language": [
      "en-US,en;q=0.8"
    ],
    "Cache-Control": [
      "max-age=0"
    ],
    "CloudFront-Forwarded-Proto": [
      "https"
    ],
    "CloudFront-Is-Desktop-Viewer": [
      "true"
    ],
    "CloudFront-Is-Mobile-Viewer": [
      "false"
    ],
    "CloudFront-Is-SmartTV-Viewer": [
      "false"
    ],
    "CloudFront-Is-Tablet-Viewer": [
      "false"
    ],
    "CloudFront-Viewer-Country": [
      "US"
    ],
    "Host": [
      "0123456789.execute-api.us-east-1.amazonaws.com"
    ],
    "Upgrade-Insecure-Requests": [
      "1"
    ],
    "User-Agent": [
      "Custom User Agent String"
    ],
    "Via": [
      "1.1 08f323deadbeefa7af34d5feb414ce27.cloudfront.net (CloudFront)"
    ],
    "X-Amz-Cf-Id": [
      "cDehVQoZnx43VYQb9j2-nvCh-9z396Uhbp027Y2JvkCPNLmGJHqlaA=="
    ],
    "X-Forwarded-For": [
      "127.0.0.1, 127.0.0.2"
    ],
    "X-Forwarded-Port": [
      "443"
    ],
    "X-Forwarded-Proto": [
      "https"
    ]
  },
  "requestContext": {
    "accountId": "123456789012",
    "resourceId": "123456",
    "stage": "prod",
    "requestId": "c6af9ac6-7b61-11e6-9a41-93e8deadbeef",
    "requestTime": "09/Apr/2015:12:34:56 +0000",
    "requestTimeEpoch": 1428582896000,
    "identity": {
      "cognitoIdentityPoolId": null,
      "accountId": null,
      "cognitoIdentityId": null,
      "caller": null,
      "accessKey": null,
      "sourceIp": "127.0.0.1",
      "cognitoAuthenticationType": null,
      "cognitoAuthenticationProvider": null,
      "userArn": null,
      "userAgent": "Custom User Agent String",
      "user": null
    },
    "path": "/prod/hey",
    "resourcePath": "/{proxy+}",
    "httpMethod": "POST",
    "apiId": "1234567890",
    "protocol": "HTTP/1.1"
  }
}

You can run the Haskell application built with sam build by running:

$ sam local invoke --event ./events/event2.json

If you do this, you'll see errors that aws-lambda-haskell-runtime is expecting more fields than are present in the above input event file. For example, aws-lambda-haskell-runtime expects the extendedRequestId field, but that is not defined in this event. There are a few other fields like this as well.

[cdepillabout]

Problem 2: sam local start-api leaves out some fields that aws-lambda-haskell-runtime is expecting

(I've sent a fix for this at https://github.com/theam/aws-lambda-haskell-runtime/pull/119.)

AWS SAM provides a command sam local start-api for running an AWS APIGateway endpoint locally for development. This is described in the document Running API Gateway locally. The requests/events that are passed to aws-lambda-haskell-runtime do not have all the fields that aws-lambda-haskell-runtime is expecting.

For instance, in one terminal run:

$ sam local start-api

and then in another terminal run:

$ curl -v -X POST -H 'Content-Type: application/json' -d '{"name": "helll", "age": 39}' http://127.0.0.1:3000/hey

The Haskell application gets a request event that looks like the following:

{
  "body": "{\"name\": \"helll\", \"age\": 39}",
  "headers": {
    "Accept": "*/*",
    "Content-Length": "28",
    "Content-Type": "application/json",
    "Host": "127.0.0.1:3000",
    "User-Agent": "curl/7.79.1",
    "X-Forwarded-Port": "3000",
    "X-Forwarded-Proto": "http"
  },
  "httpMethod": "POST",
  "isBase64Encoded": false,
  "multiValueHeaders": {
    "Accept": [
      "*/*"
    ],
    "Content-Length": [
      "28"
    ],
    "Content-Type": [
      "application/json"
    ],
    "Host": [
      "127.0.0.1:3000"
    ],
    "User-Agent": [
      "curl/7.79.1"
    ],
    "X-Forwarded-Port": [
      "3000"
    ],
    "X-Forwarded-Proto": [
      "http"
    ]
  },
  "multiValueQueryStringParameters": null,
  "path": "/hey",
  "pathParameters": null,
  "queryStringParameters": null,
  "requestContext": {
    "accountId": "123456789012",
    "apiId": "1234567890",
    "domainName": "127.0.0.1:3000",
    "extendedRequestId": null,
    "httpMethod": "POST",
    "identity": {
      "accountId": null,
      "apiKey": null,
      "caller": null,
      "cognitoAuthenticationProvider": null,
      "cognitoAuthenticationType": null,
      "cognitoIdentityPoolId": null,
      "sourceIp": "127.0.0.1",
      "user": null,
      "userAgent": "Custom User Agent String",
      "userArn": null
    },
    "path": "/hey",
    "protocol": "HTTP/1.1",
    "requestId": "eb84ba71-128a-494d-bfce-b5985e9146d8",
    "requestTime": "07/May/2022:11:30:08 +0000",
    "requestTimeEpoch": 1651923008,
    "resourceId": "123456",
    "resourcePath": "/hey",
    "stage": "Prod"
  },
  "resource": "/hey",
  "stageVariables": null,
  "version": "1.0"
}

You can see that this is missing some fields that aws-lambda-haskell-runtime is expecting (in addition to things like requestContext.extendedRequestId being null, when aws-lambda-haskell-runtime expects it to not be null).

[cdepillabout]

Problem 3: body of API Gateway requests always a String?

From the AWS documentation (and playing around with sam local generate-event and sam local start-api) it appears that the body field of an APIGatewayRequest will always be a JSON String (possibly base64-encoded). I couldn't find a specification of exactly what an ApiGatewayRequest needs to look like anywhere on the AWS site, but there are two pages that at least have examples of requests:

• https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/serverless-getting-started-hello-world.html
• https://docs.aws.amazon.com/apigateway/latest/developerguide/set-up-lambda-proxy-integrations.html

The previous comment also has an example of the body field.

This isn't a problem, per se, but it was quite confusing that there is a instance FromJSON body => FromJSON ApiGatewayRequest body, when body can't be any arbitrary JSON blob, but body really needs to be a JSON String. You can then potentially take the JSON String and decode it as JSON, but aws-lambda-haskell-runtime doesn't seem to provide any support for this.

It seems like ApiGateway also has a feature for base64-encoding the body field (and setting the corresponding field isBase64Encoded to true), but aws-lambda-haskell-runtime doesn't have any sort of helper functions for decoding the body field based on the value of isBase64Encoded.