Make writing of own runtimes easier

[fuzzy-id]

Hey there,

I used this library to write a couple of handlers exposed via API gateway. It works really nice, but it is awfully slow. So I started to write the "runtimes" directly, i.e. I put together an executable which I put as bootstrap into a zip and simply use it as a provided runtime without an additional layer. That works quite nice too and is as fast as it should be. I adapted a few lines in the code in order to work with it better. Most of it involves not having to pack/unpack from different string formats but use the ones that aeson wants. Maybe you want to cherry-pick some adaptions…

Cheers

[NickSeagull]

Thanks @fuzzy-id ! Do you have any kind of benchmark info for this? I'd like to see some rough numbers please 😄

We can probably merge this altogether if it is much faster

[fuzzy-id]

I don't have any benchmarks, yet. And I probably won't have the time to add these in the near-time future. (Lambdas are hard to benchmark anyways.) I'll see what I can do about it.

It would probably make sense to add those benchmarks together with a couple of tests to this repository. What machinery would you prefer for test-/benchmark-deployment? Terraform?

[tainguyenbui]

hi @fuzzy-id! Thanks a lot for your hard work 🚀

I would suggest that we go with the Serverless Framework

Pre-requisites: having installed and configured aws-cli with your credential

• yarn add global aws-cli having installed serverless
• yarn add global serverless

An example serverless.yml template would look like:

service: my-haskell-benchmark

provider:
    name: aws
    memorySize: 1024
    runtime: provided
    stage: ${opt:stage, 'dev'}
    region: ${opt:region, 'us-east-1'}
    role: { Fn::GetAtt: [HaskellCrudRole, Arn] }
    environment:
        MY_ENV_VARIABLE: 'Some stuff if you want here'

package:
    individually: true

functions:
    helloWorld:
        description: "[${self:provider.stage}] My Hello World function"
        handler: src/Lib.handler
        package:
            artifact: "<your-directory>/<your-function-package>.zip"
        events:
            - http:
                  path: hello
                  method: get
                  cors: true

resources:
    Resources:   
        HaskellCrudRole:
            Type: AWS::IAM::Role
            Properties:
                RoleName: benchmark-haskell-role-${self:provider.stage}
                AssumeRolePolicyDocument:
                    Statement:
                        - Action: sts:AssumeRole
                          Effect: Allow
                          Principal:
                              Service: lambda.amazonaws.com
                    Version: "2012-10-17"
                Path: /
                Policies:
                    - PolicyDocument:
                          Statement:
                              - Action:
                                    - logs:CreateLogGroup
                                    - logs:CreateLogStream
                                    - logs:PutLogEvents
                                Effect: Allow
                                Resource: "*"
                          Version: "2012-10-17"
                      PolicyName: BenchMarkHaskellRoles${self:provider.stage}

after you build your function and package it in a zip

you could deploy your stack by using sls deploy

With the above, we will get a function deployed with an endpoint to being able to hit it. We would also have a CloudWatch Log group to gather information about our function

We would then use something like serverless artillery or artillery to hit our endpoint with something like:

artilllery quick --duration 100 --rate 30 -n 1 <your-function-apigateway-endpoint>

or something like slsart invoke --path myscrpt.yml

The slsart version will require that you create a testing script.

The above will allow you to do some load testing on your function. The same can be done for the previous version for comparison

Thanks a lot for your contributions!

[tainguyenbui]

With the above, you could then create AWS CloudWatch Dashboards for analyzing the results

the above is an example of the execution durations for a Haskell function

[tainguyenbui]

Hi @fuzzy-id @NickSeagull,

I've done some benchmarking on your modifications against the current implementation. Here are the results:

36k requests (1200 seconds, 20 rps)

We can see a slight decrease in the minimum duration. On the other hand, the average has gone up in the same way that the cold start has too. The below graph shows a hint of what could that be caused by:

In terms of memory usage we cannot appreciate major changes:

Thanks for your time to contribute and let's the discussion start :D

Note: @NickSeagull had to do some small changes reflected in the PR he openend.

[fuzzy-id]

Wow, cool! Thanks for benchmarking @tainguyenbui. Although the results aren't as good as I first indicated. I guess there was some kind of misunderstanding going on anyways. (Or better, talking past each other.) As said above, I used this code here as a library rather than as a runtime layer. I think that was the part that gave me the biggest performance improvement. But as I noticed the other day, you made this switch in design choice yourself already. So I guess that's the reason why we don't see much of a difference here.

Does this make sense?

[tainguyenbui]

@fuzzy-id yes, it totally makes sense. Great work from you too! whilst benchmarking our own runtime we found that the Layer could be delaying our execution durations as well as the cold starts, which later on we both, you and The Agile Monkeys have been able to prove.

It is great having this baseline because it allows us to continue improving and perhaps work with AWS in order to get some improvements. As an example, I could talk about the improvements in Go lang which has improved almost 3x since last year.

It would be great seeing what has caused higher durations in your change in order to have it into consideration when doing the next iterations of this runtime. As an open-source project, please feel free to contribute anytime.

Soon, I will be releasing an article and a project so you are also able to perform your benchmarks easily :)

[NickSeagull]

What's missing for this PR?

[NickSeagull]

Closing as stale