4.0.0 - Simpler lambdas and the death of Template Haskell

[NickSeagull]

After the discussion at #78 , it is clear that the way to go is to have a simpler runLambda function that users can use in order to make better decisions regarding their applications.

Apart from that, with the new additions for AWS SAM it makes even less sense to have a dynamic dispatch function. This autogenerated function not only was horrible for the users to debug, but the Template Haskell code that generated this function was horrible to maintain, and test, as well.

The roadmap for the 4.0.0 version is:

• [x] Remove the Template Haskell code
• [x] Fix runLambda so it is easier to call from a main function in an executable by the user
• [ ] Provide a better way to wrap the user code, so it receives different event types from different services, e.g. API GateWay
• [ ] Initialize amazonka on cold starts, so users can write their code either in the AWST monad, IO or anything in between
• [ ] Document the new version, including how to use it with AWS SAM

Inviting @dnikolovv @lrworth @endgame for discussing this.

[dnikolovv]

Sounds pretty awesome to me! Let me know if I can help with anything.

[endgame]

Will look at the code a bit later, but some pointers to get you started on the documentation:

• AWS SAM page on building custom runtimes: https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/building-custom-runtimes.html
• Each AWS::Serverless::Function (e.g. Foo) declared with Metadata: { BuildMethod: Makefile } in template.yaml (or .json) will cause sam to run make build-Foo in the directory specified by that function's CodeUri property.
• Each target is responsible for populating $(ARTIFACTS_DIR), an environment variable set by sam when it triggers make. For most haskell projects, building one executable target and copying it to $(ARTIFACTS_DIR)/bootstrap seems to be sufficient.

[lrworth]

Wow, thanks @NickSeagull — this sounds fantastic! I'll also have a look later. I'm more than happy to contribute any work that needs doing and test stuff, just let me know.

(Also it looks like you're using ormolu for formatting, excellent choice IMO)

[NickSeagull]

Thanks everyone for the comments, I'm gonna try pushing forward a minimal version of this for 4.0.0, I'd need some help with writing unit tests and integration too.

[endgame]

Something we have discovered w.r.t. the custom runtime support: sam copies the source (i.e., the CodeUri) directory into a temporary location to do this build. It does this to maintain a pristine source tree.

This has some implications that should be documented. Our needs are a bit more complicated than the norm - we have a single repo with multiple services and multiple libraries; simpler setups can probably put template.yaml and Makefile next to the top-level cabal.project or stack.yaml. I'll explain them below so other people don't fall into the same holes we did:

1. Builds that use relative paths pointing outside the build directory won't work - see https://github.com/awslabs/aws-sam-cli/issues/2077 (which contains a sample directory tree by way of example). There are a few ways to deal with this:
   • Symlink the build directories so the symlink gets copied along with the source.
   • Make sure CodeUri points to a common ancestor directory of any libraries referenced locally. This does mean that you may copy a lot of irrelevant stuff to the build directory, and you may have to put the Makefile in the same directory. This also means that you may have a bunch of different project targets in the same Makefile.
   • Use tricks to find the real source directory, and fix up references (e.g., seding cabal.project or stack.yaml) before building. This workaround is described in the linked issue.
2. If your build tool caches its output under the source directory, you lose a lot of that when the tempdir goes away (i.e., after each target it built), meaning that each build target rebuilds all the local libraries, as well as common code. This is harder to fix, and does impact people with less complicated use-cases, if they're reusing code in a library stanza:
   • Use a build tool which caches elsewhere, so you can get reuse the local libraries that you build (e.g., use nix to cache your local library dependencies?); and
   • Make code that is shared between endpoints a separate cabal package, so that it can get this sharing too. Something like a servicename-core containing the library/tests, and a servicename-endpoints containing the code for each executable.

We avoided problem No. 2 by using shell scripts to prepare our deployment packages. This means we still get the benefit of simpler and smaller endpoint executables.

Sorry, that was a bit of a braindump; I hope that it helps. LMK if anything's unclear and I'll have another go at explaining.

[dnikolovv]

@endgame We have solved the relative build paths issue by building using Docker and passing in the root of the repo as the context.

Our Makefiles look something like this.

all:
    @rm -rf ./build/*
    DOCKER_BUILDKIT=1 docker build --file Dockerfile -t service ../../.
    id=$$(docker create service); docker cp $$id:/root/output ./build; docker rm -v $$id

[NickSeagull]

Hey, just posting a message to say that I haven't abandoned this, just that I didn't have time due to life reasons. Will soon resume work on this :)

[talw]

I wonder if this is in progress?

[NickSeagull]

This is pretty much stalled right now, would be happy to help if someone wants to pick this up

[talw]

This is pretty much stalled right now, would be happy to help if someone wants to pick this up

I see. Unfortunately my hands are full at the moment as well.

Hopefully someone will pick this up :)

[fcracker79]

Hi there,

I am trying to implement multi-handler lambda and the template expansion does not allow it, as some modules are not exposed. Will this fix solve my use case?

In addition to that, while waiting for this pull request to be merged, I could work on a simpler change that would expose ApiGatewayLambdaError. Does it make sense?

[NickSeagull]

@fcracker79 can you open an issue for each of those topics so we can discuss over there?

[fcracker79]

@NickSeagull thanks for your response. There is already an issue related to my question (https://github.com/theam/aws-lambda-haskell-runtime/issues/82#issuecomment-659753726). I was just asking if this pull request will solve that issue.

In addition to that, since I would like to proceed with my project and this pull request has been WIP for weeks, I was thinking of creating a new pull request with a simple change to export ApiGatewayLambdaError; I just need a piece of advice: wait for the completion of this pull request or create a far simpler one that temporarily solves the above issue but will be deprecated by this pull request?

[NickSeagull]

@fcracker79 I'm not actively working on this at this moment, so your PR would be well accepted. Regarding to #82 note that the issue is that the handler generator is hardcoded to look for the string handler :: in the project files, as the issue OP solved

[fcracker79]

Thanks @NickSeagull , currently it would be more than enough for my project to be able to copy and paste that generated code and then change all the hardcoded strings I want so that to implement multi handler entry point.

[NickSeagull]

Closing as #97 did this and more in a better way :)