sam-goodwin / eventual

Build scalable and durable micro-services with APIs, Messaging and Workflows
https://docs.eventual.ai
MIT License
173 stars 4 forks source link

Bundling: Support bundling customization. #232

Open thantos opened 1 year ago

thantos commented 1 year ago

I want to:

  1. Bundle in additional files (ex: native libraries, config files)
  2. Support external dependencies
  3. Customize how bundling is done

Note: Docker lambda runtimes here - https://github.com/functionless/eventual/issues/231

sam-goodwin commented 1 year ago

Handle native libraries?

What about code splitting? For example, creating a whole separate Lambda Function with only the dependencies for a specific API? Is that related to this or a separate issue?

thantos commented 1 year ago

native libraries

Could you give an example?

What about code splitting? For example, creating a whole separate Lambda Function with only the dependencies for a specific API? Is that related to this or a separate issue?

This is a separate issue, I think, and one which would complicate/be complicated by developer runtime overriding.

This Issue is for how a developer can customize the runtime their activities run on. Dynamically splitting lambdas is separate. As would manually splitting activity targets.

Manually splitting

new ActivityLambdaRuntime(this, "myActivityRuntime", {
   activities: [], // list of activity names to route here
   bundling: {}
})
sam-goodwin commented 1 year ago

Could you give an example?

I just meant any library that relies on native C code being compiled and included in the bundle. I don't have any examples on hand - I think some SQL libraries require it. I also know of a friend who has a dependency on a very old library that's only available in C.

thantos commented 1 year ago

And there are other cases like old AMD dependencies that ESBUILD didn't support at one point. I know we had some at AMZN that I had to make external.

sam-goodwin commented 1 year ago

Had a brief discussion around how we might provide control over bundling, splitting into separate lambdas, controlling memory/timeout config, etc.

this.service = new Service(this, "test-service", {
      name: "test-service",
      entry: require.resolve("test-service"),

      // base

      // does this inherit all env variables?
      // -> lean towards yes, override when not
      // @ts-ignore
      environments: {
        // option-1 - modify current lambdas
        // test-service-event-handler-expensive-shit
        // test-service-activity-handler-expensive-shit

        // option-2 - do-away with current lambdas and just have named groups
        "expensive-shit": {
          entry: require.resolve("test-service2"),
          // test-service-expensive-shit
          include: [
            formatMessage,
          ],
          // workflow can't be included
        },

        "my-java-stuff": {
          includes: ["com.amazon.ABC"],
          function: new lambda.Function(), 
        },

        "my-ecs-activity": {
          include: ["myECSActivity"],
          runtime: aws_ecs.Fargate,
          vpc: myVpc,
        },
      },
    });