search

aws / containers-roadmap

This is the public roadmap for AWS container services (ECS, ECR, Fargate, and EKS).
https://aws.amazon.com/about-aws/whats-new/containers/
Other
5.2k stars 315 forks source link

[ECS] [request]: Lambda integration with Service Connect #1960

Open nick-kang opened 1 year ago

nick-kang commented 1 year ago

Community Note

Tell us about your request What do you want us to build?

Which service(s) is this request for? This could be Fargate, ECS, EKS, ECR

Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard? What outcome are you trying to achieve, ultimately, and why is it hard/impossible to do right now? What is the impact of not having this problem solved? The more details you can provide, the better we'll be able to understand and solve the problem.

Are you currently working around this issue? How are you currently solving this problem?

Additional context Anything else we should know?

Attachments If you think you might have additional information that you'd like to include via an attachment, please do - we'll take a look. (Remember to remove any personally-identifiable information.)

herrhound commented 1 year ago

@nick-kang - can you tell us more about your use case? How do you envision using Lambdas in the context of ECS Service Connect?

RichiCoder1 commented 1 year ago

@nick-kang - can you tell us more about your use case? How do you envision using Lambdas in the context of ECS Service Connect?

I'm not the reporter, but I imagined this in two ways:

1) Client support, allowing Lambdas to seamlessly call registered services within a namespace 2) Server support, allow registering Lambdas (via Function URLs or a similar mechanism) as targets

I imagine (1) would be the nicer to immediately have (and also likely simpler to implement).

nick-kang commented 1 year ago

Yes, our use case would be (1) from @RichiCoder1 's comment above.

We have ECS services that communicate with each other and Lambda functions that communicate with ECS services. We're using CloudMap with our ECS services to allow the microservice communication. If we were to introduce ECS Service Connect today, we'd still have to keep CloudMap to allow our Lambda functions to communicate with the ECS services, which is a little redundant.

herrhound commented 1 year ago

That makes sense, thank you @RichiCoder1 and @nick-kang. Just to outline an existing alternative – Service Connect still uses Cloud Map as a service registry. You can use Cloud Map's discover-instances API to resolve the IP addresses of your ECS workloads, and connect to them directly. That experience will be equivalent to how Lambdas communicate with the ECS services today, @nick-kang. You won't get the ECS telemetry and automatic retries, though, so adding Lambdas as native Service Connect clients will definitely be beneficial.

I'd like to dive deeper in the Lambda-as-a-server support. Do you see Lambdas being called as HTTP endpoints, or you'd prefer to keep the existing invocation method (synchronous or asynchronous invoke, or via API Gateway API?), and just get the consistent metrics / retry experience?

RichiCoder1 commented 1 year ago

I'd like to dive deeper in the Lambda-as-a-server support. Do you see Lambdas being called as HTTP endpoints, or you'd prefer to keep the existing invocation method (synchronous or asynchronous invoke, or via API Gateway API?), and just get the consistent metrics / retry experience?

I was just thinking being treated as HTTP Endpoints in a similar fashion as Function URLs are today, but with the metrics, discovery, and retry experience provided by connect. For other cases, I'd likely just use the AWS SDK directly (or go to an API Gateway endpoint).

herrhound commented 1 year ago

Got it, @RichiCoder1. Just want to confirm that you won't mind migrating your applications from invoking Lambdas directly to making HTTP calls. One of the guiding principles for Service Connect was the simplicity of migrations, and we optimized for changing configurations instead of the application code. An alternative approach would be getting a tighter integration with Lambda's invocation process and/or API Gateway – in that case, you won't need to change your applications.

RichiCoder1 commented 1 year ago

We already large interface with Lambdas primarily via API Gateway/Function URLs, but I would be interested to see what other modes of integration would look like with service connect.

tmokmss commented 1 month ago

The fact that Service Connect does not support private DNS for service discovery makes it difficult to integrate it with an existing Lambda app.

Service Connect doesn't use or create DNS hosted zones in Amazon Route 53. https://docs.aws.amazon.com/AmazonECS/latest/developerguide/service-connect-concepts.html

Sometimes it is difficult to use CloudMap API instead of ordinary DNS resolution in an exsiting Lambda. We want to access an ECS service with Service Connect enabled from a Lambda function, but were forced to use ECS Service Discovery instead due to this limitation.