search

awslabs / generative-ai-cdk-constructs

AWS Generative AI CDK Constructs are sample implementations of AWS CDK for common generative AI patterns.
https://awslabs.github.io/generative-ai-cdk-constructs/
Apache License 2.0
309 stars 43 forks source link

(New Construct): LangServ Support #249

Open tech4life87 opened 5 months ago

tech4life87 commented 5 months ago

Describe the feature

This construct deploys Langserv a FastAPI based framework that helps developers deploy LangChain's Runnables and Chains as REST API in production.

Use Case

Langserv simplifies the deployment of Large language Models from a prototype to production with industry best practices such as

  1. API docs page with JSONSchema and Swagger
  2. Efficient /invoke/, /batch/ and /stream/ endpoints with support for many concurrent requests on a single server
  3. All built with battle-tested open-source Python libraries like FastAPI, Pydantic, uvloop and asyncio
  4. FastAPI is a modern, fast (high-performance), web framework for building APIs with Python 3.8+ based on standard Python type hints.
  5. Observability and Analytics
  6. LLM Provider Agnostic

Proposed Solution

At a high level this construct will deploy LangServ and LangFuse on AWS using Services such as Fargate, ECS and RDS but will depend on existing constructs such as aws_ecs_patterns. Developers will be able to make custom REST api calls to FastAPI endpoints with support for streaming, batch and invoke. The construct will be LLM provider agnostic such as Bedrock, SageMaker or VLLM on EC2 Instances giving developers flexibility to experiment with different model providers. The construct will also provide monitoring and observability capability using LangFuse giving developers the flexibility to trace and debug LLM applications, monitor latency and model evaluation capabilities.

No response

Other Information

No response

Acknowledgements

laithalsaadoon commented 5 months ago

Thank you for the suggestion @tech4life87 ! Do you mind describing a few things for us:

  1. At a high level, how do you propose deploying and running LangServe on AWS? (i.e. what resources, services, etc.)
  2. Is a LangServe construct related to or enhanced by existing constructs, and if so, how?
  3. At a high level, do you propose that users should be able to make custom REST APIs with LangServe, and if so, what would be the "entrypoint" for providing their Chains and Runnables?
tech4life87 commented 5 months ago

@laithalsaadoon I just updated the issue with these details.

krokoko commented 5 months ago

Hi @tech4life87 , thank your for this suggestion, looks very interesting !

  1. Do you have an idea of the interface for this construct ?
  2. Could you walk me through the developer workflow for this construct ?
krokoko commented 4 months ago

@tech4life87 any update on this one ?

tech4life87 commented 4 months ago

@krokoko

  1. Yes I have an idea of how this would work from my previous work on the same thing.

  2. From a developer experience/flow perspective there are two possible paths and I am open to refining it.

    1. Developer imports the constructs which default to pulling the container from a public ecr repo.
    2. They do local development and local testing.
    3. during CDK deploy the container is packaged and deployed to ECS.

    options2:

    1. Developer create a local langserve application.
    2. The developer then does local development and testing.
    3. Using the local Dockefile the construct packages the code and deploys to ECS.

    I open to ideas of what other developers have done to improve dev experience when it comes to fargate based constructs.

krokoko commented 4 months ago

great @tech4life87 , thanks ! I see you mentioned you may be able to implement this feature, would you be able to create a draft of this construct in a branch ?

tech4life87 commented 4 months ago

great @tech4life87 , thanks ! I see you mentioned you may be able to implement this feature, would you be able to create a draft of this construct in a branch ?

Yes I will start working on it

krokoko commented 4 months ago

Thank you @tech4life87 ! I will assign the ticket to you for now

krokoko commented 2 months ago

Hi @tech4life87 , any update on this ? Thank you !

github-actions[bot] commented 2 weeks ago

This issue is now marked as stale because it hasn't seen activity for a while. Add a comment or it will be closed soon. If you wish to exclude this issue from being marked as stale, add the "backlog" label.

tech4life87 commented 2 weeks ago

@krokoko sorry been super busy with customer project that just completed last week. will resume work on this starting next week