search

i-dot-ai / caddy-chatbot

Caddy is an AI powered co-pilot for customer service functions everywhere.
https://ai.gov.uk/projects/caddy/
MIT License
8 stars 3 forks source link

Migration to FastAPI #83

Closed kurtismassey closed 3 months ago

kurtismassey commented 3 months ago

Draft for discussion

Migrating to FastAPI and deploying via AWS App Runner ECS Fargate would allow us to simplify architecture and take advantage of Poetry for dependency management over current separate lambda approach

Using Background Tasks to perform some of the longer running processing we can still return a fast acceptance response to Google Chat to get around the not responding notice

This aligns with a move to having caddy_core components too - I've started some initial work of passing a chat_client through which should enable us to replicate functionality and formatting across different chat clients

Let me know if we think if this is a potential solution - I still need to go through and add docstrings and types to the functions we've migrated in from the seperate lambdas as well as redo the tests but may need help with naming conventions and file structures for the caddy core components

Have tested in google chat dev and seems to be working!

kurtismassey commented 3 months ago

Have just updated the template to use AWS Fargate over App Runner - app runner seems to heavily throttle the cpu while not actively serving requests so fargate performs better for background tasks

AndreasThinks commented 3 months ago

I've flagged this one to @lmwilkigov as would be good to get his thoughts on how we're structuring the new fastAPI approach.

kurtismassey commented 3 months ago

After doing a bit of stress testing with FastAPI Background Tasks, ie sending like 10 requests in at the same time it appears to be blocking processes so after the first request or two it would just stop processing any further I've reverted to using a manual Thread approach for running the handle message and now getting responses on 100% of requests - more than likely a better way of doing this but it's reliable so is a start!