anuj-ti
commented
1 year ago Similar problem with another execution.
question: Give me a bullet point list of all decisions to use Aws Lambda to build Devflows with the date on which the decision was made.
answer: There is no bullet point list of all decisions to use AWS Lambda to build Devflows with the date on which the decision was made.
id: f557e747218b445390ed428e3c7b5051
Comment: Retried the same question and got proper answer
answer:
Decisions to use AWS Lambda to build Devflows were made on the following dates:
- On February 22, 2021, it was decided to move to Lambda and build DevFlows Actions so they can be deployed as Typescript functions in Lambda with the proxy functionality built into a Lambda Layer (source: 2).
- On the same day, it was decided to use AWS services, including Lambda, SNS, etc. (source: 3).
- On February 26, 2021, it was decided to use AWS Lambda global variables (source: 6).
- On February 22, 2021, it was decided to use Lambda and Fargate ECS (source: 9).
- On February 22, 2021, it was decided to change the programming model to be an API model in TypeScript, deploy in Lambda, and handle the DevFlows platform work in a Lambda layer (source: 11).
- On February 10, 2021, it was decided that moving off of Knative and onto Lambda would help (source: 20).
id: 66e1511b1e24783553a9a124e503ef3bAgain, chunks are there but the information was not extracted.
Tried the above question without asking for dates and it did a good job
question: Give me a bullet point list of all decisions to use Aws Lambda to build Devflows
answer:
- IT developers, Growth Engineers (in Marketing), FP&A Analysts who use VBA and anyone else who develops scripts and code to solve problems can use DevFlows to quickly build applications by stitching together SaaS applications and AWS services in an easy, visual way.
- AWS experts could build solutions using Lambda functions, Step Functions and/or Fargate.
- Rather than try and optimize sidecars, I would build DevFlows Actions so they can be deployed as Typescript functions in Lambda with the proxy functionality built into a Lambda Layer.
- This goes away with the move to Lambda.
- The abstraction is a REST endpoint per Action (where each node can have an action).
- The design of actions being http servers means that to run on Lambda, the abstraction will need to change and every action will need to be updated to be compatible.
- This design works for any node which is a HTTP server. It does not fit with a mode where we want to deploy in Lambda.
- We need a higher level abstraction that can be applied either to deployment of containers or serverless Lambda functions.
- The CN rebuild of invocables to work on Lambda introduced the concept of “functions”.
- Use AWS Lambda global variables
- DB connection pooling using lambda global variables is easy to implement and, with sensible defaults, can scale to 100+ concurrent lambda invocations.
- ScaleArc provides connection pool support but also requires management overhead.
- A DevFlows deployment would manually require setting up a ScaleArc cluster and performance tuning.
- Additionally, a rough benchmark between a ScaleArc cluster (with 1 server and default settings) vs. using lambda global variables indicated that ScaleArc generated significantly higher error rates (i.e. connection timeouts).
- Using these libraries requires the Lambda code to be written using Node.js.
- This spec uses a lambda function to query an RDS RDBMS based on the provided connection settings.
- It provides data to the flow in an event stream of 1 row per event.
- The system uses Lambda global variables and open-source libraries to support connection pooling.
- It only supports MySQL and PostgreSQL based on availability of open-source libraries, but other databases can be supported by extending these libraries relatively easily.
- The change to run actions in Lambda in preference to Knative Serving
- Execution duration is longer than 15 minutes
- Memory requirement is greater than 10GB
- CPU requirement is greater than 6 vCPU
- I/O requirements are very high
- Support data ingestion in batches using DevFlows batch capability
- Batch_Size is set and controlled at the platform level but each action can set an upper limit on Batch_Size, based on the largest batch it can handle.
- The default value for Batch Window is 60 seconds (1 min), configurable for each workspace.
- Use __bulk_ in data ingestion requests for bulk update.
- Build a scheduled Lambda function, triggered by EventBridge rules, to poll feed destinations.
- Prioritize Lambda implementations.
- Make long running applications be separately deployed outside of the core of DevFlows, but allow communication with them by writing Lambda wrappers that can make the external HTTP requests.
- Have DevFlows actions call an API to generate events.
- The approach though the team is taking with the introduction of Lambda support is to make each proxy be a Lambda function, which then synchronously calls the action as a second lambda function.
- Change the programming model to be an API model in TypeScript, deploy in Lambda, and handle the DevFlows platform work in a Lambda layer.
- Using this model the DevFlows platform would be free to combine multiple Actions into a single Lambda function as an optimization.
- Bet fully on AWS - Lambda for serverless compute and SNS or EventBridge (needs a deeper dive with AWS) for eventing. This transformation is currently being executed by the DevFlows Code Nation team with a focus on Lambda and SNS, though the plan is to keep Kubernetes also.
- Move the programming metaphor for Actions (invocables) to be functions rather than REST endpoints. This would make it easier to run the proxy logic in the same instance as the Action, would allow us to optimize and combine different functions into a single runtime activity and would standardize our Action development on a single language (Typescript). The Lambda change currently occurring will add a function paradigm, but will preserve the existing approach in parallel.
- Re-platforming from Knative eventing to AWS SNS and from Knative serving to Lambda functions.
- Schedule time with AWS to get deep on whether SNS or EventBridge make the most sense here.
- Reportedly the EventBridge team said we should use their service over SNS, but the SNS team may say the exact opposite.
- Use existing 429 handling and exponential backoff to throttle resource consumption.
- Constrain AWS resource usage at certain levels.
- Batch events when they backup.
- Allow adapters to process batches.
- Use AWS services - AppFlow, Step Functions, EventBridge, Lambda, etc - to build their own data flows.
- DevFlows provides a simple UI and abstracts all the plumbing away from the user, automating provisioning and monitoring.
- Using AWS services directly requires a much more technical user who can handle these concerns.
- DevFlows has recently undergone a major change from Knative to Lambda functions and SNS topics to provide the runtime and inter-node communications.
- Automatically determine and use the best execution environment for each action in a flow (i.e. Lambda vs Kubernetes), to provide scalability while avoiding the execution time hard limit of Lambda.
- Actions are packaged as Docker containers, so that they are portable between both Kubernetes and Lambda.
- Use a Proxy Lambda function to provide a standard interface to actions via an SNS topic for passing events between nodes.
- Expose public HTTP endpoints using API Gateway, and use AWS services for other supporting functionality.
- Use INPUT nodes to receive pushed events into flows
- Call AWS APIs directly to create resources needed for an input node
- Include only a Lambda Listener
- Create SQL Actions, which will include drivers to the most common databases - such as Oracle, MySQL, SQL Server, and PostgreSQL
- DevFlows currently uses one single AWS account for all workspaces in a given installation.
- Our throttling approach will reduce concurrent Lambda invocations through batching and reducing the number of proxies.
- Our AWS Accounts approach will allow each workspace to have its own AWS Account, reducing the noisy neighbor problem of one flow interrupting others.
- In the event that concurrency limits are still reached under load, the primary concern is http requests from external sources that may not have retry logic.
- API limits are a normal way of handling these types of increases in API level access.
id: 75f927438ecad499aa605a3a45d7ba3b
Question: "give me a numbered list with dates about decisions on devflows?" Answer: There is no information available about decisions on devflows.
Chunks:
Intermediate Steps:
The reduce prompt is not able to extract the information.
Same question asked without date
Question: "give me a list of decision on devflows?" Answer:
Chunks:
Intermediate steps:
Most of the extracted chunks are the same:
We are losing a lot of information in the reduce step