[coepi-backend-aws]: Enable web server to run in a docker container and deploy-able to a kubernetes cluster

[ramnanib2]

Currently the server endpoints are strongly coupled with Lambda as the compute engine. This prevents the server from being hosted within a Kubernetes cluster or run on EC2. It will be desirable to refactor the front-end handlers so that most of the business logic is abstracted away in cloud-agnostic controllers. Then we can have two shims, one for handling lambda events and another a regular web server framework (ktor or spring). We can then dockerize the server and have an alternate terraform stack deploying it to an EKS cluster.

All the controller business logic is here: https://github.com/Co-Epi/coepi-backend-aws/blob/master/src/main/kotlin/org/coepi/api/v4/TCNCloudAPIHandler.kt

https://github.com/Co-Epi/coepi-backend-aws/blob/master/src/main/kotlin/org/coepi/api/CoEpiHandler.kt

[jmccance]

What do people think about using a multi-module approach for this? This would be especially helpful for keeping the Lambda implementation lean dependency-wise.

Here's a proposed directory structure. I can knock up a strawman PR as well.

coepi-backend-aws
|-- api-aws-lambda  # AWS Lambda API implementation
|-- api-ktor        # Theoretical Ktor API implementation
`-- core            # Core business logic

(The api- prefixes are so things sort nicely, in case anyone was wondering.)

[mihn]

Hey - new guy here. Bit stronger on JVM side, not so much on all cloud things. But I learn fast. :)

Multi-module sounds great - should allow us to use internal scope from Kotlin.

Are we going with reactive/async solution here?

[jmccance]

@mihn @bhushanRamnani See #26 for a partial implementation of what I'm proposing. Some TODOs remaining, but really it's 80-90% there.

@mihn I'd be in favor of taking a reactive and/or async approach, but, too, am a new guy. :) I've had a lot of success with Kotlin's coroutine support, as well as Arrow Fx. Might be worth opening another issue to discuss the project's stance on this.

[mihn]

I wouldnt experiment with FP for such simple domain. At this moment this API is pretty straight forward without any problems. Lets restructure the module for lambda and core and then see where we are.

Back in slack there is some talk of retrieving back Go version of this lamba.

[ramnanib2]

@jmccance Your approach looks sane. I see you've moved the API handler (TCNCloudAPIHandler) as-is. There's a lot of business logic in there which would need to be pulled out into core action handlers, including input validation and response. Lambda Shim can then convert the output of that into APIGatewayProxyResponseEvent. We also need a way to re-use common terraform code between the two implementations, possibly by using TF modules. We can do that in a separate task.

As a side note, v3 needs to be deprecated so better to remove all the code related to v3 to take and reduce tech debt. You might wanna wait for that before refactoring. Was planning to do the cleanup today or tomorrow.

With respect of reactive API, this back-end is pretty straight-forward in its functionality just puts some records to a single DDB table and retrieves it for a fixed time period. For now, let's keep it as lean in its implementation as possible.

[ramnanib2]

Removed all v3 related code in master and replaced lambda logger with log4j2. This should simplify the task a bit. Only need to worry about TCNCloudApiHandler.kt

[jmccance]

@mihn Not really experimentation for me, but I take your point. :) We want to maximize the number of devs that can contribute to this without having to ramp up on unfamiliar concepts.

@bhushanRamnani Thanks for the feedback! Agree there's still a bunch of core logic to move around; I can tackle that in a follow-up PR. The current one already has a lot of files moving around, so want to get it merged in quickly to minimize future merge headaches.

[jmccance]

Now that #26 has been merged, going to start work on extracting some of the business logic from TCNCloudAPIHandler.

[ramnanib2]

@jmccance Are you also planning to take the ktor front-end and docker file ? Also recommend you to join the CoEpi slack group if you haven't already. Please provide an email address so we can send you the invite.

[jmccance]

I certainly can; was taking things one step at a time. 😄

I'll shoot you a DM on TCN Slack with my email.

[jmccance]

PR out for the logic extraction (#29). Once that's merged I'll open a PR for the Ktor work, which I've started here: https://github.com/jmccance/coepi-backend-aws/commit/8e8c8f93e6bc623be105c031d31846f0db9294f4

[jmccance]

Keep forgetting to reference the ticket from the PRs. For posterity, the Ktor PR is #30.

[jmccance]

@bhushanRamnani Did you have any thoughts on where you'd like the Ktor frontend to deploy to? EKS + Fargate seems like a fair solution, though from what I can tell it'll incur some costs to the devs.

I suppose a related question is who the audience of the Terraform deploys is. Are we building this for demo purposes, or is it meant to be something people would be using in production?

[ramnanib2]

@jmccance Yes, EKS and Fargate seem like an ideal choice. We'll be working with Fairwinds to productionize this, with a CI/CD pipeline in Gitlab. Let's take this conversation to slack for that.

Currently, the terraform template is used for teams to set up the server for dev-test quickly. I'm thinking we can keep the server agnostic of deployment scripts and create a separate repo for the IaC and deployment code, considering different teams may have their own toolsets.

[bambash]

Awesome work on #30. I'll start digging in and see what we need to do to get this running on EKS. @bhushanRamnani @jmccance feel free to message me on slack - it may be easier to talk through things there.