Investigate KernelCI API and pipeline production deployment options

[mgalka]

Goal

The goal of this issue is to collect information and analyze suitability of different deployment options for KernelCI API and pipeline production environment.

Current state

New KernelCI API and pipeline are getting final shape, so we need to plan how to deploy them in a production environment. At the moment there is a staging instance running in docker containers. Both kernelci-api and kernelci-pipeline have docker-compose.yml definitions that facilitate the staging deployment.

API

KernelCI API staging deployment consists of several containers:

• api - kernelci-api FastAPI application
• db - MongoDB 5.0 container used as database for test data
• redis - Redis 6.2 container used for pubsub
• storage and ssh - containers used for storing and serving artifacts in a staging deployment.

KernelCI api container is accessed through a reverse proxy, which is not part of the docker-compose based deployment

Pipeline

KernelCI Pipeline deployment consists of several (10+) docker containers each running single pipeline service. Each service is a part of a workflow responsible for building, running tests and storing test data and artifacts. All services utilize KernelCI API as the data and artifacts storage. They also communicate with events transported over the API's pubsub system.

Production deployment

It seems reasonable to introduce some changes to the building blocks and deployment scenario so they're more suitable for production run than development and testing purposes.

Building blocks adjustments

• As KernelCI uses the Azure subscription it may make more sense to utilize the managed services it offers to create a reliable production deployment
• It makes more sense to have the storage changed to something S3-like rather than sticking to ssh+nginx solution from staging deployment, which is easy to set up in the dev environment, but doesn't scale well

Deployment solutions

At the moment 3 possible solutions have been discussed, but the list is still open for additional options.

docker-compose

This seems to be the most straightforward option. It's about mimicking the current staging setup with different storage and reverse proxy/load balancer.

Pros:

• Similar to staging/dev environment
• Less differences in the configuration files between production and staging, hence less error prone deployments

Cons:

• Infrastructure administration, configuration and scaling stays with KernelCI sysadmins

Azure Containers

Azure Containers is a managed service to run containarized applications. It should be relatively straightforward to run current services with Azure Containers

Pros:

• Managed service
• Relatively easy to use
• Can be automated with Terraform

Cons:

• Microsoft's proprietary solution, difficult migration if cloud provider changes

Kubernetes

This seems to be the most robust solution, however it has possibly the most steep learning curve.

Pros:

• Vendor independent
• Azure's managed k8s cluster can be used
• Can be automated with Terraform

Cons:

• Most difficult solution
• Totally different from staging/dev environment

Based on the above information further investigation of the possible deployment options can be performed.

• [ ] Investigate docker-compose based deployment
• [ ] Investigate Azure Container based deployment
• [ ] Investigate Kubernetes based deployment
• [ ] Prepare comparison of above solutions

[gctucker]

Another pro-Kubernetes reason would be that since it's not tied to any vendor it should be possible for other people to directly deploy it in their Cloud for a production environment. At least for the pipeline and main API services, not sure if we should also support deploying all the services in Kubernetes (Redis, MongoDB, storage).

[sbdtu5498]

Hey everyone, have been pretty deep into k8s. If you have any plans on moving to k8s for deployment purposes, it would be great and I would love to give y'all a helping hand. On top of that, it would ultimately lead to the elimination of infra. limitations as using Kubernetes cluster locally via Kubectl is extremely simple and development speed and ease of use for the new contributors can be reduced as well by using some developer experience-enhancing tools such as Okteto, etc. so that they do not need to know even the basics of anything to get up and running with just a few commands.

[mgalka]

Hey everyone, have been pretty deep into k8s. If you have any plans on moving to k8s for deployment purposes, it would be great and I would love to give y'all a helping hand. On top of that, it would ultimately lead to the elimination of infra. limitations as using Kubernetes cluster locally via Kubectl is extremely simple and development speed and ease of use for the new contributors can be reduced as well by using some developer experience-enhancing tools such as Okteto, etc. so that they do not need to know even the basics of anything to get up and running with just a few commands.

Hi @sbdtu5498, Thank you for your comment and offering help. We'd be happy to discuss things further. Would you mind joining #kernelci IRC channel on libera.chat? We could talk about k8s deployment of kernelci-api and pipeline more interactively.

[sbdtu5498]

Sure thing @mgalka!

[gctucker]

I've now broken this down into several tasks and added them to the roadmap:

• 258

• 259

• 262

• 269

So I think we can close this one as the new ones were based on the early findings found as part of the investigation carried out so far. The next phases of investigation and implementation are now split into the different areas.