Introduction

Our primary goal was to build a scalable and efficient backend for a ticketing service that could handle high loads with minimal latency. The project focuses exclusively on the backend and infrastructure aspects, omitting a frontend interface to concentrate on the underlying mechanics and performance. This repository highlights the infrastructure conponents, illustrating our journey through building infrastructure, creating CI/CD pipelines, and managing containers.

Programming Language & Framework: We chose Kotlin and Spring Boot for their expressive syntax and the powerful suite of tools for building web applications efficiently.

Database: We utilized MySQL for its powerful database locking functions, which are essential in managing concurrent operations effectively within our ticketing system. This capability ensures data integrity and consistent performance under high-load scenarios.

Containerization & Orchestration: Kubernetes was used to manage our containerized applications, enabling easy scaling and management across multiple servers hosted on AWS.

Configuration Management: Helm charts helped us streamline the installation and management of our Kubernetes applications.

Continuous Deployment: ArgoCD was employed to automate the deployment process, ensuring our changes were seamlessly and reliably pushed to production.

Infrastructure as Code: Terraform allowed us to define our infrastructure using configuration files, which helped in maintaining consistency and ease of deployment across environments.

Performance Testing: We employed K6 to conduct spike tests, simulating scenarios with excessive simultaneous access to evaluate the performance and robustness of our system under extreme conditions.

Monitoring: We integrated Prometheus and Grafana to monitor our applications and infrastructure, ensuring high availability and performance through real-time insights.

Additional Repository

• Backend

Infrastructure

In the course of developing our infrastructure, we tackled a range of infrastructure challenges and optimizations. Below are key resources and discussions that provide insights into our decision-making process and the solutions we implemented:

Deployment and Configuration with Terraform

• Building the Deployment Environment with Terraform: This issue tracks our use of Terraform to automate the provisioning of our entire cloud environment, focusing on reliability and scalability.

Cost Optimization Strategies

• Migration from AWS ALB to Nginx Ingress (Baremetal): To reduce costs, we replaced AWS ALB with a more cost-effective Nginx Ingress setup on bare metal. This discussion details the reasons behind the change and the implementation process.
• Using Public Subnet Node Group to Address NAT Gateway Cost Issues: We opted to configure our EKS cluster using a public subnet node group to avoid high costs associated with NAT gateways.

Monitoring and Metrics

• How to Scrape Metrics from Multiple Pods Using Spring Actuator: This article explains how we set up Prometheus, via Helm, to scrape metrics from multiple pods, enhancing our monitoring capabilities using Spring Boot’s Actuator.

Security Enhancements

• Injecting Secrets into EKS Pods Using Terraform: We explored methods to securely inject secrets into our Kubernetes pods using Terraform, ensuring sensitive data is managed safely and effectively.

Backend

• Queue System Design Issues: Discusses considerations for preventing update losses, implementing non-blocking APIs, and choosing data structures for the queue system.

Project Package Structure and Conventions

• Project Package Structure Considerations: Deliberations on how to organize the project's package structure effectively.
• Convention Documentation: Defines conventions for branch naming, commit messages, HTTP response structures, serialization, testing, and more.

API Optimization and Testing

• API Enhancement Considerations: Detailed discussion on time conventions, data transfer between layers (errors, responses), logging best practices, and their implementation.
• Maintaining Over 80% Test Coverage with Jacoco and Codecov: Outlines strategies and efforts to maintain a high level of test coverage using Jacoco and Codecov.
• Integration Testing Environment with Testcontainers and MySQL Container: Describes the setup of an integrated testing environment using Testcontainers and a MySQL Docker container to enhance testing reliability and consistency.

CD Pipeline

Performance Test

• Considerations for Building the Performance Test Environment: A detailed discussion on the setup and challenges of creating a suitable environment for performance testing.
• Detailed Performance Test Scenarios: This link provides a thorough description of the performance test scenarios used in our project.

Cost Management and Test Scripts

• Calculating Costs for Spike Testing Using ALB LCU: An analysis of cost implications when using AWS ALB Load Capacity Units (LCU) for spike testing.
• Creating K6 Performance Test Scripts: Discussion and documentation on how we developed K6 scripts for our performance testing.

Database and Monitoring Setup

• Database Setup for Test Data and Large-Scale Data Insertions: Outlines our approach to preparing the database for testing, including the creation of large datasets.
• Building a Monitoring Environment with Prometheus and Grafana: Details on how we configured Prometheus and Grafana to monitor our application and infrastructure during the performance tests.

Performance Test Report

SignIn Spike Test Report

• Improved Slow Queries by adding an index to a single-column with 1 million records.
• Observed changes in CPU performance due to encryption: increased CPU core count and observed changes in encryption difficulty based on encryption level adjustments.

JVM Warm Up Test Report

• Observed changes in JVM CodeHeap and performance by repeating the same test after process creation.

3000 Requests Per Minute Spike Test Report

• Improved performance of SELECT COUNT(*) on ten million records by implementing NoOffset.
• Introduced a queue system after considering competition for locks on a single resource (=Event) and observed tests.

6000 Requests Per Minute Spike Test Report

• Improved CPU resource usage by modifying thread pool strategy for thread creation.
• Improved Pending Connection by modifying DB Connection Pool strategy.
• Improved latency by applying Redis caching.

Contributors