MVP - Hybrid Microservices Architecture

[jennb33]

Product Outline

Link

Summary

This epic captures our strategic shift from a Rails monolithic framework to a hybrid microservices architecture. This transformation aims to streamline development autonomy across VFS (Veteran-Facing Services) teams and facilitate the isolation of distinct application functionalities. Automation will be pivotal to the seamless integration and operation of monolithic and microservices components within our ecosystem.

In our shift from a Rails monolithic framework to a hybrid microservices architecture, we aim to optimize our development processes and enhance VFS team autonomy. This transformation will enable us to isolate distinct application and product functionalities, allowing each team to develop, deploy, and scale their services independently. Automation and templating will play a crucial role in integrating and operating both monolithic and microservices components within our ecosystem seamlessly.

Scope

The scope of this document pertains to Platform and VFS products/applications, specifically focusing in on the Vets API Rails monolith. It aims to explore design strategies, discovery items, open questions, and outline the role of the Platform vs VFS teams. This document will define how the Platform team will undertake the responsibilities of development, templating, maintenance, and support the hybrid microservices architecture initiative.

To review the detailed design of the hybrid microservices approach, address security considerations, identify potential risks and mitigations, and list any open issues or questions related to the implementation go to this design document. The objective is to ensure a comprehensive understanding for the transition and integration of the hybrid microservices architecture.

Objectives

• Isolate Business Logic: Segment distinct functionalities and business logic into dedicated microservices.
• Boost Team Autonomy: Empower VFS teams with the capability to independently manage the lifecycle of their services — from development through to deployment.
• Forge Robust Automation: Establish comprehensive automation strategies for continuous integration and deployment (CI/CD), alongside service discovery and operational monitoring.
• Safeguard System Integrity: Ensure the new architectural model preserves or enhances the application’s performance, security, and reliability.

Key Benefits

Enhanced Scalability

Leveraging microservices enables targeted scaling of application components, isolation and optimizing both resource utilization and operational costs.

Accelerated Development Velocity

The independence afforded by microservices architecture allows teams to rapidly develop, test, and deploy, significantly speeding up release cycles.

Technological Flexibility

Microservices architecture facilitates the exploration and adoption of new technologies within isolated components, without necessitating broad architectural overhauls.

Increased System Resilience

Isolating services limits the scope of potential failures, thereby improving the overall resilience and reliability of the system.

Milestones

1. Assessment and Strategy Formulation

   • Conduct a comprehensive review of the current application to pinpoint potential microservice candidates.
   • Clearly define service boundaries and delineate responsibilities.
   • Draft a detailed migration and implementation roadmap.

2. Infrastructure and Tooling Configuration

   • Establish a Kubernetes based infrastructure to host the microservices.
   • Identify and configure essential tools for CI/CD, along with monitoring and logging systems.

3. Pilot Microservice Implementation

   • Select a low-risk module for initial migration to validate the microservices model.
   • Develop, test, and deploy the pilot, refining infrastructure and operational practices based on learnings.

4. Systematic Component Migration

   • Methodically migrate identified components to microservices, prioritizing based on complexity and impact.
   • Implement effective inter-service communication protocols and service discovery mechanisms.

5. Automation and CI/CD Optimization

   • Automate deployment workflows for each microservice to facilitate continuous delivery.
   • Implement comprehensive monitoring and alerting systems for both individual services and the collective architecture.
   • Create automation for VFS (and internal) teams to leverage for the creation and maintenance of a given microservice.
   • Ideas for automation: helm charts, datadog charts, CI/CD, Argo configurations, logging, app specific (rails) defaults, etc.

6. Knowledge Sharing and Documentation

   • Thoroughly document architectural decisions, design paradigms, and operational procedures.
   • Host educational sessions for development teams to share microservices best practices and operational insights.

7. Performance Evaluation and Refinement

   • Monitor the system’s performance and gather feedback from development teams.
   • Continuously adjust practices, tools, and service designs based on empirical evidence and team insights.

Challenges and Strategic Considerations

• Maintaining Data Consistency: Develop strategies to ensure data consistency and integrity across disparate services.
• Managing Network Complexity: Mitigate the complexities introduced by increased service-to-service communications with strategic architectural patterns (e.g., API Gateways, Circuit Breakers).
• Upholding Security Standards: Maintain a strong security posture through service isolation, secured inter-service communications, and centralized identity and access management.

High Level User Story/ies

As a developer, I need to transition our application from a monolithic architecture to a hybrid microservices architecture so I can enable individual teams to deploy updates faster, reduce overall system downtime, and improve scalability.

Hypothesis or Bet

If we restructure our application into a hybrid microservices architecture then we expect to achieve faster feature deployment times, product/application isolation, lower the risk of system-wide failures during updates, and enhance our application's scalability to handle growing user demand.

OKR

O2. Our platforms are the best way to deliver products at VA OKR 1: Our platforms hit the "elite" level (as defined by DORA) on Deployment Frequency, Lead Time for Changes, Change Failure Rate, and Time to Restore Service.

Definition of done

Definition of Done

• [ ] Kubernetes-based infrastructure is fully operational for hosting microservices.
• [ ] Components identified from the monolith are successfully transitioned into microservices and operational in production.
• [ ] CI/CD pipelines for microservices are established for automatic builds, tests, and deployments.
• [ ] Comprehensive monitoring and logging systems are implemented for real-time performance and health insights of each microservice.
• [ ] Load testing confirms the system's scalability and performance under peak loads.
• [ ] Documentation for the microservices architecture, including design decisions and operational procedures, is comprehensive and accessible.
• [ ] Training sessions for development, operations, and support teams on the new architecture have been conducted.
• [ ] The system shows stable performance and reliability over a significant period with no critical issues from the architecture transition.
• [ ] Scalability and load handling capabilities of the application are validated and improved compared to pre-transition benchmarks.
• [ ] Feedback from development teams indicates improved development efficiency and satisfaction.
• [ ] Any areas for improvement identified are addressed, or a plan is in place for future iterations.
• [ ] Measurable improvements in deployment times, reduced downtime, and enhanced scalability are achieved, aligning with stated OKR.
• [ ] The product team confirms that the architecture supports the application's long-term vision and growth objectives.

Take into consideration Accessibility/QA needs as well as Product, Technical, and Design requirements.

### Tasks
- [ ] https://github.com/department-of-veterans-affairs/va.gov-team/issues/64977
- [ ] https://github.com/department-of-veterans-affairs/va.gov-team/issues/88366
- [ ] https://github.com/department-of-veterans-affairs/va.gov-team/issues/88704
- [ ] https://github.com/department-of-veterans-affairs/va.gov-team/issues/90815
- [ ] https://github.com/department-of-veterans-affairs/va.gov-team/issues/90816
- [ ] https://github.com/department-of-veterans-affairs/va.gov-team/issues/90819
- [ ] https://github.com/department-of-veterans-affairs/va.gov-team/issues/90818
- [ ] https://github.com/department-of-veterans-affairs/va.gov-team/issues/90840
- [ ] https://github.com/department-of-veterans-affairs/va.gov-team/issues/90820
- [ ] https://github.com/department-of-veterans-affairs/va.gov-team/issues/91023
- [ ] https://github.com/department-of-veterans-affairs/va.gov-team/issues/91046
- [ ] https://github.com/department-of-veterans-affairs/va.gov-team/issues/91341
- [ ] https://github.com/department-of-veterans-affairs/va.gov-team/issues/91348
- [ ] https://github.com/department-of-veterans-affairs/va.gov-team/issues/91017

[jennb33]

@LindseySaari we need to communicate this to Jeff K and see what other ideas need to be included in this Epic and that he is aligned with it. Jeff should also perhaps synch with Bill to make sure we are breaking the right pieces into Microservices. Aparna to make sure that Bill and Jeff are in alignment.