RFC: Contract-Based Decoupling for Damavand

[shahinism]

RFC: Contract-Based Decoupling for Damavand

1. Abstract

This RFC proposes an evolution of Damavand's architecture to implement a contract-based approach for decoupling application and infrastructure. This proposal aims to enhance Damavand's core promises while simplifying the developer experience and maintaining its cloud-agnostic nature.

2. Background

Damavand is a Python framework implementing the Application Resource Controller (ARC) pattern, designed to simplify cloud development and increase developer productivity. The current implementation tightly couples application and infrastructure code, which, while powerful, can present challenges in terms of complexity and learning curve.

3. Proposal

We propose to evolve Damavand's architecture by moving the abstraction layer (controller) out of the application and introducing a contract-based approach. This new architecture will consist of:

1. A versioned contract data structure
2. Black-box treatment of application source code
3. Shared configuration management system
4. Runtime configuration parsing based on the contract

4. Detailed Design

4.1 Contract Data Structure

• Define a versioned data structure as a contract on the Damavand side
• This contract is shared and managed independently of applications
• Implemented in Python, maintaining the native Python implementation promise

4.2 Application Input

• Damavand will only require the application source code as input
• The application is treated as a black box, allowing developers to focus on business logic

4.3 Deployment Process

• Damavand deploys the application based on defined requirements
• After deployment, Damavand populates the contract data structure
• The populated structure is stored in a shared config management system

4.4 Runtime Configuration

• Applications retrieve necessary information at runtime by parsing the configuration based on the contract

4.5 Configuration Contexts

1. Application-level configuration: Managed by application developers
2. Infrastructure configuration: Managed through the contract-based system

4.6 Damavand Abstraction Layers

1. Layer 1: One-to-one mapping for seasoned engineers
   • Direct interface for creating cloud resources with best practices
   • Utilizes Pulumi/TFCDK under the hood, maintaining the "no extra dependencies" promise
2. Layer 2: High-level abstraction
   • Built on top of Layer 1
   • Cloud-agnostic interface based on application type

5. Alignment with Damavand's Promises

1. Open/Closed Principle: The contract-based approach enhances this by allowing infrastructure modifications without changing application code.
2. Template-oriented Development: Maintained through Layer 2 abstractions.
3. No Extra Dependencies: Continues to use only Pulumi/TFCDK.
4. Native Python Implementation: The entire system remains implemented in Python.
5. Multi-cloud Targets: Enhanced through the contract-based approach, providing a more flexible way to abstract cloud-specific details.

6. Benefits

1. Simplified developer experience while maintaining Damavand's core strengths
2. Clear separation of concerns between application and infrastructure
3. Enhanced flexibility in cloud resource management
4. Improved cloud-agnostic application development
5. Easier maintenance and updates of Damavand

7. Challenges and Considerations

1. Designing a flexible yet comprehensive contract structure
2. Ensuring backward compatibility of the contract versions
3. Security considerations for shared configuration management
4. Performance implications of runtime configuration parsing (network could be the only bottleneck I assume).

8. Implementation Plan

1. Develop the contract data structure in Python
2. Create adapters for existing Pulumi/TFCDK components to work with the new contract system
3. Implement the shared configuration management system
4. Develop Layer 1 and Layer 2 abstractions

9. Backwards Compatibility

• We don't expect current production users, so the focus is on future completeness.
• Migration tools will be provided to convert (upgrade or downgrade) contract versions.

10. Security Considerations

• The shared configuration management system will use encryption at rest and in transit
• Access to the configuration system will be strictly controlled using fine-grained permissions
• Sensitive information in the contracts will be handled using secure secret management practices

11. Multi-cloud Enhancement

• The contract-based approach allows for more flexible abstraction of cloud-specific details
• Layer 2 abstractions will provide cloud-agnostic interfaces for common application types
• Contracts can be designed to accommodate cloud-specific features while maintaining a consistent interface

12. Future Work

1. Develop tooling to assist in contract creation and validation
2. Explore integration with existing configuration management systems
3. Create templates for common application types using Layer 2 abstraction
4. Enhance multi-cloud capabilities through advanced contract designs

13. Conclusion

This proposal evolves Damavand into a more flexible and developer-friendly framework while maintaining its core benefits of cloud abstraction and best practice implementation. It upholds Damavand's original promises while addressing challenges in the current implementation.

[shahinism]

RFC: Contract-Based Decoupling for Damavand - Extension

14. Architectural Layers Refinement

After further discussion and analysis, we've refined our understanding of how the proposed changes align with and enhance the ARC (Application Resource Controller) design. This section elaborates on the architectural layers and their roles in the Damavand ecosystem.

14.1 Layer Structure

1. Layer 0: Landing Zone

   • Closest to the cloud provider architecture
   • Highly coupled with each cloud provider
   • Manages users, roles, access control, cost monitoring, and general monitoring setup

2. Layer 1: Infrastructure (Resource Layer in ARC)

   • Subdivided into two abstraction levels: a. Level 1: Close to cloud resources, implementing best practices b. Level 2: First step of cloud-agnostic infrastructure, abstracting based on application requirements

3. Layer 2: Control (Controller Layer in ARC)

   • Handles application layer abstraction
   • Removes dependency requirements from developers
   • Manages how data is read and written, allowing developers to focus on business logic

4. Layer 3: Application

   • Where the business logic resides

14.2 Key Terminology

• Application: The business logic component
• Product: A set of infrastructure, controller, and application components

15. Alignment with ARC Design

The proposed contract-based approach enhances the ARC design by providing clearer boundaries between system components while maintaining the core ARC principles:

1. Landing-zone: enhances the communication between layer 0, and layer 1.
2. Controller: enhances the communication between layer 1 and layer 2.

16. Interchangeability and Customization

A key feature of this architecture is its modularity and flexibility:

1. Component Interchangeability:

   • Users can choose to use all or part of the ARC framework
   • Each layer (Infrastructure, Control, Application) can be used independently or in combination

2. Customization Options:

   • Infrastructure Layer: Users can customize resource configurations or extend the provided abstractions
   • Control Layer: Custom controllers can be implemented to handle specific application requirements
   • Application Layer: Developers have full freedom in implementing business logic

3. Mix and Match:

   • Organizations can use Damavand's Infrastructure Layer with their own custom Control and Application Layers
   • Alternatively, they might use ARC's Control Layer with custom Infrastructure and Application components, as long as their layers comply with the relative contracts.

4. Gradual Adoption:

   • The modular nature allows for gradual adoption of ARC components
   • Teams can start with one layer (e.g., Infrastructure) and progressively adopt others as needed

5. Extension Points:

   • Each layer provides extension points for adding custom functionality
   • Users can extend the contract definitions to include organization-specific requirements

17. Benefits of the Refined Architecture

1. Clear Separation of Concerns: Each layer has a well-defined responsibility
2. Flexibility: Users can adopt parts of the framework that best suit their needs
3. Scalability: The layered approach allows for easier scaling of individual components
4. Maintainability: Clear boundaries make it easier to update and maintain each layer independently
5. Cloud Agnosticism: The abstraction layers provide a path to true cloud-agnostic development

18. Conclusion

This extension to the RFC clarifies how the proposed contract-based approach aligns with and enhances the ARC design. By providing clear architectural layers and emphasizing interchangeability and customization, Damavand offers a flexible, powerful framework for cloud-native development that can adapt to a wide range of organizational needs and preferences.

[farbodahm]

• Damavand deploys the application based on defined requirements

@shahinism How Damavand is going to specify and deploy the resources described in the contract?

Assume we are talking about a Spark application. In the contract, application mentions that, I want:

1. 2 Input databases (Also with more detail, I want Iceberg databases).
2. 2 Input buckets (for any reason).

Now Damavand is going to:

1. Determine and create the resources on runtime based on contract?
2. Or developer needs to specify the buckets, databases, ... somewhere in the Python source code explicitly?

If second, how we are going to map that contract to the explicit code written by the developer?

[kkiani]

Thank you, @shahinism. This is much clearer now. I think we should adding a section on the extraction of the ARC design pattern from Damavand. This addition is important because, in scenarios like mix and match or gradual adoption, users or organizations might choose to partially use a subset of the frameworks or none at all, while still independently following the ARC design pattern.

[shahinism]

@farbodahm the answer to your question, comes down to the contract design itself, and the level of abstraction we consider. Currently for the control layer, as mentioned in the RFC, the intention is to limit the optins as much as possible, and only provide the developer, with what they need, a data frame. any extra information like the storage layer, will be abstracted in the control layer (in Spark example it'd be Sparkle).

The final outcome would be very opinionated, but the goals are clear:

1. Enable fast.
2. keep developers focused on the business logic.

[kkiani]

I am closing this for main two reasons:

1. Mainly, this is not a priority for time being.
2. The proposal has conflict with the pillars of the project. As a result, making such change means refactoring everything from ground up. If this become needed in feature, would be better to start fresh.