[Escalation] e2e pipeline blocker

[sfreudenthaler]

Placeholder

Raised by @bryanboza, his e2e rock is blocked because he's having trouble getting them into the pipeline. This needs significant refinement before we can pick it up.

[spbolton]

Configuration Discrepancies and Testing Strategy

Context

Currently, there is a discrepancy between the environment configuration used during CI (integration/Postman tests) and the Docker Compose setup provided for running dotCMS. While Maven-based configurations ensure consistency and repeatability in CI, the Docker Compose file, used for local development or example setups, does not undergo similar testing. This raises concerns about configuration drift, reliability, and how the environment aligns with what customers might use.

---

Key Issues and Questions

1. Configuration Consistency Across Testing and Deployment

• Problem: The Maven configuration in CI is distinct from the Docker Compose setup. Without testing the latter, there’s no assurance that it works as expected with default configurations.
• Questions:
  • Should integration and Postman tests use the same configuration as a default customer setup (via Docker Compose)?
  • If different configurations are needed, where and how should these be tested?
  • Are we tailoring the tests to ensure they pass, instead of reflecting real-world setups?

2. Centralized Source of Truth for Configuration

• Problem: Configuration exists in multiple places—Maven POM, Docker Compose, and potentially individual developer setups. This creates the risk of divergence and inconsistencies.
• Questions:
  • Can we generate a Docker Compose file dynamically from the Maven configuration to establish a single source of truth?
  • Alternatively, can the Maven configuration rely on a base Docker Compose file and extend it for testing purposes?

3. Alignment Between Parent POM and Runtime Behavior

• Problem: The parent POM defines a base configuration for running dotCMS, extended in submodules like Postman and E2E tests. However, these configurations may differ from runtime behavior in Docker Compose.
• Questions:
  • Should we standardize around default values in the parent POM and remove overrides in test modules?
  • How do these runtime behaviors differ from Docker Compose, and what adjustments are necessary?

4. Testing the Delivered Docker-Compose Configuration

• Problem: The Docker Compose file provided to customers is not explicitly tested. There is no mechanism to ensure that this configuration remains functional as changes are introduced.
• Questions:
  • Do we need separate testing to validate the Docker Compose setup?
  • Should E2E tests be used to verify this configuration, or should it be tested through another mechanism?
  • How do we ensure the Docker Compose file is always in a working state?

---

Key Decisions to Address

1. Validity of Current Testing Configurations

   • Are we testing with a configuration that reflects real-world use cases or customer environments? If not, adjustments should be prioritized to improve reliability.

2. Synchronization of Configuration

   • How do we ensure synchronization between CI, Docker Compose, and development environments?
   • Centralizing configuration management (e.g., environmental profile-based setups) could eliminate duplication and reduce complexity.

3. Eliminating Custom Configurations

   • Can we remove one-off configurations in favor of standardized, profile-driven setups that adapt based on the environment (e.g., CI vs. local development)?

4. Testing the Docker Compose Configuration

   • Should we introduce automated tests (e.g., E2E or smoke tests) to validate the Docker Compose configuration as part of CI/CD?
   • Alternatively, should there be a dedicated process or task to ensure the Docker Compose file is functional and in sync with Maven-based configurations?

---

Possible Solutions

1. Unified Testing Approach

• Extend integration and Postman tests to include scenarios run against Docker Compose.
• Define a process to validate default Docker Compose setups in CI.

2. Centralized Configuration

• Introduce a script or mechanism to generate Docker Compose files from Maven configurations (or vice versa) to maintain a single source of truth.
• Modify the Maven setup to extend from a baseline Docker Compose configuration.

3. Environmental Profiles

• Adopt a profile-based configuration system where defaults are defined centrally, and overrides are applied for specific environments (e.g., CI, local dev).

4. Testing Workflow Refinement

• Establish a task to review and synchronize configurations between Maven and Docker Compose during development.
• Automate checks for consistency to avoid manual tracking.

5. Testing Delivered Docker Compose

• Option 1: Use E2E tests to validate the Docker Compose configuration explicitly. This ensures that the customer-delivered setup works as expected.
• Option 2: Create a lightweight testing process (e.g., smoke tests) focused solely on verifying that the Docker Compose file is functional and can start services correctly.
• Option 3: Include Docker Compose validation as part of the CI/CD pipeline to ensure every release is tested against the delivered configuration.

---

Questions to Resolve

• Which configuration should be the primary source of truth—Maven or Docker Compose?
• How much flexibility is needed in configurations to accommodate different testing and runtime environments?
• Should developers' local environments default to Docker Compose, with CI ensuring compatibility?
• How do we best integrate testing for the Docker Compose file into our workflow to ensure it remains functional and reliable?
• How do we maintain a consistent composable design that allows for structured override of default values only where needed and in limited sets of environments to prevent configuration insconsitencies and inability to change.