CI for source-postgres <= 15min

[alafanechere]

Tech spec available here.

Here are the planned tracks:

Track A: Rationalize the tests: identify test coverage overlaps

• Enable high test strictness level on source-postgres CAT -Identify which tests are redundant between Unit/Integration/CAT
• Refactor the test structure to reduce overlap
• Profile the java test execution using JMX to identify bottlenecks and address them.

Definition of done: We have a clean test suite structure without overlap + documentation of what is tested and where for source-postgres and it’s not being more wasteful of time than we’re comfortable with.

Estimated duration: 3 weeks

Track B: Improve Gradle execution in airbyte-ci

• Optimize the dind pattern to maximize resource utilization and make testcontainers work properly in this pattern
• Make sure the Gradle caching setup is optimal

Definition of done: We’re happy with the execution of gradle within airbyte-ci: gradle caching is great and the dind pattern is not impeding performances. Estimated duration: 1 week

Track C: Improve airbyte-ci observability and resource usage

• We want to be able to monitor Dagger engine resource (CPU/memory/disk) usage both locally and remotely.
• We want to be able to evaluate if a pipeline is under resource contention
• We want to give sufficient resources to remote runners to maximize performance

Definition of done: We are aware of the current resource usage of the dagger-engine when running connector test. We can make a reasonable cost/performance tradeoff. Our cluster resources are correctly sized to match this tradeoff Estimated duration: 1 week

Operating Plan

• Daily pairing between Augustin Lafanechere and Marius Posta
• Daily test performance measurement
• Weekly Tiger-team meeting led by Prateek Mukhedkar

Who does what

• Marius will focus on the connector test improvements
• Augustin will focus on the airbyte-ci / infrastructure improvements
• Augustin will help Marius fix CAT
• Marius will help Augustin improving Gradle in airbyte-ci

[alafanechere]

Weekly project update [source-postgres CI runs < 15mn] (EPIC)

:white_check_mark: What We've Shipped:

• A nearly ratified tech-spec
• A “benchmarking” environment to get unbiased CI duration
• A 10mn speed improvement by running test suites concurrently

:scientist: Experiments

• Proxying java dependencies on our cloudrepo. It worked but did not speed up the build.

  :construction_worker: What We Want to Ship this Week:

• [Connector effort] Flatten all postgres tests and remove overlap
• [Connector effort] Enable the full CAT test suite: requires fixing the incremental test set up
• [Infra effort] Provision a runner with more CPUs to check if we’re under resource constraints
• [Infra effort] Get monitoring recommendations from Dagger to make their engine more observable and identify bottlenecks.

Performance update

Last week CI duration: 32mn Today's CI duration: 22mn Speed gain: -10mn / -31%

[alafanechere]

Weekly Project Update

✅ What We've Shipped:

• Enabling the full CAT test suite on source-postgres. We fixed a bug in CAT which made it incapable of passing valid states to the connector (CAT did not produce namespaced stream state).
• Gradle caching improvement:
  • Gradle dependencies are cached in a persistent Dagger volume, avoiding costly downloads (~1mn speed gain).
  • The Gradle S3 cache is enabled to benefit from a shared build cache and incremental builds.
• We've made CAT execution faster, especially for source-postgres. CAT can now run its test suite in parallel. We also optimized CDC config used in CAT to use the test mode and avoid the Debezium wait time. CAT for source-postgres originally ran in ~7mn; we lowered this to ~2mn.
• We profiled source-postgres Java tests and confirmed the bottlenecks we anticipated:
  • Spawning 1 test database with test containers is costly.
  • The Debezium wait time makes some tests wait for 1mn doing nothing.
  • Starting the connector container is slow.
• A startup check in airbyte-ci to warn developers when they're not allocating all their CPUs to Docker.

👩‍🔬 Experiments:

• Provisioning a CI runner with 32 CPUs did not make the test execution faster.
• Try out cAdvisor to locally observe Dagger container resource usage: same results as the Java test profiling, with, of course, lower "resolution."

👷‍♀️ What We Want to Ship this Week:

• [Connector effort] Better orchestrate test containers and Debezium to lower Unit and Integration test duration.
• [Infra effort] Make sure Dagger engine metrics (CPU/RAM/Disk) can be monitored on Datadog: it will help us assess resource contention in the CI.
• [Infra effort] Sync with the Dagger team about their remote caching (magicache) logics. We want to make sure to understand cache eviction strategy and how we can tweak them if necessary.

🏃‍♂️ Performance Update:

• Original CI duration: 32mn
• Last week CI duration: 22mn
• Today's CI duration: 20mn
• Speed gain: -2mn / ~ -10%
• The speed gain is mainly due to the gradle caching improvement. The bottleneck remains the long unit test execution that we're tackling this week. All the improvements we've shipped so far should be observed on other Java connectors.

[alafanechere]

Weekly Project Update

TLDR: The target is reached, source-postgres test run in 12mn (14mn with airbyte-ci overhead)

:warning: Next week is our final project week!

:white_check_mark: What We've Shipped:

• [source-postgres specific effort]: We revisited some Debezium orchestration logic in source-postgres to limit the time-based logic that made tests wait a lot for debezium shutdown. 8mn speed gain!
• [infra effort] We’ve noticed a regression in Dagger distributed cache infrastructure. The read/write on the S3 bucket storing these caches was failing. Dagger still investigates the root cause but we believe they made an internal change as our cachemounts are now warmer than ever.
• [monitoring effort] Thanks to Datadog support we’re now collecting detailed resource usage metrics for our dagger-engine on our CI cluster.
• [monitoring effort] Create a Metabase question tracking CI performance on main Java connectors on PRs

What Got in the Way:

• [monitoring effort] On Datadog we can’t distinguish dagger’s engine used by the platform team from the connector ones. We opened a PR to create connector/platform-specific dagger engines.
• [infra effort] We’ve yet to receive the conclusion about the Dagger infra “incident” on their distributed caching feature.
• [infra effort] We still face occasional rate limit error on docker image pulls during test.

:construction_worker: What We Want to Ship in the Final Project Week:

• [global java connector effort] Better manage CDK dependencies: fat jars in connector image adds overhead on connector startup. Improving the management of CDK dependencies can reduce the image size which can improve build and test duration and connector performances in production.
• [source-postgres specific efforts] Refactoring the tests setup/teardown logic to re-use testcontainers test database will boost test performances: no need to wait for postgres container warm-up time on each test.
• [global java connector effort] Produce documentation on what we did at the connector test level for postgres and how other connectors can “copy/paste” these improvements.
• [monitoring effort] Create a Datadog dashboard to monitor our CI Cluster for connectors. It will help us assess the health of the cluster and adapt our instance sizes if needed.
• [monitoring effort] Create alerts on this Metabase question when source-postgres CI performance regresses.
• [infra effort] Fix the docker pull rate limiting issue (again).
• [infra effort] Adapt our resource sizing on the CI cluster following monitoring observation.
• [infra effort] Gain visibility with the prod-eng team about the roadmap for our CI cluster.

🏃‍♂️ Performance Update

• Original CI duration: 32mn
• Last week's CI duration: 19mn
• Today's CI duration: 12mn
• Week to week speed gain: -8mn / ~ -40%
• Overall speed gain: 20mn / ~ -60%

The speed gain is due to the Debezium orchestration improvement.