search

risingwavelabs / risingwave

Best-in-class stream processing, analytics, and management. Perform continuous analytics, or build event-driven applications, real-time ETL pipelines, and feature stores in minutes. Unified streaming and batch. PostgreSQL compatible.
https://go.risingwave.com/slack
Apache License 2.0
7.07k stars 581 forks source link

make ci e2e scripts meta backend agnostic #19527

Open BugenZhao opened 3 days ago

BugenZhao commented 3 days ago

In https://github.com/risingwavelabs/risingwave/pull/19152, we enable running regular e2e tests with Postgres and MySQL meta backend in CI. This is achieved by passing different risedev profile in different CI steps.

However, there are other e2e tests like e2e-source-test, e2e-sink-test, and slow-e2e-test, where multiple profiles might be used in the same script. This makes it difficult to cover them using the same approach. It's also easy to miss a step.

We may consider making e2e scripts meta backend agnostic to address this. Here's my proposal:

  1. Specify meta backend to ci and do not use concrete meta backend service. 
      ci-kafka:
    config-path: src/config/ci.toml
    steps:
      - use: minio
-     - use: sqlite
      - use: meta-node
-       meta-backend: sqlite
+       meta-backend: ci
      - use: compute-node
        enable-tiered-cache: true
      - use: frontend
      - use: compactor
      - use: kafka
        user-managed: true
        address: message_queue
        port: 29092
      - use: schema-registry
        user-managed: true
        address: schemaregistry
        port: 8082
  2. Run e2e tests with different env var RW_CI_META_STORE_URL. Use DB services from the docker-compose CI environment.
  3. risedev-dev will parse the env var to generate CLI args for meta-node service.
  4. Before launching services, risedev-dev executes database-specific commands to reset the database for meta store.