search

delta-io / delta

An open-source storage framework that enables building a Lakehouse architecture with compute engines including Spark, PrestoDB, Flink, Trino, and Hive and APIs
https://delta.io
Apache License 2.0
7.62k stars 1.71k forks source link

[Infra] [WIP] Add test report listener to delta-spark and bucket test suites by estimated runtime #3694

Open scottsand-db opened 2 months ago

scottsand-db commented 2 months ago

Which Delta project/connector is this regarding?

Description

Adds a test report listener to delta-spark to record test runtimes. This listener logs test runtime metrics and writes metrics to a csv file (per CI shard, per attempt, per jvm, per thread) and uploads it as a test artifact.

We can use the output of this job to learn how to better bucket our slowest + largest tests.

https://github.com/delta-io/delta/actions/runs/11002528077?pr=3694

image

How was this patch tested?

Locally and GitHub CI.

Does this PR introduce any user-facing changes?

No.

scottsand-db commented 1 month ago

Another observation of mine: not sure if our hashing is even/fair enough.

The max difference between any two threads in a given shard is 68% (i.e. one thread has 68% more tests assigned to it than the other).

DSL: Scala 2.12.18, Shard 0

[info] Test group 0 contains 40 tests [info] Test group 1 contains 34 tests [info] Test group 2 contains 33 tests [info] Test group 3 contains 36 tests

DSL: Scala 2.12.18, Shard 1

[info] Test group 0 contains 24 tests [info] Test group 1 contains 37 tests [info] Test group 2 contains 35 tests [info] Test group 3 contains 28 tests

DSL: Scala 2.12.18, Shard 2

[info] Test group 0 contains 32 tests [info] Test group 1 contains 19 tests [info] Test group 2 contains 29 tests [info] Test group 3 contains 31 tests

scottsand-db commented 1 month ago

image

24 minute disparity between first and last