Adjust TPC-H testing on CI

coiled / benchmarks

BSD 3-Clause "New" or "Revised" License

32 stars 17 forks source link

Adjust TPC-H testing on CI #1530

Closed hendrikmakait closed 2 months ago

hendrikmakait commented 3 months ago

I don't see a point in testing 10 GB workloads. If we're cost-conscious, I'd rather reduce the frequency of testing.

fjetter commented 3 months ago

how long is one test run (compared to the rest of the suite)?

hendrikmakait commented 3 months ago

Coiled estimates that the cluster cost for test_dask.py increase from ~$0.14 to $2.70 per run, which will add up to an increase of ~$77/month. test_optimization.py increases from ~$0.10 to ~$0.34, adding up to an increase of ~$7/month.

hendrikmakait commented 2 months ago

https://github.com/coiled/benchmarks/pull/1534 entirely disables all automated tests for PRs and reduces scheduled testing to once per week. This should mitigate cost concerns.