cubed-dev / cubed-benchmarks

Automated benchmark suite for testing the performance of cubed
Apache License 2.0
2 stars 0 forks source link

Have separate conda envs for different executors and clouds #11

Closed tomwhite closed 6 months ago

tomwhite commented 6 months ago

Initial work for #5.

I think it would be useful to standardise the naming of the config and conda env files. Something like {executor-name}_{cloud}, so we'd have:

If we wanted more options within a config (e.g. because we wanted to try a different serverless runtime), we could just add more modifiers, e.g. lithops_gcp_cloudrun. But I don't think we need that yet.

tomwhite commented 6 months ago

Comparing GCP with AWS from yesterday:

$ duckdb benchmark.db
D select name, duration from test_run where name like '%lithops%' and duration is not null;
┌────────────────────────────────────────────────────────────┬────────────────────┐
│                            name                            │      duration      │
│                          varchar                           │       double       │
├────────────────────────────────────────────────────────────┼────────────────────┤
│ test_quadratic_means_xarray[configs/lithops_aws.yaml-50]   │  34.91558289527893 │
│ test_quadratic_means_xarray[configs/lithops_aws.yaml-500]  │ 56.183167934417725 │
│ test_quadratic_means_xarray[configs/lithops_aws.yaml-5000] │  66.49491095542908 │
│ test_quadratic_means_xarray[configs/lithops_gcp.yaml-50]   │  43.74802899360657 │
│ test_quadratic_means_xarray[configs/lithops_gcp.yaml-500]  │   93.0142970085144 │
│ test_quadratic_means_xarray[configs/lithops_gcp.yaml-5000] │ 132.36643505096436 │
└────────────────────────────────────────────────────────────┴────────────────────┘
tomwhite commented 6 months ago

I've updated this PR so that one runtime (spec) is run per pytest invocation. This works well with a GitHub Actions matrix (as suggested by @TomNicholas here: https://github.com/cubed-dev/cubed-benchmarks/issues/5#issue-2163640415), which I've also included for a single-threaded executor and Lithops on AWS. These are passing: https://github.com/cubed-dev/cubed-benchmarks/actions/runs/8188522733

There's more work to do of course (more benchmarks, merge db files, make config naming consistent, run on GCP, ...), but I think this is at a good point to merge. Any thoughts @TomNicholas?