Closed tomwhite closed 6 months ago
Comparing GCP with AWS from yesterday:
$ duckdb benchmark.db
D select name, duration from test_run where name like '%lithops%' and duration is not null;
┌────────────────────────────────────────────────────────────┬────────────────────┐
│ name │ duration │
│ varchar │ double │
├────────────────────────────────────────────────────────────┼────────────────────┤
│ test_quadratic_means_xarray[configs/lithops_aws.yaml-50] │ 34.91558289527893 │
│ test_quadratic_means_xarray[configs/lithops_aws.yaml-500] │ 56.183167934417725 │
│ test_quadratic_means_xarray[configs/lithops_aws.yaml-5000] │ 66.49491095542908 │
│ test_quadratic_means_xarray[configs/lithops_gcp.yaml-50] │ 43.74802899360657 │
│ test_quadratic_means_xarray[configs/lithops_gcp.yaml-500] │ 93.0142970085144 │
│ test_quadratic_means_xarray[configs/lithops_gcp.yaml-5000] │ 132.36643505096436 │
└────────────────────────────────────────────────────────────┴────────────────────┘
I've updated this PR so that one runtime (spec) is run per pytest invocation. This works well with a GitHub Actions matrix (as suggested by @TomNicholas here: https://github.com/cubed-dev/cubed-benchmarks/issues/5#issue-2163640415), which I've also included for a single-threaded executor and Lithops on AWS. These are passing: https://github.com/cubed-dev/cubed-benchmarks/actions/runs/8188522733
There's more work to do of course (more benchmarks, merge db files, make config naming consistent, run on GCP, ...), but I think this is at a good point to merge. Any thoughts @TomNicholas?
Initial work for #5.
I think it would be useful to standardise the naming of the config and conda env files. Something like
{executor-name}_{cloud}
, so we'd have:single-threaded
(there's no cloud for this one, so we just drop it, alternatively we could havesingle-threaded_nocloud
or something)lithops_aws
lithops_gcp
If we wanted more options within a config (e.g. because we wanted to try a different serverless runtime), we could just add more modifiers, e.g.
lithops_gcp_cloudrun
. But I don't think we need that yet.