Dask/GCP Kubernetes tests now work correctly. So far we have a parameterized write test that tries various combinations of writing a 10 GB Dask/Xarray dataset with varying workers, chunk sizes, FUSE/gcsfs, and others. Preliminary results have been posted on from my fork: https://kaipak.github.io/storage-benchmarks/
I've moved results folder to a tracked location. Might have to move this again and find a nicer workflow depending on how messy this gets.
Made some needed updates on code styling and comments. Added a bunch of docstrings and comments where we seemed to really need it.
A bit of a warning. Even though the Dask GCP test has a (very basic) check to not run if its not running in Pangeo GCP, it still takes a long time to bypass the benchmarks. Not sure why this is, but the tests won't actually run.
Lots of updates on this PR.
A bit of a warning. Even though the Dask GCP test has a (very basic) check to not run if its not running in Pangeo GCP, it still takes a long time to bypass the benchmarks. Not sure why this is, but the tests won't actually run.