:busts_in_silhouette: Compare data loading between kvikIO and Zarr engine

[weiji14]

Getting the hard numbers on how fast the GPU-based kvikIO engine is over the CPU-based Zarr engine.

Formula for calculation:

$$ \frac{Slower - Faster}{Slower} = \frac{16.0 - 11.9}{16.0} = 0.25625 = 25.53\%$$

I.e. kvikio engine takes ~25% less time than zarr engine to load the ERA5 subset dataset.

Note that:

• $Speed = \frac{Epochs}{Time}$
• Reported time taken is using the Median statistic (instead of the Mean) to better account for random variations due to cold start or caching.

Preview results in 2_compare_results.ipynb notebook.

TODO:

• [x] Get average data loading time over 10 epochs
• [x] Initialize Jupyter Notebook with kvikIO vs Zarr engine benchmark code
• [x] Plot bar graphs showing absolute time difference

[RichardScottOZ]

Have you tried this one with COGs?

[weiji14]

Have you tried this one with COGs?

No, kvikIO doesn't work with GeoTIFFs yet unfortunately, someone needs to implement that with cuFile and all. But I'm wondering if there's a way to hack a way together using kerchunk's tiff_to_zarr.