search

pangeo-data / storage-benchmarks

testing performance of different storage layers
Apache License 2.0
12 stars 1 forks source link

using different dask.get options #21

Open rabernat opened 6 years ago

rabernat commented 6 years ago

I just thought I would share some of my own (messy) experiments with benchmarking file uploading. (I currently really need to upload a bunch of data so I am motivated to improve this.)

https://gist.github.com/rabernat/ba26802071271f088f5b4f8f9f5db81d

A key issue is to try operations with different dask.get options (single-threaded, multithreaded, multiprocessing, distributed). They can behave very differently.

kaipak commented 6 years ago

@rabernat will take a look at these and include a form of this in the Dask tests which I hope to start working on in the next week.