search

pangeo-data / storage-benchmarks

testing performance of different storage layers
Apache License 2.0
12 stars 1 forks source link

Run benchmarks on AWS #46

Open jreadey opened 6 years ago

jreadey commented 6 years ago

Run benchmarks on AWS with a Kubernetes cluster.

jreadey commented 6 years ago

Hey @rsignell-usgs - Could @kaipak use your AWS cluster for running these benchmarks?

rabernat commented 6 years ago

Yes, this would be great.

Rich, we are eventually writing a paper about this, and we would be happy to include you as an author.

kaipak commented 6 years ago

Had a long conversation with @jreadey about this and other related topics last night and it should be relatively straightforward to run the tests in AWS with only minor changes in code. Not having to set up a new cluster for all this could potentially save a lot of time.

rsignell-usgs commented 6 years ago

Absolutely. I will invite @kaipak to a whitelisted org when I get to a computer.

rsignell-usgs commented 6 years ago

I have invited @kaipak to the whitelisted cdi-workshops org, which should give access to http://pangeo.esipfed.org. We added autoscaling last week, thanks to help from @jacobtomlinson, so you should be able to request up to 80 workers, which should give you 160 cpus. Is that sufficient, or do we need to bump that up?

Also, please be advised it can take up to 10 minutes to get the workers you request.

kaipak commented 6 years ago

Thanks much @rsignell-usgs