search

scisprints / 2018_05_sklearn_skimage_dask

BSD 3-Clause "New" or "Revised" License
6 stars 2 forks source link

Computational infrastructure #3

Open mrocklin opened 6 years ago

mrocklin commented 6 years ago

During the sprint I expect that people will work most of the time on their laptops. However it might also be useful to have some distributed services available, either on the cloud with a JupyterHub-Kubernetes deployment or on an HPC system. Is this something that we want to manage centrally or should we expect people to have their own preferred computational systems?

NelleV commented 6 years ago

I can probably try to have something set up here at UC Berkeley, based on jupyterhub or the xsede computing services.

NelleV commented 6 years ago

@yuvipanda happens to have a group of machines waiting for a group of users to use them.

Yuvi, do you mind describing what you could maybe provide?

yuvipanda commented 6 years ago

The Pacific Research Platform has a Kubernetes Cluster they're eager to put to test with real world use cases. This gives you a namespace in a Kubernetes cluster with CPU, Memory, disk and some GPUs. I already have a JupyterHub running there, and I'm also experimenting with providing SSH Based access(check it out with ssh -t <github-username>@kubessh.nautilus.optiputer.net -p 32222 -- --image=jupyter/base-notebook, with your github ssh key & any docker image).

If y'all can give me more details on what kinda workloads you'd likely be running I can say more :)

mrocklin commented 6 years ago

I would default to something Pangeo-like. People will want the common set of analytics libraries like scikit-learn, scikit-image, numpy, pandas, etc..

They'll also want to use something like dask_kubernetes to scale these out on the cluster. This scaling will likely be very bursty, with use somewhere between minutes and hours. I expect that most distributed deployments will use something like 50 cores, though we will likely want to expand out to 1000 or so if available.

On Thu, May 3, 2018 at 5:51 PM, Yuvi Panda notifications@github.com wrote:

The Pacific Research Platform has a Kubernetes Cluster they're eager to put to test with real world use cases. This gives you a namespace in a Kubernetes cluster with CPU, Memory, disk and some GPUs. I already have a JupyterHub running there, and I'm also experimenting with providing SSH Based access(check it out with ssh -t @kubessh. nautilus.optiputer.net -p 32222 -- --image=jupyter/base-notebook, with your github ssh key & any docker image).

If y'all can give me more details on what kinda workloads you'd likely be running I can say more :)

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/scisprints/2018_05_sklearn_skimage_dask/issues/3#issuecomment-386448372, or mute the thread https://github.com/notifications/unsubscribe-auth/AASszBlEsPfzsZ-eNMyEExJqWhGLhRt7ks5tu3vSgaJpZM4TrxZV .