ML Notebook Image - Githubissues

informatics-lab / our-pangeo

A helm chart for deploying a custom flavour Pangeo stack

10 stars 5 forks source link

ML Notebook Image #28

Open jacobtomlinson opened 5 years ago

jacobtomlinson commented 5 years ago

Recently we added a new image to the spawner called ML Notebook. This is intended to be an environment which is more optimised for ML workloads.

Currently the only difference between it and the default environment is that is has 7 CPU and 28GB RAM compared to the standard 2 CPU 6GB RAM.

The intention in the long run is for the ML notebook to also have:

[ ] 1 GPU
[ ] ML specific libraries
- [ ] CuPy
- [ ] CuDF
- [ ] Tensorflow
- [ ] PyTorch
- [ ] GPy
- ... etc
[ ] Better auditing and cost allocation (these will be expensive and we need more accountability)

jacobtomlinson commented 5 years ago

@RPrudden I would love your input on this.

RPrudden commented 5 years ago

GPy would be another useful library to include.

A general comment: I think this image is addressing two kinds of workload. The first is deep learning, which is fairly intensive on both compute and memory. The second is heavy linear algebra workloads, which are possibly even more compute intensive, but don't tend to need as much memory. Extra CPUs and/or GPUs are useful for both, but there it might be helpful to consider them separately.

jacobtomlinson commented 5 years ago

Thanks for that.

Memory and CPU tends to be linked at the infrastructure level, if we increase one it increases the other. We could introduce some other hardware families, but it's unlikely to be worth the investment. Therefore we may as well treat it as "free" extra memory.