search

coiled / data-science-at-scale

A Pythonic introduction to methods for scaling your data science and machine learning work to larger datasets and larger models, using the tools and APIs you know and love from the PyData stack (such as numpy, pandas, and scikit-learn).
MIT License
112 stars 38 forks source link

Running Coiled from local workstation #7

Closed hugobowne closed 4 years ago

hugobowne commented 4 years ago

Ideally, on Sept 15, I could run this all on Coiled (from NB // Lab in Coiled!) and learners spin up binder

If this isn't possible, next best is running Coiled clusters from a local NB // Lab

@jrbourbeau suggested

  1. Create a software environment from the conda environment.yml file in the binder/ directory
  2. Create a cluster configuration that uses that software environment
  3. In each notebook, instead of just calling Client(…) with various parameters like n_workers=4, etc., create a coiled.Cluster with your new cluster configuration and then do client = Client(cluster)

also look here: https://docs.coiled.io/user_guide/jupyter.html

3rd option: all local