Convert github blogpost to live demo

dask / old-dask-examples

Collection of dask example notebooks

57 stars 22 forks source link

Closed mrocklin closed 8 years ago

mrocklin commented 8 years ago

This blogpost http://continuum.io/blog/dask-distributed-cluster by @cowlicks might make a fun live notebook with binder.

This work would include the following:

Integrate with cluster developers to host the github data on google storage
Recreate some of the analysis and prose in the blogpost as a notebook within this repository
Think about how these examples should be changed to be more interesting to interactive users.

cowlicks commented 8 years ago

Binder requires we host the data on google storage?

You can spin up a cluster with binder? (wow)

mrocklin commented 8 years ago

I suggest that we don't yet worry about the cluster but just do this on a single machine. They're beefy

At the moment binder is hosted on GCE, so GS is free.

cowlicks commented 8 years ago

This is live, but there are a few probelms:

I have an incorrect comment in In[3] stating the size is 4.6GB when it is 2.7GB. Some evidence for not using comments cc @danielfrg
Working with one month of data, makes things too slow to feel "interactive". Doing a frequency count on the event type takes around 8 minutes to complete. This is the first real compute. Options are:
- Reduce the amount of data. I used 2 days when I did this during a live demo during a talk
- Use a better storage format. Maybe just unziping all the data? What is a good on disk store for json?

cowlicks commented 8 years ago

With raw json, the same computation above takes 5 minutes. Which is still too long.

mrocklin commented 8 years ago

The binder machines shrunk down quite a bit recently. They used to be really powerful.