How to do multi-threading within a jupyter notebook

[paulroth3d]

Is there a way we can make dividing up work among threads so we can make especially long running processes faster, and take advantage of multiple cores?

In some ways Jupyter is really good at sketches, to understand data and to massage it to something you can work with.

Would be interesting to show how we can handle larger sets of data.

[paulroth3d]

There are some libraries that can help here:

• josdejong/workerpool being one of them - especially because they don't require a pre-compiled js file to handle the results.

• Need an example data set that could help with testing

• Memory management is very important to this discussion, including how the data is passed to the workers, rehydrated, and then mapped back. Also, some people that have tried this just have way too many things in memory, and don't allow for garbage collection.

[paulroth3d]

Discuss webworker-threads library https://github.com/audreyt/node-webworker-threads