nteract / coffee_boat

☕⛵WIP PySpark dependency management
BSD 3-Clause "New" or "Revised" License
22 stars 10 forks source link

Add dynamic package support (e.g installing a package into an already launched ship) #12

Closed holdenk closed 6 years ago

holdenk commented 6 years ago

We have a wrapper shell script (at least for conda) that we could source files in and pip install we could then use sc.addFile + foreach to install the new packages at runtime

rgbkrk commented 6 years ago

🤔

holdenk commented 6 years ago

I mean it's certainly less good than shipping the pre-packaged conda env, but while your iterating would make it faster to work. Might do a warn to remind people to move it up later.

Also since we do the addFile we'd cover the executor restart + dynamic scaling cases.

holdenk commented 6 years ago

Ok we now (in theory) support this. I'm slightly terrified for the first people who use it, buuut that's what Alpha's all about :)