Fix conda python reference issue (showed up in yarn), add example with standalone mode, make logging super verbose

nteract / coffee_boat

☕⛵WIP PySpark dependency management

BSD 3-Clause "New" or "Revised" License

22 stars 10 forks source link

Fix conda python reference issue (showed up in yarn), add example with standalone mode, make logging super verbose #37

Closed holdenk closed 6 years ago

holdenk commented 6 years ago

Conda python reference issue handled with our good friend sed.... The conda clone doesn't work well on a real cluster so avoid that.

Add a sample nteract notebook for Spark standalone deployments

Make logging really super verbose because it's probably going to explode once we run it in other places and if they send us their notebook might as well see everything.

holdenk commented 6 years ago

So I've tested it on spark standalone & Spark+yarn on Dataproc and it seems to work. If it passes CI and no one objects I'll merge it.