search

awslabs / aws-glue-libs

AWS Glue Libraries are additions and enhancements to Spark for ETL operations.
Other
648 stars 305 forks source link

Integration with Jupyter/Notebooks #27

Closed shewbaka closed 4 years ago

shewbaka commented 5 years ago

Seems there is no really easy way to hook up the local setup for use in a Jupyter notebook. I think I have made every possible attempt to get this to work. Is this intentional or is there a way to do this. It would be fantastic if local testing in a notebook environment was possible.

shewbaka commented 5 years ago

Eh, maybe fake-awsglue is the way to go?

Saziba commented 4 years ago

I was able to install a Glue Python 3.6 ipython kernel.

Did you try following these steps: https://ipython.readthedocs.io/en/latest/install/kernel_install.html#kernels-for-different-environments ?

shewbaka commented 4 years ago

Totally forgot about this until now. To answer your question, yes. To be clear, that wasn't the issue. Kernels are a dime a dozen -- I think what I was dealing with was getting glue to play nice with my local spark dist while idiomatically interacting with aws. In any case, at some point a year ago I was able to do just that, but shortly thereafter realized trying to use glue in the first place was basically putting lipstick on a pig. As such, I decided to mostly forego working with glue, instead opting to "just" stick with spark.