search

qtangs / sagemaker-notebook-container

Docker images that replicate the Amazon SageMaker Notebook instance.
MIT License
58 stars 30 forks source link

PySpark kernell #4

Closed oleksandr-lysenko closed 2 years ago

oleksandr-lysenko commented 4 years ago

Hi Quy Tang.

That is a great job, thank you.

Could you add (or provide instructions on how to do that on Windows) "Sparkmagic (PySpark)" kernel for Jupiter? Unfortunately, I am completely new in Docker technology and was able just to run a container you provided, not to build my own. But I need a kernel that will run Spark applications like it is done in AWS.

P.S. Attempt to include pyspark module in existing python kernel leads to that error: ModuleNotFoundError: No module named 'pyspark'

Thank you.

qtangs commented 3 years ago

You can try adding pyspark to https://github.com/qtangs/sagemaker-notebook-container/blob/master/envs/include_libraries.txt, it controls what libraries are added to the python environments.