search

StatCan / aaw-contrib-jupyter-notebooks

Jupyter Notebooks to be used with Advanced Analytics Workspace platform
Other
10 stars 13 forks source link

WIP: Databricks with KFP #7

Closed sylus closed 4 years ago

sylus commented 4 years ago

This should resolve issue #5 when is done.

blairdrummond commented 4 years ago

Can you show @ca-scribner how to translate some of this to a pyspark example? It looks like Steve actually already did this

https://medium.com/@szinck/setting-up-pyspark-jupyter-and-minio-on-kubeflow-kubernetes-aab98874794f

I think Andrew just needs a hand learning about the kubernetes plumbing involved here. That way people will be able to both send Spark jobs with pipelines & use it interactively as a pandas replacement with spark dataframes

sylus commented 4 years ago

@blairdrummond can u get Steven to handle that part after this commit I have to move onto some security controls but this should serve as a good base.

blairdrummond commented 4 years ago

Sure thing!

ca-scribner commented 4 years ago

Yeah Steve and I were actually talking about this when Blair @'d me but I hadn't seen this commit. We'll talk it through