question on where is spark in the full version?

EMCP commented 2 years ago

according to this list of kubeflow bundle , it should contain spark.. but I fail to find it in either my deployed Applications or in the bundle file

https://charmed-kubeflow.io/docs/operators-and-bundles

Full Kubeflow bundle - [Store](https://charmhub.io/kubeflow) | [YAML](https://github.com/canonical/bundle-kubeflow/blob/master/bundle-kubeflow.yaml)

    [admission-webhook](https://charmhub.io/admission-webhook)
    [argo-controller](https://charmhub.io/argo-controller)
    [dex-auth](https://charmhub.io/dex-auth)
    [istio-ingressgateway](https://jaas.ai/u/kubeflow-charmers/istio-ingressgateway)
    [istio-pilot](https://charmhub.io/istio-pilot)
    [jupyter-controller](https://charmhub.io/jupyter-controller)
    [jupyter-ui](https://charmhub.io/jupyter-ui)
    [katib-controller](https://charmhub.io/katib-controller)
    [katib-db](https://charmhub.io/charmed-osm-mariadb-k8s)
    [katib-db-manager](https://charmhub.io/katib-db-manager)
    [katib-ui](https://charmhub.io/katib-ui)
    [kfp-api](https://charmhub.io/kfp-api)
    [kfp-db](https://charmhub.io/charmed-osm-mariadb-k8s)
    [kfp-persistence](https://charmhub.io/kfp-persistence)
    [kfp-schedwf](https://charmhub.io/kfp-schedwf)
    [kfp-ui](https://charmhub.io/kfp-ui)
    [kfp-viewer](https://charmhub.io/kfp-viewer)
    [kfp-viz](https://charmhub.io/kfp-viz)
    [kubeflow-dashboard](https://charmhub.io/kubeflow-dashboard)
    [kubeflow-profiles](https://charmhub.io/kubeflow-profiles)
    [mlmd](https://charmhub.io/mlmd)
    [minio](https://charmhub.io/minio)
    [oidc-gatekeeper](https://charmhub.io/oidc-gatekeeper)
    [pytorch-operator](https://charmhub.io/pytorch-operator)
    [spark](https://charmhub.io/spark-charmers-spark)
    [seldon-controller-manager](https://charmhub.io/seldon-core)
    [tfjob-operator](https://charmhub.io/tfjob-operator)

https://github.com/canonical/bundle-kubeflow/blob/master/bundle-kubeflow.yaml

EMCP commented 2 years ago

perhaps I just need a tutorial how to get started and it's already installed.. but when I looked for spark in a new notebook I hadn't a clue where to start

EMCP commented 2 years ago

just tried the spark notebook image but.. seemed to fail

so maybe my first goal is to confirm.. should I have spark available? and if so.. how can I get started ? if it's omitted from the full kubeflow bundle that's just fine.. and then I can try to help add it ?

DomFleischmann commented 2 years ago

Hello @EMCP, you can install spark separately by executing the following command in the kubeflow juju model: juju deploy spark-k8s We removed spark from the default bundle as it wasn't entirely related to Kubeflow, sorry for the inconvenience.

EMCP commented 2 years ago

thank you @DomFleischmann , I need to start over from scratch.. as my k8s controller is now acting up.. the bug is already reported here https://bugs.launchpad.net/juju/+bug/1968105

but I will retry deploying this in a moment, thanks for the insights.. now its clear why it is not included

Edit: I've successfully reconnected my juju client and run juju deploy spark-k8s , unsure where or how to proceed from here but will experiment with some basic notebook code to connect to spark.. the documentation on that charm seems to point back to the kubeflow docs

canonical / bundle-kubeflow

question on where is spark in the full version? #453