jupyter-server / enterprise_gateway

A lightweight, multi-tenant, scalable and secure gateway that enables Jupyter Notebooks to share resources across distributed clusters such as Apache Spark, Kubernetes and others.
https://jupyter-enterprise-gateway.readthedocs.io/en/latest/
Other
615 stars 221 forks source link

[feature] add support for manage service and configmap in kernel-pod #1181

Closed jeremythuon closed 1 year ago

jeremythuon commented 1 year ago

Problem Statement

We need to create service and configmap per kernel pod for launch spark driver without cluster mode (because we cannot use SparkContext to add jars.package option and other things)

Feature Description

The PR allow add kind service and configmap in kernel-pod.yaml.j2 and add ownerReference to kernel pod in new object (usefull when delete kernel) Delete all namespace can resolv the problèm but we prefer not to keep a ghost object

Exemple kernel-pod.yaml :

<default pod template>
---
apiVersion: v1
kind: Service
metadata:
  name: svc-{{ kernel_pod_name }}
  namespace: "{{ kernel_namespace }}"
  labels:
    kernel_id: "{{ kernel_id }}"
    app: enterprise-gateway
    component: kernel
    source: kernel-pod.yaml
spec:
  selector:
    app: enterprise-gateway
    component: kernel
    kernel_id: "{{ kernel_id }}"
    spark-role: driver
  ports:
  - name: spark-driver-rpc-port
    protocol: TCP
    port: 7078
    target: 7078
  - name: blockmanager
    port: 7079
    protocol: TCP
    targetPort: 7079
  - name: spark-ui
    port: 4040
    protocol: TCP
    targetPort: 4040
  sessionAffinity: None
  type: ClusterIP
welcome[bot] commented 1 year ago

Thanks for submitting your first pull request! You are awesome! :hugs:
If you haven't done so already, check out Jupyter's Code of Conduct. Also, please make sure you followed the pull request template, as this will help us review your contribution more quickly. welcome You can meet the other Jovyans by joining our Discourse forum. There is also a intro thread there where you can stop by and say Hi! :wave:
Welcome to the Jupyter community! :tada:

welcome[bot] commented 1 year ago

Congrats on your first merged pull request in this project! :tada: congrats Thank you for contributing, we are very proud of you! :heart: