Test to see if the incore-lab can access the cluster's internal network

[ywkim312]

To enable a new DataWolf instance to work within IN-CORE Studio, DataWolf needs access to the internal network of the Kubernetes cluster. This requires PyINCORE to use the internal URLs of the services within the cluster. However, allowing this setup introduces a potential security risk, as it may enable IN-CORE Lab users to access the cluster's internal services. This could lead to security vulnerabilities.

Test whether IN-CORE Lab can access the internal network of the cluster. Additionally, if access is possible, explore ways to block this route to enhance security.

[ywkim312]

According to various web searches, it is possible to access the Kubernetes internal network from a JupyterLab instance spawned by JupyterHub in a Kubernetes cluster, provided the networking configuration and policies allow it.

The key points to consider:

1. Networking Inside the Cluster JupyterLab runs as a pod within the Kubernetes cluster, so it should be able to communicate with other pods and services in the same namespace or cluster. Use the Kubernetes DNS service to resolve other services. For example, a service named incore-svc-data in namespace incore can typically be accessed at incore-svc-data.incore.svc.cluster.local.

2. Accessing Cluster Resources Cluster IP Services: You can directly access ClusterIP services using their DNS names as described above. Node Ports and Load Balancers: If you need to interact with services exposed via NodePort or LoadBalancer, you might need the external IPs or specific port configurations.

3. Service Account and Role-Based Access Control (RBAC) If your JupyterLab notebook needs to interact with the Kubernetes API (e.g., to manage pods, fetch resource data), ensure the pod has an appropriate service account with the necessary permissions.

4. Network Policies If Kubernetes NetworkPolicies are enforced in your cluster, ensure they allow traffic between the JupyterLab pod and the other pods or services it needs to access.

5. Configuring Environment Variables Certain Kubernetes-related configurations might already be injected into the JupyterLab environment via environment variables like KUBERNETES_SERVICE_HOST and KUBERNETES_SERVICE_PORT.

6. Debugging Network Access Use tools like ping, curl, or Python libraries (e.g., requests, kubernetes) within the JupyterLab terminal to test connectivity

[ywkim312]

I tested checking numbers 1 and 2, and both of them were unsuccessful, which means that they don't work with those methods.

For number 1

http://incore-doc-api.incore.svc.cluster.local:80

For number 2, the following command will give the IP

kubectl get svc -n incore incore-svc-data

Also, checked with other various way but all of them failed to access the services

[ywkim312]

This yaml file does

• Traffic to private IP ranges is blocked.
• Traffic to the public internet is allowed.

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: jupyterlab-isolate
  namespace: incore
spec:
  podSelector:
    matchLabels:
      app: jupyterhub
  policyTypes:
    - Egress
  egress:
    - to:
        - ipBlock:
            cidr: 0.0.0.0/0
            except:
              - 10.0.0.0/8
              - 192.168.0.0/16
              - 172.16.0.0/12
      ports:
        - protocol: TCP
          port: 80
        - protocol: TCP
          port: 443

To deploy this

kubectl apply -f jupyterlab-isolate.yaml

[ywkim312]

This yaml file use NamespaceSelector to deny access from JupyterLab pods to all pods in a specific namespace. However, in this case, jupyterhub should be located in DIFFERENT NAMESPACE, not incore

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: block-incore-namespace
  namespace: jupyterhub
spec:
  podSelector:
    matchLabels:
      app: jupyterhub
  policyTypes:
    - Egress
  egress:
    - to:
        namespaceSelector:
          matchLabels:
            kubernetes.io/metadata.name: incore

To deploy this

kubectl apply -f block-incore-namespace.yaml

[ywkim312]

This yaml file can block access to specific pods in the same or another namespace using PodSelector

First, verify the labels of the services (do all the services)

kubectl get pods -n incore -l app.kubernetes.io/name=incore-svc-data --show-labels

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: block-jupyterlab-to-incore-services
  namespace: incore
spec:
  podSelector:
    matchLabels:
      app: jupyterhub  # Matches JupyterLab pods
      component: singleuser-server  # Targets only the JupyterLab pods
  policyTypes:
    - Egress
  egress:
    - to:
        podSelector:
          matchExpressions:
            - key: app.kubernetes.io/name
              operator: NotIn
              values:
                - incore-svc-data
                - incore-svc-hazard
                - incore-svc-dfr3
                - incore-svc-space
                - incore-svc-plotting
                - incore-svc-sema
                - incore-svc-project

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: block-jupyterlab-to-services
  namespace: incore
spec:
  podSelector:
    matchLabels:
      app: jupyterlab
  policyTypes:
    - Egress
  egress:
    - to:
        podSelector:
          matchLabels:
            app.kubernetes.io/name: "incore-svc"

[ywkim312]

This denies all

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: deny-all-egress
  namespace: incore
spec:
  podSelector:
    matchLabels:
      app: jupyterlab
  policyTypes:
    - Egress
  egress: []

[ywkim312]

I found that there was a network policy already in there named singleuser NetworkPolicy, and it is fairly restrictive, and here are the key points affecting traffic to incore-svc-data or other internal services:

kubectl describe networkpolicy singleuser -n incore

This is the content of this network policy

kubectl describe networkpolicy singleuser -n incore

Name:         singleuser
Namespace:    incore
Created on:   2023-10-04 16:29:12 -0500 CDT
Labels:       app=jupyterhub
              app.kubernetes.io/managed-by=Helm
              chart=jupyterhub-3.3.8
              component=singleuser
              heritage=Helm
              release=jupyterhub
Annotations:  meta.helm.sh/release-name: jupyterhub
              meta.helm.sh/release-namespace: incore
Spec:
  PodSelector:     app=jupyterhub,component=singleuser-server,release=jupyterhub
  Allowing ingress traffic:
    To Port: notebook-port/TCP
    From:
      PodSelector: hub.jupyter.org/network-access-singleuser=true
  Allowing egress traffic:
    To Port: 8081/TCP
    To:
      PodSelector: app=jupyterhub,component=hub,release=jupyterhub
    ----------
    To Port: 8000/TCP
    To:
      PodSelector: app=jupyterhub,component=proxy,release=jupyterhub
    ----------
    To Port: 8080/TCP
    To Port: 8443/TCP
    To:
      PodSelector: app=jupyterhub,component=autohttps,release=jupyterhub
    ----------
    To Port: 53/UDP
    To Port: 53/TCP
    To:
      IPBlock:
        CIDR: 169.254.169.254/32
        Except:
    To:
      NamespaceSelector: kubernetes.io/metadata.name=kube-system
    To:
      IPBlock:
        CIDR: 10.0.0.0/8
        Except:
    To:
      IPBlock:
        CIDR: 172.16.0.0/12
        Except:
    To:
      IPBlock:
        CIDR: 192.168.0.0/16
        Except:
    ----------
    To Port: <any> (traffic allowed to all ports)
    To:
      IPBlock:
        CIDR: 0.0.0.0/0
        Except: 10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16, 169.254.169.254/32
  Policy Types: Ingress, Egress

Key Observations from the Policy

• Egress Traffic Rules:
  • Egress is allowed to specific pods and ports:
  • Pods with labels like:
    • app=jupyterhub, component=hub
    • app=jupyterhub, component=proxy
    • app=jupyterhub, component=autohttps
  • DNS traffic (port 53/TCP and 53/UDP) to kube-system and private IP ranges (e.g., 10.0.0.0/8).
  • Traffic to public IPs (0.0.0.0/0) is allowed, except for:
    • Private ranges (10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16).
    • Metadata IP (169.254.169.254/32).
  • Traffic to incore-svc-data:
  • Internal service IPs like 10.42.7.209 fall within the blocked CIDR ranges (10.0.0.0/8), so egress traffic to this service is explicitly denied by the policy.

[ywkim312]

This is not necessary, but if you want to connect it to incore-svc-data, this kind of yaml should work (not tested)

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: singleuser
  namespace: incore
spec:
  podSelector:
    matchLabels:
      app: jupyterhub
      component: singleuser-server
      release: jupyterhub
  policyTypes:
    - Egress
    - Ingress
  ingress:
    - from:
        - podSelector:
            matchLabels:
              hub.jupyter.org/network-access-singleuser: "true"
      ports:
        - protocol: TCP
          port: notebook-port
  egress:
    - to:
        - podSelector:
            matchLabels:
              app.kubernetes.io/name: incore-svc-data
      ports:
        - protocol: TCP
          port: 8888
    - to:
        podSelector:
          matchLabels:
            app=jupyterhub
            component=hub
            release=jupyterhub
      ports:
        - protocol: TCP
          port: 8081
    - to:
        podSelector:
          matchLabels:
            app=jupyterhub
            component=proxy
            release=jupyterhub
      ports:
        - protocol: TCP
          port: 8000
    - to:
        podSelector:
          matchLabels:
            app=jupyterhub
            component=autohttps
            release=jupyterhub
      ports:
        - protocol: TCP
          port: 8080
        - protocol: TCP
          port: 8443
    - to:
        ipBlock:
          cidr: 169.254.169.254/32
      ports:
        - protocol: UDP
          port: 53
        - protocol: TCP
          port: 53
    - to:
        namespaceSelector:
          matchLabels:
            kubernetes.io/metadata.name: kube-system
    - to:
        ipBlock:
          cidr: 0.0.0.0/0
          except:
            - 10.0.0.0/8
            - 172.16.0.0/12
            - 192.168.0.0/16
            - 169.254.169.254/32