rstudio workbench kubernetes object templating

[samssann]

The newest version of workbench suggests that users should use kubernetes object templating instead of the job-json-overrides method. Will this be supported via this helm chart (meaning that you can edit job.tpl and service.tpl in helm values)?

[colearendt]

Yes!! I am super excited about this 😄 And really excited to see that you / others are excited haha.

I'm curious to hear your thoughts. It definitely seems reasonable to me that we would have values to edit job.tpl and service.tpl directly in the helm chart.

However, using helm templates inside of helm values can also be kinda irritating sometimes... (i.e. having to escape things, use strings, etc. It would be even worse if we evaluated the string as a template 😱 ) Do you have thoughts on the following potential implementation paths (or others):

• "common" modifications exposed as more "standard" values (i.e. imagePullSecrets, annotations, etc.) so you do not have to pass the entire job.tpl file
• escape hatch where you just pass the entire job.tpl / service.tpl files (they are kinda big)
• an escape hatch for important "parts" of the job.tpl (i.e. a string / YAML type of extraObjects that gets dumped into the "defaultContainer" context)
• an easy path to referencing an existing configmap that holds a job.tpl / service.tpl file? (i.e. something controlled / maintained outside of this helm release)

My initial thought is that all might be important, with a preference for the first two... and then just setting the appropriate precedence / warnings to make the UX decent. The third one seems the most "flexible without swallowing the ocean," but not sure if that would be a desirable interface or not.

EDIT: For future reference, I also think a path to testing these changes easily will be important too.

[colearendt]

@samssann I realize it has been a while here, but we are on our way to landing template support in the helm chart. The initial pass is intended to make a "values file" type of approach so that users do not have to modify custom templates directly. Would you be able to share what job-json-overrides you are currently using? We would love to be sure that commonly used values are available 😄

[samssann]

Hi @colearendt,

Apologies for the (very) late response. Here is a sample of our job-json-overrides.

launcher.kubernetes.profiles.conf:
  "*":
    allow-unknown-images: 0
    container-images:
     - xxxxx.azurecr.io/r-session-base:latest
     - xxxxx.azurecr.io/r-session-protobuf:latest

  "@group1":
    allow-unknown-images: 0
    container-images:
      - xxxxx.azurecr.io/r-session-base:latest
      - xxxxx.azurecr.io/r-session-ext:latest
      - xxxxx.azurecr.io/r-session-protobuf:latest
    default-container-image: xxxxx.azurecr.io/r-session-base:latest
    job-json-overrides:
      - target: "/spec/template/spec/containers/0/imagePullPolicy"
        json: "Always"
        name: imagePullPolicy
      - target: "/spec/template/metadata/annotations/cluster-autoscaler.kubernetes.io~1safe-to-evict"
        json: "false"
        name: evictAnnotation
      - target: "/spec/template/metadata/labels/aadpodidbinding"
        json: xxxxxxxxxxxx
        name: aadPodBinding
      - target: "/spec/template/spec/containers/0/imagePullPolicy"
        json: "Always"
        name: imagePullPolicy
      - target: "/spec/template/spec/containers/0/volumeMounts/-"
        json: 
          name: storage
          mountPath: "/data"
          readOnly: true
        name: volumeMount00
      - target: "/spec/template/spec/containers/0/volumeMounts/-"
        json: 
          name: storage-dev
          mountPath: "/data"
          readOnly: true
        name: volumeMount00dev
      - target: "/spec/template/spec/containers/0/volumeMounts/-"
        json: 
          name: secret-provider-01
          mountPath: /mnt/secrets-store
          readOnly: true
        name: volumeMount01
      - target: "/spec/template/spec/containers/0/volumeMounts/-"
        json: 
          name: secret-provider-02
          mountPath: /mnt/secrets-store-02
          readOnly: true
        name: volumeMount02
      - target: "/spec/template/spec/volumes/-"
        json:
          name: storage
          csi:
            driver: file.csi.azure.com
            readOnly: true
            volumeAttributes:
              resourceGroup: xxxxx
              storageAccount: xxxxxxx
              shareName: xxxxxx
              protocol: smb
              mountOptions: "dir_mode=0777,file_mode=0777,uid=0,gid=0,mfsymlinks,cache=strict,nosharesock"
              secretName: secret
              secretNamespace: xxxxxx
        name: volume00
      - target: "/spec/template/spec/volumes/-"
        json:
          name: storage-dev
          csi:
            driver: file.csi.azure.com
            readOnly: true
            volumeAttributes:
              resourceGroup: xxxxxxxxxx
              storageAccount: xxxxxxxxx
              shareName: xxxxxxxxx
              protocol: smb
              mountOptions: "dir_mode=0777,file_mode=0777,uid=0,gid=0,mfsymlinks,cache=strict,nosharesock"
              secretName: secret-dev
              secretNamespace: xxxxxxxx
        name: volume00dev
      - target: "/spec/template/spec/volumes/-"
        json:
          name: secret-provider-01
          csi:
            driver: secrets-store.csi.k8s.io
            readOnly: true
            volumeAttributes:
              secretProviderClass: secret-provider-01
        name: volume01
      - target: "/spec/template/spec/volumes/-"
        json:
          name: secret-provider-02
          csi:
            driver: secrets-store.csi.k8s.io
            readOnly: true
            volumeAttributes:
              secretProviderClass: secret-provider-02
        name: volume02
      - target: "/spec/template/spec/affinity"
        json: 
          nodeAffinity:
            requiredDuringSchedulingIgnoredDuringExecution:
              nodeSelectorTerms:
              - matchExpressions:
                - key: beta.kubernetes.io/instance-type
                  operator: In
                  values:
                  - Standard_E4as_v5
        name: nodeAffinity
  "@group2":
    allow-unknown-images: 1

Is there documentation on how to use the templates feature?

---

I know this is not the correct platform to post this, but i just want your view on this. After using this product for some months now, I feel like there is one aspect missing: The ability to configure different kubernetes configurations for different users/groups. I acknowledge that this feature would also require additional work in the in the workbench itself. The general idea is that a user can choose which kubernetes configurations to apply (from a drop down menu, or even upload yaml file?) to a new session the same way the user can choose which image to use (see illustrative image below). Available kubernetes configurations could be defined in the config file launcher.kubernetes.conf. Each kubernetes configuration has its own unique name. In launcher.kubernetes.profiles.conf using a variable kubernetes-configs, users/groups can be given permissions to use a subset/all of the kubernetes configurations. See example yaml below.

launcher.kubernetes.conf:
  "base":
      - target: "/spec/template/spec/nodeSelector"
        json: 
          nodeSelector:
            kubernetes.azure.com/agentpool: default
        name: nodeSelector
  "app1-production":
      - target: "/spec/template/spec/nodeSelector"
        json: 
          nodeSelector:
            kubernetes.azure.com/agentpool: prod-pool
        name: nodeSelector
      - target: "/spec/template/spec/containers/0/volumeMounts/-"
        json: 
          name: storage-prod
          mountPath: "/data"
          readOnly: true
        name: volumeMount
      - target: "/spec/template/spec/volumes/-"
        json:
          name: storage-prod
          csi:
            driver: file.csi.azure.com
            readOnly: true
            volumeAttributes:
              resourceGroup: xxxxx
              storageAccount: xxxxxxx
              shareName: xxxxxx
              protocol: smb
              mountOptions: "dir_mode=0777,file_mode=0777,uid=0,gid=0,mfsymlinks,cache=strict,nosharesock"
              secretName: secret
              secretNamespace: xxxxxx
        name: volume
  "app1-development":
      - target: "/spec/template/spec/nodeSelector"
        json: 
          nodeSelector:
            kubernetes.azure.com/agentpool: dev-pool
        name: nodeSelector
      - target: "/spec/template/spec/containers/0/volumeMounts/-"
        json: 
          name: storage-dev
          mountPath: "/data"
          readOnly: true
        name: volumeMount
      - target: "/spec/template/spec/volumes/-"
        json:
          name: storage-dev
          csi:
            driver: file.csi.azure.com
            readOnly: true
            volumeAttributes:
              resourceGroup: xxxxx
              storageAccount: xxxxxxx
              shareName: xxxxxx
              protocol: smb
              mountOptions: "dir_mode=0777,file_mode=0777,uid=0,gid=0,mfsymlinks,cache=strict,nosharesock"
              secretName: secret
              secretNamespace: xxxxxx
        name: volume
launcher.kubernetes.profiles.conf:
  "*":
    allow-unknown-images: 0
    allow-unknown-kubernetes-configs: 0
    container-images:
     - xxxxx.azurecr.io/r-session-base:latest
    default-kubernetes-config: base
    kubernetes-configs:
     - base
  "@group1":
    allow-unknown-images: 1
    allow-unknown-kubernetes-configs: 1
    default-kubernetes-config: app1-development
    container-images:
     - xxxxx.azurecr.io/r-app1:dev
    kubernetes-configs:
     - base
     - app1-development
   "@group2":
     allow-unknown-images: 0
     allow-unknown-kubernetes-configs: 0
    container-images:
     - xxxxx.azurecr.io/r-app1:latest
     kubernetes-configs:
     - app1-production

This would enable users to e.g.:

• choose on which node pool to run their computations
• doens't force to mount everything (as we are required to do now)
• overall gives better control to leverage kubernetes