[TESTING][GCP][GKE] Chart on GCP

[taktakpeops]

Following a chat on the issue https://github.com/jitsi/docker-jitsi-meet/issues/565 for Jitsi Meet in K8S using this chart, I am moving the discussion specifically related to GKE here.

@ChrisTomAlx - could you share your findings here?

I will also provide you some support.

[taktakpeops]

[ChrisTomAlx]

Hey @taktakpeops This looks fine to me except for the jibri part, because each jibri pod only supports one recording at a time and the cpu / ram optimal requirements for jibri are much more than other pods (4vcpu 8GB Ram is what I have read on the community forums). Also jibri requires some node level changes. Refer to this pr.

I was thinking we should maybe continue this discussion on your jitsi issue because the jitsi team is super helpful and they could help if we run into problems.

I would love for them to see the architecture you have created here, looks impressive..

[taktakpeops]

OK

[taktakpeops]

One thing to add is that this architecture works only if you have multiple nodes in your cluster and each JVB instance has its own node because JVB will bind the public IP of the node. So some affinity rules are needed for the deployment of multiple JVB

[ChrisTomAlx]

Sorry I have been AWOL for a bit. So to do node level changes for proper jibri functioning (atleast in GKE), I am planning to create a daemonset which will run on all nodes before they start up any other pods.

https://stackoverflow.com/questions/57005971/how-to-use-customise-ubuntu-image-for-node-when-gke-cluster-create

Also I am creating the jibri yaml file with container post-start and pre-stop scripts to correctly change the /home/jibri/.asoundrc file in each of the containers. I will probably use hostpath to read and write the same file within all the containers on a node so they all know which sound cards are available and which aren't.

As I mentioned earlier the Jibri in the architecture can possibly be a group like JVB is correct?

[taktakpeops]

Yes, indeed it can. Even if I think that it’s strongly tied to Jicofo + Prosody and I would see all of it in one pod.

About the multiple JVB, it used some pod anti-affinity rules to ensure the dispatch of the pods across nodes. Since each JVB deployment deploys every instance to its own pod with a port unique to the deployment it works with many Jitsi servers.

About Jibri, one issue remain: not all cluster can access the sound device of the VM.

I am starting to explore how to use a dummy sound driver that can be used as a loop back for recording sound from a pod without needing the physical device. I will most likely develop a K8S plugin device for that.

Let me know if you have more questions and feel free to push a PR with your scripts for GKE if you are using the chart :)

[ChrisTomAlx]

Sure I will start looking into this from tomorrow again. Will raise a PR if there are any changes required.. Regarding Jibri, a dummy sound device would be amazing. Jibri uses ALSA. If there was a way to add that to the container directly then most the jibri problems would be solved, but I am not entirely sure that is possible.

[ChrisTomAlx]

Hey @taktakpeops I noticed that you are using

  - name: DOCKER_HOST_ADDRESS
      valueFrom:
        fieldRef:
          fieldPath: status.hostIP

status.hostIP is only available in kubernetes version 1.17.0 and above, but it seems like GCP does not have a stable release for 1.17.0 yet, There is a rapid release available but google specifically mentions not to use it in production.. Does aws support 1.17.0 already ?

[taktakpeops]

Hi @ChrisTomAlx,

Looking into the documentation here, it's available since v1.7. Checking here, it seems that GCP is currently supporting 1.15.9, 1.16.5, 1.17 and higher for GKE.

In AWS, I am using the latest version supported by EKS which is v1.16.8 (you can find all supported versions of K8S for EKS here: https://docs.aws.amazon.com/eks/latest/userguide/kubernetes-versions.html).

Have you tried deploying and noticed that the IP wasn't exported in the environment?

[ChrisTomAlx]

Hi @taktakpeops Oh.. You are absolutely right. GKE does support 1.17 but it is not on a stable channel. But it does not really matter now that I know hostIP was actually implemented in 1.7.0+ and not 1.17.0+

Thanks for your quick response. You just saved me a whole lot of cluster reconfiguration

[ChrisTomAlx]

Hey @taktakpeops ..

I have a couple of questions. If time permits please do get back to me.

1. The status.hostIP uses the nodes IP address correct? But this IP address must be accessible from outside through an external IP. So my question is did you have to manually set up the nodes external IP ?

2. Each Prosody / Jicofo combo uses different xmpp domains right?

   - name: XMPP_DOMAIN
     value: jitsi.meet
   - name: XMPP_AUTH_DOMAIN
     value: auth.jitsi.meet
   - name: XMPP_INTERNAL_MUC_DOMAIN
     value: internal-muc.jitsi.meet
   - name: XMPP_MUC_DOMAIN
     value: muc.jitsi.meet
   - name: XMPP_GUEST_DOMAIN
     value: guest.jitsi.meet

   For example all the domains above should change based on the prosody / jicofo combo dynamically right? How did you manage to do that ?

3. I see some config files in your repo. Are those just to show that these are the configs within the containers? Or is it somehow being used in the helm chart ?

[taktakpeops]

Hello,

For question 1: yes, you need to be able to remotely access your nodes which means that they need a public IP. In AWS, I did so by enabling the SSH access in my node group. The public IP are being detected using STUN servers.

Question 2, yes, you are right but for that I am writing the logic in helm which will generate the domains for each Jicofo / prosody couple (so work in progress).

Question 3, the config are there for testing. I want to start using octo and enable some modules in posody.

[ChrisTomAlx]

Awesome. Thanks for the quick response. There seems to be some issue on my end when using NodePort. The JVB is not accessible. Tried with LoadBalancer and it works.

kubectl describe pod {jvb pod} gives me :-

```yaml Name: jitsi-meet-jvb-0-7c5cdc6f54-slfn5 Namespace: default Priority: 0 Node: gke-neumeet-pool-1-4768d01b-xd50/10.128.0.26 Start Time: Fri, 22 May 2020 17:15:34 +0530 Labels: app.kubernetes.io/instance=neumeet app.kubernetes.io/name=jitsi-meet-jvb-0 pod-template-hash=7c5cdc6f54 Annotations: kubernetes.io/limit-ranger: LimitRanger plugin set: cpu request for container jitsi-meet-jvb Status: Running IP: 10.56.4.199 IPs: Controlled By: ReplicaSet/jitsi-meet-jvb-0-7c5cdc6f54 Containers: jitsi-meet-jvb: Container ID: docker://7819c812939676fe72a614256ec57ce8f83a11ab0aa71c2121f9501878a90daf Image: asia.gcr.io/webrtcvideo-274408/neutrinos/jvb:v20 Image ID: docker-pullable://asia.gcr.io/webrtcvideo-274408/neutrinos/jvb@sha256:a2e405eb74e33accc0e62ec1a798d918874b505fe58cd4f0e77bc0ac719fff43 Port: 30300/UDP Host Port: 0/UDP State: Running Started: Fri, 22 May 2020 17:15:37 +0530 Ready: True Restart Count: 0 Requests: cpu: 100m Environment: XMPP_SERVER: jitsi-meet-prosody PROSODY_INSTANCE: 0 JICOFO_AUTH_USER: focus JICOFO_AUTH_PASSWORD: Optional: false JVB_AUTH_USER: jvb JVB_AUTH_PASSWORD: Optional: false JICOFO_COMPONENT_SECRET: Optional: false JVB_PORT: 30300 JVB_STUN_SERVERS: stun.l.google.com:19302,stun1.l.google.com:19302,stun2.l.google.com:19302 JVB_TCP_HARVESTER_DISABLED: true DOCKER_HOST_ADDRESS: (v1:status.hostIP) JVB_OPTS: --apis=xmpp,rest ENABLE_STATISTICS: true XMPP_DOMAIN: testmeet.neutrinos.co XMPP_AUTH_DOMAIN: auth.jitsi-meet-prosody.default.svc XMPP_INTERNAL_MUC_DOMAIN: internal-muc.jitsi-meet-prosody.default.svc XMPP_MUC_DOMAIN: muc.testmeet.neutrinos.co XMPP_GUEST_DOMAIN: guest.jitsi-meet-prosody.default.svc JVB_BREWERY_MUC: jvbbrewery TZ: Europe/Amsterdam Mounts: /defaults from config (rw) /var/run/docker.sock from dockersock (rw) /var/run/secrets/kubernetes.io/serviceaccount from neumeet-jitsi-meet-token-tclkm (ro) Conditions: Type Status Initialized True Ready True ContainersReady True PodScheduled True Volumes: dockersock: Type: HostPath (bare host directory volume) Path: /var/run/docker.sock HostPathType: config: Type: ConfigMap (a volume populated by a ConfigMap) Name: jitsi-meet-jvb-config-cm Optional: false neumeet-jitsi-meet-token-tclkm: Type: Secret (a volume populated by a Secret) SecretName: neumeet-jitsi-meet-token-tclkm Optional: false QoS Class: Burstable Node-Selectors: Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s node.kubernetes.io/unreachable:NoExecute for 300s Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 30m default-scheduler Successfully assigned default/jitsi-meet-jvb-0-7c5cdc6f54-slfn5 to gke-neumeet-pool-1-4768d01b-xd50 Normal Pulling 30m kubelet, gke-neumeet-pool-1-4768d01b-xd50 Pulling image "asia.gcr.io/webrtcvideo-274408/neutrinos/jvb:v20" Normal Pulled 30m kubelet, gke-neumeet-pool-1-4768d01b-xd50 Successfully pulled image "asia.gcr.io/webrtcvideo-274408/neutrinos/jvb:v20" Normal Created 30m kubelet, gke-neumeet-pool-1-4768d01b-xd50 Created container jitsi-meet-jvb Normal Started 30m kubelet, gke-neumeet-pool-1-4768d01b-xd50 Started container jitsi-meet-jvb ```

Anyhow just dropped this here in case you find something off. I will look into stun servers I think that must be why I am having issues. Because I am able to ping the external ip of the node from an external network but not with UDP which is strange. So from a first look stun servers should solve this issue if I am not wrong.

[taktakpeops]

@ChrisTomAlx, for the stun + public IP detection, you should see that in the logs of JVB.

Did you make sure that the security group for your nodes accept traffic on port 30300?

[ChrisTomAlx]

@taktakpeops Yea will look through the logs for some hints Its not a private cluster so I assumed nodes will be accepting traffic without problems.. let me recheck that. Thanks for the suggestions!!

[taktakpeops]

@ChrisTomAlx : it’s not sure that the nodes will have the port open. In EKS, for example, the public network for the nodes has its own security group which has to be modified accordingly after deploying Jitsi to allow UDP traffic on the node.

[ChrisTomAlx]

Hey @taktakpeops you were right. There was a firewall blocking UDP traffic, trying to get around it. In the mean time I see someone else is also trying to replace ALSA dependency from Jibri with pulseAudio. Look at this comment.. Here is his github repo. I am not sure if it works, but since you said you were looking at removing node level changes from jibri, I thought you would be interested..

[taktakpeops]

Hey @ChrisTomAlx - thanks for the suggestion. After playing around with Alsa (used currently in Jitsi) - I ended up realizing that it requires a modification of the node OS, therfore specialized nodes for running Jibri.

I saw quite a few other people also modifying the config of Jibri for using PulseAudio. After going through issues / pull-requests and the forum, I found out that PulseAudio was originally used in Jibri. The Jitsi team dropped it because of stability issues, while recording calls with a large amount of participants they had drops / packet losts. They replaced it by Alsa.

I want to keep the Helm chart with the official containers.

For now, I am looking into how the AMIs for EKS are configured and how to enable some specific kernel modules as the one missing currently: soundcore.

Once soundcore is enabled, we can run the node with an access to a Alsa dummy soundcard (ideally supporting X recordings at a time).

The custom AMI will be then used for all the nodes. If X recordings at a time aren't possible, we can still run one Jibri per nodes.

I thought also about another (weird and terrible) thing: running Jibri inside a VM running inside a container running Qemu 🤣

[ChrisTomAlx]

I thought also about another (weird and terrible) thing: running Jibri inside a VM running inside a container running Qemu 🤣

This is intriguing, but I am even scared to ask about it after reading up a bit on Qemu. Although it could work right? It will be one heavy pod but I think anything is better than making node level changes

---

I saw quite a few other people also modifying the config of Jibri for using PulseAudio. After going through issues / pull-requests and the forum, I found out that PulseAudio was originally used in Jibri. The Jitsi team dropped it because of stability issues, while recording calls with a large amount of participants they had drops / packet losts. They replaced it by Alsa.

Nice catch, would have spend a lot of time going down that rabbit hole otherwise.

---

I want to keep the Helm chart with the official containers.

Agree. I would prefer the official ones as well. Then we can simply change container images and stay up to date with new features..

---

For now, I am looking into how the AMIs for EKS are configured and how to enable some specific kernel modules as the one missing currently: soundcore.

For GCP I could change the node images to ubuntu then ssh into each of the VMs and change them manually as suggested by the Jitsi Team and this pr. Although this won't work in the case of node auto scaling. Hence my plan is to create a daemonset that run on all nodes and adjusts them to run jibri. Just a side note remember one jibri pod can only do one recording at a time. This video gave me a lot of insight into jibri. Leaving it here in case you want to watch it as well.

[ChrisTomAlx]

Also thanks a ton for the firewall tip.. Got JVB to run as expected.. Testing scaling next

[taktakpeops]

This is intriguing, but I am even scared to ask about it after reading up a bit on Qemu. Although it could work right? It will be one heavy pod but I think anything is better than making node level changes

Yes - it could work but it would also be a hell to debug - it's a high level of inception :D

Nice catch, would have spend a lot of time going down that rabbit hole otherwise.

Agreed - it's pretty tricky to gather info about jitsi as it's all over the place

For GCP I could change the node images to ubuntu then ssh into each of the VMs and change them manually as suggested by the Jitsi Team and this pr. Although this won't work in the case of node auto scaling. Hence my plan is to create a daemonset that run on all nodes and adjusts them to run jibri. Just a side note remember one jibri pod can only do one recording at a time. This video gave me a lot of insight into jibri. Leaving it here in case you want to watch it as wel

For GCP, we might have the same issue as with AWS which is that the VM don't have a sound device. I am looking further into dummy-snd but it remains a kernel module and not included in Ubuntu on AWS EC2.

Will keep you posted once I find a workaround for that

[ChrisTomAlx]

I just followed the below steps as shown in the pr for each node on GCP and I was able to get jibri to run.

For each node in the cluster do the following:

### Install generic kernel image ``` sudo apt-get update sudo apt-get install linux-image-generic ``` #### Change grub to start node VM with generic kernel ``` # GKE Node sudo vim /etc/default/grub.d/50-cloudimg-settings.cfg # Other Setup sudo vim /etc/default/grub ``` Replace #GRUB_DEFAULT=0 with ``` # Make the latest generic kernel the default release=$(linux-version list | grep -e '-generic$' | sort -V | tail -n1) GRUB_DEFAULT="Advanced options for Ubuntu>Ubuntu, with Linux $release" ``` ``` sudo update-grub ``` #### Reboot VM ``` sudo reboot ``` ### Setup virtual sound device in the node #### install the module ``` sudo apt update && sudo apt install linux-image-extra-virtual ``` #### configure 5 capture/playback interfaces ``` sudo su echo "options snd-aloop enable=1,1,1,1,1 index=0,1,2,3,4" > /etc/modprobe.d/alsa-loopback.conf ``` #### setup autoload the module ``` echo "snd-aloop" >> /etc/modules ``` #### Reboot ``` sudo reboot ``` #### check that the module is loaded ``` lsmod | grep snd_aloop ```

[taktakpeops]

Hey @ChrisTomAlx, thank you for sharing.

I don’t see in your steps anything related to setting up the snd-dummy component, is it present in your nodes?

[ChrisTomAlx]

The Setup virtual sound device in the node section handles that I assume.

You can also refer this. The jitsi official documentation mentions how to set it up on AWS. Although I guess they meant it for VM's. In GKE the the kubernetes nodes are also VM's. I am not sure if its the same in AWS

For GCP if node auto scaling and node auto repair are on these changes won't stick. So I had to turn these off as well.

[ChrisTomAlx]

Here is my jibri yaml file. You might want to change it according to your needs slightly.. This is a one time only download link.

[taktakpeops]

If you run « aplay -L » in your terminal, what does it print?

Also, master and nodes are VM. Inside of the node VM, kubelet must be running to subscribe to the master. Then, pod scheduling happens in this VM.

[ChrisTomAlx]

If you run « aplay -L » in your terminal, what does it print?

It just says -bash: aplay: command not found. But I do have a jibri pod running and working on this node. And all I did was follow the steps above and made sure the node image was ubuntu instead of Google's own node image

---

Also, master and nodes are VM. Inside of the node VM, kubelet must be running to subscribe to the master. Then, pod scheduling happens in this VM.

Yes, makes sense, similar to GKE.

[taktakpeops]

Ah oki, i installed alsa on the VM to test my OS setup.

Got it to work on AWS. Now, for Jibri, how do you mount the device for the pods?

[taktakpeops]

For the EC2 instance, I had to use a different GRUB_DEFAULT. Instead of the menu title, I used gnulinux-advanced-6156ec80-9446-4eb1-95e0-9ae6b7a46187>gnulinux-4.15.0-101-generic-advanced-6156ec80-9446-4eb1-95e0-9ae6b7a46187.

Could you share the logs from Jibri to see how alsa gets init and so on?

[ChrisTomAlx]

how do you mount the device for the pods?

Look at my jibri yaml file. Also /home/jibri/.asoundrc contains the loopback device being used. Refer the Set interface in file /home/jibri/.asoundrc inside a docker container section here

---

Could you share the logs from Jibri to see how alsa gets init and so on?

Sorry. I don't think its safe sharing, since I have jibri currently up on a UAT environment, so my hands are tied.

---

Jisti has mentioned how to change the GRUB file in the same link :-

# open the grub file in editor
nano /etc/default/grub
# Modify the value of GRUB_DEFAULT from "0" to "1>2"
# Save and exit from file

# Update grub
update-grub
# Reboot the machine
reboot now

[ChrisTomAlx]

Look at my jibri yaml file.

The volumes section has these two mounted from the host node as hostpath volumes which basically just read the host nodes directories at those paths.

      volumes:
        - name: dev-snd
          hostPath:
            path: "/dev/snd"
            type: Directory
        - name: dev-shm
          hostPath:
            path: "/dev/shm"
            type: Directory

[taktakpeops]

Sorry - checked everything, looks good.

Regarding the logs, I was just wondering if alsa is being loaded correctly or not. Because there are no health checks for Jibri, it can be in crashed inside of the pod but still considered as healthy as the process keeps running (in all the Jitsi Docker containers, the software runs as a daemon).

I am going to script the setup for the EKS nodes.

One issue remain: currently, only 1 jibri per nodes can run. 2 instances on one node would create a conflict for accessing the device

[ChrisTomAlx]

no worries 😄

---

One issue remain: currently, only 1 jibri per nodes can run. 2 instances on one node would create a conflict for accessing the device

If you look at the Set interface section as I had mentioned above (quoting the same below), you can actually have more than one jibri pods in one node. But this is because you set echo "options snd-aloop enable=1,1,1,1,1 index=0,1,2,3,4" > /etc/modprobe.d/alsa-loopback.conf in your node. The index as per my understanding is saying you can have 5 jibri recordings running simlutaneously in this node.

Look at my jibri yaml file. Also /home/jibri/.asoundrc contains the loopback device being used. Refer the Set interface in file /home/jibri/.asoundrc inside a docker container section here

---

I actually did run two jibri pods in the same node. with different interfaces. One as Loopback and the other as Loopback_1. I think the best way to do this, since it has to happen dynamically per pod is to have a file at hostpath which keeps a list of interfaces already being used and then adjust the file as per the postStart and preStop lifecycle hooks of the pods. So all the pods can access the same file and edit it as required. What do you think about this approach ?

---

Regarding the logs, I was just wondering if alsa is being loaded correctly or not. Because there are no health checks for Jibri, it can be in crashed inside of the pod but still considered as healthy as the process keeps running (in all the Jitsi Docker containers, the software runs as a daemon).

You are right, health checks are an issue as well I guess...

[taktakpeops]

I think the best approach remains having a device plugin for managing the sound card. Still work in progress on my side 😄

[VengefulAncient]

Hi guys! Awesome work on this so far, thank you both so much. I'm just coming here to point out that on GKE, it seems that you don't need all the trickery with GRUB and a non-GCP boot image. If you are running Ubuntu, you can simply apt-get install linux-modules-extra-gcp and reboot. You will then get an up-to-date GCP image with snd-aloop module.

[ChrisTomAlx]

Hey @VengefulAncient Thanks for chipping in. This is brilliant. This just reduced the work of my Daemonset by a lot. There might be something similar for AWS as well

Although I will still have to do the following I assume

#### configure 5 capture/playback interfaces ``` sudo su echo "options snd-aloop enable=1,1,1,1,1 index=0,1,2,3,4" > /etc/modprobe.d/alsa-loopback.conf ``` #### setup autoload the module ``` echo "snd-aloop" >> /etc/modules ``` #### Reboot ``` sudo reboot ```

Maybe autoload and reboot isn't required but configuring playback must be correct.?

[taktakpeops]

@VengefulAncient : thank you for the tip ! I assume that using the following for AWS does the same: https://packages.ubuntu.com/bionic/linux-modules-extra-aws

Testing it now and will update the documentation accordingly.

[VengefulAncient]

@ChrisTomAlx Unfortunately, it looks like the other steps are still required. The module is only installed for the new GCP image, which is not in use until the node is rebooted. I haven't gotten around to the DaemonSet yet, but perhaps its script could check if the snd-aloop module is loaded, and if not, install the package, perform the configuration and reboot the node? I'm still fairly new to this so not entirely sure whether that's possible.

[ChrisTomAlx]

@VengefulAncient Agreed. I am a bit new too. Daemonsets should be able to handle this as per my understanding.

[taktakpeops]

@VengefulAncient, the script in the docker container takes care of it: https://github.com/jitsi/docker-jitsi-meet/blob/master/jibri/rootfs/etc/cont-init.d/10-config#L25

[VengefulAncient]

@taktakpeops Sadly, that only seems to exit if snd-aloop is not loaded - for true automatic scalability, we'd want the configuration performed on the node automatically (by a privileged DaemonSet?) if the module is not loaded. Could start with the same snippet you linked, but instead of exit, run apt-get update && apt-get install linux-modules-extra-gcp && echo "options snd-aloop enable=1,1,1,1,1 index=0,1,2,3,4" > /etc/modprobe.d/alsa-loopback.conf && echo "snd-aloop" >> /etc/modules && reboot (obviously replace linux-modules-extra-gcp with linux-modules-extra-aws for AWS).

[taktakpeops]

I see - but I don't think it needs to be a modification of the node by kubernetes. Instead, wouldn't it be easier to customize the script starting the VM? At the end, we need the update of the VM + reboot before launching the kubelet

[ChrisTomAlx]

Here is the Daemonset I am working with.

```yaml kind: DaemonSet apiVersion: apps/v1 metadata: name: daejibri spec: selector: matchLabels: name: daejibri # Label selector that determines which Pods belong to the DaemonSet template: metadata: labels: name: daejibri # Pod template's label selector spec: hostPID: true # nodeSelector: # type: jibri containers: - name: daejibri image: asia.gcr.io/google-containers/startup-script:v1 securityContext: privileged: true resources: limits: memory: 200Mi requests: cpu: 100m memory: 200Mi env: - name: STARTUP_SCRIPT value: | ! /bin/bash mkdir -p /chris/hey/ # apt-get install linux-modules-extra-gcp # sudo reboot # echo done ```

The asia.gcr.io/google-containers/startup-script:v1 image already exists in GCP by default. I think something similar might be there for AWS.

I ran the the daemonset and noticed node level changes after the daemonset ran (meaning I did ls / on the node and I saw the chris folder which did not exist before). So the script I ran was mkdir -p /chris/hey/. I assume you can replace that with anything you want and we can have truly scaleable jibri.

One thing I would add is Node selectors for both jibri and daemonset pods. So these nodes will have only jibri running on them

[ChrisTomAlx]

Instead, wouldn't it be easier to customize the script starting the VM? At the end, we need the update of the VM + reboot before launching the kubelet

@taktakpeops I am not sure that is possible on GKE. But I recall reading that is possible on AWS. This is from memory so I could be wrong

[taktakpeops]

It seems possible using a gcloud config file (https://cloud.google.com/container-optimized-os/docs/how-to/create-configure-instance#using_cloud-init_with_the_cloud_config_format)

All the setup of the node is managed through system.d (https://cloud.google.com/kubernetes-engine/docs/concepts/node-images#system_initialization) - so the script checking the setup and exiting in case of issues would then be a system.d service.

For installing the dependencies and doing the reboot (when needed), the flag --metadata-from-file allows to upload a local file which will be executed on startup (a Bash script) (https://cloud.google.com/sdk/gcloud/reference/container/clusters/create#--metadata-from-file and https://cloud.google.com/compute/docs/startupscript#startupscriptlocalfile).

[taktakpeops]

It's to be tested, I don't have a GKE cluster available currently to confirm

[ChrisTomAlx]

I will look into it. But I am not sure this will work. Because GKE VM's and Compute engine VM's don't always follow the same rules. I ran into a similar issue with firewall, because I could not add firewall rules to VM's within kubernetes so I had to go around it by editing an already existing rule.

In https://cloud.google.com/sdk/gcloud/reference/container/clusters/create#--metadata, It specifically mentions the startup-script is reserved for use, which is understandable since GKE sets most of it up. The master worker setup and all the kubernetes components

Additionally, the following keys are reserved for use by Kubernetes Engine:

cluster-location
cluster-name
cluster-uid
configure-sh
enable-os-login
gci-update-strategy
gci-ensure-gke-docker
instance-template
kube-env
startup-script
user-data

Still thanks for the in-depth research and all the links. Very helpful. Let me check if its doable.

[taktakpeops]

Let me know how it goes.

On another hand, a custom AMI or base Image seems the best pattern to follow: https://github.com/kubernetes/kops/issues/387

[ChrisTomAlx]

Yes I agree. Unforunately I don't think GKE VM's support custom base images while GCP's compute engine VM's do.

[VengefulAncient]

@ChrisTomAlx I don't think it's necessarily such a bad thing. Frankly, I'd prefer it if the configuration was done by the DaemonSet - this approach lends itself much better to IaC (Infrastructure as Code). Doing additional configuration fetching scripts that have to be stored somewhere that will differ between providers is less maintainable. Ideally, all I'd want to do is add a node pool with label/taint that will prevent other pods from being scheduled there, deploy the chart with a few customized values, and have everything configure itself. The privileged DaemonSet configuring the nodes for us plays into that nicely - if it can do that.