Adding multiple tolerations on vizier-pems

Is your feature request related to a problem? Please describe. Hi Pixie Team! I'm a bit new with Pixie and wanted to be able to add nodeSelector and tolerations on my vizier-pem pods. I'm currently using Helm values.yaml installation via Terraform. So far I've seen the New relic documentation about Pixie patch here in which suggests to use patches. This works perfectly on nodeSelector however when using tolerations it doesn't work. Furthermore, I can't seem to find a way to add multiple tolerations

pixie-chart:
  enabled: true
    patches:
      vizier-pem: '{"spec": {"template": {"spec": { "nodeSelector": {"pixie": "allowed" }}}}}'

Describe the solution you'd like The ability to add multiple tolerations on vizier-pem pods and can be retrieved dynamically for example the snippet below is where I apply tolerations on other components of nri-bundle. I'm using a list of tolerations that differ depending on the environment being used.

kube-state-metrics:
  enabled: true
  tolerations:
    %{for tolerationKey in tolerationKeys}
    - key: ${tolerationKey}
      operator: Exists
      effect: NoSchedule
    %{endfor}

Describe alternatives you've considered Since I'm using Terraform, I've tried using resource local_file to generate the patch file with dynamic toleration values

# vizier-pem patch via kubernetes_manifest
resource "local_file" "vizier_patch_file" {
  filename   = "${path.module}/pixie-patch.yaml "
  content    = <<-EOT
  spec:
    template:
      spec:
        tolerations:
          %{for tolerationKey in local.node_selectors}
          - key: ${tolerationKey}
            operator: Exists
            effect: NoSchedule
          %{endfor}
  EOT
}

And then follow it up with null_resource local-exec to execute kubectl patch.

# Pixie Vizier-pem toleration patch
resource "null_resource" "vizier_pem_patch" {
  provisioner "local-exec" {
    interpreter = ["/bin/bash", "-c"]
    command     = <<EOT
    kubectl patch daemonset vizier-pem --patch-file ${path.module}/pixie-patch.yaml -n monitoring-newrelic
    EOT
  }
}

This solution works however, the resource local_exec creates noise as it re-create the vizier patch file every time it doesn't match the tolerations present in the environment.

Additional context I've checked on #598 however, it didn't work on my end. Any help or insight is greatly appreciated.

pixie-io / pixie

Adding multiple tolerations on vizier-pems #1939