In order to round out the kubernetes experience with bicep, it seems clear that we will need a helm provider.

DISCLAIMER: I don't use helm or kubernetes day-to-day, so there are probably some statements below that don't make sense! I'm using this issue to get thoughts down on paper and support future discussions.

Scenarios

Helm serves three distinct use cases today:

I need dynamic kubernetes manifests, which helm allows me to do via chart templates (YAML files in the templates/ directory)

Helm seems like overkill if this is the only need. There are seemingly lots and lots of options for templating manifests.
With the kubernetes provider for bicep, we are introducing yet another way of templating out a manifest. If we built a helm provider, it would not be to address this scenario.

I need to install helm charts

Without a helm provider, this won't be possible with bicep, other than with a deployment script. We actually have a dedicated aks-run-helm module in the public registry for this purpose.
While it's possible some users may want to convert their helm chart to bicep, there are many reasons why you would still want to author a helm chart.
- I need my helm chart to be installable in bicep and non-bicep scenarios.
- If I am not the owner/maintainer of the chart, then I have no way of getting the helm chart converted (nor may I want to)
- This potentially could help with migration from helm chart to bicep k8s provider (if needed)

Declarative deletion

helm has stacks-like behavior. Need to fill in details...

Some pseudo-code for a helm provider might look like the following (based on how this is implemented in TF):

resource chartInstall 'helm/release@v1' = {
  repository: 'https://charts.bitnami.com/bitnami' // I would assume we can support "local" charts in addition to ones in a registry?
  chart: 'nginx-ingress-controller'

  // could support keys/values or loadYamlContent() assuming there is already a `values.yaml` file
  values: { 
    foo: 'bar'
  }
  // values: loadYamlContent('values.yaml') 
}

tf uses set because helm CLI uses set
need to support set_sensitive for secrets

Implementation

One of the barriers to implementation is the lack of a .NET client for Helm. The canonical way to build helm "apps" would be to use the Go SDK.

Some implementation options that have not been given much thought:

Build first-class support creating providers in Go
Support container-based providers using a defined RPC. We built a prototype of this for our most recent hackathon.
- this is more desirable than Option 1
Call the helm CLI via a .NET bicep provider. This is the hackiest option, but also the lowest cost.
- helm CLI supports a JSON output, so there would be some element of maintainability/supportability
- helm CLI would need support for multi-tenancy, isolation, etc.

Open questions

Do we actually need this? How many use cases exist for Bicep + k8s if we don't support helm?
Do we need this so urgently that we should implement via the hacky solution?
How do we support local charts? Need to somehow package the chart using loadFileAsBase64 or something like that..
- this is a very important scenario for anyone authoring their own helm charts
Need to provide an async experience, which bicep providers do not support yet today.

LoadYamlContent would also need to provide ability to use parameters as argument which is not possible currently (tracked by #3816)

Today, I use this module for running Helm Scripts. https://github.com/Azure/bicep-registry-modules/tree/main/modules/deployment-scripts/aks-run-helm

Running Helm Commands

module helmInstallIngressController 'br/public:deployment-scripts/aks-run-helm:1.0.1' = {
  name: 'helmInstallIngressController'
  params: {
    aksName: aksName
    location: location
    helmApps: [
      {
        helmApp: 'bitnami/contour'
        helmAppName: 'contour-ingress'
        helmParams: '--version 7.7.1 --namespace ingress-basic --create_namespace --set envoy.kind=deployment --set contour.service.externalTrafficPolicy=cluster'
      }
    ]
  }
}

Proposed

With loadYamlContent, I will be able to directly interact with the value.yaml files that are already in use by helm today.

envoy:
  kind: deployment
contour:
  service:
    externalTrafficPolicy: cluster

var values = loadYamlContent('values.yaml')

module helmInstallIngressController 'br/public:deployment-scripts/aks-run-helm:1.0.1' = {
  name: 'helmInstallIngressController'
  params: {
    aksName: aksName
    location: location
    helmApps: [
      {
        helmApp: 'bitnami/contour'
        helmAppName: 'contour-ingress'
        helmParams: '--version 7.7.1 --namespace ingress-basic --create_namespace --set-json ${string(values)}'
      }
    ]
  }
}

We can use the union operation for more advanced cases where some values need to be dynamically set.

param envoyName string = 'envoy'
var override = {
  envoy:
    name: envoyName 
}
var values = union(loadYamlContent(values.yaml), override)

Here are some examples of where I am using a Bicep module while waiting for custom object types to organize very common scripts for AKS.

(None of this is a recommendation for how to do things. Instead, this would be the code I replace with the outcome of this proposal)

Ingress

@description('The name of the Azure Resource Group')
param resourceGroupName string = resourceGroup().name

@description('The IP of the Azure Public IP Address')
param staticIP string

var helmRepo = 'ingress-nginx'
var helmRepoURL = 'https://kubernetes.github.io/ingress-nginx'
var helmChart = 'ingress-nginx/ingress-nginx'
var helmName = 'ingress-nginx'
var namespace = 'ingress-basic'

var helmArgs = [
  'ingress-nginx.controller.replicaCount=2'
  'ingress-nginx.controller.labels.azure\\.workload\\.identity/use="true"'
  'ingress-nginx.controller.nodeSelector.kubernetes\\.io/os=linux'
  'ingress-nginx.controller.nodeSelector.kubernetes\\.io/arch=linux'
  'ingress-nginx.controller.image.repository=mcr.microsoft.com/oss/kubernetes/ingress/nginx-ingress-controller'
  'ingress-nginx.controller.image.tag=v1.0.4'
  'ingress-nginx.controller.image.digest=""'
  'ingress-nginx.controller.admissionWebhooks.patch.nodeSelector.kubernetes\\.io/os=linux'
  'ingress-nginx.controller.admissionWebhooks.patch.nodeSelector.kubernetes\\.io/arch=amd64'
  'ingress-nginx.controller.admissionWebhooks.patch.image.repository=mcr.microsoft.com/oss/kubernetes/ingress/nginx-ingress-controller'
  'ingress-nginx.controller.admissionWebhooks.patch.image.tag=v1.1.1'
  'ingress-nginx.controller.admissionWebhooks.patch.image.digest=""'
  'ingress-nginx.controller.defaultBackend.nodeSelector.kubernetes\\.io/os=linux'
  'ingress-nginx.controller.defaultBackend.nodeSelector.kubernetes\\.io/arch=amd64'
  'ingress-nginx.controller.defaultBackend.image.repository=mcr.microsoft.com/oss/kubernetes/defaultbackend'
  'ingress-nginx.controller.defaultBackend.image.tag=1.4'
  'ingress-nginx.controller.defaultBackend.image.digest=""'
  'ingress-nginx.controller.service.loadBalancerIP=${staticIP}'
  'ingress-nginx.controller.service.annotations.service\\.beta\\.kubernetes\\.io/azure-load-balancer-resource-group="${resourceGroupName}"'
]
var helmArgsString = replace(replace(string(helmArgs), '[', ''), ']', '')

var helmCharts = {
  helmRepo: helmRepo
  helmRepoURL: helmRepoURL
  helmChart: helmChart
  helmName: helmName
  helmNamespace: namespace
  helmValues: helmArgsString
  version: '4.1.3'
}

output helmChart object = helmCharts

Workload Identity

param azureTenantID string = subscription().tenantId
var namespace = 'azure-workload-identity-system'

var helmCharts = {
  helmRepo: 'azure-workload-identity'
  helmRepoURL: 'https://azure.github.io/azure-workload-identity/charts'
  helmChart: 'azure-workload-identity/workload-identity-webhook'
  helmName: 'workload-identity-webhook'
  helmNamespace: namespace
  helmValues: 'azureTenantID=${azureTenantID}'
}

output helmChart object = helmCharts

Secret Store

@description('Enable the syncSecret setting for the CSI Driver')
param enableSync bool = true

@description('Enable the secret rotation setting for the CSI Driver')
param enableRotation bool = true

var helmRepo = 'csi-secrets-store-provider-azure'
var helmRepoURL = 'https://azure.github.io/secrets-store-csi-driver-provider-azure/charts'
var helmChart = 'csi-secrets-store-provider-azure/csi-secrets-store-provider-azure'
var helmName = 'csi'
var namespace = 'kube-system'

var helmJson = {
  'csi-secrets-store-provider-azure': {
    'secrets-store-csi-driver': {
      'syncSecret': { 'enabled': enableSync }
      'enableSecretRotation': enableRotation
    }
  }
}

var helmArgs = [
  'secrets-store-csi-driver.syncSecret.enabled=${helmJson[helmRepo]['secrets-store-csi-driver'].syncSecret.enabled}'
  'secrets-store-csi-driver.enableSecretRotation=${helmJson[helmRepo]['secrets-store-csi-driver'].enableSecretRotation}'
]
var helmArgsString = replace(replace(string(helmArgs), '[', ''), ']', '')

var helmCharts = {
  helmRepo: helmRepo
  helmRepoURL: helmRepoURL
  helmChart: helmChart
  helmName: helmName
  helmNamespace: namespace
  helmValues: helmArgsString
  helmJson: helmJson
}

output helmChart object = helmCharts

Running the Scripts

In order to run these all in one script action, I load them all together and use a for loop.

param aksName string
param location string
param staticIP string = ''
param additionalCharts array = []

param enableWorkloadIdentity bool = true
#disable-next-line secure-secrets-in-params 
param enableSecretStore bool = true
param enableIngress bool = true
param azureTenantID string = subscription().tenantId

module helmInstallWorkloadID 'workload-id.bicep' = if(enableWorkloadIdentity) { 
  name: 'helmInstallWorkloadID-${uniqueString(aksName, location, resourceGroup().name)}'
  params: {
    azureTenantID: azureTenantID
  }
}

module helmInstallSecretStore 'csi-secret-store.bicep' = if(enableSecretStore) { name: 'helmInstallSecretStore-${uniqueString(aksName, location, resourceGroup().name)}' }

resource publicIP 'Microsoft.Network/publicIPAddresses@2021-03-01' existing = {
  name: staticIP
}

module helmInstallIngress 'nginx-ingress.bicep' = if(enableIngress) {
  name: 'helmInstallIngress-${uniqueString(aksName, location, resourceGroup().name)}'
  params: { staticIP: publicIP.properties.ipAddress }
}

var helmCharts = union(enableWorkloadIdentity ? [helmInstallWorkloadID.outputs.helmChart] : [], enableIngress ? [helmInstallIngress.outputs.helmChart] : [], enableSecretStore ? [helmInstallSecretStore.outputs.helmChart] : [], additionalCharts)

module combo 'br/public:deployment-scripts/aks-run-helm:1.0.1' = {
  name: 'helmInstallCombo-${uniqueString(aksName, location, resourceGroup().name)}'
  params: {
    aksName: aksName
    location: location
    helmCharts: helmCharts
  }
}

@alex-frankel - I think your description of the two scenarios are spot on. This matches how we've thought about Helm + Bicep.

@dciborow what I am trying to understand with this proposal is less about the introduction of loadYamlContent() and more about the introduction of a first-class provider for Helm. IOW, what is the best way to obsolete your helm deployment script?

Just a few comments based your OP. For a side project I've tried bicep with AKS, as I thought the kubernetes provider would've been fully available and documented by now, but I went for a start with the deployment script module for Helm. I haven't figured the kubernetes provider out yet tbh :smile:

Disclosure: I'm not a bicep fan, active contributor for the Terraform Provider for Azure(azurerm).

In order to round out the kubernetes experience with bicep, it seems clear that we will need a helm provider.

DISCLAIMER: I don't use helm or kubernetes day-to-day, so there are probably some statements below that don't make sense! I'm using this issue to get thoughts down on paper and support future discussions.

Scenarios

Helm serves three distinct use cases today:

I need dynamic kubernetes manifests, which helm allows me to do via chart templates (YAML files in the templates/ directory)

Helm seems like overkill if this is the only need. There are seemingly lots and lots of options for templating manifests.

With the kubernetes provider for bicep, we are introducing yet another way of templating out a manifest. If we built a helm provider, it would not be to address this scenario.

I need to install helm charts

Without a helm provider, this won't be possible with bicep, other than with a deployment script. We actually have a dedicated aks-run-helm module in the public registry for this purpose.

Not to bump into anyone, but if that worked, that would already have made it a lot easier. I've copied the bicep code and optimised it a bit, as at this moment in time it's completely useless (as in: it failed my scenarios) and unclear whether the module was successful or not. It would be awesome to be able to let fail a script module when the script is failing, not sure whether it is the implementation or bicep in general.

While it's possible some users may want to convert their helm chart to bicep, there are many reasons why you would still want to author a helm chart.

I need my helm chart to be installable in bicep and non-bicep scenarios.

A different scenario not thought of yet is also maintenance: inspection, daily operation and upgrade and deletion operations. So not only install scenario's, also inspect, update, upgrade, uninstall. I'd expect that the Helm CLI is expected for some inspection when things going wrong, so cross-tool use of Helm.

If I am not the owner/maintainer of the chart, then I have no way of getting the helm chart converted (nor may I want to)

Not sure if that is covered by this, but cross-team use of base templates which are provided by a platform team and used within app team helm charts as dependencies

This potentially could help with migration from helm chart to bicep k8s provider (if needed)

Declarative deletion

helm has stacks-like behavior. Need to fill in details...

Some pseudo-code for a helm provider might look like the following (based on how this is implemented in TF):
resource chartInstall 'helm/release@v1' = {
  repository: 'https://charts.bitnami.com/bitnami' // I would assume we can support "local" charts in addition to ones in a registry?
  chart: 'nginx-ingress-controller'

  // could support keys/values or loadYamlContent() assuming there is already a `values.yaml` file
  values: { 
    foo: 'bar'
  }
  // values: loadYamlContent('values.yaml') 
}
tf uses set because helm CLI uses set

need to support set_sensitive for secrets

Not sure about CRDs and CR provisioning yet. Is that implemented in the kubernetes provider already? Support for that took a long time in Terraform to become available and was a huge pitfall for proper adoption of K8s and dependant Helm charts (like cert-manager) with Terraform.

Implementation

One of the barriers to implementation is the lack of a .NET client for Helm. The canonical way to build helm "apps" would be to use the Go SDK.

Some implementation options that have not been given much thought:

Build first-class support creating providers in Go

Would be awesome:

we could create a bicep terraform provider 💯
we would disclose the bicep provider development to Go developers (👋🏽 )

Support container-based providers using a defined RPC. We built a prototype of this for our most recent hackathon.

this is more desirable than Option 1

This sounds awesome, but in what way is this different from the deployment script module? Can we give WhatIf feedback in that protocol and fail gracefully? I believe that this would be my direction from an outsider perspective, as this would (as I see it) also make building providers in Go possible.

Call the helm CLI via a .NET bicep provider. This is the hackiest option, but also the lowest cost.

helm CLI supports a JSON output, so there would be some element of maintainability/supportability

helm CLI would need support for multi-tenancy, isolation, etc.

Open questions

Do we actually need this? How many use cases exist for Bicep + k8s if we don't support helm?

I'm not sure about the numbers, but whenever I start with an AKS cluster I'd need a few helm charts to get dependencies into place, like cert-manager and nginx-ingress. Both are possible to deploy in other ways of course..

Do we need this so urgently that we should implement via the hacky solution?

The deployment script module is already hacky, if that works it is not as urgent. Time and speed is an issue there, as deployment scripts take their time. Any hacky solution we'd go with should be failing clear and gracefully.

How do we support local charts? Need to somehow package the chart using loadFileAsBase64 or something like that..

this is a very important scenario for anyone authoring their own helm charts

💯

Need to provide an async experience, which bicep providers do not support yet today.

Helm is huge, the cases where people deploy "vanilla" resources to Kubernetes using standard Kubernetes manifests is dwarfed by the usage of Helm. It's the defacto way to deploy to Kubernetes IMO. This my way of saying - this is very much needed!

We absolutely need to support local Helm charts as well as those in remote repos

I'd love to see first class support for extensions/providers via Go. Weirdly Dapr has the opposite problem where the .NET community can't add custom components to Dapr as they need to be in Go. However I can see getting Go support (or some other native binary interface to support other languages) might be a pretty far out long term roadmap item which will be hard to prioritize. So perhaps the hacky route is the way to go for now

Happy to provide more inputs, as a fairly heavy Kubernetes and Helm & Go user

Any update on this at all?

Azure / bicep

Helm provider for bicep #9088

Scenarios

Implementation

Open questions

Running Helm Commands

Proposed

Ingress

Workload Identity

Secret Store

Running the Scripts

Scenarios

Implementation

Open questions