helm templates with lookup function not working with skaffold v2

Context

In our Helm templates, we heavily rely on the lookup function to apply data to our application templates.

One use case is that we use the lookup function to fetch the ClusterIP from our internal namespaced service to add hostAliases to our deployments and statefulsets to route http data internally instead of externally.

Expected behavior

In skaffold v1, when skaffold would directly execute helm upgrade --install, this lookup method would work as expected. With skaffold v2, we expect our use case to be still available. So when executing skaffold build and skaffold deploy, skaffold should not return an error about the lookup function or data is unavailable.

Maybe we are missing something, and is the option still available. However, I could not find any information in the documentation to use the old behaviour.

Actual behavior

Because of the helm template action in the manifest step, the lookup function is unavailable (restriction of helm template). Then executing our deployment strategy using skaffold build and skaffold deploy we now get the error that the template could not be executed properly and fails on the lookup in the templates.

The error we get is the following

Error: template: example/templates/application.yaml:31:45: executing "example/templates/application.yaml" at <$internalLoadBalancerService.spec.clusterIP>: nil pointer evaluating interface {}.clusterIP

Use --debug flag to render out invalid YAML
)

Information

Skaffold version: v2.13.1
Operating system: Mixed. macOS for development, Linux for CI/CD process
Installed via: Google Cloud SDK (GitHub releases)
Contents of skaffold.yaml:

# skaffold.yaml
apiVersion: skaffold/v4beta11
kind: Config
metadata:
  name: example
build:
  local:
    push: true
    useDockerCLI: false
    useBuildkit: true
    concurrency: 0
  tagPolicy:
    inputDigest: { }
  artifacts:
    - image: registry.example.com/example.com
manifests:
  helm:
    releases:
      - name: example
        chartPath: .helm
        wait: true
        valuesFiles:
          - .helm/values.yaml
        setValueTemplates:
          example.image: registry.example.com/example.com
    flags:
      install:
        - --timeout=120m
        - --render-subchart-notes
      upgrade:
        - --timeout=5m
        - --render-subchart-notes
deploy:
  helm:
    releases:
      - name: example
        chartPath: .helm
        wait: true
        valuesFiles:
          - .helm/values.yaml
        setValueTemplates:
          example.image: registry.example.com/example.com
    flags:
      install:
        - --timeout=120m
        - --render-subchart-notes
      upgrade:
        - --timeout=5m
        - --render-subchart-notes
profiles:
  - name: dev
    activation:
      - command: dev
    patches:
      - op: add
        path: /manifests/helm/releases/0/valuesFiles
        value:
          - .helm/values.dev.yaml
      - op: add
        path: /deploy/helm/releases/0/valuesFiles
        value:
          - .helm/values.dev.yaml
      - op: replace
        path: /build/artifacts/0/image
        value: registry.example.com/dev/example.com
      - op: replace
        path: /manifests/helm/releases/0/setValueTemplates/example.image
        value: registry.example.com/dev/example.com
      - op: replace
        path: /deploy/helm/releases/0/setValueTemplates/example.image
        value: registry.example.com/dev/example.com

Steps to reproduce the behavior

At first, create a simple helm chart using the files below

# values.yaml
global:
  internalLoadBalancerService: private-http-proxy

# Chart.yaml
apiVersion: v2
name: example-chart
version: 1.0.0

# templates/deployment.yaml
---
apiVersion: apps/v1
kind: Deployment
metadata:
  annotations:
    meta.helm.sh/release-name: example-chart
  labels:
    app.kubernetes.io/instance: example-chart
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: example-app
    helm.sh/chart: example-1.0.0
  name: example-app
spec:
  replicas: 0
  selector:
    matchLabels:
      app.kubernetes.io/instance: example-chart
      app.kubernetes.io/name: example-app
  template:
    metadata:
      labels:
        app.kubernetes.io/instance: example-chart
        app.kubernetes.io/name: example-app
    spec:
      containers:
      - env:
        - name: CONFIG_CHECKSUM
          value: 1fed86344f801e7f1a1e5490202fade968f1fd54
        envFrom:
        - configMapRef:
            name: example-application
        - secretRef:
            name: example-application
        image: registry.example.com/example.com:e57801aa5fb3576ec30407301963bad85be187cca670dbbfadcf4982962e33e4
        imagePullPolicy: IfNotPresent
        name: example-app
        ports:
        - containerPort: 54663
          name: http
          protocol: TCP
      {{- if .Values.global.internalLoadBalancerService }}
      {{- $internalLoadBalancerService := (lookup "v1" "Service" .Release.Namespace .Values.global.internalLoadBalancerService) }}
      hostAliases:
        - ip: {{ $internalLoadBalancerService.spec.clusterIP }}
          hostnames:
          {{- $hosts := list "example.com" }}
          {{- range $host := $hosts }}
            - {{ $host | quote }}
          {{- end}}
      {{- end }}

Apply the service below using `kubectl apply -f service.yaml -n [NAMESPACE]`

# service.yaml
---
apiVersion: v1
kind: Service
metadata:
  name: private-http-proxy
spec:
  clusterIP: 172.18.0.254
  clusterIPs:
    - 172.18.0.254
  internalTrafficPolicy: Cluster
  ipFamilies:
    - IPv4
  ipFamilyPolicy: SingleStack
  ports:
    - name: http
      port: 80
      protocol: TCP
      targetPort: 980
    - name: https
      port: 443
      protocol: TCP
      targetPort: 9443
  type: ClusterIP

Use helm to deploy the chart created earlier

helm upgrade --install example .helm --debug

The correct ClusterIP should be set in the applications hostAliases settings.

Use skaffold to apply the helm chart, using the `skaffold.build-artifacts.json` below and the skaffold.yaml provided above.

skaffold.build-artifacts.json

{"builds": [{"imageName": "registry.example.com/example.com","tag": "registry.example.com/example.com:e57801aa5fb3576ec30407301963bad85be187cca670dbbfadcf4982962e33e4"}]}

skaffold deploy --build-artifacts skaffold.build-artifacts.json -n [NAMESPACE]

See that you get the error

std out err: %!(EXTRA *errors.errorString=Error: template: example-chart/templates/deployment.yaml:44:45: executing "example-chart/templates/deployment.yaml" at <$internalLoadBalancerService.spec.clusterIP>: nil pointer evaluating interface {}.clusterIP

Use --debug flag to render out invalid YAML
)

When using skaffold v1.39.18, and the skaffold-v1.yaml provided below, the skaffold deploy will succeed and the correct ClusterIP should be set in the applications hostAliases settings.

# skaffold-v1.yaml
apiVersion: skaffold/v2beta29
kind: Config
metadata:
  name: example
build:
  local:
    push: true
    useDockerCLI: false
    useBuildkit: true
    concurrency: 0
  tagPolicy:
    inputDigest: { }
  artifacts:
    - image: registry.example.com/example.com
deploy:
  helm:
    releases:
      - name: example
        chartPath: .helm
        wait: true
        valuesFiles:
          - .helm/values.yaml
        artifactOverrides:
          example.image: registry.example.com/example.com
    flags:
      install:
        - --timeout=120m
        - --render-subchart-notes
      upgrade:
        - --timeout=5m
        - --render-subchart-notes
profiles:
  - name: dev
    activation:
      - command: dev
    patches:
      - op: add
        path: /manifests/helm/releases/0/valuesFiles
        value:
          - .helm/values.dev.yaml
      - op: add
        path: /deploy/helm/releases/0/valuesFiles
        value:
          - .helm/values.dev.yaml
      - op: replace
        path: /build/artifacts/0/image
        value: registry.example.com/dev/example.com
      - op: replace
        path: /manifests/helm/releases/0/artifactOverrides/example.image
        value: registry.example.com/dev/example.com
      - op: replace
        path: /deploy/helm/releases/0/artifactOverrides/example.image
        value: registry.example.com/dev/example.com

skaffold deploy --build-artifacts skaffold.build-artifacts.json -f skaffold-v1.yaml -n [NAMESPACE]

GoogleContainerTools / skaffold