traefik / traefik-helm-chart

Traefik Proxy Helm Chart
https://traefik.io
Apache License 2.0
1.05k stars 745 forks source link

Docs issues -values.yml ports.$foo.exposed #1127

Open srcreigh opened 1 month ago

srcreigh commented 1 month ago

Welcome!

What version of the Traefik's Helm Chart are you using?

traefik-25.0.2_up25.0.0

What version of Traefik are you using?

2.10.5

What did you do?

after reading values.yaml, I tried to setup a UDP ingress route for a DNS server via the helm chart by applying this config

apiVersion: helm.cattle.io/v1
kind: HelmChartConfig
metadata:
  name: traefik
  namespace: kube-system
spec:
  valuesContent: |-
    ports:
      dns:
        protocol: UDP
        port: 53
        expose:
          default: true
        exposedPort: 53

What did you see instead?

First thing was that dig google.com @$CLUSTER_IP was unreachable.

Traefik access logs didn't show any UDP access attempts, but i later found that traefik access logs doesn't support UDP traffic anyways.

Next I found the helm chart pod was failing.

kubectl get pods -n kube-system
NAME                                      READY   STATUS             RESTARTS       AGE
...
helm-install-traefik-cqt89                0/1     CrashLoopBackOff   5 (18s ago)    3m25s

Looking at the logs for the failing pod it had to do with the exposed.default (see end)

kubectl logs helm-install-traefik-cqt89 -n kube-system
if [[ ${KUBERNETES_SERVICE_HOST} =~ .*:.* ]]; then
        echo "KUBERNETES_SERVICE_HOST is using IPv6"
        CHART="${CHART//%\{KUBERNETES_API\}%/[${KUBERNETES_SERVICE_HOST}]:${KUBERNETES_SERVICE_PORT}}"
else
        CHART="${CHART//%\{KUBERNETES_API\}%/${KUBERNETES_SERVICE_HOST}:${KUBERNETES_SERVICE_PORT}}"
fi

set +v -x
+ [[ '' != \t\r\u\e ]]
+ export HELM_HOST=127.0.0.1:44134
+ HELM_HOST=127.0.0.1:44134
+ helm_v2 init --skip-refresh --client-only --stable-repo-url https://charts.helm.sh/stable/
+ tiller --listen=127.0.0.1:44134 --storage=secret
$HELM_HOME has been configured at /home/klipper-helm/.helm.
Not installing Tiller due to 'client-only' flag having been set
++ ++ jq -r '.Releases | length'
timeout -s KILL 30 helm_v2 ls --all '^traefik$' --output json
[main] 2024/07/16 03:25:47 Starting Tiller v2.17.0 (tls=false)
[main] 2024/07/16 03:25:47 GRPC listening on 127.0.0.1:44134
[main] 2024/07/16 03:25:47 Probes listening on :44135
[main] 2024/07/16 03:25:47 Storage driver is Secret
[main] 2024/07/16 03:25:47 Max history per release is 0
[storage] 2024/07/16 03:25:47 listing all releases with filter
+ V2_CHART_EXISTS=
+ [[ '' == \1 ]]
+ [[ '' == \v\2 ]]
+ shopt -s nullglob
+ [[ -f /config/ca-file.pem ]]
+ [[ -f /tmp/ca-file.pem ]]
+ [[ -n '' ]]
+ helm_content_decode
+ set -e
+ ENC_CHART_PATH=/chart/traefik.tgz.base64
+ CHART_PATH=/tmp/traefik.tgz
+ [[ ! -f /chart/traefik.tgz.base64 ]]
+ return
+ [[ install != \d\e\l\e\t\e ]]
+ helm_repo_init
+ grep -q -e 'https\?://'
chart path is a url, skipping repo update
+ echo 'chart path is a url, skipping repo update'
+ helm_v3 repo remove stable
Error: no repositories configured
+ true
+ return
+ helm_update install --set-string global.systemDefaultRegistry=
+ [[ helm_v3 == \h\e\l\m\_\v\3 ]]
++ helm_v3 ls --all -f '^traefik$' --namespace kube-system --output json
++ jq -r '"\(.[0].app_version),\(.[0].status)"'
++ tr '[:upper:]' '[:lower:]'
+ LINE=v2.10.5,deployed
+ IFS=,
+ read -r INSTALLED_VERSION STATUS _
+ VALUES=
+ for VALUES_FILE in /config/*.yaml
+ VALUES=' --values /config/values-01_HelmChart.yaml'
+ for VALUES_FILE in /config/*.yaml
+ VALUES=' --values /config/values-01_HelmChart.yaml --values /config/values-10_HelmChartConfig.yaml'
+ [[ install = \d\e\l\e\t\e ]]
+ [[ v2.10.5 =~ ^(|null)$ ]]
+ [[ deployed =~ ^(pending-install|pending-upgrade|pending-rollback)$ ]]
+ [[ deployed == \d\e\p\l\o\y\e\d ]]
+ echo 'Already installed traefik'
Already installed traefik
+ [[ helm_v3 == \h\e\l\m\_\v\3 ]]
+ helm_v3 mapkubeapis traefik --namespace kube-system
2024/07/16 03:25:47 Release 'traefik' will be checked for deprecated or removed Kubernetes APIs and will be updated if necessary to supported API versions.
2024/07/16 03:25:47 Get release 'traefik' latest version.
2024/07/16 03:25:48 Check release 'traefik' for deprecated or removed APIs...
2024/07/16 03:25:48 Finished checking release 'traefik' for deprecated or removed APIs.
2024/07/16 03:25:48 Release 'traefik' has no deprecated or removed APIs.
2024/07/16 03:25:48 Map of release 'traefik' deprecated or removed APIs to supported versions, completed successfully.
+ echo 'Upgrading helm_v3 chart'
Upgrading traefik
+ echo 'Upgrading traefik'
+ shift 1
+ helm_v3 upgrade --set-string global.systemDefaultRegistry= traefik https://10.43.0.1:443/static/charts/traefik-25.0.2+up25.0.0.tgz --values /config/values-01_HelmChart.yaml --values /config/values-10_HelmChartConfig.yaml
Error: UPGRADE FAILED: template: traefik/templates/service.yaml:21:12: executing "traefik/templates/service.yaml" at <eq $config.expose true>: error calling eq: incompatible types for comparison

What is your environment & configuration?

k3s 1.29.1+k3s2-1

Tailwind

$ uname -a
Linux ... 6.9.9-arch1-1 #1 SMP PREEMPT_DYNAMIC Fri, 12 Jul 2024 00:06:53 +0000 x86_64 GNU/Linux
Name:                     traefik
Namespace:                kube-system
Labels:                   app.kubernetes.io/instance=traefik-kube-system
                          app.kubernetes.io/managed-by=Helm
                          app.kubernetes.io/name=traefik
                          helm.sh/chart=traefik-25.0.2_up25.0.0
Annotations:              meta.helm.sh/release-name: traefik
                          meta.helm.sh/release-namespace: kube-system
Selector:                 app.kubernetes.io/instance=traefik-kube-system,app.kubernetes.io/name=traefik
Type:                     LoadBalancer
IP Family Policy:         PreferDualStack
IP Families:              IPv4
IP:                       10.43.144.96
IPs:                      10.43.144.96
LoadBalancer Ingress:     192.168.0.12, 192.168.0.13, 192.168.0.14, 192.168.0.15, 192.168.0.16, 192.168.0.17
Port:                     web  80/TCP
TargetPort:               web/TCP
NodePort:                 web  32062/TCP
Endpoints:                10.42.1.72:8000
Port:                     websecure  443/TCP
TargetPort:               websecure/TCP
NodePort:                 websecure  31071/TCP
Endpoints:                10.42.1.72:8443
Port:                     dns  53/UDP
TargetPort:               dns/UDP
NodePort:                 dns  31989/UDP
Endpoints:                10.42.1.72:53
Session Affinity:         None
External Traffic Policy:  Cluster

Additional Information

I ended up just sort of guessing upon a working helm chart config


apiVersion: helm.cattle.io/v1
kind: HelmChartConfig
metadata:
  name: traefik
  namespace: kube-system
spec:
  valuesContent: |-
    ports:
      dns:
        protocol: UDP
        port: 53
        expose: true
        exposedPort: 53

I'm posting this report to request some better documentation about the values. I was able to get this to work only because of lucky guesswork.

The link to the values (which are the only docs for configuring this chart AFAICT) are buried in the 2-3rd page of README under deployment section.

The expose: true option that I chanced upon which works doesn't seem to be documented at all (at least not reachable from README), and the expose.default: true option which is documented errors out.

darkweaver87 commented 1 month ago

Hello @srcreigh ,

Thanks for reaching out.

First, I tried to extract the chart from k3s v1.29.1+k3s2:

mkdir /tmp/charts && cd /tmp/charts
git clone git@github.com:k3s-io/k3s.git --depth 1 --branch v1.29.1+k3s2 .

cat <<EOF > test.go
package main

import "github.com/k3s-io/k3s/pkg/static"

func main(){
  static.Stage("/tmp")
}
EOF

go run test.go
tar xfvz /tmp/charts/traefik-25.0.2+up25.0.0.tgz

curl -L https://github.com/traefik/traefik-helm-chart/archive/refs/tags/v25.0.0.tar.gz -o v25.0.0.tar.gz
tar xfvz v25.0.0.tar.gz 

diff -rq traefik-helm-chart-25.0.0/traefik/ traefik/
Files traefik-helm-chart-25.0.0/traefik/Chart.yaml and traefik/Chart.yaml differ
Only in traefik-helm-chart-25.0.0/traefik/: crds
Only in traefik/: EXAMPLES.md
Only in traefik/: LICENSE
Only in traefik/: README.md
Files traefik-helm-chart-25.0.0/traefik/templates/_helpers.tpl and traefik/templates/_helpers.tpl differ
Files traefik-helm-chart-25.0.0/traefik/templates/_podtemplate.tpl and traefik/templates/_podtemplate.tpl differ
Only in traefik/templates: validate-install-crd.yaml
Only in traefik-helm-chart-25.0.0/traefik/: tests
Files traefik-helm-chart-25.0.0/traefik/values.yaml and traefik/values.yaml differ

This looks like a modified version of the one we are releasing (and I don't know why since we can override parameters which are changed by running --set at the install/upgrade stage), so I strongly encourage you to use --disable=traefik k3s option and use upstream chart :-)

You'll benefit from an up to date traefik and docs :-)

That's said, it seems you were looking at latest v29.0.1 values.yaml which differs from v25.0.0 especially in ports exposition which were refactored in v27.0.0 (cf. release note).

Do you think we can improve the doc or it's the change between v25.0.0 and v29.0.1 you didn't notice ?

Rémi

srcreigh commented 1 month ago

thank you very much Rémi. I see my situation was a bit confused.

I haven’t had the time to check at this again, but I’m planning to and will update later :-)

On Tue, Jul 16, 2024 at 3:39 AM Rémi BUISSON @.***> wrote:

Hello @srcreigh https://github.com/srcreigh ,

Thanks for reaching out.

First, I tried to extract the chart from k3s v1.29.1+k3s2:

mkdir /tmp/charts && cd /tmp/charts git clone @.***:k3s-io/k3s.git --depth 1 --branch v1.29.1+k3s2 .

cat < test.gopackage mainimport "github.com/k3s-io/k3s/pkg/static"func main(){ static.Stage("/tmp")}EOF

go run test.go tar xfvz /tmp/charts/traefik-25.0.2+up25.0.0.tgz

curl -L https://github.com/traefik/traefik-helm-chart/archive/refs/tags/v25.0.0.tar.gz -o v25.0.0.tar.gz tar xfvz v25.0.0.tar.gz

diff -rq traefik-helm-chart-25.0.0/traefik/ traefik/

Files traefik-helm-chart-25.0.0/traefik/Chart.yaml and traefik/Chart.yaml differ Only in traefik-helm-chart-25.0.0/traefik/: crds Only in traefik/: EXAMPLES.md Only in traefik/: LICENSE Only in traefik/: README.md Files traefik-helm-chart-25.0.0/traefik/templates/_helpers.tpl and traefik/templates/_helpers.tpl differ Files traefik-helm-chart-25.0.0/traefik/templates/_podtemplate.tpl and traefik/templates/_podtemplate.tpl differ Only in traefik/templates: validate-install-crd.yaml Only in traefik-helm-chart-25.0.0/traefik/: tests Files traefik-helm-chart-25.0.0/traefik/values.yaml and traefik/values.yaml differ

This look like a modified version of the one we are releasing https://github.com/traefik/traefik-helm-chart/releases/tag/v25.0.0 (and I don't know why since we can override parameters which are changed by running --set at the install/upgrade stage), so I strongly encourage you to use --disable=traefik k3s option and use upstream chart :-)

You'll benefit from an up to date traefik and docs :-)

That's said, it seems you were looking at latest v29.0.1 values.yaml which differs from v25.0.0 especially in ports exposition which were refactored in v27.0.0 https://github.com/traefik/traefik-helm-chart/releases/tag/v27.0.0 (cf. release note).

Rémi

— Reply to this email directly, view it on GitHub https://github.com/traefik/traefik-helm-chart/issues/1127#issuecomment-2230229007, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAU6W4J2UUDPNGKJLE6Z2ZLZMTE2NAVCNFSM6AAAAABK5XV6HKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEMZQGIZDSMBQG4 . You are receiving this because you were mentioned.Message ID: @.***>