Server doesn't have a resource type "serviceaccount"

[Falx]

We have a buildserver that deploys to a kubernetes cluster with helmsman. As of late we complete helmsman apply anymore.

When doing - helmsman --debug --apply --keep-untracked-releases --force-upgrades -f .deployment/our_desired_state_file, we get this output:

...
 ------------------- 
 2020/05/15 08:24:19 INFO: validating that tiller is installed.
 2020/05/15 08:24:19 INFO: namespace validation -- Either no or invalid certs/keys provided for DEPLOYING Tiller with TLS in namespace [ kube-system ].
 2020/05/15 08:24:19 INFO: namespace validation -- Tiller is NOT desired to be deployed in namespace [ kafka ].
 2020/05/15 08:24:19 INFO: namespace validation -- Tiller is NOT desired to be deployed in namespace [ deps ].
 2020/05/15 08:24:19 INFO: namespace validation -- Tiller is NOT desired to be deployed in namespace [ auth ].
 2020/05/15 08:24:19 INFO: namespace validation -- Tiller is NOT desired to be deployed in namespace [ ingress-nginx ].
 2020/05/15 08:24:19 INFO: setting kubectl context to [ idlab-k8s-production ]
 2020/05/15 08:24:19 INFO: initializing helm on the client only.
 2020/05/15 08:24:19 INFO: adding repo ingress-nginx
 2020/05/15 08:24:20 INFO: adding repo stable
 2020/05/15 08:24:22 INFO: adding repo incubator
 2020/05/15 08:24:23 INFO: adding repo idlab
 2020/05/15 08:24:23 INFO: adding repo codecentric
 2020/05/15 08:24:23 INFO: updating helm repos
 2020/05/15 08:24:26 INFO: creating namespace  kafka
 2020/05/15 08:24:26 WARN: I could not create namespace [ kafka ]. It already exists. I am skipping this.
 2020/05/15 08:24:26 INFO: creating namespace  deps
 2020/05/15 08:24:27 WARN: I could not create namespace [ deps ]. It already exists. I am skipping this.
 2020/05/15 08:24:27 INFO: creating namespace  auth
 2020/05/15 08:24:27 WARN: I could not create namespace [ auth ]. It already exists. I am skipping this.
 2020/05/15 08:24:27 INFO: creating namespace  ingress-nginx
 2020/05/15 08:24:27 WARN: I could not create namespace [ ingress-nginx ]. It already exists. I am skipping this.
 2020/05/15 08:24:27 INFO: creating namespace  kube-system
 2020/05/15 08:24:27 WARN: I could not create namespace [ kube-system ]. It already exists. I am skipping this.
 2020/05/15 08:24:27 INFO: deploying Tiller in namespace [ kube-system ].
 2020/05/15 08:24:27 INFO: validating if serviceaccount [ tiller ] exists in namespace [ kube-system ].
 2020/05/15 08:24:27 ERROR: while validating/creating service account [ tiller ] in namespace [kube-system]: error: the server doesn't have a resource type "serviceaccount"
Running after_script
00:01
Uploading artifacts for failed job
00:02
 ERROR: Job failed: exit code 1

We have no idea how to fix this, things we've done:

• Checked all RBAC files multiple times (we have a serviceaccount tiller, that has clusterrolebinding to the cluster-admin clusterrole)
• helm install a release from client pc to k8s server (this works, so it isn't helm not working)
• updated k8s from 1.13 to 1.14 (upgrade successfull, all service running, still no luck on deploying new releases)
• use the latest available helmsman for the k8s-helm combo
• helm reset --force followed by helm init --service-account tiller

Any help would be greatly appreciated!

We are using:

• kubernetes 1.14
• helm 2.14.3 (both on service and client, and buildserver)
• helmsman 1.12.0
• buildserver uses docker image: "praqma/helmsman:v1.12.0-helm-v2.14.3"

Relevant parts of our DSF:

settings:
  kubeContext: "idlab-k8s-production"

helmRepos:
  stable: "https://kubernetes-charts.storage.googleapis.com"
  incubator: "http://storage.googleapis.com/kubernetes-charts-incubator"
  ...
  codecentric: "https://codecentric.github.io/helm-charts"
  ingress-nginx: "https://kubernetes.github.io/ingress-nginx/"

namespaces:
  kafka:
  deps:
  auth:
  ingress-nginx:
  kube-system:
    installTiller: true
    tillerServiceAccount: tiller

apps:
  traefik:
    namespace: "kube-system"
    enabled: true
    chart: "stable/traefik"
    version: "1.61.1"
    valuesFile: "configs/production-traefik.yaml"
  kafka:
    namespace: "kafka"
    enabled: true
    chart: "incubator/kafka"
    version: "0.13.10"
    valuesFile: "configs/production-kafka.yaml"
  mongodb:
    namespace: "deps"
    enabled: true
    chart: "stable/mongodb"
    version: "5.3.2"
    valuesFile: "configs/production-mongodb.yaml"
  redis:
    namespace: "deps"
    enabled: true
    chart: "stable/redis"
    version: "6.4.5"
    valuesFile: "configs/redis.yaml"
    set:
      metrics.serviceMonitor.namespace: "monitoring"
  ingress-nginx:
    namespace: "ingress-nginx"
    enabled: true
    chart: "ingress-nginx/ingress-nginx"
    version: "2.1.0"
    valuesFile: "configs/ingress-nginx.yaml"

[sami-alajrami]

@Falx can you run with --verbose to log the kubectl command leading to this error (Note this may log sensitive data you pass to your charts).

[Falx]

Thanks for your quick reaction @sami-alajrami! I've redacted the sensitive information. Output is below (a few commands added to show that serviceaccounts resource works)

[...]
 $ kubectl auth can-i get serviceaccounts
 yes
 $ kubectl -n kube-system get serviceaccount
 NAME                                 SECRETS   AGE
 attachdetach-controller              1         36d
 bootstrap-signer                     1         36d
 certificate-controller               1         36d
 clusterrole-aggregation-controller   1         36d
 coredns                              1         36d
 cronjob-controller                   1         36d
 daemon-set-controller                1         36d
 default                              1         36d
 deployment-controller                1         36d
 disruption-controller                1         36d
 endpoint-controller                  1         36d
 expand-controller                    1         36d
 flannel                              1         36d
 generic-garbage-collector            1         36d
 horizontal-pod-autoscaler            1         36d
 job-controller                       1         36d
 kube-proxy                           1         36d
 namespace-controller                 1         36d
 node-controller                      1         36d
 persistent-volume-binder             1         36d
 pod-garbage-collector                1         36d
 pv-protection-controller             1         36d
 pvc-protection-controller            1         36d
 replicaset-controller                1         36d
 replication-controller               1         36d
 resourcequota-controller             1         36d
 service-account-controller           1         36d
 service-controller                   1         36d
 statefulset-controller               1         36d
 tiller                               1         35d
 token-cleaner                        1         36d
 ttl-controller                       1         36d
 $ helmsman --debug --verbose --apply --keep-untracked-releases --force-upgrades -f .deployment/${DSF}
  _          _ 
 | |        | | 
 | |__   ___| |_ __ ___  ___ _ __ ___   __ _ _ __
 | '_ \ / _ \ | '_ ` _ \/ __| '_ ` _ \ / _` | '_ \ 
 | | | |  __/ | | | | | \__ \ | | | | | (_| | | | | 
 |_| |_|\___|_|_| |_| |_|___/_| |_| |_|\__,_|_| |_| version: v1.12.0
 A Helm-Charts-as-Code tool.
 2020/05/15 09:18:32 INFO: checking Helm version 
 2020/05/15 09:18:32 INFO: checking kubectl version 
 2020/05/15 09:18:32 VERBOSE: kubectl client version: v1.11.3
 2020/05/15 09:18:32 VERBOSE: Helm client version: v2.14.3+g0e7f3b6
 2020/05/15 09:18:32 INFO: validating that kubectl is installed.
 2020/05/15 09:18:32 INFO: validating that helm is installed.
 2020/05/15 09:18:32 INFO: validating that diff is installed.
 2020/05/15 09:18:32 INFO: substituting env variables in values and secrets files ...
 2020/05/15 09:18:32 INFO: Parsed YAML [[ .deployment/idlab-k8s-production.yaml ]] successfully and found [ 13 ] apps.
 Metadata: 
 --------- 
 description  :  Desired state file for automated deployment of Obelisk (production environment)
 Certificates: 
 --------- 
 Settings: 
 --------- 
 {KubeContext:idlab-k8s-production Username: Password: ClusterURI: ServiceAccount: StorageBackend: SlackWebhook: ReverseDelete:false BearerToken:false BearerTokenPath: Tillerless:false}
 Namespaces: 
 ------------- 
 deps  : protected =  {false false false    0      [] map[] map[]}
 auth  : protected =  {false false false    0      [] map[] map[]}
 ingress-nginx  : protected =  {false false false    0      [] map[] map[]}
 kube-system  : protected =  {false true false tiller   0      [] map[] map[]}
 obelisk  : protected =  {false false false    0      [] map[] map[]}
 saturnus  : protected =  {false false false    0      [] map[] map[]}
 kafka  : protected =  {false false false    0      [] map[] map[]}
 Repositories: 
 ------------- 
 incubator  :  http://storage.googleapis.com/kubernetes-charts-incubator
 idlab  :  https://charts.limeds.be/
 codecentric  :  https://codecentric.github.io/helm-charts
 ingress-nginx  :  https://kubernetes.github.io/ingress-nginx/
 stable  :  https://kubernetes-charts.storage.googleapis.com
 Applications: 
 --------------- 
    name :  
    description :  
    namespace :  obelisk
    enabled :  true
    chart :  /builds/ibcndevs/iot-stack/obelisk-backend/.deployment/charts/obelisk-commons
    version :  0.1.0
    valuesFile :  
    valuesFiles :  
    purge :  false
    test :  false
    protected :  false
    wait :  false
    priority :  0
    tiller namespace :  
    no-hooks :  false
    timeout :  0
    values to override from env:
        dockerImageConfig.registry  :  [REDACTED]
        dockerImageConfig.username  :  gitlab+deploy-token-23
        dockerImageConfig.password  :  [MASKED]
 ------------------- 
    name :  
    description :  
    namespace :  deps
    enabled :  true
    chart :  stable/redis
    version :  6.4.5
    valuesFile :  .helmsman-tmp/tmp859437677/redis.yaml
    valuesFiles :  
    purge :  false
    test :  false
    protected :  false
    wait :  false
    priority :  0
    tiller namespace :  
    no-hooks :  false
    timeout :  0
    values to override from env:
        metrics.serviceMonitor.namespace  :  monitoring
 ------------------- 
    name :  
    description :  
    namespace :  obelisk
    enabled :  true
    chart :  idlab/obelisk-public-api
    version :  1.2.36
    valuesFile :  .helmsman-tmp/tmp694455386/production-iot-stack.yaml
    valuesFiles :  
    purge :  false
    test :  false
    protected :  false
    wait :  false
    priority :  0
    tiller namespace :  
    no-hooks :  false
    timeout :  0
    values to override from env:
        dockerImageConfig.registry  :  [REDACTED]
        apiVersion  :  v1
 ------------------- 
    name :  
    description :  
    namespace :  kafka
    enabled :  true
    chart :  incubator/kafka
    version :  0.13.10
    valuesFile :  .helmsman-tmp/tmp213495924/production-kafka.yaml
    valuesFiles :  
    purge :  false
    test :  false
    protected :  false
    wait :  false
    priority :  0
    tiller namespace :  
    no-hooks :  false
    timeout :  0
    values to override from env:
 ------------------- 
    name :  
    description :  
    namespace :  auth
    enabled :  true
    chart :  codecentric/keycloak
    version :  5.1.7
    valuesFile :  .helmsman-tmp/tmp557915078/production-keycloak.yaml
    valuesFiles :  
    purge :  false
    test :  false
    protected :  false
    wait :  false
    priority :  0
    tiller namespace :  
    no-hooks :  false
    timeout :  0
    values to override from env:
        postgresql.postgresqlPassword  :  [REDACTED]
        keycloak.ingress.hosts[0]  :  [REDACTED]
        keycloak.password  :  [REDACTED]
 ------------------- 
    name :  
    description :  
    namespace :  kube-system
    enabled :  true
    chart :  stable/traefik
    version :  1.61.1
    valuesFile :  .helmsman-tmp/tmp525663464/production-traefik.yaml
    valuesFiles :  
    purge :  false
    test :  false
    protected :  false
    wait :  false
    priority :  0
    tiller namespace :  
    no-hooks :  false
    timeout :  0
    values to override from env:
 ------------------- 
    name :  
    description :  
    namespace :  obelisk
    enabled :  true
    chart :  /builds/ibcndevs/iot-stack/obelisk-backend/.deployment/charts/obelisk-public-api
    version :  0.1.0
    valuesFile :  .helmsman-tmp/tmp872377639/production-iot-stack.yaml
    valuesFiles :  
    purge :  false
    test :  false
    protected :  false
    wait :  false
    priority :  0
    tiller namespace :  
    no-hooks :  false
    timeout :  0
    values to override from env:
        dockerImageConfig.registry  :  [REDACTED]
        imageVersion  :  2.2.9
        commitSha  :  0aba987a5e0259c4fbb6942c81b9be7dab41e656
        statusMonitor.apiKey  :  [REDACTED]
 ------------------- 
    name :  
    description :  
    namespace :  obelisk
    enabled :  true
    chart :  /builds/ibcndevs/iot-stack/obelisk-backend/.deployment/charts/obelisk-core-pipeline
    version :  0.1.0
    valuesFile :  .helmsman-tmp/tmp457175025/production-iot-stack.yaml
    valuesFiles :  
    purge :  false
    test :  false
    protected :  false
    wait :  false
    priority :  0
    tiller namespace :  
    no-hooks :  false
    timeout :  0
    values to override from env:
        dockerImageConfig.registry  :  [REDACTED]
        imageVersion  :  2.2.9
        commitSha  :  0aba987a5e0259c4fbb6942c81b9be7dab41e656
 ------------------- 
    name :  
    description :  
    namespace :  deps
    enabled :  true
    chart :  /builds/ibcndevs/iot-stack/obelisk-backend/.deployment/charts/obelisk-commons
    version :  0.1.0
    valuesFile :  
    valuesFiles :  
    purge :  false
    test :  false
    protected :  false
    wait :  false
    priority :  0
    tiller namespace :  
    no-hooks :  false
    timeout :  0
    values to override from env:
        dockerImageConfig.password  :  [MASKED]
        dockerImageConfig.registry  :  [REDACTED]
        dockerImageConfig.username  :  gitlab+deploy-token-23
 ------------------- 
    name :  
    description :  
    namespace :  saturnus
    enabled :  true
    chart :  /builds/ibcndevs/iot-stack/obelisk-backend/.deployment/charts/obelisk-influxdb-cluster
    version :  0.1.0
    valuesFile :  
    valuesFiles :  
    purge :  false
    test :  false
    protected :  false
    wait :  false
    priority :  0
    tiller namespace :  
    no-hooks :  false
    timeout :  0
    values to override from env:
        nodes.amount  :  6
        config.coordinator.query_timeout  :  5s
        persistence.size  :  250Gi
 ------------------- 
    name :  
    description :  
    namespace :  deps
    enabled :  true
    chart :  /builds/ibcndevs/iot-stack/obelisk-backend/.deployment/charts/obelisk-dep-dashboards
    version :  0.1.0
    valuesFile :  .helmsman-tmp/tmp652042571/production-iot-stack.yaml
    valuesFiles :  
    purge :  false
    test :  false
    protected :  false
    wait :  false
    priority :  0
    tiller namespace :  
    no-hooks :  false
    timeout :  0
    values to override from env:
        mongodbDashboard.targetHost  :  mongodb.deps
        mongodbDashboard.user  :  [REDACTED]
        mongodbDashboard.password  :  [REDACTED]
 ------------------- 
    name :  
    description :  
    namespace :  deps
    enabled :  true
    chart :  stable/mongodb
    version :  5.3.2
    valuesFile :  .helmsman-tmp/tmp262262595/production-mongodb.yaml
    valuesFiles :  
    purge :  false
    test :  false
    protected :  false
    wait :  false
    priority :  0
    tiller namespace :  
    no-hooks :  false
    timeout :  0
    values to override from env:
 ------------------- 
    name :  
    description :  
    namespace :  ingress-nginx
    enabled :  true
    chart :  ingress-nginx/ingress-nginx
    version :  2.1.0
    valuesFile :  .helmsman-tmp/tmp261854364/ingress-nginx.yaml
    valuesFiles :  
    purge :  false
    test :  false
    protected :  false
    wait :  false
    priority :  0
    tiller namespace :  
    no-hooks :  false
    timeout :  0
    values to override from env:
 ------------------- 
 2020/05/15 09:18:32 INFO: validating that tiller is installed.
 2020/05/15 09:18:32 INFO: namespace validation -- Tiller is NOT desired to be deployed in namespace [ obelisk ].
 2020/05/15 09:18:32 INFO: namespace validation -- Tiller is NOT desired to be deployed in namespace [ saturnus ].
 2020/05/15 09:18:32 INFO: namespace validation -- Tiller is NOT desired to be deployed in namespace [ kafka ].
 2020/05/15 09:18:32 INFO: namespace validation -- Tiller is NOT desired to be deployed in namespace [ deps ].
 2020/05/15 09:18:32 INFO: namespace validation -- Tiller is NOT desired to be deployed in namespace [ auth ].
 2020/05/15 09:18:32 INFO: namespace validation -- Tiller is NOT desired to be deployed in namespace [ ingress-nginx ].
 2020/05/15 09:18:32 INFO: namespace validation -- Either no or invalid certs/keys provided for DEPLOYING Tiller with TLS in namespace [ kube-system ].
 2020/05/15 09:18:32 INFO: setting kubectl context to [ idlab-k8s-production ]
 2020/05/15 09:18:32 VERBOSE: kubectl config use-context idlab-k8s-production
 2020/05/15 09:18:32 INFO: initializing helm on the client only.
 2020/05/15 09:18:32 VERBOSE: helm init --client-only 
 2020/05/15 09:18:32 INFO: adding repo stable
 2020/05/15 09:18:32 VERBOSE: helm repo add  stable "https://kubernetes-charts.storage.googleapis.com"
 2020/05/15 09:18:35 INFO: adding repo incubator
 2020/05/15 09:18:35 VERBOSE: helm repo add  incubator "http://storage.googleapis.com/kubernetes-charts-incubator"
 2020/05/15 09:18:36 INFO: adding repo idlab
 2020/05/15 09:18:36 VERBOSE: helm repo add  idlab "https://charts.limeds.be/"
 2020/05/15 09:18:36 INFO: adding repo codecentric
 2020/05/15 09:18:36 VERBOSE: helm repo add  codecentric "https://codecentric.github.io/helm-charts"
 2020/05/15 09:18:36 INFO: adding repo ingress-nginx
 2020/05/15 09:18:36 VERBOSE: helm repo add  ingress-nginx "https://kubernetes.github.io/ingress-nginx/"
 2020/05/15 09:18:36 INFO: updating helm repos
 2020/05/15 09:18:36 VERBOSE: helm repo update 
 2020/05/15 09:18:39 INFO: creating namespace  ingress-nginx
 2020/05/15 09:18:39 VERBOSE: kubectl create namespace ingress-nginx
 2020/05/15 09:18:39 WARN: I could not create namespace [ ingress-nginx ]. It already exists. I am skipping this.
 2020/05/15 09:18:39 INFO: creating namespace  kube-system
 2020/05/15 09:18:39 VERBOSE: kubectl create namespace kube-system
 2020/05/15 09:18:40 WARN: I could not create namespace [ kube-system ]. It already exists. I am skipping this.
 2020/05/15 09:18:40 INFO: creating namespace  obelisk
 2020/05/15 09:18:40 VERBOSE: kubectl create namespace obelisk
 2020/05/15 09:18:40 WARN: I could not create namespace [ obelisk ]. It already exists. I am skipping this.
 2020/05/15 09:18:40 INFO: creating namespace  saturnus
 2020/05/15 09:18:40 VERBOSE: kubectl create namespace saturnus
 2020/05/15 09:18:40 WARN: I could not create namespace [ saturnus ]. It already exists. I am skipping this.
 2020/05/15 09:18:40 INFO: creating namespace  kafka
 2020/05/15 09:18:40 VERBOSE: kubectl create namespace kafka
 2020/05/15 09:18:40 WARN: I could not create namespace [ kafka ]. It already exists. I am skipping this.
 2020/05/15 09:18:40 INFO: creating namespace  deps
 2020/05/15 09:18:40 VERBOSE: kubectl create namespace deps
 2020/05/15 09:18:40 WARN: I could not create namespace [ deps ]. It already exists. I am skipping this.
 2020/05/15 09:18:40 INFO: creating namespace  auth
 2020/05/15 09:18:40 VERBOSE: kubectl create namespace auth
 2020/05/15 09:18:40 WARN: I could not create namespace [ auth ]. It already exists. I am skipping this.
 2020/05/15 09:18:40 INFO: deploying Tiller in namespace [ kube-system ].
 2020/05/15 09:18:40 INFO: validating if serviceaccount [ tiller ] exists in namespace [ kube-system ].
 2020/05/15 09:18:40 VERBOSE: kubectl get serviceaccount tiller -n kube-system
 2020/05/15 09:18:40 ERROR: while validating/creating service account [ tiller ] in namespace [kube-system]: error: the server doesn't have a resource type "serviceaccount"
Running after_script
Uploading artifacts for failed job
00:02
 ERROR: Job failed: exit code 1

[sami-alajrami]

What happen when you run the command kubectl get serviceaccount tiller -n kube-system from the same process/machine? All helmsman is doing here is passing that kubectl command and expect a zero exit code, which does not happen.

The suggestions here might be useful to find the root cause.

[Falx]

We've just traced it down to a single user-account which wasn't updated in our CI .kube/config file. This can be closed. Thanks for trying to help anyways!