Missing GCE default credentials on Control Plane for Hetzner cluster

willihr commented 10 months ago

/kind bug

1. What kops version are you running? The command kops version, will display this information. 1.28.0

2. What Kubernetes version are you running? kubectl version will print the version if a cluster is running or provide the Kubernetes version specified as a kops flag. 1.27.3

3. What cloud provider are you using? Hetzner

4. What commands did you run? What is the simplest way to reproduce this issue?

gcloud auth login
gcloud auth application-default login

export HCLOUD_TOKEN=xxxxxxxxx

kops create cluster --name=main.k8s.local --ssh-public-key=~/.ssh/id_rsa.pub --cloud=hetzner --zones=fsn1 --networking=calico --network-cidr=10.10.0.0/16 --node-size cax11 --control-plane-size cax11 --state gs://willihr-k8s-clusters
kops update cluster main.k8s.local --state gs://willihr-k8s-clusters --yes --admin

5. What happened after the commands executed? Control-plane node Nodeup went into an error loop due to missing Google Cloud credentials.

Jan 06 22:30:56 control-plane-fsn1-1d120563d7ede239 systemd[1]: Starting Run kOps bootstrap (nodeup)...
Jan 06 22:30:56 control-plane-fsn1-1d120563d7ede239 nodeup[945]: nodeup version 1.28.0 (git-v1.28.0)
Jan 06 22:30:56 control-plane-fsn1-1d120563d7ede239 nodeup[945]: I0106 22:30:56.045999     945 gsfs.go:278] Reading file "gs://willihr-k8s-clusters/main.k8s.local/igconfig/control-plane/control-plane-fsn1/nodeupconfig.yaml"
Jan 06 22:30:56 control-plane-fsn1-1d120563d7ede239 nodeup[945]: I0106 22:30:56.048015     945 context.go:310] retrying after error error building GCS client: google: could not find default credentials. See https://cloud.google.com/docs/authentication/external/set-up-adc for more information
Jan 06 22:30:57 control-plane-fsn1-1d120563d7ede239 nodeup[945]: I0106 22:30:57.068414     945 gsfs.go:278] Reading file "gs://willihr-k8s-clusters/main.k8s.local/igconfig/control-plane/control-plane-fsn1/nodeupconfig.yaml"
Jan 06 22:30:57 control-plane-fsn1-1d120563d7ede239 nodeup[945]: I0106 22:30:57.068545     945 context.go:310] retrying after error error building GCS client: google: could not find default credentials. See https://cloud.google.com/docs/authentication/external/set-up-adc for more information
Jan 06 22:30:58 control-plane-fsn1-1d120563d7ede239 nodeup[945]: I0106 22:30:58.656729     945 gsfs.go:278] Reading file "gs://willihr-k8s-clusters/main.k8s.local/igconfig/control-plane/control-plane-fsn1/nodeupconfig.yaml"
Jan 06 22:30:58 control-plane-fsn1-1d120563d7ede239 nodeup[945]: I0106 22:30:58.656848     945 context.go:310] retrying after error error building GCS client: google: could not find default credentials. See https://cloud.google.com/docs/authentication/external/set-up-adc for more information
Jan 06 22:31:01 control-plane-fsn1-1d120563d7ede239 nodeup[945]: I0106 22:31:01.008961     945 gsfs.go:278] Reading file "gs://willihr-k8s-clusters/main.k8s.local/igconfig/control-plane/control-plane-fsn1/nodeupconfig.yaml"
Jan 06 22:31:01 control-plane-fsn1-1d120563d7ede239 nodeup[945]: I0106 22:31:01.009073     945 context.go:314] hit maximum retries 4 with error error building GCS client: google: could not find default credentials. See https://cloud.google.com/docs/authentication/external/set-up-adc for more information
Jan 06 22:31:01 control-plane-fsn1-1d120563d7ede239 nodeup[945]: W0106 22:31:01.009142     945 main.go:133] got error running nodeup (will retry in 30s): error loading NodeupConfig "gs://willihr-k8s-clusters/main.k8s.local/igconfig/control-plane/control-plane-fsn1/nodeupconfig.yaml": error building GCS client: google: could not find default credentials. See https://cloud.google.com/docs/authentication/external/set-up-adc for more information
Jan 06 22:31:31 control-plane-fsn1-1d120563d7ede239 nodeup[945]: I0106 22:31:31.012683     945 gsfs.go:278] Reading file "gs://willihr-k8s-clusters/main.k8s.local/igconfig/control-plane/control-plane-fsn1/nodeupconfig.yaml"
Jan 06 22:31:31 control-plane-fsn1-1d120563d7ede239 nodeup[945]: I0106 22:31:31.012782     945 context.go:310] retrying after error error building GCS client: google: could not find default credentials. See https://cloud.google.com/docs/authentication/external/set-up-adc for more information
Jan 06 22:31:32 control-plane-fsn1-1d120563d7ede239 nodeup[945]: I0106 22:31:32.111564     945 gsfs.go:278] Reading file "gs://willihr-k8s-clusters/main.k8s.local/igconfig/control-plane/control-plane-fsn1/nodeupconfig.yaml"
Jan 06 22:31:32 control-plane-fsn1-1d120563d7ede239 nodeup[945]: I0106 22:31:32.111673     945 context.go:310] retrying after error error building GCS client: google: could not find default credentials. See https://cloud.google.com/docs/authentication/external/set-up-adc for more information
Jan 06 22:31:33 control-plane-fsn1-1d120563d7ede239 nodeup[945]: I0106 22:31:33.690096     945 gsfs.go:278] Reading file "gs://willihr-k8s-clusters/main.k8s.local/igconfig/control-plane/control-plane-fsn1/nodeupconfig.yaml"
Jan 06 22:31:33 control-plane-fsn1-1d120563d7ede239 nodeup[945]: I0106 22:31:33.690210     945 context.go:310] retrying after error error building GCS client: google: could not find default credentials. See https://cloud.google.com/docs/authentication/external/set-up-adc for more information
Jan 06 22:31:36 control-plane-fsn1-1d120563d7ede239 nodeup[945]: I0106 22:31:36.036566     945 gsfs.go:278] Reading file "gs://willihr-k8s-clusters/main.k8s.local/igconfig/control-plane/control-plane-fsn1/nodeupconfig.yaml"
Jan 06 22:31:36 control-plane-fsn1-1d120563d7ede239 nodeup[945]: I0106 22:31:36.036697     945 context.go:314] hit maximum retries 4 with error error building GCS client: google: could not find default credentials. See https://cloud.google.com/docs/authentication/external/set-up-adc for more information
Jan 06 22:31:36 control-plane-fsn1-1d120563d7ede239 nodeup[945]: W0106 22:31:36.036769     945 main.go:133] got error running nodeup (will retry in 30s): error loading NodeupConfig "gs://willihr-k8s-clusters/main.k8s.local/igconfig/control-plane/control-plane-fsn1/nodeupconfig.yaml": error building GCS client: google: could not find default credentials. See https://cloud.google.com/docs/authentication/external/set-up-adc for more information
Jan 06 22:32:06 control-plane-fsn1-1d120563d7ede239 nodeup[945]: I0106 22:32:06.041616     945 gsfs.go:278] Reading file "gs://willihr-k8s-clusters/main.k8s.local/igconfig/control-plane/control-plane-fsn1/nodeupconfig.yaml"
Jan 06 22:32:06 control-plane-fsn1-1d120563d7ede239 nodeup[945]: I0106 22:32:06.041701     945 context.go:310] retrying after error error building GCS client: google: could not find default credentials. See https://cloud.google.com/docs/authentication/external/set-up-adc for more information
Jan 06 22:32:07 control-plane-fsn1-1d120563d7ede239 nodeup[945]: I0106 22:32:07.135405     945 gsfs.go:278] Reading file "gs://willihr-k8s-clusters/main.k8s.local/igconfig/control-plane/control-plane-fsn1/nodeupconfig.yaml"
Jan 06 22:32:07 control-plane-fsn1-1d120563d7ede239 nodeup[945]: I0106 22:32:07.135513     945 context.go:310] retrying after error error building GCS client: google: could not find default credentials. See https://cloud.google.com/docs/authentication/external/set-up-adc for more information
Jan 06 22:32:08 control-plane-fsn1-1d120563d7ede239 nodeup[945]: I0106 22:32:08.744268     945 gsfs.go:278] Reading file "gs://willihr-k8s-clusters/main.k8s.local/igconfig/control-plane/control-plane-fsn1/nodeupconfig.yaml"
Jan 06 22:32:08 control-plane-fsn1-1d120563d7ede239 nodeup[945]: I0106 22:32:08.744372     945 context.go:310] retrying after error error building GCS client: google: could not find default credentials. See https://cloud.google.com/docs/authentication/external/set-up-adc for more information
Jan 06 22:32:11 control-plane-fsn1-1d120563d7ede239 nodeup[945]: I0106 22:32:11.176460     945 gsfs.go:278] Reading file "gs://willihr-k8s-clusters/main.k8s.local/igconfig/control-plane/control-plane-fsn1/nodeupconfig.yaml"
Jan 06 22:32:11 control-plane-fsn1-1d120563d7ede239 nodeup[945]: I0106 22:32:11.176573     945 context.go:314] hit maximum retries 4 with error error building GCS client: google: could not find default credentials. See https://cloud.google.com/docs/authentication/external/set-up-adc for more information
Jan 06 22:32:11 control-plane-fsn1-1d120563d7ede239 nodeup[945]: W0106 22:32:11.176601     945 main.go:133] got error running nodeup (will retry in 30s): error loading NodeupConfig "gs://willihr-k8s-clusters/main.k8s.local/igconfig/control-plane/control-plane-fsn1/nodeupconfig.yaml": error building GCS client: google: could not find default credentials. See https://cloud.google.com/docs/authentication/external/set-up-adc for more information

6. What did you expect to happen? Control plane nodeup run correctly and show a success message

7. Please provide your cluster manifest. Execute kops get --name my.example.com -o yaml to display your cluster manifest. You may want to remove your cluster name and other sensitive information.

apiVersion: kops.k8s.io/v1alpha2
kind: Cluster
metadata:
  creationTimestamp: "2024-01-06T22:26:53Z"
  name: main.k8s.local
spec:
  api:
    loadBalancer:
      type: Public
  authorization:
    rbac: {}
  channel: stable
  cloudProvider: hetzner
  configBase: gs://willihr-k8s-clusters/main.k8s.local
  etcdClusters:
  - cpuRequest: 200m
    etcdMembers:
    - instanceGroup: control-plane-fsn1
      name: etcd-1
    manager:
      backupRetentionDays: 90
    memoryRequest: 100Mi
    name: main
  - cpuRequest: 100m
    etcdMembers:
    - instanceGroup: control-plane-fsn1
      name: etcd-1
    manager:
      backupRetentionDays: 90
    memoryRequest: 100Mi
    name: events
  iam:
    allowContainerRegistry: true
    legacy: false
  kubelet:
    anonymousAuth: false
  kubernetesApiAccess:
  - 0.0.0.0/0
  - ::/0
  kubernetesVersion: 1.28.5
  networkCIDR: 10.10.0.0/16
  networking:
    calico: {}
  nonMasqueradeCIDR: 100.64.0.0/10
  sshAccess:
  - 0.0.0.0/0
  - ::/0
  subnets:
  - name: fsn1
    type: Public
    zone: fsn1
  topology:
    dns:
      type: None

---

apiVersion: kops.k8s.io/v1alpha2
kind: InstanceGroup
metadata:
  creationTimestamp: "2024-01-06T22:26:55Z"
  labels:
    kops.k8s.io/cluster: main.k8s.local
  name: control-plane-fsn1
spec:
  image: ubuntu-22.04
  machineType: cax11
  maxSize: 1
  minSize: 1
  role: Master
  subnets:
  - fsn1

---

apiVersion: kops.k8s.io/v1alpha2
kind: InstanceGroup
metadata:
  creationTimestamp: "2024-01-06T22:26:57Z"
  labels:
    kops.k8s.io/cluster: main.k8s.local
  name: nodes-fsn1
spec:
  image: ubuntu-22.04
  machineType: cax11
  maxSize: 1
  minSize: 1
  role: Node
  subnets:
  - fsn1

**8. Please run the commands with most verbose logging by adding the -v 10 flag. I believe the problem is not with the executed commands, so I'll save space here. But if necessary, I'll for sure post.

9. Anything else do we need to know? I'm not sure, but the function below seems to collect credentials from multiple Clouds to insert them into the controlPlane/node cloud servers configuration, but Google Cloud credentials aren't being collected. https://github.com/kubernetes/kops/blob/0162f39aaec7e6b4e0d00b29047d1d140807c4bc/nodeup/pkg/bootstrap/install.go#L80C23-L80C23

After running cat /etc/sysconfig/kops-configuration in the control plane server, got only:

HCLOUD_TOKEN=xxxxxxxx

hakman commented 10 months ago

@willihr kOps requires S3 compatible storage for Hetzner, as mentioned in the docs: https://kops.sigs.k8s.io/getting_started/hetzner/#environment-variables

You will need to create HMAC keys for authenticating to Google Cloud Storage as described in the docs: https://cloud.google.com/storage/docs/authentication/hmackeys

k8s-triage-robot commented 7 months ago

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

After 90d of inactivity, lifecycle/stale is applied
After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

Mark this issue as fresh with /remove-lifecycle stale
Close this issue with /close
Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

k8s-triage-robot commented 6 months ago

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

After 90d of inactivity, lifecycle/stale is applied
After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

Mark this issue as fresh with /remove-lifecycle rotten
Close this issue with /close
Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

k8s-triage-robot commented 5 months ago

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues according to the following rules:

After 90d of inactivity, lifecycle/stale is applied
After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

Reopen this issue with /reopen
Mark this issue as fresh with /remove-lifecycle rotten
Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close not-planned

k8s-ci-robot commented 5 months ago

@k8s-triage-robot: Closing this issue, marking it as "Not Planned".

In response to [this](https://github.com/kubernetes/kops/issues/16230#issuecomment-2149394152): >The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. > >This bot triages issues according to the following rules: >- After 90d of inactivity, `lifecycle/stale` is applied >- After 30d of inactivity since `lifecycle/stale` was applied, `lifecycle/rotten` is applied >- After 30d of inactivity since `lifecycle/rotten` was applied, the issue is closed > >You can: >- Reopen this issue with `/reopen` >- Mark this issue as fresh with `/remove-lifecycle rotten` >- Offer to help out with [Issue Triage][1] > >Please send feedback to sig-contributor-experience at [kubernetes/community](https://github.com/kubernetes/community). > >/close not-planned > >[1]: https://www.kubernetes.dev/docs/guide/issue-triage/ Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes-sigs/prow](https://github.com/kubernetes-sigs/prow/issues/new?title=Prow%20issue:) repository.

kubernetes / kops

Missing GCE default credentials on Control Plane for Hetzner cluster #16230