search

kubermatic / kubeone

Kubermatic KubeOne automate cluster operations on all your cloud, on-prem, edge, and IoT environments.
https://kubeone.io
Apache License 2.0
1.36k stars 231 forks source link

kubeone asks for hcloud_token although hcloud_token is there #1206

Closed shibumi closed 3 years ago

shibumi commented 3 years ago

What happened:

I am evaluating kubeone for my company. Actually I tried to bootstrap kubeone on bare-metal (without success + I am still waiting for the necessary infra). So i thought I could use my time and play with kubeone in hetzner cloud, but I ran into several issues with it.

I am following this guide here: https://www.kubermatic.com/blog/kubeone-oidc-authentication-audit-logging/

What I did so far is:

$ git clone https://github.com/kubermatic/kubeone
$ cd kubeone/examples/terraform/hetzner

Then I modified the variables.tf file with the following changes, because I found it annoying to always type in the hcloud token. (The environment variable HCLOUD_TOKEN was not working):

provider "hcloud" {
  token = "REDACTED"
}

variable "cluster_name" {
  description = "prefix for cloud resources"
  default = "avency"
}

Then I did the following:

$ terraform init
$ terraform plan
$ terraform apply
$ terraform output -json > output.json
$ export HCLOUD_TOKEN=(REDACTED)
$ kubeone apply --manifest kubeone.yaml -t output.json

The last command gave me this output here:

Error: failed to validate credentials: unable to validate credentials: key HCLOUD_TOKEN is required but isn't present
failed to validate credentials: unable to validate credentials: key HCLOUD_TOKEN is required but isn't present

What is the expected behavior:

Kubeone should work as expected and start installing the kubeone cluster.

How to reproduce the issue:

Follow my steps above

Anything else we need to know?

Would be cool to have better documentation for Hetzner cloud And/Or bare metal. The kubeone documentation is good, but these two seem to be very undocumented although they are "officially supported".

Information about the environment: KubeOne version (kubeone version): 

{
  "kubeone": {
    "major": "1",
    "minor": "0",
    "gitVersion": "v1.0.5",
    "gitCommit": "26aa328936715c8607859ae4405bda0776d6c405",
    "gitTreeState": "",
    "buildDate": "2020-11-14T22:42:00+00:00",
    "goVersion": "go1.15.5",
    "compiler": "gc",
    "platform": "linux/amd64"
  },
  "machine_controller": {
    "major": "1",
    "minor": "19",
    "gitVersion": "v1.19.0",
    "gitCommit": "",
    "gitTreeState": "",
    "buildDate": "",
    "goVersion": "",
    "compiler": "",
    "platform": "linux/amd64"
  }
}

Operating system: Arch Linux Provider you're deploying cluster on: Hetzner Cloud Operating system you're deploying on: Ubuntu

kron4eg commented 3 years ago

Hey,

Please make sure to setup the HCLOUD_TOKEN as suggested in https://registry.terraform.io/providers/hetznercloud/hcloud/latest/docs#argument-reference and since kubeone uses exactly the same var. For more information please refer to https://docs.kubermatic.com/kubeone/v1.0/prerequisites/credentials/#environment-variables

shibumi commented 3 years ago

@kron4eg shouldn't kubeone respect my HCLOUD_TOKEN env variable? According to your linked documentation and the tutorial by kubermatic this should be fine.

kron4eg commented 3 years ago

@shibumi it sure will respect your HCLOUD_TOKEN, if it finds one. Apparently nether terraform nor kubeone have found it in your ENV. So that's why all the problems.

shibumi commented 3 years ago

@kron4eg And you are sure this is not a bug with my kubeone version or terraform version? It's really weird. I tried 2 ways setting the env variable:

$ export HCLOUD_TOKEN=(REDACTED)
or
$ HCLOUD_TOKEN=(REDACTED) kubeone apply --manifest kubeone.yaml -t output.json
kron4eg commented 3 years ago

Absolutely. We create hetzner clusters sometimes multiple times per day.

shibumi commented 3 years ago

@kron4eg Ok. I investigated this issue further and I found the following solution for this problem.

This problem is shell related. With ZSH the right way to go is export HCLOUD_TOKEN="<token>" with double quotes instead of paranthesis. I used paranthesis, because I followed the link on the kubermatic website.

No idea if it would be nice to document this somewhere. I would be willing to donate a PR to the kubeone documentation for hetzner if I have finished bootstrapping my cluster. Are you interested? Or would you say "nah.. this is not our beer. It's shell related and the people should know about this when using zsh"?

The kubeone documentation does not mention the whole hcloud_token story at all. Might make sense to include it in the provisioning section (https://docs.kubermatic.com/kubeone/v1.0/provisioning/) what do you think?

shibumi commented 3 years ago

@kron4eg but I have another question:

When I do directly after bootstrapping the cluster and importing the kubeconfig file.. kubeone config machinedeployments --manifest kubeone.yaml -t tf.json

I get this error here:

❯ kubeone config machinedeployments --manifest kubeone.yaml -t tf.json
Error: failed to initialize State: failed to load cluster: unable to load a given KubeOneCluster object: unable to read the given terraform output file: open tf.json: no such file or directory
failed to initialize State: failed to load cluster: unable to load a given KubeOneCluster object: unable to read the given terraform output file: open tf.json: no such file or directory

I would really recommend that you provide proper documentation if you bootstrap clusters on hetzner daily. Kubeone is really awesome and I enjoy playing around with it, but the missing documentation is really a big problem during the evaluation for my company.

I am just following this guide here: https://docs.kubermatic.com/kubeone/v1.0/workers/machine_controller/

EDIT: Should kubeone update the kubeone.yaml file after provisioning the cluster? It did not..

xrstf commented 3 years ago

This problem is shell related. With ZSH the right way to go is export HCLOUD_TOKEN="" with double quotes instead of paranthesis. I used paranthesis, because I followed the link on the kubermatic website.

We can clarify this in the blog article, thanks for mentioning :-)

kron4eg commented 3 years ago

Please read the error message carefully. It says what's the problem: open tf.json: no such file or directory. From previous messages I see you've directed terraform json output to the output.json.

So the command should be kubeone config machinedeployments --manifest kubeone.yaml -t output.json.

kron4eg commented 3 years ago

The kubeone documentation does not mention the whole hcloud_token story at all. Might make sense to include it in the provisioning section (https://docs.kubermatic.com/kubeone/v1.0/provisioning/) what do you think?

The documentation section Environment Variables features tabs with every officially supported provider and its respective ENVs including hetzner and its HCLOUD_TOKEN.

shibumi commented 3 years ago

@kron4eg ah ok. Looks like I missed that part.

Thanks for your help folks. Any chance we can get bare-metal documented as well? I asked about this in the slack channel and there they told me that the bare metal means to disable the machine controller and to compose the kubeone.yaml config manually.

Also I would like to have some other documentation about getting a proper loadbalancer in front of my kubeAPI. You mention that gobetween should not be used in production but you never actually give a hint how to replace it.

Sorry for discussing multiple topics in one bug report. Maybe I should just move that discussion into the slack channel.

kron4eg commented 3 years ago