search

techno-tim / k3s-ansible

The easiest way to bootstrap a self-hosted High Availability Kubernetes cluster. A fully automated HA k3s etcd install with kube-vip, MetalLB, and more. Build. Destroy. Repeat.
https://technotim.live/posts/k3s-etcd-ansible/
Apache License 2.0
2.41k stars 1.05k forks source link

Test playbook using molecule #67

Closed sleiner closed 2 years ago

sleiner commented 2 years ago

Proposed Changes

Checklist

sleiner commented 2 years ago

@timothystewart6 should the PR checklist be changed?

twistedgrim commented 2 years ago

Excellent.... nice job!

timothystewart6 commented 2 years ago

I think so now that we have CI running. I think we can do it in a separate PR!

timothystewart6 commented 2 years ago

Why don't we see the output of molecule when testing in the Test with molecule step? Am I missing something?

timothystewart6 commented 2 years ago

Note to self, I will have to update this post. https://github.com/techno-tim/techno-tim.github.io/blob/master/_posts/2022-03-26-k3s-etcd-ansible.md

sleiner commented 2 years ago

@timothystewart6

I think so now that we have CI running. I think we can do it in a separate PR!

I'm missing a bit of context here 😅 What could be done in a separate PR?

Why don't we see the output of molecule when testing in the Test with molecule step? Am I missing something?

To debug some file permission issue on the runner itself more quickly, I had replaced the molecule invocation. The issue is now fixed and molecule is back in place 😊

timothystewart6 commented 2 years ago

I'm missing a bit of context here 😅 What could be done in a separate PR?

Oh, sorry, I thought I quoted your comment! I mean that the PR template can be done in another PR!

sleiner commented 2 years ago

@timothystewart6 have you seen these types of errors before?

TASK [k3s/post : Wait for metallb controller to be running] ********************
[496](https://github.com/techno-tim/k3s-ansible/runs/8158547698?check_suite_focus=true#step:7:504)
  failed: [control1] (item=control1) => {"ansible_loop_var": "item", "changed": false, "cmd": ["kubectl", "wait", "deployment", "-n", "metallb-system", "controller", "--for", "condition=Available=True", "--timeout=60s"], "delta": "0:00:01.729871", "end": "2022-09-02 15:06:12.479623", "item": "control1", "msg": "non-zero return code", "rc": 1, "start": "2022-09-02 15:06:10.749752", "stderr": "Error from server (ServiceUnavailable): apiserver not ready", "stderr_lines": ["Error from server (ServiceUnavailable): apiserver not ready"], "stdout": "", "stdout_lines": []}
[497](https://github.com/techno-tim/k3s-ansible/runs/8158547698?check_suite_focus=true#step:7:505)
  failed: [control1] (item=control2) => {"ansible_loop_var": "item", "changed": false, "cmd": ["kubectl", "wait", "deployment", "-n", "metallb-system", "controller", "--for", "condition=Available=True", "--timeout=60s"], "delta": "0:00:01.886031", "end": "2022-09-02 15:06:17.390877", "item": "control2", "msg": "non-zero return code", "rc": 1, "start": "2022-09-02 15:06:15.504846", "stderr": "Error from server (ServiceUnavailable): apiserver not ready", "stderr_lines": ["Error from server (ServiceUnavailable): apiserver not ready"], "stdout": "", "stdout_lines": []}
[498](https://github.com/techno-tim/k3s-ansible/runs/8158547698?check_suite_focus=true#step:7:506)
  failed: [control1] (item=control3) => {"ansible_loop_var": "item", "changed": false, "cmd": ["kubectl", "wait", "deployment", "-n", "metallb-system", "controller", "--for", "condition=Available=True", "--timeout=60s"], "delta": "0:00:02.417555", "end": "2022-09-02 15:06:23.943977", "item": "control3", "msg": "non-zero return code", "rc": 1, "start": "2022-09-02 15:06:21.526422", "stderr": "error: the server doesn't have a resource type \"deployment\"", "stderr_lines": ["error: the server doesn't have a resource type \"deployment\""], "stdout": "", "stdout_lines": []}

I could build a retry in, but the API server not being available (and afterwards not knowing deployments) seems super weird - especially since at this point we have already used the API server quite a bit 🤔

timothystewart6 commented 2 years ago

@timothystewart6 have you seen these types of errors before?

TASK [k3s/post : Wait for metallb controller to be running] ********************
[496](https://github.com/techno-tim/k3s-ansible/runs/8158547698?check_suite_focus=true#step:7:504)
  failed: [control1] (item=control1) => {"ansible_loop_var": "item", "changed": false, "cmd": ["kubectl", "wait", "deployment", "-n", "metallb-system", "controller", "--for", "condition=Available=True", "--timeout=60s"], "delta": "0:00:01.729871", "end": "2022-09-02 15:06:12.479623", "item": "control1", "msg": "non-zero return code", "rc": 1, "start": "2022-09-02 15:06:10.749752", "stderr": "Error from server (ServiceUnavailable): apiserver not ready", "stderr_lines": ["Error from server (ServiceUnavailable): apiserver not ready"], "stdout": "", "stdout_lines": []}
[497](https://github.com/techno-tim/k3s-ansible/runs/8158547698?check_suite_focus=true#step:7:505)
  failed: [control1] (item=control2) => {"ansible_loop_var": "item", "changed": false, "cmd": ["kubectl", "wait", "deployment", "-n", "metallb-system", "controller", "--for", "condition=Available=True", "--timeout=60s"], "delta": "0:00:01.886031", "end": "2022-09-02 15:06:17.390877", "item": "control2", "msg": "non-zero return code", "rc": 1, "start": "2022-09-02 15:06:15.504846", "stderr": "Error from server (ServiceUnavailable): apiserver not ready", "stderr_lines": ["Error from server (ServiceUnavailable): apiserver not ready"], "stdout": "", "stdout_lines": []}
[498](https://github.com/techno-tim/k3s-ansible/runs/8158547698?check_suite_focus=true#step:7:506)
  failed: [control1] (item=control3) => {"ansible_loop_var": "item", "changed": false, "cmd": ["kubectl", "wait", "deployment", "-n", "metallb-system", "controller", "--for", "condition=Available=True", "--timeout=60s"], "delta": "0:00:02.417555", "end": "2022-09-02 15:06:23.943977", "item": "control3", "msg": "non-zero return code", "rc": 1, "start": "2022-09-02 15:06:21.526422", "stderr": "error: the server doesn't have a resource type \"deployment\"", "stderr_lines": ["error: the server doesn't have a resource type \"deployment\""], "stdout": "", "stdout_lines": []}

I could build a retry in, but the API server not being available (and afterwards not knowing deployments) seems super weird - especially since at this point we have already used the API server quite a bit 🤔

I just pushed up a fix, not sure if this will actually fix your issue because it still works locally and in my test cluster without this. Might be worth merging in.

timothystewart6 commented 2 years ago

(I updated the branch)

timothystewart6 commented 2 years ago

@timothystewart6 have you seen these types of errors before?

TASK [k3s/post : Wait for metallb controller to be running] ********************
[496](https://github.com/techno-tim/k3s-ansible/runs/8158547698?check_suite_focus=true#step:7:504)
  failed: [control1] (item=control1) => {"ansible_loop_var": "item", "changed": false, "cmd": ["kubectl", "wait", "deployment", "-n", "metallb-system", "controller", "--for", "condition=Available=True", "--timeout=60s"], "delta": "0:00:01.729871", "end": "2022-09-02 15:06:12.479623", "item": "control1", "msg": "non-zero return code", "rc": 1, "start": "2022-09-02 15:06:10.749752", "stderr": "Error from server (ServiceUnavailable): apiserver not ready", "stderr_lines": ["Error from server (ServiceUnavailable): apiserver not ready"], "stdout": "", "stdout_lines": []}
[497](https://github.com/techno-tim/k3s-ansible/runs/8158547698?check_suite_focus=true#step:7:505)
  failed: [control1] (item=control2) => {"ansible_loop_var": "item", "changed": false, "cmd": ["kubectl", "wait", "deployment", "-n", "metallb-system", "controller", "--for", "condition=Available=True", "--timeout=60s"], "delta": "0:00:01.886031", "end": "2022-09-02 15:06:17.390877", "item": "control2", "msg": "non-zero return code", "rc": 1, "start": "2022-09-02 15:06:15.504846", "stderr": "Error from server (ServiceUnavailable): apiserver not ready", "stderr_lines": ["Error from server (ServiceUnavailable): apiserver not ready"], "stdout": "", "stdout_lines": []}
[498](https://github.com/techno-tim/k3s-ansible/runs/8158547698?check_suite_focus=true#step:7:506)
  failed: [control1] (item=control3) => {"ansible_loop_var": "item", "changed": false, "cmd": ["kubectl", "wait", "deployment", "-n", "metallb-system", "controller", "--for", "condition=Available=True", "--timeout=60s"], "delta": "0:00:02.417555", "end": "2022-09-02 15:06:23.943977", "item": "control3", "msg": "non-zero return code", "rc": 1, "start": "2022-09-02 15:06:21.526422", "stderr": "error: the server doesn't have a resource type \"deployment\"", "stderr_lines": ["error: the server doesn't have a resource type \"deployment\""], "stdout": "", "stdout_lines": []}

I could build a retry in, but the API server not being available (and afterwards not knowing deployments) seems super weird - especially since at this point we have already used the API server quite a bit 🤔

I just pushed up a fix, not sure if this will actually fix your issue because it still works locally and in my test cluster without this. Might be worth merging in.

This might have fixed it, but now there's an error further down in the pipeline :)

sleiner commented 2 years ago

@timothystewart6 have you seen these types of errors before?

TASK [k3s/post : Wait for metallb controller to be running] ********************
[496](https://github.com/techno-tim/k3s-ansible/runs/8158547698?check_suite_focus=true#step:7:504)
  failed: [control1] (item=control1) => {"ansible_loop_var": "item", "changed": false, "cmd": ["kubectl", "wait", "deployment", "-n", "metallb-system", "controller", "--for", "condition=Available=True", "--timeout=60s"], "delta": "0:00:01.729871", "end": "2022-09-02 15:06:12.479623", "item": "control1", "msg": "non-zero return code", "rc": 1, "start": "2022-09-02 15:06:10.749752", "stderr": "Error from server (ServiceUnavailable): apiserver not ready", "stderr_lines": ["Error from server (ServiceUnavailable): apiserver not ready"], "stdout": "", "stdout_lines": []}
[497](https://github.com/techno-tim/k3s-ansible/runs/8158547698?check_suite_focus=true#step:7:505)
  failed: [control1] (item=control2) => {"ansible_loop_var": "item", "changed": false, "cmd": ["kubectl", "wait", "deployment", "-n", "metallb-system", "controller", "--for", "condition=Available=True", "--timeout=60s"], "delta": "0:00:01.886031", "end": "2022-09-02 15:06:17.390877", "item": "control2", "msg": "non-zero return code", "rc": 1, "start": "2022-09-02 15:06:15.504846", "stderr": "Error from server (ServiceUnavailable): apiserver not ready", "stderr_lines": ["Error from server (ServiceUnavailable): apiserver not ready"], "stdout": "", "stdout_lines": []}
[498](https://github.com/techno-tim/k3s-ansible/runs/8158547698?check_suite_focus=true#step:7:506)
  failed: [control1] (item=control3) => {"ansible_loop_var": "item", "changed": false, "cmd": ["kubectl", "wait", "deployment", "-n", "metallb-system", "controller", "--for", "condition=Available=True", "--timeout=60s"], "delta": "0:00:02.417555", "end": "2022-09-02 15:06:23.943977", "item": "control3", "msg": "non-zero return code", "rc": 1, "start": "2022-09-02 15:06:21.526422", "stderr": "error: the server doesn't have a resource type \"deployment\"", "stderr_lines": ["error: the server doesn't have a resource type \"deployment\""], "stdout": "", "stdout_lines": []}

I could build a retry in, but the API server not being available (and afterwards not knowing deployments) seems super weird - especially since at this point we have already used the API server quite a bit 🤔

I just pushed up a fix, not sure if this will actually fix your issue because it still works locally and in my test cluster without this. Might be worth merging in.

This might have fixed it, but now there's an error further down in the pipeline :)

That makes sense, I forgot to commit something 😄 Retrying it now...

sleiner commented 2 years ago

fyi: I just realized that the overrides are specific to the default scenario and modified the respective commit :-)

timothystewart6 commented 2 years ago

Are you ok if I merge @sleiner ?

sleiner commented 2 years ago

Yes 🙌🏻