search

xtruder / kubenix

Replaced by https://github.com/hall/kubenix
MIT License
300 stars 34 forks source link

Flaky tests #17

Closed xavierzwirtz closed 1 year ago

xavierzwirtz commented 4 years ago

I think I am seeing flakiness with the kube node getting spun up during test runs. I don't know enough to give more information, the output of nix build -f ./. test-script rapidly flashes incomplete information. It looks like its repeatedly running ("kubectl get node kube.my.xzy | grep -w Ready"). Any guidance to getting more info on this?

xavierzwirtz commented 4 years ago

I can reproduce this with the nginx-deployment example. Steps to repro:

git clone https://github.com/xtruder/kubenix/
cd kubenix/examples/nginx-deployment
#run this command till failure, success happens quickly, failure takes 900 seconds before timeout
rm -f result && nix-collect-garbage -d && nix-build . -A test-script
xavierzwirtz commented 4 years ago

hung output 1 hung output 2 successful output

offlinehacker commented 4 years ago

Thank you for reporting issue. There are issues with e2e tests as kubetnetes module in nixos is having issues. I am planning to replace testing infra with k3s (or probably make different test profiles so you can choose how to run tests), which would also much shorten test time and resources used. If you can debug what is causing issues pull requests are much appreciated.

offlinehacker commented 4 years ago

There seems to be permission issues:

Failed to list *v1.Node: nodes is forbidden: User "system:kube-scheduler" cannot list resource "nodes" in API group "" at the cluster scope
xavierzwirtz commented 4 years ago

Thanks for taking a look. I didn't realize the tests themselves were flaky at first, was making minor changes to one of my tests and watching them hang. Thought I was going crazy.

Using k3s sounds lovely, faster test times would be great. Ill see if I can figure out whats going on with the existing tests, I want to move forward using kubenix.

offlinehacker commented 4 years ago

Yes, it would be also nice to have backend independent testing abstraction, so you can also run it on local kubernetes outside nix build process, using nix-shell or nix run on any k8s cluster. If you have any ideas how to implement/integrate something like this, it would be nice to discuss.

offlinehacker commented 1 year ago

This repo has been deprecated, since I stopped maintaining it some time ago. There is a fork maintained by @hall available at https://github.com/hall/kubenix, that has better documentation and looks like a way further.