search

k3s-io / kine

Run Kubernetes on MySQL, Postgres, sqlite, dqlite, not etcd.
Apache License 2.0
1.49k stars 226 forks source link

Kubeadm + Kine + NATS possible bug: illegal resource version from storage: 0 #274

Closed jrovira-kumori closed 2 months ago

jrovira-kumori commented 4 months ago

I have a minimal reproduction of deploying a cluster with Kind (kubeadm) and replacing ETCD with Kine + NATS KV store. It works when Kine uses the default SQLite store (omitting the --endpoint=... flag).

Minimal repoduction: https://github.com/jrovira-kumori/kine-nats-kv-bug-minimal-repro

Running the minimal repoduction you will find that the KubeAPI server is stuck in a loop emitting a high volume of logs stating illegal resource version from storage: 0.

The only issue that I have found with a similiar problem is #74 - when using CockroachDB. Not much help here. I have been seriously stuck with this for a while. Any clarification if it is still a WIP feature would be greatly appreciated.

brandond commented 4 months ago

What version of kine are you using?

The NATS stuff is supported by the Synadia folks. We test it on every PR using k3s; I'm not sure what kubeadm might be doing differently.

brandond commented 4 months ago

cc @bruth

jrovira-kumori commented 4 months ago

Thanks for the quick response! Software versions are as follows:

jrovira-kumori commented 4 months ago

I have updated the minimal reproduction to make it much easier to follow.

You can run k3s (with everything disabled but the KubeAPI component) and point it to the same Kine. After the initialization is done you can kill k3s and restart the Kubernetes KubeAPI and it now seems to work.

At least the stream of illegal resource version from storage: 0 no longer appears.

The new minimal reproduction shows this.

jrovira-kumori commented 4 months ago

Found the issue. Will publish a PR shortly.