search

siderolabs / omni

SaaS-simple deployment of Kubernetes - on your own hardware.
Other
547 stars 32 forks source link

[bug] After the problem that is v0.40.0-beta.0 and etcd quota, recovering from it is currently troublesome #499

Open samip5 opened 3 months ago

samip5 commented 3 months ago

Is there an existing issue for this?

Current Behavior

After running v0.40.0-beta.0 before seeing that it had an etcd related bug, the updates version without the bug cannot be started due to Error: failed to run Omni: failed to create concurrency session: etcdserver: mvcc: database space exceeded

Expected Behavior

I expected there to be a way to customize QuotaBackendBytes for etcd, but there appears to not be.

Steps To Reproduce

  1. Run v0.40.0-beta.0 for a while
  2. Update to v0.40.0
  3. See it fail to start

What browsers are you seeing the problem on?

No response

Anything else?

There is no disk pressure on the system, so that error doesn't make any sense.

Unix4ever commented 3 months ago

I expected there to be a way to customize QuotaBackendBytes for etcd, but there appears to not be.

I think exposing that won't help as Omni will try to connect the etcd instance in the bad state and as they run together in the same process both will crash. It is necessary to run the dedicated etcd to be able to fix such kind of issues.