search

apache / apisix-dashboard

Dashboard for Apache APISIX
https://apisix.apache.org/
Apache License 2.0
1k stars 527 forks source link

etcd watch error #2742

Open MarsOu1995 opened 1 year ago

MarsOu1995 commented 1 year ago

Issue description

etcd auth enable.

The etcd watch error will appear in the log one or two days after startup.

log output continues until the disk is full.

Expected behavior

How to Reproduce

etcd auth enable.

Screenshots

2023-02-10T23:55:50.138+0800 ERROR storage/etcd.go:188 etcd watch error: key: gateway-prod/ssls err: rpc error: code = Unauthenticated desc = etcdserver: invalid auth token 2023-02-10T23:55:50.138+0800 ERROR store/store.go:364 etcd watch exception closed, restarting: resource: ssl 2023-02-10T23:55:50.139+0800 ERROR storage/etcd.go:188 etcd watch error: key: gateway-prod/upstreams err: rpc error: code = Unauthenticated desc = etcdserver: invalid auth token 2023-02-10T23:55:50.139+0800 ERROR store/store.go:364 etcd watch exception closed, restarting: resource: upstream 2023-02-10T23:55:50.141+0800 ERROR storage/etcd.go:188 etcd watch error: key: gateway-prod/global_rules err: rpc error: code = Unauthenticated desc = etcdserver: invalid auth token 2023-02-10T23:55:50.141+0800 ERROR store/store.go:364 etcd watch exception closed, restarting: resource: global_rule 2023-02-10T23:55:50.142+0800 ERROR storage/etcd.go:188 etcd watch error: key: gateway-prod/stream_routes err: rpc error: code = Unauthenticated desc = etcdserver: invalid auth token 2023-02-10T23:55:50.142+0800 ERROR store/store.go:364 etcd watch exception closed, restarting: resource: stream_route 2023-02-10T23:55:50.143+0800 ERROR storage/etcd.go:188 etcd watch error: key: gateway-prod/protos err: rpc error: code = Unauthenticated desc = etcdserver: invalid auth token 2023-02-10T23:55:50.143+0800 ERROR store/store.go:364 etcd watch exception closed, restarting: resource: proto 2023-02-10T23:55:50.144+0800 ERROR storage/etcd.go:188 etcd watch error: key: gateway-prod/consumers err: rpc error: code = Unauthenticated desc = etcdserver: invalid auth token 2023-02-10T23:55:50.144+0800 ERROR store/store.go:364 etcd watch exception closed, restarting: resource: consumer 2023-02-10T23:55:50.145+0800 ERROR storage/etcd.go:188 etcd watch error: key: gateway-prod/services err: rpc error: code = Unauthenticated desc = etcdserver: invalid auth token 2023-02-10T23:55:50.145+0800 ERROR store/store.go:364 etcd watch exception closed, restarting: resource: service 2023-02-10T23:55:50.146+0800 ERROR storage/etcd.go:188 etcd watch error: key: gateway-prod/scripts err: rpc error: code = Unauthenticated desc = etcdserver: invalid auth token 2023-02-10T23:55:50.146+0800 ERROR store/store.go:364 etcd watch exception closed, restarting: resource: script 2023-02-10T23:55:50.147+0800 ERROR storage/etcd.go:188 etcd watch error: key: gateway-prod/data_plane/server_info err: rpc error: code = Unauthenticated desc = etcdserver: invalid auth token 2023-02-10T23:55:50.147+0800 ERROR store/store.go:364 etcd watch exception closed, restarting: resource: server_info 2023-02-10T23:55:50.149+0800 ERROR storage/etcd.go:188 etcd watch error: key: gateway-prod/plugin_configs err: rpc error: code = Unauthenticated desc = etcdserver: invalid auth token 2023-02-10T23:55:50.149+0800 ERROR store/store.go:364 etcd watch exception closed, restarting: resource: plugin_config 2023-02-10T23:55:50.150+0800 ERROR storage/etcd.go:188 etcd watch error: key: gateway-prod/system_config err: rpc error: code = Unauthenticated desc = etcdserver: invalid auth token 2023-02-10T23:55:50.150+0800 ERROR store/store.go:364 etcd watch exception closed, restarting: resource: system_config 2023-02-10T23:55:50.151+0800 ERROR storage/etcd.go:188 etcd watch error: key: gateway-prod/routes err: rpc error: code = Unauthenticated desc = etcdserver: invalid auth token 2023-02-10T23:55:50.151+0800 ERROR store/store.go:364 etcd watch exception closed, restarting: resource: route

Environment

Additional context

No response

SkyeYoung commented 1 year ago

Looks related to https://github.com/etcd-io/etcd/issues/13413, and, https://github.com/etcd-io/etcd/blob/19002cfc689fba2b8f56605e5797bf79f8b61fdd/client/v3/retry_interceptor.go#L153-L162

rbo54 commented 10 months ago

I'm pretty new to etcd, but I see this "invalid auth token" in a context that is very difficult to prevent/recover from I think:

panic: rpc error: code = Unauthenticated desc = etcdserver: invalid auth token

goroutine 62 [running]: go.etcd.io/etcd/client/v3/concurrency.(lockerMutex).Lock(0xc000196000?) /home/tom/go/pkg/mod/go.etcd.io/etcd/client/v3@v3.5.4/concurrency/mutex.go:154 +0x45 github.com/dell/csi-md/md.(mdsvc).leaseLock(0xc00014a690, {0x1ea8380?, 0xc0006387b0?}, {0xc000864480, 0x27}) /home/tom/csi-powerflex-rbo54/csi-md/md/service.go:522 +0x365 github.com/dell/csi-md/md.(mdsvc).reconcilerOpNodeStage(0x2c7aec0?, 0xc0009517a0, 0xc000618150) /home/tom/csi-powerflex-rbo54/csi-md/md/reconciler.go:361 +0x108 github.com/dell/csi-md/md.(mdsvc).nodeLookForDegradedVolumes.func1() /home/tom/csi-powerflex-rbo54/csi-md/md/reconciler.go:325 +0x4a7 created by github.com/dell/csi-md/md.(*mdsvc).nodeLookForDegradedVolumes in goroutine 73 /home/tom/csi-powerflex-rbo54/csi-md/md/reconciler.go:290 +0x4f

Is this believed to be a bug? If so is there a reasonable work around? If not am I doing something wrong in my use of mutex?

Here is the function I have calling it: // leaseLock returns a etcd lock based on a lease held by a session. // The lease will be held for up to 3 the time to the context deadline, // or if no context deadline is set, 300 seconds. // A unlockFunction and err are returned. // Call the unlock function with no arguments to release the lock. func (s mdsvc) leaseLock(ctx context.Context, lockID string) (func(), error) { var sessionDuration int64 deadline, ok := ctx.Deadline() if ok { duration := time.Until(deadline) log.Debugf("deadline %v duration %v", deadline, duration) sessionDuration = duration.Milliseconds() * 3 / 1000 } else { sessionDuration = 300 // if no deadline, default is 300 seconds } log.Debugf("sessionDuration: %d seconds", sessionDuration) session, err := concurrency.NewSession(s.etcdClient, concurrency.WithTTL(int(sessionDuration))) if err != nil { return nil, err } locker := concurrency.NewLocker(session, ETCD_MD_LEASELOCK_PREFIX+lockID) locker.Lock() return locker.Unlock, nil }

I'm running: go.etcd.io/etcd/api/v3 v3.5.4 h1:OHVyt3TopwtUQ2GKdd5wu3PmmipR4FTwCqoEjSyRdIc=