panic: 'put' with an unexpected smaller revision

[defp]

master 节点启动失败

日志报panic

Journal file /var/log/journal/d01909160a4443b3b23a4f7f91a94af1/system@0005fc932194d934-5812809a832afc6a.journal~ is truncated, ignoring file.
--
-- A start job for unit k8e.service has begun execution.
--
-- The job identifier is 6442.
Jun 02 07:55:46 k8s-master-133 sh[7399]: + /usr/bin/systemctl is-enabled --quiet nm-cloud-setup.service
Jun 02 07:55:46 k8s-master-133 sh[7400]: Failed to get unit file state for nm-cloud-setup.service: No such file or directory
Jun 02 07:55:46 k8s-master-133 k8e[7403]: time="2023-06-02T07:55:46Z" level=info msg="Starting k8e v1.25.5+k8e1 (7f66e3d5)"
Jun 02 07:55:46 k8s-master-133 k8e[7403]: time="2023-06-02T07:55:46Z" level=info msg="Managed etcd cluster bootstrap already complete and initialized"
Jun 02 07:55:46 k8s-master-133 k8e[7403]: time="2023-06-02T07:55:46Z" level=warning msg="Cluster CA certificate is not trusted by the host CA bundle, but the token does not include a CA hash. Use the full token from the server's node-token file to ena>
Jun 02 07:55:46 k8s-master-133 k8e[7403]: time="2023-06-02T07:55:46Z" level=info msg="Reconciling bootstrap data between datastore and disk"
Jun 02 07:55:46 k8s-master-133 k8e[7403]: time="2023-06-02T07:55:46Z" level=info msg="Successfully reconciled with datastore"
Jun 02 07:55:46 k8s-master-133 k8e[7403]: time="2023-06-02T07:55:46Z" level=info msg="Starting etcd for existing cluster member"
Jun 02 07:55:46 k8s-master-133 k8e[7403]: {"level":"info","ts":"2023-06-02T07:55:46.327Z","caller":"embed/etcd.go:131","msg":"configuring peer listeners","listen-peer-urls":["https://10.3.56.133:2380","https://127.0.0.1:2380"]}
Jun 02 07:55:46 k8s-master-133 k8e[7403]: {"level":"info","ts":"2023-06-02T07:55:46.327Z","caller":"embed/etcd.go:479","msg":"starting with peer TLS","tls-info":"cert = /var/lib/k8e/server/tls/etcd/peer-server-client.crt, key = /var/lib/k8e/server/tls>
Jun 02 07:55:46 k8s-master-133 k8e[7403]: {"level":"info","ts":"2023-06-02T07:55:46.327Z","caller":"embed/etcd.go:139","msg":"configuring client listeners","listen-client-urls":["https://10.3.56.133:2379","https://127.0.0.1:2379"]}
Jun 02 07:55:46 k8s-master-133 k8e[7403]: {"level":"info","ts":"2023-06-02T07:55:46.328Z","caller":"embed/etcd.go:308","msg":"starting an etcd server","etcd-version":"3.5.3","git-sha":"Not provided (use ./build instead of go build)","go-version":"go1.>
Jun 02 07:55:46 k8s-master-133 k8e[7403]: {"level":"info","ts":"2023-06-02T07:55:46.334Z","caller":"etcdserver/backend.go:81","msg":"opened backend db","path":"/var/lib/k8e/server/db/etcd/member/snap/db","took":"6.339922ms"}
Jun 02 07:55:46 k8s-master-133 k8e[7403]: {"level":"warn","ts":"2023-06-02T07:55:46.334Z","caller":"wal/util.go:90","msg":"ignored file in WAL directory","path":"00000000000000e0-00000000038036ee.wal.broken"}
Jun 02 07:55:46 k8s-master-133 k8e[7403]: {"level":"info","ts":"2023-06-02T07:55:46.676Z","caller":"etcdserver/server.go:508","msg":"recovered v2 store from snapshot","snapshot-index":63888066,"snapshot-size":"78 kB"}
Jun 02 07:55:46 k8s-master-133 k8e[7403]: {"level":"info","ts":"2023-06-02T07:55:46.676Z","caller":"etcdserver/server.go:521","msg":"recovered v3 backend from snapshot","backend-size-bytes":20418560,"backend-size":"20 MB","backend-size-in-use-bytes":1>
Jun 02 07:55:46 k8s-master-133 k8e[7403]: {"level":"warn","ts":"2023-06-02T07:55:46.677Z","caller":"wal/util.go:90","msg":"ignored file in WAL directory","path":"00000000000000e0-00000000038036ee.wal.broken"}
Jun 02 07:55:46 k8s-master-133 k8e[7403]: {"level":"info","ts":"2023-06-02T07:55:46.755Z","caller":"etcdserver/raft.go:483","msg":"restarting local member","cluster-id":"6197cb83bdf5daa2","local-member-id":"ae8204c5fce5e269","commit-index":63888782}
Jun 02 07:55:46 k8s-master-133 k8e[7403]: {"level":"info","ts":"2023-06-02T07:55:46.755Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"ae8204c5fce5e269 switched to configuration voters=(9395344785431473673 12574618357969838697 152045413>
Jun 02 07:55:46 k8s-master-133 k8e[7403]: {"level":"info","ts":"2023-06-02T07:55:46.755Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"ae8204c5fce5e269 became follower at term 49"}
Jun 02 07:55:46 k8s-master-133 k8e[7403]: {"level":"info","ts":"2023-06-02T07:55:46.755Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"newRaft ae8204c5fce5e269 [peers: [8262f84dbdf62e09,ae8204c5fce5e269,d30161cc42193c89], term: 49, comm>
Jun 02 07:55:46 k8s-master-133 k8e[7403]: {"level":"info","ts":"2023-06-02T07:55:46.755Z","caller":"api/capability.go:75","msg":"enabled capabilities for version","cluster-version":"3.5"}
Jun 02 07:55:46 k8s-master-133 k8e[7403]: {"level":"info","ts":"2023-06-02T07:55:46.755Z","caller":"membership/cluster.go:278","msg":"recovered/added member from store","cluster-id":"6197cb83bdf5daa2","local-member-id":"ae8204c5fce5e269","recovered-re>
Jun 02 07:55:46 k8s-master-133 k8e[7403]: {"level":"info","ts":"2023-06-02T07:55:46.755Z","caller":"membership/cluster.go:278","msg":"recovered/added member from store","cluster-id":"6197cb83bdf5daa2","local-member-id":"ae8204c5fce5e269","recovered-re>
Jun 02 07:55:46 k8s-master-133 k8e[7403]: {"level":"info","ts":"2023-06-02T07:55:46.755Z","caller":"membership/cluster.go:278","msg":"recovered/added member from store","cluster-id":"6197cb83bdf5daa2","local-member-id":"ae8204c5fce5e269","recovered-re>
Jun 02 07:55:46 k8s-master-133 k8e[7403]: {"level":"info","ts":"2023-06-02T07:55:46.755Z","caller":"membership/cluster.go:287","msg":"set cluster version from store","cluster-version":"3.5"}
Jun 02 07:55:46 k8s-master-133 k8e[7403]: {"level":"warn","ts":"2023-06-02T07:55:46.756Z","caller":"auth/store.go:1220","msg":"simple token is not cryptographically signed"}
Jun 02 07:55:46 k8s-master-133 k8e[7403]: {"level":"info","ts":"2023-06-02T07:55:46.759Z","caller":"mvcc/kvstore.go:345","msg":"restored last compact revision","meta-bucket-name":"meta","meta-bucket-name-key":"finishedCompactRev","restored-compact-rev>
Jun 02 07:55:46 k8s-master-133 k8e[7403]: {"level":"panic","ts":"2023-06-02T07:55:46.765Z","caller":"mvcc/key_index.go:81","msg":"'put' with an unexpected smaller revision","given-revision-main":56891779,"given-revision-sub":0,"modified-revision-main">
Jun 02 07:55:46 k8s-master-133 k8e[7403]: panic: 'put' with an unexpected smaller revision
Jun 02 07:55:46 k8s-master-133 k8e[7403]: goroutine 285 [running]:
Jun 02 07:55:46 k8s-master-133 k8e[7403]: go.uber.org/zap/zapcore.(*CheckedEntry).Write(0xc000e46600, {0xc00050cd00, 0x4, 0x4})
Jun 02 07:55:46 k8s-master-133 k8e[7403]:         /go/src/github.com/xiaods/k8e/vendor/go.uber.org/zap/zapcore/entry.go:232 +0x44c
Jun 02 07:55:46 k8s-master-133 k8e[7403]: go.uber.org/zap.(*Logger).Panic(0x1da5845?, {0x4d15cc4?, 0x56f0740?}, {0xc00050cd00, 0x4, 0x4})
Jun 02 07:55:46 k8s-master-133 k8e[7403]:         /go/src/github.com/xiaods/k8e/vendor/go.uber.org/zap/logger.go:230 +0x59
Jun 02 07:55:46 k8s-master-133 k8e[7403]: go.etcd.io/etcd/server/v3/mvcc.(*keyIndex).put(0xc001da0300, 0xc0000b9cb8?, 0x3641983, 0x0)
Jun 02 07:55:46 k8s-master-133 k8e[7403]:         /go/src/github.com/xiaods/k8e/vendor/go.etcd.io/etcd/server/v3/mvcc/key_index.go:81 +0x34b
Jun 02 07:55:46 k8s-master-133 k8e[7403]: go.etcd.io/etcd/server/v3/mvcc.restoreIntoIndex.func1()
Jun 02 07:55:46 k8s-master-133 k8e[7403]:         /go/src/github.com/xiaods/k8e/vendor/go.etcd.io/etcd/server/v3/mvcc/kvstore.go:475 +0x651
Jun 02 07:55:46 k8s-master-133 k8e[7403]: created by go.etcd.io/etcd/server/v3/mvcc.restoreIntoIndex
Jun 02 07:55:46 k8s-master-133 k8e[7403]:         /go/src/github.com/xiaods/k8e/vendor/go.etcd.io/etcd/server/v3/mvcc/kvstore.go:441 +0xfa
Jun 02 07:55:46 k8s-master-133 systemd[1]: k8e.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
-- Subject: Unit process exited
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
--
-- An ExecStart= process belonging to unit k8e.service has exited.
--
-- The process' exit code is 'exited' and its exit status is 2.
Jun 02 07:55:46 k8s-master-133 systemd[1]: k8e.service: Failed with result 'exit-code'.
-- Subject: Unit failed
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
--
-- The unit k8e.service has entered the 'failed' state with result 'exit-code'.
Jun 02 07:55:46 k8s-master-133 systemd[1]: Failed to start Kubernetes Easy Engine.
-- Subject: A start job for unit k8e.service has failed
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
--
-- A start job for unit k8e.service has finished with a failure.
--
-- The job identifier is 6442 and the job result is failed.

[defp]

VERSION: v1.25.5+k8e1 (7f66e3d5)

[defp]

nm-cloud-setup.service: No such file or directory 重新安装节点后出现这个

[xiaods]

VERSION: v1.25.5+k8e1 (7f66e3d)

我之前也发现这个问题。升级到v1.25.7+k8e1 for Security Upgrade，修复了此问题。

nm-cloud-setup.service: No such file or directory , 这个可以不用关注，你可以手工在 k8e.service 里面去掉这个

[xiaods]

https://github.com/xiaods/k8e/pull/319 在安排文件中也优化去掉了nm-cloud-setup.service，

[xiaods]

/close as resolved

[defp]

升级完可以了