search

etcd-io / etcd

Distributed reliable key-value store for the most critical data of a distributed system
https://etcd.io
Apache License 2.0
47.6k stars 9.74k forks source link

ETCD is shutting down continuously restarting the pod #15941

Closed rahulbapumore closed 1 year ago

rahulbapumore commented 1 year ago

What happened?

installed etcd with 3 replicas as part of deployment and pods are restarting continuously. There are many error messages coming inside logs, not sure why its coming.

{"caller":"osutil/interrupt_unix.go:64","message":"received signal; shutting down",,"service_id":"etcd","severity":"info","signal":"terminated","timestamp":"2023-05-19T09:39:38.931+02:00","version":"1.2.0"} {"advertise-client-urls":["https://etcd-0.etcd.keep-cerberus-04:2379"],"advertise-peer-urls":["https://etcd-0.etcd-peer.keep-cerberus-04.svc.cluster.local:2380"],"caller":"embed/etcd.go:371","data-dir":"/data","message":"closing etcd server",,"name":"etcd-0","service_id":"etcd","severity":"info","timestamp":"2023-05-19T09:39:38.932+02:00","version":"1.2.0"} {"caller":"embed/config_logging.go:169","error":"read tcp 10.255.31.131:2379->172.30.22.62:55386: use of closed network connection","message":"rejected connection",,"remote-addr":"172.30.22.62:55386","server-name":"etcd.keep-cerberus-04","service_id":"etcd","severity":"warning","timestamp":"2023-05-19T09:39:38.932+02:00","version":"1.2.0"} {"caller":"zapgrpc/zapgrpc.go:174","message":"[core] Subchannel Connectivity change to IDLE",,"service_id":"etcd","severity":"info","timestamp":"2023-05-19T09:39:38.932+02:00","version":"1.2.0"} {"caller":"zapgrpc/zapgrpc.go:174","message":"[transport] transport: loopyWriter.run returning. connection error: desc = \"transport is closing\"",,"service_id":"etcd","severity":"info","timestamp":"2023-05-19T09:39:38.932+02:00","version":"1.2.0"} {"caller":"zapgrpc/zapgrpc.go:174","message":"[core] pickfirstBalancer: UpdateSubConnState: 0xc000496720, {IDLE }",,"service_id":"etcd","severity":"info","timestamp":"2023-05-19T09:39:38.932+02:00","version":"1.2.0"} {"caller":"zapgrpc/zapgrpc.go:174","message":"[core] Channel Connectivity change to IDLE",,"service_id":"etcd","severity":"info","timestamp":"2023-05-19T09:39:38.932+02:00","version":"1.2.0"} {"caller":"etcdserver/server.go:1465","current-leader-member-id":"a93f9ecf02976efa","local-member-id":"a93f9ecf02976efa","message":"skipped leadership transfer for single voting member cluster",,"service_id":"etcd","severity":"info","timestamp":"2023-05-19T09:39:38.933+02:00","version":"1.2.0"} {"caller":"zapgrpc/zapgrpc.go:174","message":"[core] Channel Connectivity change to SHUTDOWN",,"service_id":"etcd","severity":"info","timestamp":"2023-05-19T09:39:38.933+02:00","version":"1.2.0"} {"caller":"zapgrpc/zapgrpc.go:174","message":"[core] Subchannel Connectivity change to SHUTDOWN",,"service_id":"etcd","severity":"info","timestamp":"2023-05-19T09:39:38.933+02:00","version":"1.2.0"} {"address":"[::]:2380","caller":"embed/etcd.go:566","message":"stopping serving peer traffic",,"service_id":"etcd","severity":"info","timestamp":"2023-05-19T09:39:38.935+02:00","version":"1.2.0"} {"address":"[::]:2380","caller":"embed/etcd.go:571","message":"stopped serving peer traffic",,"service_id":"etcd","severity":"info","timestamp":"2023-05-19T09:39:38.935+02:00","version":"1.2.0"} {"advertise-client-urls":["https://etcd-0.etcd.keep-cerberus-04:2379"],"advertise-peer-urls":["https://etcd-0.etcd-peer.keep-cerberus-04.svc.cluster.local:2380"],"caller":"embed/etcd.go:373","data-dir":"/data","message":"closed etcd server",,"name":"etcd-0","service_id":"etcd","severity":"info","timestamp":"2023-05-19T09:39:38.935+02:00","version":"1.2.0"} {"message":"/usr/local/bin/scripts/etcd_runnerBootstrapDisabled.sh: line 87: 79 Terminated

What did you expect to happen?

ETCD should behave normally and serve the traffic

How can we reproduce it (as minimally and precisely as possible)?

Not sure, just installation

Anything else we need to know?

No response

Etcd version (please run commands below)

bash-4.4$ etcd --version etcd Version: 3.5.7 Git SHA: https://github.com/etcd-io/etcd/commit/215b53cf3b48ee761f4c40908b3874b2e5e95e9f Go Version: go1.17.13 Go OS/Arch: linux/amd64 bash-4.4$ etcdctl version etcdctl version: 3.5.7 API version: 3.5 bash-4.4$

Etcd configuration (command line flags or environment variables)

bash-4.4$ env | grep ETCD ETCD_INITIAL_CLUSTER_TOKEN=etcd ETCD_MAX_SNAPSHOTS=3 ETCD_LISTEN_CLIENT_URLS=https://0.0.0.0:2379 ETCD_HEARTBEAT_INTERVAL=100 ETCD_AUTO_COMPACTION_RETENTION=100 ETCD_TRUSTED_CA_FILE=/data/combinedca/cacertbundle.pem ENTRYPOINT_RESTART_ETCD=true ETCDCTL_CERT=/run/sec/certs/client/clicert.pem ETCD_LOG_LEVEL=info ETCD_LISTEN_PEER_URLS=https://0.0.0.0:2380 ETCD_AUTO_COMPACTION_MODE=revision ETCD_LOGGER=zap ETCD_CERT_FILE=/run/sec/certs/server/srvcert.pem ETCD_PEER_AUTO_TLS=true ETCD_DATA_DIR=/data ETCD_CLIENT_CERT_AUTH=true ETCDCTL_ENDPOINTS=etcd.zmorrah:2379 ETCD_METRICS=basic ETCDCTL_API=3 ETCD_SNAPSHOT_COUNT=5000 ETCD_MAX_WALS=3 ETCD_INITIAL_ADVERTISE_PEER_URLS=https://etcd-0.etcd-peer.zmorrah.svc.cluster.local:2380 ETCD_KEY_FILE=/run/sec/certs/server/srvprivkey.pem ETCD_ELECTION_TIMEOUT=1000 ETCDCTL_CACERT=/data/combinedca/cacertbundle.pem ETCD_NAME=etcd-0 ETCD_QUOTA_BACKEND_BYTES=268435456 ETCD_ADVERTISE_CLIENT_URLS=https://etcd-0.etcd.zmorrah:2379 ETCDCTL_KEY=/run/sec/certs/client/cliprivkey.pem

Etcd debug information (please run commands below, feel free to obfuscate the IP address or FQDN in the output)

NA

Relevant log output

{"caller":"auth/store.go:1146","common-name":"eric-sec-key-management","message":"found command name",,"revision":9,"service_id":"etcd","severity":"debug","timestamp":"2023-05-19T09:39:38.537+02:00","user-name":"eric-sec-key-management","version":"1.2.0"} {"caller":"etcdserver/server.go:2134","message":"Applying entries",,"num-entries":1,"service_id":"etcd","severity":"debug","timestamp":"2023-05-19T09:39:38.538+02:00","version":"1.2.0"} {"caller":"etcdserver/server.go:2137","index":637,"message":"Applying entry",,"service_id":"etcd","severity":"debug","term":4,"timestamp":"2023-05-19T09:39:38.538+02:00","type":"EntryNormal","version":"1.2.0"} {"caller":"etcdserver/server.go:2196","consistent-index":636,"entry-index":637,"message":"apply entry normal",,"service_id":"etcd","severity":"debug","should-applyV3":true,"timestamp":"2023-05-19T09:39:38.538+02:00","version":"1.2.0"} {"caller":"etcdserver/server.go:2223","message":"applyEntryNormal",,"raftReq":"header:<ID:7996853840735518914 username:\"eric-sec-key-management\" auth_revision:9 > put:<key:\"/shelter/logical/73265fc9-2e14-b494-6b01-e8d27ed77476/config/key/11b9bc99-9ad6-08af-3f88-824dc0496c08\" value:\"\\000\\000\\000\\001\\002\\357\\207\\210o\\301\\334\\036\\277WM\\360\\001\\335^$h\\3665\\242\\007\\315D\\364\\004\\213\\266\\212qj\\211=\\016B\\313k\\206\\217,\\355\\351\\3458F0\\016\\253`0z-\\257!\\020\\032\\027[\\255\\240\\261N44n!\\242D\\363\\023<\\322\\323\\024\\206e\\352\\373P\\356\\311\\377\\267\\306\\217\\2610\\300\\226\\334\\225\\215\\350)\\016w\\260\\327\\030\\300\\216\\321\\255sg\\215I\\245\\002\\363GrC\\303\\310\\324\\t\\023\\261\\n\\\\A\\030\\345\\315\\346\\212\\341)\\232\\323\\363\\262Q\\346v\\263\\347\\360V\\024\\351k}}3\\320\\2678(\\3000\\310\\367_\\326CS\\331\\010\\277\\214\\275o\\201\\022\\314(v\\305\\320'\\205\\206\\3363\\223\\254.\\304N\\024qp\\342n\\033\\027\\250\\264A{\\345\\365\\347\\360\\335$\\232\\t\\247\\003C\\272\\026^\\023Z\\324k\\316\\304Tn\\276\\345\\353\\366V\\274uDv\\004\\231\\245\\002\\262\\n0\\320\\253^@N\\360\\266\\267\\250G\\261[Sy\\3037)%\\\\\\0059\\241\\274\\315\\211\\022+\\323Qk\\310\\2415\\261Xt\\352\\353Z\\210\\254\\221Dv\\312\\242\\347\\370u5\\320nQ\\232C\\000\\357\\215\\205\\017\\224\\241\\317\\365\\007c\\256\\320\\014\\265\\341Hj\\\\I1\\216\\241<x:\\214\\331\\271\\353Ge\\341\\224\\233\\336\\\"\\240\\323\\235\\031\\016\\261\\253\\262\\372L\\333S:\\267+\\223\\t\\200\\312s\\257E5%\\270\\347\\333\\277\\250k\\320 u<A`N\" > ","service_id":"etcd","severity":"debug","timestamp":"2023-05-19T09:39:38.539+02:00","version":"1.2.0"} {"caller":"v3rpc/interceptor.go:182","message":"request stats",,"remote":"10.255.55.80:58520","request content":"key:\"/shelter/logical/73265fc9-2e14-b494-6b01-e8d27ed77476/config/key/11b9bc99-9ad6-08af-3f88-824dc0496c08\" value_size:362 ","request count":1,"request size":468,"response count":0,"response size":29,"response type":"/etcdserverpb.KV/Put","service_id":"etcd","severity":"debug","start time":"2023-05-19T09:39:38.537+0200","time spent":"785.649µs","timestamp":"2023-05-19T09:39:38.539+02:00","version":"1.2.0"} {"caller":"auth/store.go:1146","common-name":"eric-sec-key-management","message":"found command name",,"revision":9,"service_id":"etcd","severity":"debug","timestamp":"2023-05-19T09:39:38.539+02:00","user-name":"eric-sec-key-management","version":"1.2.0"} {"caller":"v3rpc/interceptor.go:182","message":"request stats",,"remote":"10.255.55.80:58520","request content":"key:\"/shelter/logical/73265fc9-2e14-b494-6b01-e8d27ed77476/config/issuer/\" range_end:\"/shelter/logical/73265fc9-2e14-b494-6b01-e8d27ed77476/config/issuer0\" ","request count":0,"request size":140,"response count":0,"response size":29,"response type":"/etcdserverpb.KV/Range","service_id":"etcd","severity":"debug","start time":"2023-05-19T09:39:38.538+0200","time spent":"75.641µs","timestamp":"2023-05-19T09:39:38.539+02:00","version":"1.2.0"} {"caller":"auth/store.go:1146","common-name":"eric-sec-key-management","message":"found command name",,"revision":9,"service_id":"etcd","severity":"debug","timestamp":"2023-05-19T09:39:38.539+02:00","user-name":"eric-sec-key-management","version":"1.2.0"} {"caller":"v3rpc/interceptor.go:182","message":"request stats",,"remote":"10.255.55.80:58520","request content":"key:\"/shelter/logical/73265fc9-2e14-b494-6b01-e8d27ed77476/config/issuers\" ","request count":0,"request size":70,"response count":0,"response size":29,"response type":"/etcdserverpb.KV/Range","service_id":"etcd","severity":"debug","start time":"2023-05-19T09:39:38.539+0200","time spent":"61.238µs","timestamp":"2023-05-19T09:39:38.539+02:00","version":"1.2.0"} {"caller":"auth/store.go:1146","common-name":"eric-sec-key-management","message":"found command name",,"revision":9,"service_id":"etcd","severity":"debug","timestamp":"2023-05-19T09:39:38.539+02:00","user-name":"eric-sec-key-management","version":"1.2.0"} {"caller":"v3rpc/interceptor.go:182","message":"request stats",,"remote":"10.255.55.80:58520","request content":"key:\"/shelter/logical/73265fc9-2e14-b494-6b01-e8d27ed77476/config/keys\" ","request count":0,"request size":67,"response count":0,"response size":29,"response type":"/etcdserverpb.KV/Range","service_id":"etcd","severity":"debug","start time":"2023-05-19T09:39:38.539+0200","time spent":"63.193µs","timestamp":"2023-05-19T09:39:38.539+02:00","version":"1.2.0"} {"caller":"auth/store.go:1146","common-name":"eric-sec-key-management","message":"found command name",,"revision":9,"service_id":"etcd","severity":"debug","timestamp":"2023-05-19T09:39:38.539+02:00","user-name":"eric-sec-key-management","version":"1.2.0"} {"caller":"etcdserver/server.go:2134","message":"Applying entries",,"num-entries":1,"service_id":"etcd","severity":"debug","timestamp":"2023-05-19T09:39:38.540+02:00","version":"1.2.0"} {"caller":"etcdserver/server.go:2137","index":638,"message":"Applying entry",,"service_id":"etcd","severity":"debug","term":4,"timestamp":"2023-05-19T09:39:38.540+02:00","type":"EntryNormal","version":"1.2.0"} {"caller":"etcdserver/server.go:2196","consistent-index":637,"entry-index":638,"message":"apply entry normal",,"service_id":"etcd","severity":"debug","should-applyV3":true,"timestamp":"2023-05-19T09:39:38.540+02:00","version":"1.2.0"} {"caller":"etcdserver/server.go:2223","message":"applyEntryNormal",,"raftReq":"header:<ID:7996853840735518918 username:\"eric-sec-key-management\" auth_revision:9 > put:<key:\"/shelter/logical/73265fc9-2e14-b494-6b01-e8d27ed77476/config/keys\" value:\"\\000\\000\\000\\001\\002\\362\\033\\333\\025\\221\\n\\310\\016\\213\\306\\212\\341\\226'\\273u\\014\\342u\\251\\\"\\306s\\201\\212\\202\\375\\020\\013w\\323\\357p\\365p\\000\\301A\\3118\\2754w;\\264KU+T\\365\\231\\247&j\\357\\316\\344\\255\\376\\234)\\325\\312\\201\\244\\345\\242\\356.\\247I\\313q\\322\\032\\357\\254\\257\\013\" > ","service_id":"etcd","severity":"debug","timestamp":"2023-05-19T09:39:38.540+02:00","version":"1.2.0"} {"caller":"v3rpc/interceptor.go:182","message":"request stats",,"remote":"10.255.55.80:58520","request content":"key:\"/shelter/logical/73265fc9-2e14-b494-6b01-e8d27ed77476/config/keys\" value_size:84 ","request count":1,"request size":153,"response count":0,"response size":29,"response type":"/etcdserverpb.KV/Put","service_id":"etcd","severity":"debug","start time":"2023-05-19T09:39:38.539+0200","time spent":"756.508µs","timestamp":"2023-05-19T09:39:38.540+02:00","version":"1.2.0"} {"caller":"auth/store.go:1146","common-name":"eric-sec-key-management","message":"found command name",,"revision":9,"service_id":"etcd","severity":"debug","timestamp":"2023-05-19T09:39:38.540+02:00","user-name":"eric-sec-key-management","version":"1.2.0"} {"caller":"v3rpc/interceptor.go:182","message":"request stats",,"remote":"10.255.55.80:58520","request content":"key:\"/shelter/logical/73265fc9-2e14-b494-6b01-e8d27ed77476/config/issuer/\" range_end:\"/shelter/logical/73265fc9-2e14-b494-6b01-e8d27ed77476/config/issuer0\" ","request count":0,"request size":140,"response count":0,"response size":29,"response type":"/etcdserverpb.KV/Range","service_id":"etcd","severity":"debug","start time":"2023-05-19T09:39:38.540+0200","time spent":"91.407µs","timestamp":"2023-05-19T09:39:38.541+02:00","version":"1.2.0"} {"caller":"auth/store.go:1146","common-name":"eric-sec-key-management","message":"found command name",,"revision":9,"service_id":"etcd","severity":"debug","timestamp":"2023-05-19T09:39:38.541+02:00","user-name":"eric-sec-key-management","version":"1.2.0"} {"caller":"v3rpc/interceptor.go:182","message":"request stats",,"remote":"10.255.55.80:58520","request content":"key:\"/shelter/logical/73265fc9-2e14-b494-6b01-e8d27ed77476/config/key/\" range_end:\"/shelter/logical/73265fc9-2e14-b494-6b01-e8d27ed77476/config/key0\" ","request count":0,"request size":134,"response count":1,"response size":510,"response type":"/etcdserverpb.KV/Range","service_id":"etcd","severity":"debug","start time":"2023-05-19T09:39:38.541+0200","time spent":"71.263µs","timestamp":"2023-05-19T09:39:38.541+02:00","version":"1.2.0"} {"caller":"auth/store.go:1146","common-name":"eric-sec-key-management","message":"found command name",,"revision":9,"service_id":"etcd","severity":"debug","timestamp":"2023-05-19T09:39:38.541+02:00","user-name":"eric-sec-key-management","version":"1.2.0"} {"caller":"v3rpc/interceptor.go:182","message":"request stats",,"remote":"10.255.55.80:58520","request content":"key:\"/shelter/logical/73265fc9-2e14-b494-6b01-e8d27ed77476/config/issuer/\" range_end:\"/shelter/logical/73265fc9-2e14-b494-6b01-e8d27ed77476/config/issuer0\" ","request count":0,"request size":140,"response count":0,"response size":29,"response type":"/etcdserverpb.KV/Range","service_id":"etcd","severity":"debug","start time":"2023-05-19T09:39:38.541+0200","time spent":"61.645µs","timestamp":"2023-05-19T09:39:38.541+02:00","version":"1.2.0"} {"caller":"auth/store.go:1146","common-name":"eric-sec-key-management","message":"found command name",,"revision":9,"service_id":"etcd","severity":"debug","timestamp":"2023-05-19T09:39:38.541+02:00","user-name":"eric-sec-key-management","version":"1.2.0"} {"caller":"etcdserver/server.go:2134","message":"Applying entries",,"num-entries":1,"service_id":"etcd","severity":"debug","timestamp":"2023-05-19T09:39:38.543+02:00","version":"1.2.0"} {"caller":"etcdserver/server.go:2137","index":639,"message":"Applying entry",,"service_id":"etcd","severity":"debug","term":4,"timestamp":"2023-05-19T09:39:38.543+02:00","type":"EntryNormal","version":"1.2.0"} {"caller":"etcdserver/server.go:2196","consistent-index":638,"entry-index":639,"message":"apply entry normal",,"service_id":"etcd","severity":"debug","should-applyV3":true,"timestamp":"2023-05-19T09:39:38.544+02:00","version":"1.2.0"} {"caller":"etcdserver/server.go:2223","message":"applyEntryNormal",,"raftReq":"header:<ID:7996853840735518922 username:\"eric-sec-key-management\" auth_revision:9 > put:<key:\"/shelter/logical/73265fc9-2e14-b494-6b01-e8d27ed77476/config/issuer/98e2161d-0e54-4ec0-7523-6ece7018511f\" value:\"\\000\\000\\000\\001\\002\\033,f\\304f\\332I\\363\\320\\2021\\311\\213\\307\\310Y\\035\\222nd+\\006>\\340\\207\\312R\\035\\034\\204*\\262\\2307\\222^\\324\\367\\213\\211D\\253\\202'U\\210\\352\\275>\\376\\024\\022&\\275\\316\\225\\3774\\341\\216\\251\\300\\340Q8\\364\\266\\327\\345t\\200G\\364\\247\\322\\2359\\352q\\323\\363ze+\\300\\243\\236\\216\\367H\\257\\314\\260\\3426u\\367ZB\\002\\372\\257\\335\\232\\230\\027\\247\\374\\020\\003\\343!^\\346\\315\\370\\353H\\271!\\316\\276^\\317\\370\\253\\351\\305%\\220\\240#\\316\\203J}\\213\\254\\316\\252\\306\\261X\\005\\0328\\363\\362\\355!a\\273c\\321\\372?*\\260\\367\\035\\032\\263\\375!\\005qS\\2201|\\222\\255\\242\\203\\n\\035)\\321=\\265y\\332XP%\\230\\tvm\\253\\353\\375^\\302\\232\\213\\232\\026\\001\\326uN,\\033%\\221b\\007\\246\\005\\317K\\320\\n\\225p\\270gD\\310UWWz=\\312\\031\\n1\\223\\232\\345\\036\\243\\321u\\036\\323\\226f\\r\\361b\\252-\\201\\243\\214\\367\\250g\\202\\331\\240\\243+\\250\\303\\243\\317/(\\036\\300\\357\\204+&!)\\250,\\371\\352\\326\\345\\223\\321\\216\\0263\\324Cn9@_\\334 \\331\\034Bc\\201\\354\\213\\004\\350\\212v\\213\\352\\331|\\253_\\247\\356?Y\\231\\316\\313\\300\\304\\333\\202o|\\306\\277,R\\263\\222\\273J\\313\\371L[f\\352\\367z\\240\\005~g:\\361#4@7\\027W\\352;-\\007\\220\\001\\221\\016\\215\\323\\201\\376\\351[_\\261\\247\\024dC\\301\\002\\31446\\337\\376\\351ws\\333~\\375\\256\\365\\216\\3046\\025<5\\200\\334l\\246\\246t\\373\\254\\313a|\\353\\373\\235>\\036\\251\\273G\\024t\\362U-\\376\\1776p<\\235\\220\\307\\303y\\303\\241\\310^\\237?\\014\\345(\\246\\362c\\326\\347\\360:7\\335g\\234\\020R\\236\\332OJ\\0339o}\\276\\350y6\\311\\014\\001\\342\\003y\\333\\220VO\\333\\365\\256%\\222\\366\\242\\331\\035\\3716b\\211\\003\\243t9\\374p[\\t\\260\\352b\\356\\264\\375f\\271@<\\352\\277\\210\\035\\005Mwp\\365\\372\\302[P\\227\\316\\035X\\005\\250\\314\\2527=\\204?!@\\231\\213jT\\207\\217%\\333C\\311\\322B~J\\255\\236\\346&\\276\\203`k\\255\\324m\\2640\\022\\276\\353\\221\\177\\017\\337s?\\324\\022\\371\\305 \\310\\330\\232O\\030\\354h\\274\\245u>\\375\\005\\246vC\\375H~\\266d,\\356\\307-N;\\242l;\\240\\365X\\365\\337\\023P~\\332\\264\\010\\345H\\264\\300\\273\\213Um\\317\\303.\\006\\254\\310$\\306\\023\\014\\226\\347\\223\\225\\244\\304\\360C{#Z\\316\\336\\362\\344\\266F\\010\\324\\0233\\354\\247i2vG\\371\\322\\033\\315\\317E.:'\\3479\\242\\236\\277[\\221\\241L\\202,\\312\\005\\326\\370\\035I\\0011\\303\\204\\021\\333\\330\\332\\267\\344~\\205\\205M\\251lq\\262o\\003\\024\\345@\\263 \\305\\316\\006\\267\\005\\036\\256\\244OC\\213\\317M\\177\\207\\327\\367\\245[,\\207\\001\\034\\274\\2408Tm\\260\\224\\207d\\2072Y\\371\\000A\\352\\031R\\032Bo\\276&Z68I\\n\\031?\\373\\351\\017\\004\\237t\\333\\016\\254&\\244m2rT\\361+) \\274o\\266s\\245\\200\\210u\\222x\\247\\256\\262\\340,\\357\\213\\371\\354\\343n\\236\\017\\007\\315\\276\\236\\004\\312>]\\360\\260\\n\\002\\026\\263\\314Z%\\302v\\277C5`#qbgm\\217\\353\\200\\363\\003`\\nj\\321\\306Ws\\037\\361_}/\\004x\\307\\030\\230\\360`AR'\\325Kn+\\306L\\341\\303\\177\\317h\\251!\\007\\016j;'I\\235}\\362\\303\\214\\030`t+\\317\\357\\256\\262\\305\\300\\021\\030\\\\.Zm.\\013\\227\\014\\345\\320C4\\234E\\206I\\301i0RPL)L\\2766\\001\\016_\\321\\307\\267\\2359\\030\\021\\274-\\010\\t\\304\\253M\\253\\032 ,\\014\\346\\217\\014\\251\\241\\315\\226\\n\\330\\202;}\\354\\034\\322\\\"\\335\\245\\375\\204\\252\\367\\225kC\\322a\\375\\276\\212\\033\\\\~\\353\\214\\023\\037ZN\\240\\355\\232yV\\245B\\277K\\351\\224\\224\\001D\\003\\324C\\353\\020X.]\\236\\335\\353\\233\\345\\024\\267`W\\\\Gb%4\\223\\026+\\013\\324\\003r3qQ\\247:\\304}DTRM^\\006\\376\\274\\016\\315\\265\\275-H\\021:kq\\210>\\243t\\231\\210c\\345\\242\\325|V8\\342\\023\\207\\3051\\260\\346\\224\\357\\353\\325\\033$\\360\\026L>2\\212\\234\\276\\206\\271{#\\335\\262\\322\\243R\\263\\0055\\022\\313\\013\\272\\256\\255\\251I-\\254\\241\\301\\032\\032\\027\\316\\374\\006\\343\\t\\251\\312 \\332\\327\\363\\013m2\\016|G\\264\\242\\367a\\352\\344\\256\\362\\224\\\"s\\215dI\\300\\244L\\035\\330b\\t\\226\\327\\255\\347\\321\\005\\275\\327\\320\\303;Z\\030\\254\\242V\\273\\335$.\\320\\325\\334}uL\\004\\326\\377\\232\\317Ux\\266.\\024\\232\\026yK\\221\\344y]D[\\336\\265\\237\\241\\333q\\3310\\374Kt\\225L\\351\\260s\\307\\3779l@\\203Y4\\223\\366`\\331\\256\\3516\\252\\000\\376\\362\\t\\313\\3731\\366_\\267\\257\\\\\\336\\347OS\\014\\247\\317`'\\261\\246\\277\\300\\325\\216\\373n3\\311\\215\\007\\003\\225\\304`\\002T\\276Y>O\\0370t\\274,\\277\\004\\037\\t\\245V\\255\\371\\014\\314\\001\\353\\355\\274\\275\\232\\006\\020}\\210\\036\\252\\336v\\326\\220aBr'p\\211\\331\\273\\220]\\367+\\336=OS\\337\\035z\\302g\\236&\\374\\310(\\237|\\207\\344\\017\\026l\\253-A\\n\\333M\\335\\241U\\017qH\\317\\266Q\\255\\330\\341x\\212u\\250\\356\\022\\324\\021\\366\\326\\246q4/vh\\271\\376\\210\\345\\365!\\227\\204\\256\\264\\207\\002P\\030sZ\\201\\200\\232\\001\\027\\374\\234\\235\\260C\\3067T\\270\\177\\217\\005d\\227\\2205\\332\\272\\257\\030o#\\254\\351SS\\222\\0143?\\355\\241;I\\316q\\254\\331\\265\\245K\\223\\327u\\206<\\025\\207\\374E\\326EO\\327\\233\\357\\345E\\211\\214'\\263$}\\346\\343w>\\377h\\t\\032Jz\\206\\010\\363\\031\\331H\\005\\202p\\242f\\346\\210\\366\\0235{\\252\\203\\034)\\301\\306=\\005\\034|\\271\\272e\\365\\257\\345\\033/NtQ|\\377\\2558\\027&N\\204/\\2167Z\\252\\244\\366\\215yr\\263\\271\\333\\205AyB\\372\\002[\\307S\" > ","service_id":"etcd","severity":"debug","timestamp":"2023-05-19T09:39:38.544+02:00","version":"1.2.0"} {"caller":"v3rpc/interceptor.go:182","message":"request stats",,"remote":"10.255.55.80:58520","request content":"key:\"/shelter/logical/73265fc9-2e14-b494-6b01-e8d27ed77476/config/issuer/98e2161d-0e54-4ec0-7523-6ece7018511f\" value_size:1529 ","request count":1,"request size":1638,"response count":0,"response size":29,"response type":"/etcdserverpb.KV/Put","service_id":"etcd","severity":"debug","start time":"2023-05-19T09:39:38.541+0200","time spent":"2.279171ms","timestamp":"2023-05-19T09:39:38.544+02:00","version":"1.2.0"} {"caller":"auth/store.go:1146","common-name":"eric-sec-key-management","message":"found command name",,"revision":9,"service_id":"etcd","severity":"debug","timestamp":"2023-05-19T09:39:38.626+02:00","user-name":"eric-sec-key-management","version":"1.2.0"} {"caller":"etcdserver/server.go:2134","message":"Applying entries",,"num-entries":1,"service_id":"etcd","severity":"debug","timestamp":"2023-05-19T09:39:38.627+02:00","version":"1.2.0"} {"caller":"etcdserver/server.go:2137","index":640,"message":"Applying entry",,"service_id":"etcd","severity":"debug","term":4,"timestamp":"2023-05-19T09:39:38.627+02:00","type":"EntryNormal","version":"1.2.0"} {"caller":"etcdserver/server.go:2196","consistent-index":639,"entry-index":640,"message":"apply entry normal",,"service_id":"etcd","severity":"debug","should-applyV3":true,"timestamp":"2023-05-19T09:39:38.627+02:00","version":"1.2.0"} {"caller":"etcdserver/server.go:2223","message":"applyEntryNormal",,"raftReq":"header:<ID:7996853840735518923 username:\"eric-sec-key-management\" auth_revision:9 > put:<key:\"/shelter/logical/73265fc9-2e14-b494-6b01-e8d27ed77476/config/issuers\" value:\"\\000\\000\\000\\001\\002\\252\\001Yu\\377\\212X\\216\\265\\256\\273|\\010\\241\\036,\\336\\014'BI'\\357\\2356\\244PJ\\3460\\265\\356\\324\\2079\\215\\005\\375\\247\\334^2\\014\\222\\206\\343\\267<(\\rHH\\177p\\261\\251q\\233W\\032D%\\017\\350\\2104D\\214\\240x&a\\003\\027\\316\\\"\\2374\\201\" > ","service_id":"etcd","severity":"debug","timestamp":"2023-05-19T09:39:38.627+02:00","version":"1.2.0"} {"caller":"v3rpc/interceptor.go:182","message":"request stats",,"remote":"10.255.55.80:58520","request content":"key:\"/shelter/logical/73265fc9-2e14-b494-6b01-e8d27ed77476/config/issuers\" value_size:84 ","request count":1,"request size":156,"response count":0,"response size":29,"response type":"/etcdserverpb.KV/Put","service_id":"etcd","severity":"debug","start time":"2023-05-19T09:39:38.544+0200","time spent":"785.01µs","timestamp":"2023-05-19T09:39:38.627+02:00","version":"1.2.0"} {"caller":"auth/store.go:1146","common-name":"eric-sec-key-management","message":"found command name",,"revision":9,"service_id":"etcd","severity":"debug","timestamp":"2023-05-19T09:39:38.627+02:00","user-name":"eric-sec-key-management","version":"1.2.0"} {"caller":"etcdserver/server.go:2134","message":"Applying entries",,"num-entries":1,"service_id":"etcd","severity":"debug","timestamp":"2023-05-19T09:39:38.627+02:00","version":"1.2.0"} {"caller":"etcdserver/server.go:2137","index":641,"message":"Applying entry",,"service_id":"etcd","severity":"debug","term":4,"timestamp":"2023-05-19T09:39:38.627+02:00","type":"EntryNormal","version":"1.2.0"} {"caller":"etcdserver/server.go:2196","consistent-index":640,"entry-index":641,"message":"apply entry normal",,"service_id":"etcd","severity":"debug","should-applyV3":true,"timestamp":"2023-05-19T09:39:38.627+02:00","version":"1.2.0"} {"caller":"etcdserver/server.go:2223","message":"applyEntryNormal",,"raftReq":"header:<ID:7996853840735518924 username:\"eric-sec-key-management\" auth_revision:9 > put:<key:\"/shelter/logical/73265fc9-2e14-b494-6b01-e8d27ed77476/certs/35-de-e4-41-35-9c-16-40-3b-e5-47-23-50-a0-98-a1-d0-7d-a7-0b\" value:\"\\000\\000\\000\\001\\002\\244\\2268\\030!1n\\032\\034\\373$@~\\273\\325lbz\\007\\016fX-\\215\\344^\\010i\\304ir\\272\\022\\024c\\355n\\253\\0165\\322\\323V\\300\\\"Di\\247\\350\\332Z\\367#(\\202\\3717\\010w\\273\\010\\031\\033\\331\\203\\265wJ[Q\\214\\205\\217.\\263(Um3\\306\\034^\\305\\374\\315\\\"\\330\\030\\222wP \\200\\336\\245\\324\\316\\220 O\\263\\347~\\325a{%\\374\\332\\266\\212\\363<\\005\\262\\345\\324\\306-\\270\\355D\\375K\\0318\\241V]\\330bcNn\\305\\260\\350\\031n/\\007\\3752(N[\\230m2\\324\\226WZ\\342\\244\\025Y@C\\376\\260\\234\\330\\212\\324\\2046on^\\260\\256\\376%\\203\\313\\263\\n\\035Zb\\230\\332\\010\\032a\\336\\\\\\032\\004\\016\\352\\215_\\234M\\260\\330\\2447\\0344B!\\371\\315y\\223\\215\\356Q%\\324 \\204'\\214\\213M\\372\\2338j!\\267/\\030\\253\\271\\263v\\036m\\376<\\266+bvh\\302\\242{48\\351\\277#<\\336\\333\\023\\272\\021*r\\026\\201\\3524\\026\\341_\\212d\\265\\262\\242\\353\\225\\237\\357\\213y \\255\\003\\006\\003\\304\\236\\364\\322\\310\\353\\231\\027J\\023\\232\\307\\021\\217;(}\\021(^\\232\\002\\356\\211\\363B\\261\\364\\274q^>\\314\\026\\265\\240|\\254h)\\275\\004\\265\\217\\302\\246\\177\\372\\253\\215TJ:\\2133`\\326\\366TB5\\332\\031\\352o`\\027\\364\\365=\\217G\\3029\\366\\306_0\\342Gy\\227\\372\\333\\034\\347\\316\\360X\\031Q\\tY\\237\\261\\200\\317\\005\\231\\206\\026_\\033\\366&\\035\\231|6\\373\\\\\\275\\006\\242'9\\032\\ncH\\236\\277\\361\\233\\247F\\251\\032\\264(K6\\231\\372)]\\273\\2005g\\002\\211.\\342\\030\\265g\\036\\233H2\\237I\" > ","service_id":"etcd","severity":"debug","timestamp":"2023-05-19T09:39:38.627+02:00","version":"1.2.0"} {"caller":"v3rpc/interceptor.go:182","message":"request stats",,"remote":"10.255.55.80:58520","request content":"key:\"/shelter/logical/73265fc9-2e14-b494-6b01-e8d27ed77476/certs/35-de-e4-41-35-9c-16-40-3b-e5-47-23-50-a0-98-a1-d0-7d-a7-0b\" value_size:437 ","request count":1,"request size":561,"response count":0,"response size":29,"response type":"/etcdserverpb.KV/Put","service_id":"etcd","severity":"debug","start time":"2023-05-19T09:39:38.545+0200","time spent":"85.170077ms","timestamp":"2023-05-19T09:39:38.630+02:00","version":"1.2.0"} {"caller":"auth/store.go:1146","common-name":"eric-sec-key-management","message":"found command name",,"revision":9,"service_id":"etcd","severity":"debug","timestamp":"2023-05-19T09:39:38.631+02:00","user-name":"eric-sec-key-management","version":"1.2.0"} {"caller":"v3rpc/interceptor.go:182","message":"request stats",,"remote":"10.255.55.80:58520","request content":"key:\"/shelter/logical/73265fc9-2e14-b494-6b01-e8d27ed77476/config/issuer/\" range_end:\"/shelter/logical/73265fc9-2e14-b494-6b01-e8d27ed77476/config/issuer0\" ","request count":0,"request size":140,"response count":1,"response size":1680,"response type":"/etcdserverpb.KV/Range","service_id":"etcd","severity":"debug","start time":"2023-05-19T09:39:38.631+0200","time spent":"208.457µs","timestamp":"2023-05-19T09:39:38.631+02:00","version":"1.2.0"} {"caller":"auth/store.go:1146","common-name":"eric-sec-key-management","message":"found command name",,"revision":9,"service_id":"etcd","severity":"debug","timestamp":"2023-05-19T09:39:38.728+02:00","user-name":"eric-sec-key-management","version":"1.2.0"} {"caller":"v3rpc/interceptor.go:182","message":"request stats",,"remote":"10.255.55.80:58520","request content":"key:\"/shelter/logical/73265fc9-2e14-b494-6b01-e8d27ed77476/crls/config\" ","request count":0,"request size":67,"response count":0,"response size":29,"response type":"/etcdserverpb.KV/Range","service_id":"etcd","severity":"debug","start time":"2023-05-19T09:39:38.727+0200","time spent":"128.812µs","timestamp":"2023-05-19T09:39:38.728+02:00","version":"1.2.0"} {"caller":"auth/store.go:1146","common-name":"eric-sec-key-management","message":"found command name",,"revision":9,"service_id":"etcd","severity":"debug","timestamp":"2023-05-19T09:39:38.728+02:00","user-name":"eric-sec-key-management","version":"1.2.0"} {"caller":"v3rpc/interceptor.go:182","message":"request stats",,"remote":"10.255.55.80:58520","request content":"key:\"/shelter/logical/73265fc9-2e14-b494-6b01-e8d27ed77476/revoked/\" range_end:\"/shelter/logical/73265fc9-2e14-b494-6b01-e8d27ed77476/revoked0\" ","request count":0,"request size":128,"response count":0,"response size":29,"response type":"/etcdserverpb.KV/Range","service_id":"etcd","severity":"debug","start time":"2023-05-19T09:39:38.728+0200","time spent":"84.449µs","timestamp":"2023-05-19T09:39:38.826+02:00","version":"1.2.0"} {"caller":"auth/store.go:1146","common-name":"eric-sec-key-management","message":"found command name",,"revision":9,"service_id":"etcd","severity":"debug","timestamp":"2023-05-19T09:39:38.827+02:00","user-name":"eric-sec-key-management","version":"1.2.0"} {"caller":"v3rpc/interceptor.go:182","message":"request stats",,"remote":"10.255.55.80:58520","request content":"key:\"/shelter/logical/73265fc9-2e14-b494-6b01-e8d27ed77476/config/crl\" ","request count":0,"request size":66,"response count":0,"response size":29,"response type":"/etcdserverpb.KV/Range","service_id":"etcd","severity":"debug","start time":"2023-05-19T09:39:38.728+0200","time spent":"54.546µs","timestamp":"2023-05-19T09:39:38.827+02:00","version":"1.2.0"} {"caller":"auth/store.go:1146","common-name":"eric-sec-key-management","message":"found command name",,"revision":9,"service_id":"etcd","severity":"debug","timestamp":"2023-05-19T09:39:38.827+02:00","user-name":"eric-sec-key-management","version":"1.2.0"} {"caller":"etcdserver/server.go:2134","message":"Applying entries",,"num-entries":1,"service_id":"etcd","severity":"debug","timestamp":"2023-05-19T09:39:38.827+02:00","version":"1.2.0"} {"caller":"etcdserver/server.go:2137","index":642,"message":"Applying entry",,"service_id":"etcd","severity":"debug","term":4,"timestamp":"2023-05-19T09:39:38.827+02:00","type":"EntryNormal","version":"1.2.0"} {"caller":"etcdserver/server.go:2196","consistent-index":641,"entry-index":642,"message":"apply entry normal",,"service_id":"etcd","severity":"debug","should-applyV3":true,"timestamp":"2023-05-19T09:39:38.827+02:00","version":"1.2.0"} {"caller":"etcdserver/server.go:2223","message":"applyEntryNormal",,"raftReq":"header:<ID:7996853840735518929 username:\"eric-sec-key-management\" auth_revision:9 > put:<key:\"/shelter/logical/73265fc9-2e14-b494-6b01-e8d27ed77476/crls/d266d235-2131-b396-d3bf-8488d390bb70\" value:\"\\000\\000\\000\\001\\002i\\302\\214qy]\\326\\007\\262\\205\\014\\364HXn\\332\\265~\\321\\373\\247\\351\\216~!\\347\\274\\260\\0143$\\220af\\313\\213b\\004tY\\370H\\010\\\"\\213\\250\\205;1\\322?\\215~\\261LF\\037\\356\\377\\366\\177\\032.O\\347SZ%\\255l\\237R]+\\024\\231>\\263\\205\\321BU\\252|\\220\\262k\\\"\\306\\312\\021\\232#C\\346\\220Q|}\\253\\254Cu\\217\\354#D\\357t\\036\\257U\\016\\352\\356\\307\\271\\337\\267\\227\\031\\025\\3464\\2650\\025.d\\302Y\\263\\2720\\306's\\014\\202\\222\\033\\267\\314)\\3518\\344\\314\\333\\034\\325S\\r\\013\\214\\010c\\31118+j\\376\\227[\\234b\\3219U\\014\\345\\244\\007\\321\\031\\2630\\375\\252[\\024\\337\\026dH\\220\\212!\\337\\206\\207:\\251\\207zMFuB\\227\\247\\0245^\\304b\\311f{K\\237\\361$\\302\\360\\313\\224A\\021\\311f\\323\\037E|<\\372\\341\\261\\n\\323\\302V\\267\" > ","service_id":"etcd","severity":"debug","timestamp":"2023-05-19T09:39:38.827+02:00","version":"1.2.0"} {"caller":"v3rpc/interceptor.go:182","message":"request stats",,"remote":"10.255.55.80:58520","request content":"key:\"/shelter/logical/73265fc9-2e14-b494-6b01-e8d27ed77476/crls/d266d235-2131-b396-d3bf-8488d390bb70\" value_size:240 ","request count":1,"request size":340,"response count":0,"response size":29,"response type":"/etcdserverpb.KV/Put","service_id":"etcd","severity":"debug","start time":"2023-05-19T09:39:38.729+0200","time spent":"1.075405ms","timestamp":"2023-05-19T09:39:38.827+02:00","version":"1.2.0"} {"caller":"auth/store.go:1146","common-name":"eric-sec-key-management","message":"found command name",,"revision":9,"service_id":"etcd","severity":"debug","timestamp":"2023-05-19T09:39:38.827+02:00","user-name":"eric-sec-key-management","version":"1.2.0"} {"message":"[ /run/sec/cas/pmca ] Caught Certificate renewed event.",,"service_id":"etcd","severity":"info","timestamp":"2023-05-19T09:39:38.827+02:00","version":"1.2.0"} {"caller":"etcdserver/server.go:2134","message":"Applying entries",,"num-entries":1,"service_id":"etcd","severity":"debug","timestamp":"2023-05-19T09:39:38.828+02:00","version":"1.2.0"} {"caller":"etcdserver/server.go:2137","index":643,"message":"Applying entry",,"service_id":"etcd","severity":"debug","term":4,"timestamp":"2023-05-19T09:39:38.828+02:00","type":"EntryNormal","version":"1.2.0"} {"caller":"etcdserver/server.go:2196","consistent-index":642,"entry-index":643,"message":"apply entry normal",,"service_id":"etcd","severity":"debug","should-applyV3":true,"timestamp":"2023-05-19T09:39:38.828+02:00","version":"1.2.0"} {"caller":"etcdserver/server.go:2223","message":"applyEntryNormal",,"raftReq":"header:<ID:7996853840735518930 username:\"eric-sec-key-management\" auth_revision:9 > put:<key:\"/shelter/logical/73265fc9-2e14-b494-6b01-e8d27ed77476/crls/config\" value:\"\\000\\000\\000\\001\\002p\\216Z?N)\\321\\204s.\\342\\207\\023\\245\\2437=\\203wr \\205\\335\\241\\361\\270:\\333\\341\\034 A\\261$O^\\374L\\213\\300\\361\\\\\\353*Q\\240\\022\\210\\353~\\262I\\315\\212\\200pnOY\\227jb\\0011\\007\\241=i\\177J\\0319\\256JF\\206\\357\\374f\\240\\307|\\251\\271tz\\0308V$\\366\\201g\\375g\\332\\000\\014Kb\\304\\334s\\177\\364.\\241*\\260*\\214\\207F\\030\\177\\323\\355\\213@{\\0028\\311\\352\\201\\316r(\\332\\226\\311\\265\\372\\264\\027\\033\\226\\304\\266\\263\\025 Ih\\204/\\307\\241R\\267\\213\\256\\014\\030X\\236&\\004\\325i\\004\\313\\014\\000\\202~\\240z\\025YV\\313^)X!Cs\\300z\\227\\235f^j\\234K\\323\\373m\" > ","service_id":"etcd","severity":"debug","timestamp":"2023-05-19T09:39:38.828+02:00","version":"1.2.0"} {"caller":"v3rpc/interceptor.go:182","message":"request stats",,"remote":"10.255.55.80:58520","request content":"key:\"/shelter/logical/73265fc9-2e14-b494-6b01-e8d27ed77476/crls/config\" value_size:195 ","request count":1,"request size":265,"response count":0,"response size":29,"response type":"/etcdserverpb.KV/Put","service_id":"etcd","severity":"debug","start time":"2023-05-19T09:39:38.827+0200","time spent":"1.158658ms","timestamp":"2023-05-19T09:39:38.828+02:00","version":"1.2.0"} {"message":"Restarting ETCD while reloading certificates",,"service_id":"etcd","severity":"info","timestamp":"2023-05-19T09:39:38.928+02:00","version":"1.2.0"} {"message":"Killing ETCD process",,"service_id":"etcd","severity":"info","timestamp":"2023-05-19T09:39:38.929+02:00","version":"1.2.0"} {"caller":"osutil/interrupt_unix.go:64","message":"received signal; shutting down",,"service_id":"etcd","severity":"info","signal":"terminated","timestamp":"2023-05-19T09:39:38.931+02:00","version":"1.2.0"} {"advertise-client-urls":["https://etcd-0.etcd.keep-cerberus-04:2379"],"advertise-peer-urls":["https://etcd-0.etcd-peer.keep-cerberus-04.svc.cluster.local:2380"],"caller":"embed/etcd.go:371","data-dir":"/data","message":"closing etcd server",,"name":"etcd-0","service_id":"etcd","severity":"info","timestamp":"2023-05-19T09:39:38.932+02:00","version":"1.2.0"} {"caller":"embed/config_logging.go:169","error":"read tcp 10.255.31.131:2379->172.30.22.62:55386: use of closed network connection","message":"rejected connection",,"remote-addr":"172.30.22.62:55386","server-name":"etcd.keep-cerberus-04","service_id":"etcd","severity":"warning","timestamp":"2023-05-19T09:39:38.932+02:00","version":"1.2.0"} {"caller":"zapgrpc/zapgrpc.go:174","message":"[core] Subchannel Connectivity change to IDLE",,"service_id":"etcd","severity":"info","timestamp":"2023-05-19T09:39:38.932+02:00","version":"1.2.0"} {"caller":"zapgrpc/zapgrpc.go:174","message":"[transport] transport: loopyWriter.run returning. connection error: desc = \"transport is closing\"",,"service_id":"etcd","severity":"info","timestamp":"2023-05-19T09:39:38.932+02:00","version":"1.2.0"} {"caller":"zapgrpc/zapgrpc.go:174","message":"[core] pickfirstBalancer: UpdateSubConnState: 0xc000496720, {IDLE <nil>}",,"service_id":"etcd","severity":"info","timestamp":"2023-05-19T09:39:38.932+02:00","version":"1.2.0"} {"caller":"zapgrpc/zapgrpc.go:174","message":"[core] Channel Connectivity change to IDLE",,"service_id":"etcd","severity":"info","timestamp":"2023-05-19T09:39:38.932+02:00","version":"1.2.0"} {"caller":"etcdserver/server.go:1465","current-leader-member-id":"a93f9ecf02976efa","local-member-id":"a93f9ecf02976efa","message":"skipped leadership transfer for single voting member cluster",,"service_id":"etcd","severity":"info","timestamp":"2023-05-19T09:39:38.933+02:00","version":"1.2.0"} {"caller":"zapgrpc/zapgrpc.go:174","message":"[core] Channel Connectivity change to SHUTDOWN",,"service_id":"etcd","severity":"info","timestamp":"2023-05-19T09:39:38.933+02:00","version":"1.2.0"} {"caller":"zapgrpc/zapgrpc.go:174","message":"[core] Subchannel Connectivity change to SHUTDOWN",,"service_id":"etcd","severity":"info","timestamp":"2023-05-19T09:39:38.933+02:00","version":"1.2.0"} {"address":"[::]:2380","caller":"embed/etcd.go:566","message":"stopping serving peer traffic",,"service_id":"etcd","severity":"info","timestamp":"2023-05-19T09:39:38.935+02:00","version":"1.2.0"} {"address":"[::]:2380","caller":"embed/etcd.go:571","message":"stopped serving peer traffic",,"service_id":"etcd","severity":"info","timestamp":"2023-05-19T09:39:38.935+02:00","version":"1.2.0"} {"advertise-client-urls":["https://etcd-0.etcd.keep-cerberus-04:2379"],"advertise-peer-urls":["https://etcd-0.etcd-peer.keep-cerberus-04.svc.cluster.local:2380"],"caller":"embed/etcd.go:373","data-dir":"/data","message":"closed etcd server",,"name":"etcd-0","service_id":"etcd","severity":"info","timestamp":"2023-05-19T09:39:38.935+02:00","version":"1.2.0"} {"message":"/usr/local/bin/scripts/etcd_runnerBootstrapDisabled.sh: line 87: 79 Terminated /usr/local/bin/etcd > ${FIFO_DIR}/etcd.fifo",,"service_id":"etcd","severity":"info","timestamp":"2023-05-19T09:39:39.128+02:00","version":"1.2.0"} {"message":"/usr/local/bin/scripts/entrypoint.sh: line 64: kill: (18) - No such process",,"service_id":"etcd","severity":"info","timestamp":"2023-05-19T09:39:43.241+02:00","version":"1.2.0"} {"message":"/usr/local/bin/scripts/entrypoint.sh: line 64: kill: (18) - No such process",,"service_id":"etcd","severity":"info","timestamp":"2023-05-19T09:39:48.244+02:00","version":"1.2.0"} {"message":"Liveness probe failed: + '[' -e /data/etcd.liveness ']'\n+ grep -qi alive /data/etcd.liveness\n+ exit 1\n exit status 1",,"service_id":"etcd","severity":"warning","timestamp":"2023-05-19T09:39:51.935+02:00","version":"1.2.0"} {"message":"/usr/local/bin/scripts/entrypoint.sh: line 64: kill: (18) - No such process",,"service_id":"etcd","severity":"info","timestamp":"2023-05-19T09:39:53.248+02:00","version":"1.2.0"} {"message":"/usr/local/bin/scripts/entrypoint.sh: line 64: kill: (18) - No such process",,"service_id":"etcd","severity":"info","timestamp":"2023-05-19T09:39:58.251+02:00","version":"1.2.0"} {"message":"Liveness probe failed: + '[' -e /data/etcd.liveness ']'\n+ grep -qi alive /data/etcd.liveness\n+ exit 1\n exit status 1",,"service_id":"etcd","severity":"warning","timestamp":"2023-05-19T09:40:01.936+02:00","version":"1.2.0"} {"message":"/usr/local/bin/scripts/entrypoint.sh: line 64: kill: (18) - No such process",,"service_id":"etcd","severity":"info","timestamp":"2023-05-19T09:40:03.254+02:00","version":"1.2.0"} {"message":"/usr/local/bin/scripts/entrypoint.sh: line 64: kill: (18) - No such
rahulbapumore commented 1 year ago

@chaochn47 @ahrtr

rahulbapumore commented 1 year ago

please help

serathius commented 1 year ago

https://github.com/etcd-io/etcd/blob/main/Documentation/contributor-guide/triage_issues.md#support-requests

jmhbnz commented 1 year ago

Hi @rahulbapumore - Please take a look at the support request guidelines linked above. In future please raise any support requests as discussions rather than as bug issues.

Additionally - please don't ping individuals asking for help within moments of an issue being created. Remember that the contributors to projects like etcd are generally volunteers working on projects in their spare time who don't owe you anything.

Moving onto the issue itself, I can't see any clear reason for the pod termination in the logs you provided, only that a shutdown signal was received. Can you please provide the kubernetes events for the etcd pods as these may explain why the pods are being killed.

Edit: Can you also please provide an untruncated log for {"message":"/usr/local/bin/scripts/entrypoint.sh: line 64: kill: (18) - No such that is mentioned at the bottom.

rahulbapumore commented 1 year ago

Hi @jmhbnz , Sorry for inconvenience.

Actually below message is coming because we keep checking in loop whether other process exist or not, and that process is exited already because etcd is killed automatically. So this line can be ignored. its just zero signal passed to kill command to check whether process exists or not. so its printing no such process in the logs.

{"message":"/usr/local/bin/scripts/entrypoint.sh: line 64: kill: (18) - No such process

jmhbnz commented 1 year ago

Actually below message is coming because we keep checking in loop whether other process exist or not, and that process is exited already because etcd is killed automatically. So this line can be ignored. its just zero signal passed to kill command to check whether process exists or not. so its printing no such process in the logs.

Thanks for the clarification, ok so as I mentioned above can you please provide the kubernetes events for the etcd pod? Perhaps the pod yaml as well, that might have some clues.

Also - can you reproduce the error when running etcd outside of kubernetes, or does it only happen within your kubernetes cluster?

rahulbapumore commented 1 year ago

Hi @jmhbnz , This was reproduced once, that deployment is no longer there. We tried to reproduce the issue, but it doesnot happen now😕

jmhbnz commented 1 year ago

Thanks for the update, as we can't reproduce I will close this issue.