search

lf-edge / eve

EVE is Edge Virtualization Engine
https://www.lfedge.org/projects/eve/
Apache License 2.0
468 stars 158 forks source link

chronyd.pid watchdog on recent master #4014

Closed eriknordmark closed 4 days ago

eriknordmark commented 1 week ago

Describe the bug

I've been running various ztests in a loop to verify that the fix in #4012 (which is looking good) but when doing that I occasionally (more than one time) see chronyd watchdog as in:

AssertionError: Abnormal reboot Watchdog report for IMGA EVE version 0.0.0-bugfix_fix-memory-monitor-visible-issues-acf6861a-dirty-2024-06-26.16.48-kvm-amd64 at 2024-06-26T21:40:31+00:00: 2 /run/chronyd.pid

Does chronyd syslog stuff? If so there might be something useful in kibana about it faiing?

Recent release

running on current master + 4012.

OhmSpectator commented 1 week ago

@eriknordmark, on which device are you running this?

OhmSpectator commented 1 week ago

Found, sc-supermicro-zc1. Will grab some logs and bring here

OhmSpectator commented 1 week ago

I see this, for example:

shutting down the system because of error 254 = 'unconditional hard reset requested'

Wasn't the machine just shut down?

OhmSpectator commented 1 week ago

Ok, I see also this, 3 minutes before the previous message I posted:


Jun 26, 2024 @ 23:59:10.036086748 | ----- eve pressure event -----
-- | --

and here is more log


Jun 26, 2024 @ 23:58:57.776817719 | zedbox | Read 32 bytes
-- | -- | --

  | Jun 26, 2024 @ 23:58:57.777024823 | zedbox | response head: &{Version:6 PKTType:reply Res1:0 Res2:0 Command:14 Reply:2 Status:SUCCESS Pad1:0 Pad2:0 Pad3:0 Sequence:2 Pad4:0 Pad5:0}

  | Jun 26, 2024 @ 23:58:57.777101210 | zedbox | response data: &{NSources:1}

  | Jun 26, 2024 @ 23:58:57.778096108 | zedbox | Read 76 bytes

  | Jun 26, 2024 @ 23:58:57.778273837 | zedbox | response head: &{Version:6 PKTType:reply Res1:0 Res2:0 Command:15 Reply:3 Status:SUCCESS Pad1:0 Pad2:0 Pad3:0 Sequence:3 Pad4:0 Pad5:0}

  | Jun 26, 2024 @ 23:58:57.778382852 | zedbox | response data: &{IPAddr:{IP:[10 216 0 254 0 0 0 0 0 0 0 0 0 0 0 0] Family:1 Pad:0} Poll:6 Stratum:4 State:unreach Mode:client Flags:0 Reachability:1 SinceSample:63 OrigLatestMeas:-152828849 LatestMeas:-152828849 LatestMeasErr:-154756954}

  | Jun 26, 2024 @ 23:58:57.779011851 | zedbox | Read 152 bytes

  | Jun 26, 2024 @ 23:58:57.779175664 | zedbox | response head: &{Version:6 PKTType:reply Res1:0 Res2:0 Command:57 Reply:16 Status:SUCCESS Pad1:0 Pad2:0 Pad3:0 Sequence:4 Pad4:0 Pad5:0}

  | Jun 26, 2024 @ 23:58:57.779299201 | zedbox | response data: &{RemoteAddr:{IP:[10 216 0 254 0 0 0 0 0 0 0 0 0 0 0 0] Family:1 Pad:0} LocalAddr:{IP:[10 216 0 14 0 0 0 0 0 0 0 0 0 0 0 0] Family:1 Pad:0} RemotePort:123 Leap:0 Version:4 Mode:4 Stratum:4 Poll:6 Precision:-29 RootDelay:-121430016 RootDispersion:0 RefID:2549880062 RefTime:{SecHigh:0 SecLow:1719439017 Nsec:183766841} Offset:-149161039 PeerDelay:-321045782 PeerDispersion:-621898473 ResponseTime:0 JitterAsymmetry:0 Flags:1023 TXTssChar:75 RXTssChar:75 TotalTXCount:1 TotalRXCount:1 TotalValidCount:1 Reserved:[4294967295 4294967295 4294967295 4294967295]}

  | Jun 26, 2024 @ 23:58:57.779910902 | zedbox | Read 284 bytes

  | Jun 26, 2024 @ 23:58:57.780088616 | zedbox | response head: &{Version:6 PKTType:reply Res1:0 Res2:0 Command:65 Reply:19 Status:SUCCESS Pad1:0 Pad2:0 Pad3:0 Sequence:5 Pad4:0 Pad5:0}

  | Jun 26, 2024 @ 23:58:57.781451850 | zedbox | response data: &{Name:[49 48 46 50 49 54 46 48 46 50 53 52 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0]}

  | Jun 26, 2024 @ 23:58:58.203449422 | zedagent | GetContextForAllIntfFunctions: maxWaitDuration 360000000000 is more than limit, will use 240000000000

  | Jun 26, 2024 @ 23:59:03.095071912 | zedagent | GetContextForAllIntfFunctions: maxWaitDuration 360000000000 is more than limit, will use 240000000000

  | Jun 26, 2024 @ 23:59:04.424693605 | domainmgr | Domain metric modify

  | Jun 26, 2024 @ 23:59:04.425211831 | domainmgr | Host memory modify

  | Jun 26, 2024 @ 23:59:04.426306818 | zedmanager | Host memory modify

  | Jun 26, 2024 @ 23:59:04.426431834 | zedmanager | handleHostMemoryImpl(global) done

  | Jun 26, 2024 @ 23:59:04.426742439 | zedagent | Host memory modify

  | Jun 26, 2024 @ 23:59:04.427380867 | zedagent | Domain metric modify

  | Jun 26, 2024 @ 23:59:04.427798571 | watcher | Host memory modify

  | Jun 26, 2024 @ 23:59:04.428217641 | watcher | handleHostMemoryModify:

  | Jun 26, 2024 @ 23:59:04.770413787 | zedrouter | Network metrics modify

  | Jun 26, 2024 @ 23:59:04.774003347 | zedrouter | Network instance metrics modify

  | Jun 26, 2024 @ 23:59:04.778537208 | zedagent | Network metrics modify

  | Jun 26, 2024 @ 23:59:04.786096184 | zedagent | Network instance metrics modify

  | Jun 26, 2024 @ 23:59:06.177882209 | zedagent | GetContextForAllIntfFunctions: maxWaitDuration 360000000000 is more than limit, will use 240000000000

  | Jun 26, 2024 @ 23:59:07.144317016 | zedbox | SIGUSR2 triggered memory info:

  | Jun 26, 2024 @ 23:59:07.158787899 | zedbox | alloc 21 sites len 21

  | Jun 26, 2024 @ 23:59:07.159086763 | zedbox | alloc 1056768 bytes 1 objects total 224034816/212 at: /pillar/vendor/google.golang.org/grpc/shared_buffer_pool.go[150] google.golang.org/grpc.nopBufferPool.Get /pillar/vendor/google.golang.org/grpc/rpc_util.go[616] google.golang.org/grpc.(*parser).recvMsg /pillar/vendor/google.golang.org/grpc/rpc_util.go[734] google.golang.org/grpc.recvAndDecompress /pillar/vendor/google.golang.org/grpc/rpc_util.go[800] google.golang.org/grpc.recv /pillar/vendor/google.golang.org/grpc/stream.go[1086] google.golang.org/grpc.(*csAttempt).recvMsg /pillar/vendor/google.golang.org/grpc/stream.go[929] google.golang.org/grpc.(*clientStream).RecvMsg.func1 /pillar/vendor/google.golang.org/grpc/stream.go[778] google.golang.org/grpc.(*clientStream).withRetry /pillar/vendor/google.golang.org/grpc/stream.go[928] google.golang.org/grpc.(*clientStream).RecvMsg /pillar/vendor/github.com/containerd/containerd/api/services/content/v1/content_grpc.pb.go[174] github.com/containerd/containerd/api/services/content/v1.(*contentReadClient).Recv /pillar/vendor/github.com/containerd/containerd/content/proxy/content_reader.go[57] github.com/containerd/containerd/content/proxy.(*remoteReaderAt).ReadAt /usr/lib/go/src/io/io.go[513] io.(*SectionReader).Read /pillar/vendor/github.com/lf-edge/edge-containers/pkg/resolver/containerd.go[133] github.com/lf-edge/edge-containers/pkg/resolver.(*containerdReader).Read /usr/lib/go/src/io/io.go[332] io.ReadAtLeast /pillar/vendor/github.com/containerd/containerd/content/helpers.go[309] github.com/containerd/containerd/content.copyWithBuffer /pillar/vendor/github.com/containerd/containerd/content/helpers.go[167] github.com/containerd/containerd/content.Copy /pillar/vendor/oras.land/oras-go/pkg/oras/copy.go[126] oras.land/oras-go/pkg/oras.transferContent.func2.1 /pillar/vendor/oras.land/oras-go/pkg/oras/copy.go[137] oras.land/oras-go/pkg/oras.transferContent.func3 /pillar/vendor/github.com/containerd/containerd/images/handlers.go[59] github.com/containerd/containerd/images.HandlerFunc.Handle /pillar/vendor/github.com/containerd/containerd/images/handlers.go[69] github.com/containerd/containerd/images.Handlers.func1 /pillar/vendor/github.com/containerd/containerd/images/handlers.go[59] github.com/containerd/containerd/images.HandlerFunc.Handle /pillar/vendor/oras.land/oras-go/pkg/oras/opts.go[104] oras.land/oras-go/pkg/oras.dispatchBFS /pillar/vendor/oras.land/oras-go/pkg/oras/copy.go[151] oras.land/oras-go/pkg/oras.transferContent /pillar/vendor/oras.land/oras-go/pkg/oras/copy.go[79] oras.land/oras-go/pkg/oras.Copy /pillar/vendor/github.com/lf-edge/edge-containers/pkg/registry/pull.go[69] github.com/lf-edge/edge-containers/pkg/registry.(*Puller).Pull /pillar/zboot/zboot.go[445] github.com/lf-edge/eve/pkg/pillar/zboot.WriteToPartition /pillar/cmd/baseosmgr/worker.go[60] github.com/lf-edge/eve/pkg/pillar/cmd/baseosmgr.installWorker /pillar/worker/worker.go[145] github.com/lf-edge/eve/pkg/pillar/worker.(*Single).processWork /usr/lib/go/src/runtime/asm_amd64.s[1598] runtime.goexit

  | Jun 26, 2024 @ 23:59:07.169300179 | zedbox | alloc 1048576 bytes 1 objects total 220200960/210 at: /pillar/vendor/google.golang.org/protobuf/internal/impl/codec_gen.go[5497] google.golang.org/protobuf/internal/impl.consumeBytesNoZero /pillar/vendor/google.golang.org/protobuf/internal/impl/decode.go[148] google.golang.org/protobuf/internal/impl.(*MessageInfo).unmarshalPointer /pillar/vendor/google.golang.org/protobuf/internal/impl/decode.go[66] google.golang.org/protobuf/internal/impl.(*MessageInfo).unmarshal /pillar/vendor/google.golang.org/protobuf/proto/decode.go[105] google.golang.org/protobuf/proto.UnmarshalOptions.unmarshal /pillar/vendor/google.golang.org/protobuf/proto/decode.go[77] google.golang.org/protobuf/proto.UnmarshalOptions.UnmarshalState /pillar/vendor/github.com/golang/protobuf/proto/wire.go[67] github.com/golang/protobuf/proto.UnmarshalMerge /pillar/vendor/github.com/golang/protobuf/proto/wire.go[58] github.com/golang/protobuf/proto.Unmarshal /pillar/vendor/google.golang.org/grpc/encoding/proto/proto.go[53] google.golang.org/grpc/encoding/proto.codec.Unmarshal /pillar/vendor/google.golang.org/grpc/rpc_util.go[804] google.golang.org/grpc.recv /pillar/vendor/google.golang.org/grpc/stream.go[1086] google.golang.org/grpc.(*csAttempt).recvMsg /pillar/vendor/google.golang.org/grpc/stream.go[929] google.golang.org/grpc.(*clientStream).RecvMsg.func1 /pillar/vendor/google.golang.org/grpc/stream.go[778] google.golang.org/grpc.(*clientStream).withRetry /pillar/vendor/google.golang.org/grpc/stream.go[928] google.golang.org/grpc.(*clientStream).RecvMsg /pillar/vendor/github.com/containerd/containerd/api/services/content/v1/content_grpc.pb.go[174] github.com/containerd/containerd/api/services/content/v1.(*contentReadClient).Recv /pillar/vendor/github.com/containerd/containerd/content/proxy/content_reader.go[57] github.com/containerd/containerd/content/proxy.(*remoteReaderAt).ReadAt /usr/lib/go/src/io/io.go[513] io.(*SectionReader).Read /pillar/vendor/github.com/lf-edge/edge-containers/pkg/resolver/containerd.go[133] github.com/lf-edge/edge-containers/pkg/resolver.(*containerdReader).Read /usr/lib/go/src/io/io.go[332] io.ReadAtLeast /pillar/vendor/github.com/containerd/containerd/content/helpers.go[309] github.com/containerd/containerd/content.copyWithBuffer /pillar/vendor/github.com/containerd/containerd/content/helpers.go[167] github.com/containerd/containerd/content.Copy /pillar/vendor/oras.land/oras-go/pkg/oras/copy.go[126] oras.land/oras-go/pkg/oras.transferContent.func2.1 /pillar/vendor/oras.land/oras-go/pkg/oras/copy.go[137] oras.land/oras-go/pkg/oras.transferContent.func3 /pillar/vendor/github.com/containerd/containerd/images/handlers.go[59] github.com/containerd/containerd/images.HandlerFunc.Handle /pillar/vendor/github.com/containerd/containerd/images/handlers.go[69] github.com/containerd/containerd/images.Handlers.func1 /pillar/vendor/github.com/containerd/containerd/images/handlers.go[59] github.com/containerd/containerd/images.HandlerFunc.Handle /pillar/vendor/oras.land/oras-go/pkg/oras/opts.go[104] oras.land/oras-go/pkg/oras.dispatchBFS /pillar/vendor/oras.land/oras-go/pkg/oras/copy.go[151] oras.land/oras-go/pkg/oras.transferContent /pillar/vendor/oras.land/oras-go/pkg/oras/copy.go[79] oras.land/oras-go/pkg/oras.Copy /pillar/vendor/github.com/lf-edge/edge-containers/pkg/registry/pull.go[69] github.com/lf-edge/edge-containers/pkg/registry.(*Puller).Pull /pillar/zboot/zboot.go[445] github.com/lf-edge/eve/pkg/pillar/zboot.WriteToPartition /pillar/cmd/baseosmgr/worker.go[60] github.com/lf-edge/eve/pkg/pillar/cmd/baseosmgr.installWorker /pillar/worker/worker.go[145] github.com/lf-edge/eve/pkg/pillar/worker.(*Single).processWork

  | Jun 26, 2024 @ 23:59:07.176418942 | zedbox | alloc 1048576 bytes 1 objects total 1048576/1 at: /pillar/vendor/github.com/containerd/containerd/content/helpers.go[41] github.com/containerd/containerd/content.glob..func1 /usr/lib/go/src/sync/pool.go[151] sync.(*Pool).Get /pillar/vendor/github.com/containerd/containerd/content/helpers.go[305] github.com/containerd/containerd/content.copyWithBuffer /pillar/vendor/github.com/containerd/containerd/content/helpers.go[167] github.com/containerd/containerd/content.Copy /pillar/vendor/github.com/containerd/containerd/content/helpers.go[103] github.com/containerd/containerd/content.WriteBlob /pillar/containerd/containerd.go[179] github.com/lf-edge/eve/pkg/pillar/containerd.(*Client).CtrWriteBlob /pillar/cas/containerd.go[335] github.com/lf-edge/eve/pkg/pillar/cas.(*containerdCAS).IngestBlob /pillar/cas/containerd.go[916] github.com/lf-edge/eve/pkg/pillar/cas.(*containerdCAS).IngestBlobsAndCreateImage /pillar/cmd/volumemgr/handlework.go[241] github.com/lf-edge/eve/pkg/pillar/cmd/volumemgr.casIngestWorker /pillar/worker/worker.go[145] github.com/lf-edge/eve/pkg/pillar/worker.(*Single).processWork /usr/lib/go/src/runtime/asm_amd64.s[1598] runtime.goexit

  | Jun 26, 2024 @ 23:59:07.182526623 | zedbox | alloc 40960 bytes 1 objects total 40960/1 at: /usr/lib/go/src/runtime/map.go[363] runtime.makeBucketArray /usr/lib/go/src/runtime/map.go[1051] runtime.hashGrow /usr/lib/go/src/runtime/map_faststr.go[276] runtime.mapassign_faststr /pillar/base/stringmap.go[39] github.com/lf-edge/eve/pkg/pillar/base.(*LockedStringMap).Store /pillar/base/logobjecttypes.go[276] github.com/lf-edge/eve/pkg/pillar/base.InitLogObject /pillar/base/logobjecttypes.go[245] github.com/lf-edge/eve/pkg/pillar/base.NewLogObject /pillar/types/domainmgrtypes.go[335] github.com/lf-edge/eve/pkg/pillar/types.DomainStatus.LogCreate /pillar/pubsub/subscribe.go[258] github.com/lf-edge/eve/pkg/pillar/pubsub.handleModify /pillar/pubsub/subscribe.go[133] github.com/lf-edge/eve/pkg/pillar/pubsub.(*SubscriptionImpl).ProcessChange /pillar/cmd/msrv/msrv.go[570] github.com/lf-edge/eve/pkg/pillar/cmd/msrv.(*Msrv).Run /pillar/cmd/zedrouter/zedrouter.go[190] github.com/lf-edge/eve/pkg/pillar/cmd/zedrouter.Run.func1 /usr/lib/go/src/runtime/asm_amd64.s[1598] runtime.goexit

  | Jun 26, 2024 @ 23:59:07.187723309 | zedbox | alloc 32768 bytes 1 objects total 32768/1 at: /usr/lib/go/src/bufio/bufio.go[57] bufio.NewReaderSize /pillar/vendor/google.golang.org/grpc/internal/transport/http_util.go[401] google.golang.org/grpc/internal/transport.newFramer /pillar/vendor/google.golang.org/grpc/internal/transport/http2_client.go[336] google.golang.org/grpc/internal/transport.newHTTP2Client /pillar/vendor/google.golang.org/grpc/internal/transport/transport.go[615] google.golang.org/grpc/internal/transport.NewClientTransport /pillar/vendor/google.golang.org/grpc/clientconn.go[1393] google.golang.org/grpc.(*addrConn).createTransport /pillar/vendor/google.golang.org/grpc/clientconn.go[1340] google.golang.org/grpc.(*addrConn).tryAllAddrs /pillar/vendor/google.golang.org/grpc/clientconn.go[1275] google.golang.org/grpc.(*addrConn).resetTransport /pillar/vendor/google.golang.org/grpc/clientconn.go[930] google.golang.org/grpc.(*addrConn).connect /usr/lib/go/src/runtime/asm_amd64.s[1598] runtime.goexit

  | Jun 26, 2024 @ 23:59:07.193495379 | zedbox | alloc 16384 bytes 1 objects total 245760/15 at: /usr/lib/go/src/bytes/buffer.go[240] bytes.growSlice /usr/lib/go/src/bytes/buffer.go[142] bytes.(*Buffer).grow /usr/lib/go/src/bytes/buffer.go[170] bytes.(*Buffer).Write /pillar/vendor/google.golang.org/grpc/internal/transport/http2_client.go[1162] google.golang.org/grpc/internal/transport.(*http2Client).handleData /pillar/vendor/google.golang.org/grpc/internal/transport/http2_client.go[1622] google.golang.org/grpc/internal/transport.(*http2Client).reader /usr/lib/go/src/runtime/asm_amd64.s[1598] runtime.goexit

  | Jun 26, 2024 @ 23:59:07.196829078 | zedbox | alloc 9728 bytes 1 objects total 9728/1 at: /usr/lib/go/src/regexp/onepass.go[226] regexp.onePassCopy /usr/lib/go/src/regexp/onepass.go[498] regexp.compileOnePass /usr/lib/go/src/regexp/regexp.go[193] regexp.compile /usr/lib/go/src/regexp/regexp.go[137] regexp.Compile /usr/lib/go/src/regexp/regexp.go[317] regexp.MustCompile /pillar/vendor/github.com/go-playground/validator/v10/regexes.go[127] github.com/go-playground/validator/v10.init /usr/lib/go/src/runtime/proc.go[6506] runtime.doInit /usr/lib/go/src/runtime/proc.go[6483] runtime.doInit /usr/lib/go/src/runtime/proc.go[6483] runtime.doInit /usr/lib/go/src/runtime/proc.go[6483] runtime.doInit /usr/lib/go/src/runtime/proc.go[6483] runtime.doInit /usr/lib/go/src/runtime/proc.go[233] runtime.main /usr/lib/go/src/runtime/asm_amd64.s[1598] runtime.goexit

  | Jun 26, 2024 @ 23:59:07.202271868 | zedbox | alloc 6784 bytes 1 objects total 6784/1 at: /usr/lib/go/src/runtime/malloc.go[1281] reflect.unsafe_NewArray /usr/lib/go/src/reflect/value.go[3091] reflect.MakeSlice /usr/lib/go/src/encoding/json/decode.go[551] encoding/json.(*decodeState).array /usr/lib/go/src/encoding/json/decode.go[364] encoding/json.(*decodeState).value /usr/lib/go/src/encoding/json/decode.go[775] encoding/json.(*decodeState).object /usr/lib/go/src/encoding/json/decode.go[374] encoding/json.(*decodeState).value /usr/lib/go/src/encoding/json/decode.go[181] encoding/json.(*decodeState).unmarshal /usr/lib/go/src/encoding/json/decode.go[108] encoding/json.Unmarshal /pillar/pubsub/util.go[34] github.com/lf-edge/eve/pkg/pillar/pubsub.deepCopy /pillar/pubsub/subscribe.go[266] github.com/lf-edge/eve/pkg/pillar/pubsub.handleModify /pillar/pubsub/subscribe.go[133] github.com/lf-edge/eve/pkg/pillar/pubsub.(*SubscriptionImpl).ProcessChange /pillar/cmd/zedagent/zedagent.go[851] github.com/lf-edge/eve/pkg/pillar/cmd/zedagent.mainEventLoop /pillar/cmd/zedagent/zedagent.go[542] github.com/lf-edge/eve/pkg/pillar/cmd/zedagent.Run /pillar/zedbox/zedbox.go[241] main.startAgentAndDone /usr/lib/go/src/runtime/asm_amd64.s[1598] runtime.goexit

  | Jun 26, 2024 @ 23:59:07.210342448 | zedbox | alloc 5376 bytes 1 objects total 5376/1 at: /pillar/dpcreconciler/linux.go[616] github.com/lf-edge/eve/pkg/pillar/dpcreconciler.(*LinuxDpcReconciler).saveArgs /pillar/dpcreconciler/linux.go[569] github.com/lf-edge/eve/pkg/pillar/dpcreconciler.(*LinuxDpcReconciler).Reconcile /pillar/dpcmanager/dpcmanager.go[275] github.com/lf-edge/eve/pkg/pillar/dpcmanager.(*DpcManager).run /usr/lib/go/src/runtime/asm_amd64.s[1598] runtime.goexit

  | Jun 26, 2024 @ 23:59:07.214832483 | zedbox | alloc 4864 bytes 1 objects total 4864/1 at: /pillar/vendor/github.com/lf-edge/eve-api/go/attest/attest.pb.go[1583] github.com/lf-edge/eve-api/go/attest.init /usr/lib/go/src/runtime/proc.go[6506] runtime.doInit /usr/lib/go/src/runtime/proc.go[6483] runtime.doInit /usr/lib/go/src/runtime/proc.go[6483] runtime.doInit /usr/lib/go/src/runtime/proc.go[233] runtime.main /usr/lib/go/src/runtime/asm_amd64.s[1598] runtime.goexit

  | Jun 26, 2024 @ 23:59:07.219147838 | zedbox | alloc 4096 bytes 1 objects total 4096/1 at: /pillar/vendor/google.golang.org/protobuf/internal/filedesc/desc_lazy.go[324] google.golang.org/protobuf/internal/filedesc.(*Message).unmarshalFull /pillar/vendor/google.golang.org/protobuf/internal/filedesc/desc_lazy.go[166] google.golang.org/protobuf/internal/filedesc.(*File).unmarshalFull /pillar/vendor/google.golang.org/protobuf/internal/filedesc/desc_lazy.go[20] google.golang.org/protobuf/internal/filedesc.(*File).lazyRawInit /pillar/vendor/google.golang.org/protobuf/internal/filedesc/desc.go[142] google.golang.org/protobuf/internal/filedesc.(*File).lazyInitOnce /pillar/vendor/google.golang.org/protobuf/internal/filedesc/desc.go[134] google.golang.org/protobuf/internal/filedesc.(*File).lazyInit /pillar/vendor/google.golang.org/protobuf/internal/filedesc/desc.go[303] google.golang.org/protobuf/internal/filedesc.(*Message).lazyInit /pillar/vendor/google.golang.org/protobuf/internal/filedesc/desc.go[285] google.golang.org/protobuf/internal/filedesc.(*Message).Fields /pillar/vendor/google.golang.org/protobuf/internal/impl/message_reflect.go[57] google.golang.org/protobuf/internal/impl.(*MessageInfo).makeKnownFieldsFunc /pillar/vendor/google.golang.org/protobuf/internal/impl/message_reflect.go[42] google.golang.org/protobuf/internal/impl.(*MessageInfo).makeReflectFuncs /pillar/vendor/google.golang.org/protobuf/internal/impl/message.go[90] google.golang.org/protobuf/internal/impl.(*MessageInfo).initOnce /pillar/vendor/google.golang.org/protobuf/internal/impl/message.go[72] google.golang.org/protobuf/internal/impl.(*MessageInfo).init /pillar/vendor/google.golang.org/protobuf/internal/impl/message_reflect_gen.go[44] google.golang.org/protobuf/internal/impl.(*messageState).Range /pillar/vendor/google.golang.org/protobuf/internal/order/range.go[50] google.golang.org/protobuf/internal/order.RangeFields /pillar/vendor/google.golang.org/protobuf/encoding/prototext/encode.go[179] google.golang.org/protobuf/encoding/prototext.encoder.marshalMessage /pillar/vendor/google.golang.org/protobuf/encoding/prototext/encode.go[138] google.golang.org/protobuf/encoding/prototext.MarshalOptions.marshal /pillar/vendor/google.golang.org/protobuf/encoding/prototext/encode.go[104] google.golang.org/protobuf/encoding/prototext.MarshalOptions.Marshal /pillar/vendor/google.golang.org/protobuf/encoding/prototext/encode.go[96] google.golang.org/protobuf/encoding/prototext.MarshalOptions.Format /pillar/vendor/google.golang.org/protobuf/internal/impl/api_export.go[176] google.golang.org/protobuf/internal/impl.Export.MessageStringOf /pillar/vendor/github.com/lf-edge/eve-api/go/info/info.pb.go[5571] github.com/lf-edge/eve-api/go/info.(*ZInfoDevSW).String /pillar/cmd/zedagent/reportinfo.go[761] github.com/lf-edge/eve/pkg/pillar/cmd/zedagent.addUserSwInfo /pillar/cmd/zedagent/reportinfo.go[351] github.com/lf-edge/eve/pkg/pillar/cmd/zedagent.PublishDeviceInfoToZedCloud.func2 /pillar/cmd/zedagent/reportinfo.go[356] github.com/lf-edge/eve/pkg/pillar/cmd/zedagent.PublishDeviceInfoToZedCloud /pillar/cmd/zedagent/reportinfo.go[64] github.com/lf-edge/eve/pkg/pillar/cmd/zedagent.deviceInfoTask /usr/lib/go/src/runtime/asm_amd64.s[1598] runtime.goexit

  | Jun 26, 2024 @ 23:59:07.225876594 | zedbox | alloc 2304 bytes 1 objects total 2304/1 at: /usr/lib/go/src/encoding/json/decode.go[956] encoding/json.(*decodeState).literalStore /usr/lib/go/src/encoding/json/decode.go[388] encoding/json.(*decodeState).value /usr/lib/go/src/encoding/json/decode.go[775] encoding/json.(*decodeState).object /usr/lib/go/src/encoding/json/decode.go[374] encoding/json.(*decodeState).value /usr/lib/go/src/encoding/json/decode.go[181] encoding/json.(*decodeState).unmarshal /usr/lib/go/src/encoding/json/decode.go[108] encoding/json.Unmarshal /pillar/pubsub/util.go[49] github.com/lf-edge/eve/pkg/pillar/pubsub.parseTemplate /pillar/pubsub/subscribe.go[230] github.com/lf-edge/eve/pkg/pillar/pubsub.handleModify /pillar/pubsub/subscribe.go[97] github.com/lf-edge/eve/pkg/pillar/pubsub.(*SubscriptionImpl).populate /pillar/pubsub/subscribe.go[54] github.com/lf-edge/eve/pkg/pillar/pubsub.(*SubscriptionImpl).Activate /pillar/cmd/downloader/context.go[69] github.com/lf-edge/eve/pkg/pillar/cmd/downloader.(*downloaderContext).registerHandlers /pillar/cmd/downloader/downloader.go[98] github.com/lf-edge/eve/pkg/pillar/cmd/downloader.Run /pillar/zedbox/zedbox.go[241] main.startAgentAndDone /usr/lib/go/src/runtime/asm_amd64.s[1598] runtime.goexit

  | Jun 26, 2024 @ 23:59:07.230556902 | zedbox | alloc 1792 bytes 1 objects total 1792/1 at: /usr/lib/go/src/encoding/pem/pem.go[171] encoding/pem.Decode /usr/lib/go/src/crypto/x509/cert_pool.go[210] crypto/x509.(*CertPool).AppendCertsFromPEM /pillar/zedcloud/tls.go[109] github.com/lf-edge/eve/pkg/pillar/zedcloud.GetTlsConfig /pillar/conntester/zedcloud.go[70] github.com/lf-edge/eve/pkg/pillar/conntester.(*ZedcloudConnectivityTester).TestConnectivity /pillar/dpcmanager/verify.go[235] github.com/lf-edge/eve/pkg/pillar/dpcmanager.(*DpcManager).verifyDPC /pillar/dpcmanager/verify.go[96] github.com/lf-edge/eve/pkg/pillar/dpcmanager.(*DpcManager).runVerify /pillar/dpcmanager/dpcmanager.go[568] github.com/lf-edge/eve/pkg/pillar/dpcmanager.(*DpcManager).resumeVerifyIfAsyncDone /pillar/dpcmanager/dpcmanager.go[276] github.com/lf-edge/eve/pkg/pillar/dpcmanager.(*DpcManager).run /usr/lib/go/src/runtime/asm_amd64.s[1598] runtime.goexit

  | Jun 26, 2024 @ 23:59:07.234690906 | zedbox | alloc 1536 bytes 1 objects total 1536/1 at: /usr/lib/go/src/encoding/pem/pem.go[171] encoding/pem.Decode /usr/lib/go/src/crypto/x509/cert_pool.go[210] crypto/x509.(*CertPool).AppendCertsFromPEM /pillar/zedcloud/tls.go[109] github.com/lf-edge/eve/pkg/pillar/zedcloud.GetTlsConfig /pillar/zedcloud/tls.go[57] github.com/lf-edge/eve/pkg/pillar/zedcloud.UpdateTLSConfig /pillar/cmd/diag/diag.go[1206] github.com/lf-edge/eve/pkg/pillar/cmd/diag.tryPing /pillar/cmd/diag/diag.go[988] github.com/lf-edge/eve/pkg/pillar/cmd/diag.printOutput /pillar/cmd/diag/diag.go[744] github.com/lf-edge/eve/pkg/pillar/cmd/diag.printTask /usr/lib/go/src/runtime/asm_amd64.s[1598] runtime.goexit

  | Jun 26, 2024 @ 23:59:07.238748617 | zedbox | alloc 1536 bytes 1 objects total 1536/1 at: /usr/lib/go/src/encoding/pem/pem.go[171] encoding/pem.Decode /usr/lib/go/src/crypto/x509/cert_pool.go[210] crypto/x509.(*CertPool).AppendCertsFromPEM /pillar/zedcloud/tls.go[109] github.com/lf-edge/eve/pkg/pillar/zedcloud.GetTlsConfig /pillar/conntester/zedcloud.go[70] github.com/lf-edge/eve/pkg/pillar/conntester.(*ZedcloudConnectivityTester).TestConnectivity /pillar/dpcmanager/verify.go[235] github.com/lf-edge/eve/pkg/pillar/dpcmanager.(*DpcManager).verifyDPC /pillar/dpcmanager/verify.go[96] github.com/lf-edge/eve/pkg/pillar/dpcmanager.(*DpcManager).runVerify /pillar/dpcmanager/dpcmanager.go[568] github.com/lf-edge/eve/pkg/pillar/dpcmanager.(*DpcManager).resumeVerifyIfAsyncDone /pillar/dpcmanager/dpcmanager.go[276] github.com/lf-edge/eve/pkg/pillar/dpcmanager.(*DpcManager).run /usr/lib/go/src/runtime/asm_amd64.s[1598] runtime.goexit

  | Jun 26, 2024 @ 23:59:07.244147170 | zedbox | alloc 416 bytes 1 objects total 416/1 at: /usr/lib/go/src/runtime/proc.go[4223] runtime.malg /usr/lib/go/src/runtime/proc.go[4268] runtime.newproc1 /usr/lib/go/src/runtime/proc.go[4245] runtime.newproc.func1 /usr/lib/go/src/runtime/asm_amd64.s[496] runtime.systemstack

  | Jun 26, 2024 @ 23:59:07.251683191 | zedbox | alloc 144 bytes 1 objects total 144/1 at: /usr/lib/go/src/runtime/map.go[677] runtime.mapassign /pillar/vendor/github.com/aws/aws-sdk-go/aws/endpoints/defaults.go[27150] github.com/aws/aws-sdk-go/aws/endpoints.init /usr/lib/go/src/runtime/proc.go[6506] runtime.doInit /usr/lib/go/src/runtime/proc.go[6483] runtime.doInit /usr/lib/go/src/runtime/proc.go[6483] runtime.doInit /usr/lib/go/src/runtime/proc.go[6483] runtime.doInit /usr/lib/go/src/runtime/proc.go[6483] runtime.doInit /usr/lib/go/src/runtime/proc.go[6483] runtime.doInit /usr/lib/go/src/runtime/proc.go[6483] runtime.doInit /usr/lib/go/src/runtime/proc.go[233] runtime.main /usr/lib/go/src/runtime/asm_amd64.s[1598] runtime.goexit

  | Jun 26, 2024 @ 23:59:07.256169660 | zedbox | alloc 144 bytes 1 objects total 144/1 at: /usr/lib/go/src/runtime/map.go[677] runtime.mapassign /pillar/vendor/github.com/aws/aws-sdk-go/aws/endpoints/defaults.go[357] github.com/aws/aws-sdk-go/aws/endpoints.init /usr/lib/go/src/runtime/proc.go[6506] runtime.doInit /usr/lib/go/src/runtime/proc.go[6483] runtime.doInit /usr/lib/go/src/runtime/proc.go[6483] runtime.doInit /usr/lib/go/src/runtime/proc.go[6483] runtime.doInit /usr/lib/go/src/runtime/proc.go[6483] runtime.doInit /usr/lib/go/src/runtime/proc.go[6483] runtime.doInit /usr/lib/go/src/runtime/proc.go[6483] runtime.doInit /usr/lib/go/src/runtime/proc.go[233] runtime.main /usr/lib/go/src/runtime/asm_amd64.s[1598] runtime.goexit

  | Jun 26, 2024 @ 23:59:07.260429072 | zedbox | alloc 96 bytes 1 objects total 96/1 at: /usr/lib/go/src/runtime/chan.go[106] runtime.makechan /pillar/pubsub/pubsub.go[103] github.com/lf-edge/eve/pkg/pillar/pubsub.(*PubSub).NewSubscription /pillar/cmd/baseosmgr/baseosmgr.go[371] github.com/lf-edge/eve/pkg/pillar/cmd/baseosmgr.initializeNodeAgentHandles /pillar/cmd/baseosmgr/baseosmgr.go[103] github.com/lf-edge/eve/pkg/pillar/cmd/baseosmgr.Run /pillar/zedbox/zedbox.go[241] main.startAgentAndDone /usr/lib/go/src/runtime/asm_amd64.s[1598] runtime.goexit

  | Jun 26, 2024 @ 23:59:07.265800336 | zedbox | alloc 80 bytes 1 objects total 80/1 at: /pillar/vendor/google.golang.org/protobuf/internal/impl/message_reflect.go[65] google.golang.org/protobuf/internal/impl.(*MessageInfo).makeKnownFieldsFunc /pillar/vendor/google.golang.org/protobuf/internal/impl/message_reflect.go[42] google.golang.org/protobuf/internal/impl.(*MessageInfo).makeReflectFuncs /pillar/vendor/google.golang.org/protobuf/internal/impl/message.go[90] google.golang.org/protobuf/internal/impl.(*MessageInfo).initOnce /pillar/vendor/google.golang.org/protobuf/internal/impl/message.go[72] google.golang.org/protobuf/internal/impl.(*MessageInfo).init /pillar/vendor/google.golang.org/protobuf/internal/impl/decode.go[89] google.golang.org/protobuf/internal/impl.(*MessageInfo).unmarshalPointer /pillar/vendor/google.golang.org/protobuf/internal/impl/codec_field.go[503] google.golang.org/protobuf/internal/impl.consumeMessageSliceInfo /pillar/vendor/google.golang.org/protobuf/internal/impl/decode.go[148] google.golang.org/protobuf/internal/impl.(*MessageInfo).unmarshalPointer /pillar/vendor/google.golang.org/protobuf/internal/impl/codec_field.go[252] google.golang.org/protobuf/internal/impl.consumeMessageInfo /pillar/vendor/google.golang.org/protobuf/internal/impl/decode.go[148] google.golang.org/protobuf/internal/impl.(*MessageInfo).unmarshalPointer /pillar/vendor/google.golang.org/protobuf/internal/impl/decode.go[66] google.golang.org/protobuf/internal/impl.(*MessageInfo).unmarshal /pillar/vendor/google.golang.org/protobuf/proto/decode.go[105] google.golang.org/protobuf/proto.UnmarshalOptions.unmarshal /pillar/vendor/google.golang.org/protobuf/proto/decode.go[55] google.golang.org/protobuf/proto.Unmarshal /pillar/cmd/upgradeconverter/parseconfig.go[84] github.com/lf-edge/eve/pkg/pillar/cmd/upgradeconverter.readSavedProtoMessage /pillar/cmd/upgradeconverter/parseconfig.go[26] github.com/lf-edge/eve/pkg/pillar/cmd/upgradeconverter.parseConfig /pillar/cmd/upgradeconverter/persistlayout.go[37] github.com/lf-edge/eve/pkg/pillar/cmd/upgradeconverter.convertPersistVolumes /pillar/cmd/upgradeconverter/upgradeconverter.go[141] github.com/lf-edge/eve/pkg/pillar/cmd/upgradeconverter.runHandlers /pillar/cmd/upgradeconverter/upgradeconverter.go[246] github.com/lf-edge/eve/pkg/pillar/cmd/upgradeconverter.runPhase /pillar/cmd/upgradeconverter/upgradeconverter.go[212] github.com/lf-edge/eve/pkg/pillar/cmd/upgradeconverter.RunPostVaultHandlers /usr/lib/go/src/runtime/asm_amd64.s[1598] runtime.goexit

  | Jun 26, 2024 @ 23:59:07.274214851 | zedbox | alloc 32 bytes 1 objects total 32/1 at: /usr/lib/go/src/regexp/onepass.go[185] regexp.mergeRuneSets.func2 /usr/lib/go/src/regexp/onepass.go[194] regexp.mergeRuneSets /usr/lib/go/src/regexp/onepass.go[343] regexp.makeOnePass.func1 /usr/lib/go/src/regexp/onepass.go[449] regexp.makeOnePass /usr/lib/go/src/regexp/onepass.go[501] regexp.compileOnePass /usr/lib/go/src/regexp/regexp.go[193] regexp.compile /usr/lib/go/src/regexp/regexp.go[137] regexp.Compile /usr/lib/go/src/regexp/regexp.go[317] regexp.MustCompile /pillar/vendor/github.com/go-playground/validator/v10/regexes.go[127] github.com/go-playground/validator/v10.init /usr/lib/go/src/runtime/proc.go[6506] runtime.doInit /usr/lib/go/src/runtime/proc.go[6483] runtime.doInit /usr/lib/go/src/runtime/proc.go[6483] runtime.doInit /usr/lib/go/src/runtime/proc.go[6483] runtime.doInit /usr/lib/go/src/runtime/proc.go[6483] runtime.doInit /usr/lib/go/src/runtime/proc.go[233] runtime.main /usr/lib/go/src/runtime/asm_amd64.s[1598] runtime.goexit

  | Jun 26, 2024 @ 23:59:07.781486061 | zedbox | Read 32 bytes

  | Jun 26, 2024 @ 23:59:07.781658679 | zedbox | response head: &{Version:6 PKTType:reply Res1:0 Res2:0 Command:14 Reply:2 Status:SUCCESS Pad1:0 Pad2:0 Pad3:0 Sequence:2 Pad4:0 Pad5:0}

  | Jun 26, 2024 @ 23:59:07.781738460 | zedbox | response data: &{NSources:1}

  | Jun 26, 2024 @ 23:59:07.783933638 | zedbox | Read 76 bytes

  | Jun 26, 2024 @ 23:59:07.784877099 | zedbox | response head: &{Version:6 PKTType:reply Res1:0 Res2:0 Command:15 Reply:3 Status:SUCCESS Pad1:0 Pad2:0 Pad3:0 Sequence:3 Pad4:0 Pad5:0}

  | Jun 26, 2024 @ 23:59:07.785601236 | zedbox | response data: &{IPAddr:{IP:[10 216 0 254 0 0 0 0 0 0 0 0 0 0 0 0] Family:1 Pad:0} Poll:6 Stratum:4 State:unreach Mode:client Flags:0 Reachability:3 SinceSample:8 OrigLatestMeas:-152814232 LatestMeas:-152814232 LatestMeasErr:-154707747}

  | Jun 26, 2024 @ 23:59:07.786913476 | zedbox | Read 152 bytes

  | Jun 26, 2024 @ 23:59:07.787113051 | zedbox | response head: &{Version:6 PKTType:reply Res1:0 Res2:0 Command:57 Reply:16 Status:SUCCESS Pad1:0 Pad2:0 Pad3:0 Sequence:4 Pad4:0 Pad5:0}

  | Jun 26, 2024 @ 23:59:07.787330448 | zedbox | response data: &{RemoteAddr:{IP:[10 216 0 254 0 0 0 0 0 0 0 0 0 0 0 0] Family:1 Pad:0} LocalAddr:{IP:[10 216 0 14 0 0 0 0 0 0 0 0 0 0 0 0] Family:1 Pad:0} RemotePort:123 Leap:0 Version:4 Mode:4 Stratum:4 Poll:6 Precision:-29 RootDelay:-121430016 RootDispersion:0 RefID:2549880062 RefTime:{SecHigh:0 SecLow:1719439017 Nsec:183766841} Offset:-149175656 PeerDelay:-293172320 PeerDispersion:-594539205 ResponseTime:-620768855 JitterAsymmetry:0 Flags:1023 TXTssChar:75 RXTssChar:75 TotalTXCount:2 TotalRXCount:2 TotalValidCount:2 Reserved:[4294967295 4294967295 4294967295 4294967295]}

  | Jun 26, 2024 @ 23:59:07.788493649 | zedbox | Read 284 bytes

  | Jun 26, 2024 @ 23:59:07.792908778 | zedbox | response head: &{Version:6 PKTType:reply Res1:0 Res2:0 Command:65 Reply:19 Status:SUCCESS Pad1:0 Pad2:0 Pad3:0 Sequence:5 Pad4:0 Pad5:0}

  | Jun 26, 2024 @ 23:59:07.793071791 | zedbox | response data: &{Name:[49 48 46 50 49 54 46 48 46 50 53 52 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0]}

  | Jun 26, 2024 @ 23:59:07.922772604 | zedagent | Cipher metrics for downloader: {AgentName:downloader FailureCount:0 SuccessCount:1 LastFailure:0001-01-01 00:00:00 +0000 UTC LastSuccess:2024-06-26 21:57:53.415558617 +0000 UTC TypeCounters:[0 0 0 0 0 0 0 0]}

  | Jun 26, 2024 @ 23:59:07.922940505 | zedagent | Cipher metrics for domainmgr: {AgentName:domainmgr FailureCount:0 SuccessCount:0 LastFailure:0001-01-01 00:00:00 +0000 UTC LastSuccess:0001-01-01 00:00:00 +0000 UTC TypeCounters:[0 0 0 0 0 0 0 0]}

  | Jun 26, 2024 @ 23:59:07.923032019 | zedagent | Cipher metrics for nim: {AgentName:nim FailureCount:0 SuccessCount:0 LastFailure:0001-01-01 00:00:00 +0000 UTC LastSuccess:0001-01-01 00:00:00 +0000 UTC TypeCounters:[0 0 0 0 0 0 0 0]}

  | Jun 26, 2024 @ 23:59:07.923116567 | zedagent | Cipher metrics for : {AgentName: FailureCount:0 SuccessCount:0 LastFailure:0001-01-01 00:00:00 +0000 UTC LastSuccess:0001-01-01 00:00:00 +0000 UTC TypeCounters:[]}

  | Jun 26, 2024 @ 23:59:07.923201744 | zedagent | Cipher metrics for wwan: {AgentName:wwan FailureCount:0 SuccessCount:0 LastFailure:0001-01-01 00:00:00 +0000 UTC LastSuccess:0001-01-01 00:00:00 +0000 UTC TypeCounters:[0 0 0 0 0 0 0 0]}

  | Jun 26, 2024 @ 23:59:07.923817441 | zedagent | GetContextForAllIntfFunctions: maxWaitDuration 360000000000 is more than limit, will use 240000000000

  | Jun 26, 2024 @ 23:59:08.159012613 | zedagent | maybeUpdateMetricsTimer: currentMetricInterval 60, latestMetricsInterval 60, forceUpdate: true

  | Jun 26, 2024 @ 23:59:08.438417961 | zedagent | GetContextForAllIntfFunctions: maxWaitDuration 360000000000 is more than limit, will use 240000000000

  | Jun 26, 2024 @ 23:59:08.659954998 | domainmgr | pidFile /run/chronyd.pid read error open /run/chronyd.pid: no such file or directory

  | Jun 26, 2024 @ 23:59:09.018942351 | domainmgr | getProcessMetric failed: open /proc/23972/status: no such file or directory

  | Jun 26, 2024 @ 23:59:09.317069990 | zedbox | Listening failed: http: Server closed

  | Jun 26, 2024 @ 23:59:10.035521666 | memory-monitor | Handler script exited with status 2

  | Jun 26, 2024 @ 23:59:10.036086748 | memory-monitor | ----- eve pressure event -----

  | Jun 26, 2024 @ 23:59:10.036516987 | memory-monitor | Running handler script

  | Jun 26, 2024 @ 23:59:12.233340960 | zedagent | GetContextForAllIntfFunctions: maxWaitDuration 360000000000 is more than limit, will use 240000000000

  | Jun 26, 2024 @ 23:59:12.386036885 | memory-monitor | Handler script exited with status 1

  | Jun 26, 2024 @ 23:59:13.990795004 | loguploader | GetContextForAllIntfFunctions: maxWaitDuration 360000000000 is more than limit, will use 240000000000

  | Jun 26, 2024 @ 23:59:15.875304091 | domainmgr | Domain metric modify

  | Jun 26, 2024 @ 23:59:15.879263206 | domainmgr | Host memory modify

  | Jun 26, 2024 @ 23:59:15.882588954 | zedmanager | Host memory modify

  | Jun 26, 2024 @ 23:59:15.883832968 | zedmanager | handleHostMemoryImpl(global) done

  | Jun 26, 2024 @ 23:59:15.886236371 | zedagent | Host memory modify

  | Jun 26, 2024 @ 23:59:15.886879458 | watcher | Host memory modify

  | Jun 26, 2024 @ 23:59:15.890306643 | zedagent | Domain metric modify

  | Jun 26, 2024 @ 23:59:15.896256769 | watcher | handleHostMemoryModify:

  | Jun 26, 2024 @ 23:59:16.024699213 | zedagent | GetContextForAllIntfFunctions: maxWaitDuration 360000000000 is more than limit, will use 240000000000

  | Jun 26, 2024 @ 23:59:16.790632950 | pillar.out | not replacing BootReason BootReasonWatchdogPid

  | Jun 26, 2024 @ 23:59:16.790651512 | pillar.out | with BootReasonUpdate

  | Jun 26, 2024 @ 23:59:16.799359884 | nodeagent | allDomainsHalted: All Domains Halted.

  | Jun 26, 2024 @ 23:59:16.801738383 | nodeagent | Nodeagent status modify

  | Jun 26, 2024 @ 23:59:16.807635560 | baseosmgr | Nodeagent status modify

  | Jun 26, 2024 @ 23:59:16.810139834 | baseosmgr | handleNodeAgentStatusImpl(nodeagent) done

  | Jun 26, 2024 @ 23:59:16.813124637 | zedagent | Nodeagent status modify

  | Jun 26, 2024 @ 23:59:16.815139130 | zedagent | handleNodeAgentStatusImpl: updateInProgress false rebootReason Watchdog report for IMGA EVE version 0.0.0-bugfix_fix-memory-monitor-visible-issues-acf6861a-dirty-2024-06-26.16.48-kvm-amd64 at 2024-06-26T21:40:31+00:00: 2 /run/chronyd.pid  bootReason BootReasonWatchdogPid

  | Jun 26, 2024 @ 23:59:16.817380742 | zedagent | Triggered PublishDeviceInfo

  | Jun 26, 2024 @ 23:59:16.817834348 | zedagent | Triggering POST for /api/v1/devinfo to local server

  | Jun 26, 2024 @ 23:59:16.819060426 | zedagent | handleNodeAgentStatusImpl: done.

  | Jun 26, 2024 @ 23:59:16.822370075 | zedagent | deviceInfoTask got message

  | Jun 26, 2024 @ 23:59:16.823827325 | zedagent | PublishDeviceInfoToZedCloud uuid 0ba325bd-e0f8-4d90-9125-62386982a6cc

  | Jun 26, 2024 @ 23:59:16.841835133 | zedagent | Device swInfo: activated:true  partitionLabel:"IMGA"  partitionDevice:"/dev/sda2"  partitionState:"active"  status:INSTALLED  shortVersion:"0.0.0-bugfix_fix-memory-monitor-visible-issues-acf6861a-dirty-2024-06-26.16.48-kvm-amd64"  downloadProgress:100

  | Jun 26, 2024 @ 23:59:16.841993959 | zedagent | Device swInfo: partitionLabel:"IMGB"  partitionDevice:"/dev/sda3"  partitionState:"updating"  status:INSTALLED  shortVersion:"0.0.0-bugfix_fix-memory-monitor-visible-issues-fd950df7-dirty-2024-06-26.21.39-kvm-amd64"  downloadProgress:100

  | Jun 26, 2024 @ 23:59:16.842083961 | zedagent | Device swInfo: status:INSTALLED  shortVersion:"0.0.0-bugfix_fix-memory-monitor-visible-issues-acf6861a-dirty-2024-06-26.16.48-kvm-amd64"

  | Jun 26, 2024 @ 23:59:16.862417308 | nodeagent | Doing a sync..

  | Jun 26, 2024 @ 23:59:16.884106074 | nodeagent | Rebooting... Starting timer for Duration(secs): 30

  | Jun 26, 2024 @ 23:59:16.884252887 | nodeagent | Timer started. Wait to expire

  | Jun 26, 2024 @ 23:59:17.007459546 | zedagent | SetDeferred(0ba325bd-e0f8-4d90-9125-62386982a6cc) size 3003 items 0

  | Jun 26, 2024 @ 23:59:17.007594927 | zedagent | startTimer()

  | Jun 26, 2024 @ 23:59:17.007667521 | zedagent | deviceInfoTask done with message

  | Jun 26, 2024 @ 23:59:17.007781863 | zedagent | handleDeferred items 1

  | Jun 26, 2024 @ 23:59:17.007891660 | zedagent | GetContextForAllIntfFunctions: maxWaitDuration 360000000000 is more than limit, will use 240000000000

  | Jun 26, 2024 @ 23:59:17.007966211 | zedagent | handleDeferred: Trying to send for 0ba325bd-e0f8-4d90-9125-62386982a6cc

  | Jun 26, 2024 @ 23:59:17.418264107 | zedagent | stopTimer()

  | Jun 26, 2024 @ 23:59:17.686962571 | zedrouter | Network metrics modify

  | Jun 26, 2024 @ 23:59:17.697946039 | zedagent | Network metrics modify

  | Jun 26, 2024 @ 23:59:17.703220695 | zedrouter | Network instance metrics modify

  | Jun 26, 2024 @ 23:59:17.710282862 | zedagent | Network instance metrics modify

  | Jun 26, 2024 @ 23:59:17.777980682 | zedbox | Read 32 bytes

  | Jun 26, 2024 @ 23:59:17.778467720 | zedbox | response head: &{Version:6 PKTType:reply Res1:0 Res2:0 Command:14 Reply:2 Status:SUCCESS Pad1:0 Pad2:0 Pad3:0 Sequence:2 Pad4:0 Pad5:0}

  | Jun 26, 2024 @ 23:59:17.780599162 | zedbox | response data: &{NSources:1}

  | Jun 26, 2024 @ 23:59:17.782788397 | zedbox | Read 76 bytes

  | Jun 26, 2024 @ 23:59:17.784779457 | zedbox | response head: &{Version:6 PKTType:reply Res1:0 Res2:0 Command:15 Reply:3 Status:SUCCESS Pad1:0 Pad2:0 Pad3:0 Sequence:3 Pad4:0 Pad5:0}

  | Jun 26, 2024 @ 23:59:17.785634719 | zedbox | response data: &{IPAddr:{IP:[10 216 0 254 0 0 0 0 0 0 0 0 0 0 0 0] Family:1 Pad:0} Poll:6 Stratum:4 State:unreach Mode:client Flags:0 Reachability:3 SinceSample:18 OrigLatestMeas:-152814232 LatestMeas:-152814232 LatestMeasErr:-154707747}

  | Jun 26, 2024 @ 23:59:17.787045966 | zedbox | Read 152 bytes

  | Jun 26, 2024 @ 23:59:17.787952871 | zedbox | response head: &{Version:6 PKTType:reply Res1:0 Res2:0 Command:57 Reply:16 Status:SUCCESS Pad1:0 Pad2:0 Pad3:0 Sequence:4 Pad4:0 Pad5:0}

  | Jun 26, 2024 @ 23:59:17.789216514 | zedbox | response data: &{RemoteAddr:{IP:[10 216 0 254 0 0 0 0 0 0 0 0 0 0 0 0] Family:1 Pad:0} LocalAddr:{IP:[10 216 0 14 0 0 0 0 0 0 0 0 0 0 0 0] Family:1 Pad:0} RemotePort:123 Leap:0 Version:4 Mode:4 Stratum:4 Poll:6 Precision:-29 RootDelay:-121430016 RootDispersion:0 RefID:2549880062 RefTime:{SecHigh:0 SecLow:1719439017 Nsec:183766841} Offset:-149175656 PeerDelay:-293172320 PeerDispersion:-594539205 ResponseTime:-620768855 JitterAsymmetry:0 Flags:1023 TXTssChar:75 RXTssChar:75 TotalTXCount:2 TotalRXCount:2 TotalValidCount:2 Reserved:[4294967295 4294967295 4294967295 4294967295]}

  | Jun 26, 2024 @ 23:59:17.793137429 | zedbox | Read 284 bytes

  | Jun 26, 2024 @ 23:59:17.794024682 | zedbox | response head: &{Version:6 PKTType:reply Res1:0 Res2:0 Command:65 Reply:19 Status:SUCCESS Pad1:0 Pad2:0 Pad3:0 Sequence:5 Pad4:0 Pad5:0}

  | Jun 26, 2024 @ 23:59:17.794910891 | zedbox | response data: &{Name:[49 48 46 50 49 54 46 48 46 50 53 52 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0]}

  | Jun 26, 2024 @ 23:59:20.535216794 | zedagent | GetContextForAllIntfFunctions: maxWaitDuration 360000000000 is more than limit, will use 240000000000

  | Jun 26, 2024 @ 23:59:20.706100156 | domainmgr | pidFile /run/chronyd.pid read error open /run/chronyd.pid: no such file or directory
eriknordmark commented 1 week ago

The error 254 is just an artifact of the watchdog (and the fact that we use the watchdog repair script functionality to save watchdog info and never repair anything but let the device shutdown aka reboot).

The pidFile /run/chronyd.pid read error open /run/chronyd.pid: no such file or directory is interesting if it indicates that the watchdog deamon has been told (via /run/watchdog/pid) to check for that process, but the pid file hasn't even been created. @rouming can you check if the chronyd.pid might be added to the watchdog pid directory before that pidfile is created?

eriknordmark commented 1 week ago

I saw a total of 6 crashes/watchdogs to this overnight (running on two different devices in the lab - sc-supermicro-zc1 and sc-supermicro-zc2). So I think we need to figure this out before we cut the next EVE release.

rouming commented 1 week ago

Looking into that.

eriknordmark commented 4 days ago

Fixed by PR #4016