[Bug] binary v3.1.0(Golang map concurrency issue)

[kimheeje12]

• Bug Description

  fatal error: concurrent map iteration and map write
  - Issue: Golang map concurrency issue.

• Expected Behavior

  Block sync should proceed smoothly without interruptions.

• Actual Behavior

  Occasional unintended and abnormal restarts occur.

• Environment

  OS: Ubuntu 22.04.5 LTS (baremetal)
  CPU: AMD EPYC 7313P (16 cores, 32 threads, 3.73 GHz max)
  Virtualization: AMD-V
  used cosmovisor

• logs:

  Nov 24 02:07:07 s12217972 cosmovisor_dymension_1100-1[443359]: fatal error: concurrent map iteration and map write
  Nov 24 02:07:07 s12217972 cosmovisor_dymension_1100-1[443359]: goroutine 6014837474 [running]:
  Nov 24 02:07:07 s12217972 cosmovisor_dymension_1100-1[443359]: http://github.com/cosmos/iavl.NewUnsavedFastIterator ({0xc24bb06a00, 0x1e, 0x20}, {0xc24bb06a20, 0x1e, 0x20}, 0x1, 0xc000aaa800, 0xc119a346c0, 0xc119a34750)
  Nov 24 02:07:07 s12217972 cosmovisor_dymension_1100-1[443359]: #011github.com/cosmos/iavl@v0.19.6/unsaved_fast_iterator.go:55 +0x1f6
  Nov 24 02:07:07 s12217972 cosmovisor_dymension_1100-1[443359]: http://github.com/cosmos/iavl.(*MutableTree ).Iterator(0xc007bcbbd0, {0xc24bb06a00, 0x1e, 0x20}, {0xc24bb06a20, 0x1e, 0x20}, 0x1)
  Nov 24 02:07:07 s12217972 cosmovisor_dymension_1100-1[443359]: #011github.com/cosmos/iavl@v0.19.6/mutable_tree.go:216 +0x105
  Nov 24 02:07:07 s12217972 cosmovisor_dymension_1100-1[443359]: http://github.com/cosmos/cosmos-sdk/store/iavl.(*Store ).Iterator(0xc10b72c500?, {0xc24bb06a00?, 0x20?, 0xc10b72c680?}, {0xc24bb06a20?, 0x20?, 0xc10b72c6a0?})
  Nov 24 02:07:07 s12217972 cosmovisor_dymension_1100-1[443359]: #011github.com/cosmos/cosmos-sdk@v0.46.16/store/iavl/store.go:249 +0x2b
  Nov 24 02:07:07 s12217972 cosmovisor_dymension_1100-1[443359]: http://github.com/cosmos/cosmos-sdk/store/cachekv.(*Store ).iterator(0xc20dcbf400, {0xc24bb06a00, 0x1e, 0x20}, {0xc24bb06a20, 0x1e, 0x20}, 0x1)
  Nov 24 02:07:07 s12217972 cosmovisor_dymension_1100-1[443359]: #011github.com/cosmos/cosmos-sdk@v0.46.16/store/cachekv/store.go:185 +0x11a
  Nov 24 02:07:07 s12217972 cosmovisor_dymension_1100-1[443359]: http://github.com/cosmos/cosmos-sdk/store/cachekv.(*Store ).Iterator(0x3?, {0xc24bb06a00?, 0x1e?, 0x1e?}, {0xc24bb06a20?, 0x64?, 0x64?})
  Nov 24 02:07:07 s12217972 cosmovisor_dymension_1100-1[443359]: #011github.com/cosmos/cosmos-sdk@v0.46.16/store/cachekv/store.go:170 +0x25
  Nov 24 02:07:07 s12217972 cosmovisor_dymension_1100-1[443359]: http://github.com/cosmos/cosmos-sdk/store/cachekv.(*Store ).iterator(0xc104b17ec0, {0xc24bb06a00, 0x1e, 0x20}, {0xc24bb06a20, 0x1e, 0x20}, 0x1)
  Nov 24 02:07:07 s12217972 cosmovisor_dymension_1100-1[443359]: #011github.com/cosmos/cosmos-sdk@v0.46.16/store/cachekv/store.go:185 +0x11a
  Nov 24 02:07:07 s12217972 cosmovisor_dymension_1100-1[443359]: http://github.com/cosmos/cosmos-sdk/store/cachekv.(*Store ).Iterator(0xc10b72c360?, {0xc24bb06a00?, 0x20?, 0xc10b72c400?}, {0xc24bb06a20?, 0x20?, 0xc21f1defc8?})
  Nov 24 02:07:07 s12217972 cosmovisor_dymension_1100-1[443359]: #011github.com/cosmos/cosmos-sdk@v0.46.16/store/cachekv/store.go:170 +0x25
  Nov 24 02:07:07 s12217972 cosmovisor_dymension_1100-1[443359]: http://github.com/cosmos/cosmos-sdk/store/cachekv.(*Store ).iterator(0xc0b38a3240, {0xc24bb06a00, 0x1e, 0x20}, {0xc24bb06a20, 0x1e, 0x20}, 0x1)
  Nov 24 02:07:07 s12217972 cosmovisor_dymension_1100-1[443359]: #011github.com/cosmos/cosmos-sdk@v0.46.16/store/cachekv/store.go:185 +0x11a
  Nov 24 02:07:07 s12217972 cosmovisor_dymension_1100-1[443359]: http://github.com/cosmos/cosmos-sdk/store/cachekv.(*Store ).Iterator(0x0?, {0xc24bb06a00?, 0x26ec260?, 0xc21f1df0a0?}, {0xc24bb06a20?, 0x20?, 0x0?})
  Nov 24 02:07:07 s12217972 cosmovisor_dymension_1100-1[443359]: #011github.com/cosmos/cosmos-sdk@v0.46.16/store/cachekv/store.go:170 +0x25
  Nov 24 02:07:07 s12217972 cosmovisor_dymension_1100-1[443359]: http://github.com/cosmos/cosmos-sdk/store/gaskv.(*Store ).iterator(0xc0f1fcd5c0, {0xc24bb06a00?, 0x64?, 0xb824440000000000?}, {0xc24bb06a20?, 0x1e?, 0x20?}, 0x20?)
  Nov 24 02:07:07 s12217972 cosmovisor_dymension_1100-1[443359]: #011github.com/cosmos/cosmos-sdk@v0.46.16/store/gaskv/store.go:98 +0x48
  Nov 24 02:07:07 s12217972 cosmovisor_dymension_1100-1[443359]: http://github.com/cosmos/cosmos-sdk/store/gaskv.(*Store ).Iterator(0xc1716b6dc8?, {0xc24bb06a00?, 0x4d6d250?, 0xc0b38a3900?}, {0xc24bb06a20?, 0x0?, 0xc0b284c1c0?})
  Nov 24 02:07:07 s12217972 cosmovisor_dymension_1100-1[443359]: #011github.com/cosmos/cosmos-sdk@v0.46.16/store/gaskv/store.go:74 +0x25
  Nov 24 02:07:07 s12217972 cosmovisor_dymension_1100-1[443359]: github.com/cosmos/cosmos-sdk/x/distribution/keeper.Keeper.IterateValidatorSlashEventsBetween({{0x4d36ee8, 0xc003b8c410}, {0x4d6d330, 0xc0009a5950}, {{0x4d6d330, 0xc0009a5950}, 0xc0000c88b0, {0x4d36ee8, 0xc003b8c440}, {0x4d36f10, ...}, ...}, ...}, ...)
  Nov 24 02:07:07 s12217972 cosmovisor_dymension_1100-1[443359]: #011github.com/cosmos/cosmos-sdk@v0.46.16/x/distribution/keeper/store.go:337 +0x14a
  Nov 24 02:07:07 s12217972 cosmovisor_dymension_1100-1[443359]: github.com/cosmos/cosmos-sdk/x/distribution/keeper.Keeper.CalculateDelegationRewards({{0x4d36ee8, 0xc003b8c410}, {0x4d6d330, 0xc0009a5950}, {{0x4d6d330, 0xc0009a5950}, 0xc0000c88b0, {0x4d36ee8, 0xc003b8c440}, {0x4d36f10, ...}, ...}, ...}, ...)
  Nov 24 02:07:07 s12217972 cosmovisor_dymension_1100-1[443359]: #011github.com/cosmos/cosmos-sdk@v0.46.16/x/distribution/keeper/delegation.go:80 +0x2d9
  Nov 24 02:07:07 s12217972 cosmovisor_dymension_1100-1[443359]: github.com/cosmos/cosmos-sdk/x/distribution/keeper.Keeper.withdrawDelegationRewards({{0x4d36ee8, 0xc003b8c410}, {0x4d6d330, 0xc0009a5950}, {{0x4d6d330, 0xc0009a5950}, 0xc0000c88b0, {0x4d36ee8, 0xc003b8c440}, {0x4d36f10, ...}, ...}, ...}, ...)
  Nov 24 02:07:07 s12217972 cosmovisor_dymension_1100-1[443359]: #011github.com/cosmos/cosmos-sdk@v0.46.16/x/distribution/keeper/delegation.go:147 +0x1e5
  Nov 24 02:07:07 s12217972 cosmovisor_dymension_1100-1[443359]: github.com/cosmos/cosmos-sdk/x/distribution/keeper.Hooks.BeforeDelegationSharesModified({{{0x4d36ee8, 0xc003b8c410}, {0x4d6d330, 0xc0009a5950}, {{0x4d6d330, 0xc0009a5950}, 0xc0000c88b0, {0x4d36ee8, 0xc003b8c440}, {0x4d36f10, ...}, ...}, ...}}, ...)
  Nov 24 02:07:07 s12217972 cosmovisor_dymension_1100-1[443359]: #011github.com/cosmos/cosmos-sdk@v0.46.16/x/distribution/keeper/hooks.go:93 +0x192
  Nov 24 02:07:07 s12217972 cosmovisor_dymension_1100-1[443359]: github.com/cosmos/cosmos-sdk/x/staking/types.MultiStakingHooks.BeforeDelegationSharesModified({_, _, _}, {{0x4d57760, 0x670c6e0}, {0x4d6d250, 0xc0b38a3900}, {{0xb, 0x0}, {0xc0b284c1c0, ...}, ...}, ...}, ...)
  Nov 24 02:07:07 s12217972 cosmovisor_dymension_1100-1[443359]: #011github.com/cosmos/cosmos-sdk@v0.46.16/x/staking/types/hooks.go:73 +0x10e
  Nov 24 02:07:07 s12217972 cosmovisor_dymension_1100-1[443359]: github.com/cosmos/cosmos-sdk/x/staking/keeper.Keeper.BeforeDelegationSharesModified(...)
  Nov 24 02:07:07 s12217972 cosmovisor_dymension_1100-1[443359]: #011github.com/cosmos/cosmos-sdk@v0.46.16/x/staking/keeper/hooks.go:62
  Nov 24 02:07:07 s12217972 cosmovisor_dymension_1100-1[443359]: github.com/cosmos/cosmos-sdk/x/staking/keeper.Keeper.Unbond({{0x4d36ee8, 0xc003b8c3f0}, {0x4d6d330, 0xc0009a5950}, {0x4d5ad00, 0xc003434280}, {0x7f06105b6000, 0xc001a27760}, {0x4d6b388, 0xc000aa9158}, ...}, ...)
  Nov 24 02:07:07 s12217972 cosmovisor_dymension_1100-1[443359]: #011github.com/cosmos/cosmos-sdk@v0.46.16/x/staking/keeper/delegation.go:710 +0x1cb
  Nov 24 02:07:07 s12217972 cosmovisor_dymension_1100-1[443359]: github.com/cosmos/cosmos-sdk/x/staking/keeper.Keeper.Undelegate({{0x4d36ee8, 0xc003b8c3f0}, {0x4d6d330, 0xc0009a5950}, {0x4d5ad00, 0xc003434280}, {0x7f06105b6000, 0xc001a27760}, {0x4d6b388, 0xc000aa9158}, ...}, ...)
  Nov 24 02:07:07 s12217972 cosmovisor_dymension_1100-1[443359]: #011github.com/cosmos/cosmos-sdk@v0.46.16/x/staking/keeper/delegation.go:811 +0x2b0
  Nov 24 02:07:07 s12217972 cosmovisor_dymension_1100-1[443359]: github.com/cosmos/cosmos-sdk/x/staking/keeper.msgServer.Undelegate({{{0x4d36ee8, 0xc003b8c3f0}, {0x4d6d330, 0xc0009a5950}, {0x4d5ad00, 0xc003434280}, {0x7f06105b6000, 0xc001a27760}, {0x4d6b388, 0xc000aa9158}, ...}}, ...)
  Nov 24 02:07:07 s12217972 cosmovisor_dymension_1100-1[443359]: #011github.com/cosmos/cosmos-sdk@v0.46.16/x/staking/keeper/msg_server.go:361 +0x473
  Nov 24 02:07:07 s12217972 cosmovisor_dymension_1100-1[443359]: github.com/cosmos/cosmos-sdk/x/staking/types._Msg_Undelegate_Handler.func1({0x4d57f78?, 0xc286c798f0?}, {0x2af6b20?, 0xc0b38a2180?})
  Nov 24 02:07:07 s12217972 cosmovisor_dymension_1100-1[443359]: #011github.com/cosmos/cosmos-sdk@v0.46.16/x/staking/types/tx.pb.go:944 +0xcb
  Nov 24 02:07:07 s12217972 cosmovisor_dymension_1100-1[443359]: github.com/cosmos/cosmos-sdk/baseapp.(*MsgServiceRouter).RegisterService.func2.1({0x4d57aa8, 0xc0f0b39348}, {0xc21f1e4480?, 0x412e05?}, 0x2a0?, 0xc0f1fbe7e0)
  Nov 24 02:07:07 s12217972 cosmovisor_dymension_1100-1[443359]: #011github.com/cosmos/cosmos-sdk@v0.46.16/baseapp/msg_service_router.go:118 +0x8a
  Nov 24 02:07:07 s12217972 cosmovisor_dymension_1100-1[443359]: github.com/cosmos/cosmos-sdk/x/staking/types._Msg_Undelegate_Handler({0x2b8d5c0, 0xc000d6f6b0}, {0x4d57aa8, 0xc0f0b39348}, 0x46f9748, 0xc1d216c980)
  Nov 24 02:07:07 s12217972 cosmovisor_dymension_1100-1[443359]: #011github.com/cosmos/cosmos-sdk@v0.46.16/x/staking/types/tx.pb.go:946 +0x143
  Nov 24 02:07:07 s12217972 cosmovisor_dymension_1100-1[443359]: github.com/cosmos/cosmos-sdk/baseapp.(*MsgServiceRouter).RegisterService.func2({{0x4d57760, 0x670c6e0}, {0x4d6d250, 0xc0b38a3900}, {{0xb, 0x0}, {0xc0b284c1c0, 0x10}, 0x4424b7, {0x2e090b81, ...}, ...}, ...}, ...)
  Nov 24 02:07:07 s12217972 cosmovisor_dymension_1100-1[443359]: #011github.com/cosmos/cosmos-sdk@v0.46.16/baseapp/msg_service_router.go:145 +0x2e3
  Nov 24 02:07:07 s12217972 cosmovisor_dymension_1100-1[443359]: github.com/cosmos/cosmos-sdk/baseapp.(*BaseApp).runMsgs(_, {{0x4d57760, 0x670c6e0}, {0x4d6d250, 0xc0b38a3900}, {{0xb, 0x0}, {0xc0b284c1c0, 0x10}, 0x4424b7, ...}, ...}, ...)
  Nov 24 02:07:07 s12217972 cosmovisor_dymension_1100-1[443359]: #011github.com/cosmos/cosmos-sdk@v0.46.16/baseapp/baseapp.go:788 +0x2c5
  Nov 24 02:07:07 s12217972 cosmovisor_dymension_1100-1[443359]: github.com/cosmos/cosmos-sdk/baseapp.(*BaseApp).runTx(0xc001b01dc0, 0x2, {0xc09db8f380, 0x16f, 0x180})
  Nov 24 02:07:07 s12217972 cosmovisor_dymension_1100-1[443359]: #011github.com/cosmos/cosmos-sdk@v0.46.16/baseapp/baseapp.go:728 +0xd05
  Nov 24 02:07:07 s12217972 cosmovisor_dymension_1100-1[443359]: github.com/cosmos/cosmos-sdk/baseapp.(*BaseApp).Simulate(...)
  Nov 24 02:07:07 s12217972 cosmovisor_dymension_1100-1[443359]: #011github.com/cosmos/cosmos-sdk@v0.46.16/baseapp/test_helpers.go:25
  Nov 24 02:07:07 s12217972 cosmovisor_dymension_1100-1[443359]: github.com/cosmos/cosmos-sdk/x/auth/tx.txServer.Simulate({{{0x0, 0x0, 0x0}, {0x4d83528, 0xc000a54000}, 0x0, {0xc001a02450, 0x10}, {0x4d75740, 0xc0009a5950}, ...}, ...}, ...)
  Nov 24 02:07:07 s12217972 cosmovisor_dymension_1100-1[443359]: #011github.com/cosmos/cosmos-sdk@v0.46.16/x/auth/tx/service.go:132 +0x12f
  Nov 24 02:07:07 s12217972 cosmovisor_dymension_1100-1[443359]: github.com/cosmos/cosmos-sdk/types/tx._Service_Simulate_Handler.func1({0x4d57f78?, 0xc28bb12f00?}, {0x2a92360?, 0xc07b23c380?})
  Nov 24 02:07:07 s12217972 cosmovisor_dymension_1100-1[443359]: #011github.com/cosmos/cosmos-sdk@v0.46.16/types/tx/service.pb.go:933 +0xcb
  Nov 24 02:07:07 s12217972 cosmovisor_dymension_1100-1[443359]: github.com/cosmos/cosmos-sdk/baseapp.(*BaseApp).RegisterGRPCServer.func1({0x4d57f78, 0xc088c786c0}, {0x2a92360, 0xc07b23c380}, 0xc07b23c3a0?, 0xc0f0272450)
  Nov 24 02:07:07 s12217972 cosmovisor_dymension_1100-1[443359]: #011github.com/cosmos/cosmos-sdk@v0.46.16/baseapp/grpcserver.go:68 +0x391
  Nov 24 02:07:07 s12217972 cosmovisor_dymension_1100-1[443359]: github.com/cosmos/cosmos-sdk/baseapp.(*BaseApp).RegisterGRPCServer.func2.ChainUnaryServer.1.1.1({0x4d57f78?, 0xc088c786c0?}, {0x2a92360?, 0xc07b23c380?})
  Nov 24 02:07:07 s12217972 cosmovisor_dymension_1100-1[443359]: #011github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25 +0x34
  Nov 24 02:07:07 s12217972 cosmovisor_dymension_1100-1[443359]: github.com/grpc-ecosystem/go-grpc-middleware/recovery.UnaryServerInterceptor.func1({0x4d57f78?, 0xc088c786c0?}, {0x2a92360?, 0xc07b23c380?}, 0x18?, 0xc07b23c3a0?)
  Nov 24 02:07:07 s12217972 cosmovisor_dymension_1100-1[443359]: #011github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/recovery/interceptors.go:33 +0xb0
  Nov 24 02:07:07 s12217972 cosmovisor_dymension_1100-1[443359]: github.com/cosmos/cosmos-sdk/baseapp.(*BaseApp).RegisterGRPCServer.func2.ChainUnaryServer.1.1.1({0x4d57f78?, 0xc088c786c0?}, {0x2a92360?, 0xc07b23c380?})
  Nov 24 02:07:07 s12217972 cosmovisor_dymension_1100-1[443359]: #011github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25 +0x34
  Nov 24 02:07:07 s12217972 cosmovisor_dymension_1100-1[443359]: github.com/cosmos/cosmos-sdk/baseapp.(*BaseApp).RegisterGRPCServer.func2.ChainUnaryServer.1({0x4d57f78, 0xc088c786c0}, {0x2a92360, 0xc07b23c380}, 0x415dc5?, 0x28?)
  Nov 24 02:07:07 s12217972 cosmovisor_dymension_1100-1[443359]: #011github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:34 +0xb8
  Nov 24 02:07:07 s12217972 cosmovisor_dymension_1100-1[443359]: github.com/cosmos/cosmos-sdk/types/tx._Service_Simulate_Handler({0x2a08940, 0xc0050b6540}, {0x4d57f78, 0xc088c786c0}, 0xc0ebf79f00, 0xc088c786f0)
  Nov 24 02:07:07 s12217972 cosmovisor_dymension_1100-1[443359]: #011github.com/cosmos/cosmos-sdk@v0.46.16/types/tx/service.pb.go:935 +0x143
  Nov 24 02:07:07 s12217972 cosmovisor_dymension_1100-1[443359]: github.com/cosmos/cosmos-sdk/baseapp.(*BaseApp).RegisterGRPCServer.func2({0x2a08940, 0xc0050b6540}, {0x4d57f78, 0xc088c786c0}, 0xc0ebf79f00, 0xc0ee7a2480?)
  Nov 24 02:07:07 s12217972 cosmovisor_dymension_1100-1[443359]: #011github.com/cosmos/cosmos-sdk@v0.46.16/baseapp/grpcserver.go:82 +0x112
  Nov 24 02:07:07 s12217972 cosmovisor_dymension_1100-1[443359]: google.golang.org/grpc.(*Server).processUnaryRPC(0xc000337400, {0x4d57f78, 0xc088c78450}, {0x4d6a680, 0xc016747040}, 0xc0ee7a2480, 0xc037723140, 0xc034aefb80, 0x0)
  Nov 24 02:07:07 s12217972 cosmovisor_dymension_1100-1[443359]: #011google.golang.org/grpc@v1.61.0/server.go:1385 +0xdd1
  Nov 24 02:07:07 s12217972 cosmovisor_dymension_1100-1[443359]: google.golang.org/grpc.(*Server).handleStream(0xc000337400, {0x4d6a680, 0xc016747040}, 0xc0ee7a2480)
  Nov 24 02:07:07 s12217972 cosmovisor_dymension_1100-1[443359]: #011google.golang.org/grpc@v1.61.0/server.go:1796 +0xfb8
  Nov 24 02:07:07 s12217972 cosmovisor_dymension_1100-1[443359]: google.golang.org/grpc.(*Server).serveStreams.func2.1()
  Nov 24 02:07:07 s12217972 cosmovisor_dymension_1100-1[443359]: #011google.golang.org/grpc@v1.61.0/server.go:1029 +0x8b
  Nov 24 02:07:07 s12217972 cosmovisor_dymension_1100-1[443359]: created by google.golang.org/grpc.(*Server).serveStreams.func2 in goroutine 5537
  Nov 24 02:07:07 s12217972 cosmovisor_dymension_1100-1[443359]: #011google.golang.org/grpc@v1.61.0/server.go:1040 +0x125
  Nov 24 02:07:07 s12217972 cosmovisor_dymension_1100-1[443359]: goroutine 1 [chan receive, 47351 minutes]:
  Nov 24 02:07:07 s12217972 cosmovisor_dymension_1100-1[443359]: github.com/cosmos/cosmos-sdk/server.WaitForQuitSignals()
  Nov 24 02:07:07 s12217972 cosmovisor_dymension_1100-1[443359]: #011github.com/cosmos/cosmos-sdk@v0.46.16/server/util.go:363 +0x86
  Nov 24 02:07:07 s12217972 cosmovisor_dymension_1100-1[443359]: github.com/evmos/ethermint/server.startInProcess(_, {{0x0, 0x0, 0x0}, {0x4d83528, 0xc000a54000}, 0xc00a5d2008, {0xc00cb492d0, 0x10}, {0x4d75740, ...}, ...}, ...)
  Nov 24 02:07:07 s12217972 cosmovisor_dymension_1100-1[443359]: #011github.com/evmos/ethermint@v0.22.0/server/start.go:644 +0x2dd6
  Nov 24 02:07:07 s12217972 cosmovisor_dymension_1100-1[443359]: github.com/evmos/ethermint/server.StartCmd.func2(0xc000d40008, {0xc0006d7100?, 0x0?, 0x2?})
  Nov 24 02:07:07 s12217972 cosmovisor_dymension_1100-1[443359]: #011github.com/evmos/ethermint@v0.22.0/server/start.go:158 +0x2e5
  Nov 24 02:07:07 s12217972 cosmovisor_dymension_1100-1[443359]: github.com/spf13/cobra.(*Command).execute(0xc000d40008, {0xc0006d70e0, 0x2, 0x2})
  Nov 24 02:07:07 s12217972 cosmovisor_dymension_1100-1[443359]: #011github.com/spf13/cobra@v1.8.0/command.go:983 +0xaca
  Nov 24 02:07:07 s12217972 cosmovisor_dymension_1100-1[443359]: github.com/spf13/cobra.(*Command).ExecuteC(0xc000d12008)
  Nov 24 02:07:07 s12217972 cosmovisor_dymension_1100-1[443359]: #011github.com/spf13/cobra@v1.8.0/command.go:1115 +0x3ff
  Nov 24 02:07:07 s12217972 cosmovisor_dymension_1100-1[443359]: github.com/spf13/cobra.(*Command).Execute(...)
  Nov 24 02:07:07 s12217972 cosmovisor_dymension_1100-1[443359]: #011github.com/spf13/cobra@v1.8.0/command.go:1039
  Nov 24 02:07:07 s12217972 cosmovisor_dymension_1100-1[443359]: github.com/spf13/cobra.(*Command).ExecuteContext(...)
  Nov 24 02:07:07 s12217972 cosmovisor_dymension_1100-1[443359]: #011github.com/spf13/cobra@v1.8.0/command.go:1032
  Nov 24 02:07:07 s12217972 cosmovisor_dymension_1100-1[443359]: github.com/cosmos/cosmos-sdk/server/cmd.Execute(0xc000d12008, {0x0, 0x0}, {0xc0009c4400, 0x1a})
  Nov 24 02:07:07 s12217972 cosmovisor_dymension_1100-1[443359]: #011github.com/cosmos/cosmos-sdk@v0.46.16/server/cmd/execute.go:36 +0x176
  Nov 24 02:07:07 s12217972 cosmovisor_dymension_1100-1[443359]: main.main()
  Nov 24 02:07:07 s12217972 cosmovisor_dymension_1100-1[443359]: #011github.com/dymensionxyz/dymension/v3/cmd/dymd/main.go:15 +0x2a
  Nov 24 02:07:07 s12217972 cosmovisor_dymension_1100-1[443359]: goroutine 5 [select, 47352 minutes]:
  Nov 24 02:07:07 s12217972 cosmovisor_dymension_1100-1[443359]: github.com/desertbit/timer.timerRoutine()
  Nov 24 02:07:07 s12217972 cosmovisor_dymension_1100-1[443359]: #011github.com/desertbit/timer@v0.0.0-20180107155436-c41aec40b27f/timers.go:119 +0xac
  Nov 24 02:07:07 s12217972 cosmovisor_dymension_1100-1[443359]: created by github.com/desertbit/timer.init.0 in goroutine 1
  Nov 24 02:07:07 s12217972 cosmovisor_dymension_1100-1[443359]: #011github.com/desertbit/timer@v0.0.0-20180107155436-c41aec40b27f/timers.go:15 +0x1a
  Nov 24 02:07:07 s12217972 cosmovisor_dymension_1100-1[443359]: goroutine 43 [chan receive, 47352 minutes]:
  Nov 24 02:07:07 s12217972 cosmovisor_dymension_1100-1[443359]: github.com/rjeczalik/notify.(*nonrecursiveTree).dispatch(0xc000202180, 0xc00038e240)
  Nov 24 02:07:07 s12217972 cosmovisor_dymension_1100-1[443359]: #011github.com/rjeczalik/notify@v0.9.3/tree_nonrecursive.go:36 +0x5a
  Nov 24 02:07:07 s12217972 cosmovisor_dymension_1100-1[443359]: created by github.com/rjeczalik/notify.newNonrecursiveTree in goroutine 1
  Nov 24 02:07:07 s12217972 cosmovisor_dymension_1100-1[443359]: #011github.com/rjeczalik/notify@v0.9.3/tree_nonrecursive.go:29 +0x132
  Nov 24 02:07:07 s12217972 cosmovisor_dymension_1100-1[443359]: goroutine 44 [chan receive, 47352 minutes]:
  Nov 24 02:07:07 s12217972 cosmovisor_dymension_1100-1[443359]: github.com/rjeczalik/notify.(*nonrecursiveTree).internal(0xc000202180, 0xc000202120)
  Nov 24 02:07:07 s12217972 cosmovisor_dymension_1100-1[443359]: #011github.com/rjeczalik/notify@v0.9.3/tree_nonrecursive.go:81 +0x4d
  Nov 24 02:07:07 s12217972 cosmovisor_dymension_1100-1[443359]: created by github.com/rjeczalik/notify.newNonrecursiveTree in goroutine 1
  Nov 24 02:07:07 s12217972 cosmovisor_dymension_1100-1[443359]: #011github.com/rjeczalik/notify@v0.9.3/tree_nonrecursive.go:30 +0x185
  Nov 24 02:07:07 s12217972 cosmovisor_dymension_1100-1[443359]: goroutine 43 [chan receive, 47352 minutes]:
  Nov 24 02:07:07 s12217972 cosmovisor_dymension_1100-1[443359]: github.com/rjeczalik/notify.(*nonrecursiveTree).dispatch(0xc000202180, 0xc00038e240)
  Nov 24 02:07:07 s12217972 cosmovisor_dymension_1100-1[443359]: #011github.com/rjeczalik/notify@v0.9.3/tree_nonrecursive.go:36 +0x5a
  Nov 24 02:07:07 s12217972 cosmovisor_dymension_1100-1[443359]: created by github.com/rjeczalik/notify.newNonrecursiveTree in goroutine 1
  Nov 24 02:07:07 s12217972 cosmovisor_dymension_1100-1[443359]: #011github.com/rjeczalik/notify@v0.9.3/tree_nonrecursive.go:29 +0x132
  Nov 24 02:07:07 s12217972 cosmovisor_dymension_1100-1[443359]: goroutine 44 [chan receive, 47352 minutes]:
  Nov 24 02:07:07 s12217972 cosmovisor_dymension_1100-1[443359]: github.com/rjeczalik/notify.(*nonrecursiveTree).internal(0xc000202180, 0xc000202120)
  Nov 24 02:07:07 s12217972 cosmovisor_dymension_1100-1[443359]: #011github.com/rjeczalik/notify@v0.9.3/tree_nonrecursive.go:81 +0x4d
  Nov 24 02:07:07 s12217972 cosmovisor_dymension_1100-1[443359]: created by github.com/rjeczalik/notify.newNonrecursiveTree in goroutine 1
  Nov 24 02:07:07 s12217972 cosmovisor_dymension_1100-1[443359]: #011github.com/rjeczalik/notify@v0.9.3/tree_nonrecursive.go:30 +0x185
  Nov 24 02:07:07 s12217972 cosmovisor_dymension_1100-1[443359]: goroutine 9 [select]:
  Nov 24 02:07:07 s12217972 cosmovisor_dymension_1100-1[443359]: go.opencensus.io/stats/view.(*worker).start(0xc0003cac80)
  Nov 24 02:07:07 s12217972 cosmovisor_dymension_1100-1[443359]: #011go.opencensus.io@v0.24.0/stats/view/worker.go:292 +0x9f
  Nov 24 02:07:07 s12217972 cosmovisor_dymension_1100-1[443359]: created by go.opencensus.io/stats/view.init.0 in goroutine 1
  Nov 24 02:07:07 s12217972 cosmovisor_dymension_1100-1[443359]: #011go.opencensus.io@v0.24.0/stats/view/worker.go:34 +0x8d
  Nov 24 02:07:07 s12217972 cosmovisor_dymension_1100-1[443359]: goroutine 152 [select, 3 minutes]:
  Nov 24 02:07:07 s12217972 cosmovisor_dymension_1100-1[443359]: github.com/syndtr/goleveldb/leveldb.(*DB).mCompaction(0xc001a80c40)
  Nov 24 02:07:07 s12217972 cosmovisor_dymension_1100-1[443359]: #011github.com/syndtr/goleveldb@v1.0.1-0.20220721030215-126854af5e6d/leveldb/db_compaction.go:773 +0x105
  Nov 24 02:07:07 s12217972 cosmovisor_dymension_1100-1[443359]: created by github.com/syndtr/goleveldb/leveldb.openDB in goroutine 1
  Nov 24 02:07:07 s12217972 cosmovisor_dymension_1100-1[443359]: #011github.com/syndtr/goleveldb@v1.0.1-0.20220721030215-126854af5e6d/leveldb/db.go:156 +0x536
  Nov 24 02:07:07 s12217972 cosmovisor_dymension_1100-1[443359]: goroutine 66 [chan receive]:
  Nov 24 02:07:07 s12217972 cosmovisor_dymension_1100-1[443359]: github.com/ethereum/go-ethereum/metrics.(*meterArbiter).tick(...)
  Nov 24 02:07:07 s12217972 cosmovisor_dymension_1100-1[443359]: #011github.com/ethereum/go-ethereum@v1.10.26/metrics/meter.go:290
  Nov 24 02:07:07 s12217972 cosmovisor_dymension_1100-1[443359]: created by github.com/ethereum/go-ethereum/metrics.NewMeterForced in goroutine 1
  Nov 24 02:07:07 s12217972 cosmovisor_dymension_1100-1[443359]: #011github.com/ethereum/go-ethereum@v1.10.26/metrics/meter.go:71 +0xb7
  Nov 24 02:07:07 s12217972 cosmovisor_dymension_1100-1[443359]: goroutine 67 [select]:
  Nov 24 02:07:07 s12217972 cosmovisor_dymension_1100-1[443359]: github.com/ethereum/go-ethereum/consensus/ethash.(*remoteSealer).loop(0xc0008780f0)
  Nov 24 02:07:07 s12217972 cosmovisor_dymension_1100-1[443359]: #011github.com/ethereum/go-ethereum@v1.10.26/consensus/ethash/sealer.go:279 +0x205
  Nov 24 02:07:07 s12217972 cosmovisor_dymension_1100-1[443359]: created by github.com/ethereum/go-ethereum/consensus/ethash.startRemoteSealer in goroutine 1
  Nov 24 02:07:07 s12217972 cosmovisor_dymension_1100-1[443359]: #011github.com/ethereum/go-ethereum@v1.10.26/consensus/ethash/sealer.go:263 +0x2b6
  Nov 24 02:07:07 s12217972 cosmovisor_dymension_1100-1[443359]: goroutine 68 [chan receive, 47352 minutes]:
  Nov 24 02:07:07 s12217972 cosmovisor_dymension_1100-1[443359]: github.com/ethereum/go-ethereum/core.(*txSenderCacher).cache(0x0?)
  Nov 24 02:07:07 s12217972 cosmovisor_dymension_1100-1[443359]: #011github.com/ethereum/go-ethereum@v1.10.26/core/tx_cacher.go:63 +0x35
  Nov 24 02:07:07 s12217972 cosmovisor_dymension_1100-1[443359]: created by github.com/ethereum/go-ethereum/core.newTxSenderCacher in goroutine 1
  Nov 24 02:07:07 s12217972 cosmovisor_dymension_1100-1[443359]: #011github.com/ethereum/go-ethereum@v1.10.26/core/tx_cacher.go:55 +0x72

[mtsitrin]

Hi @kimheeje12 Is it reproducible? On what scenario did it happened to u?