search

celestiaorg / celestia-app

Celestia consensus node
https://celestiaorg.github.io/celestia-app/
Apache License 2.0
347 stars 297 forks source link

bug: consensus node panics in concurrent map read-writes #2840

Open Bidon15 opened 1 year ago

Bidon15 commented 1 year ago

Summary of Bug

Happened on mainnet for devops infra

RAM consumption was 8Gib which was fine

Version

v1.3.0

Steps to Reproduce

run a node for a while

https://celestia.grafana.net/goto/ath5JDSIg?orgId=1


2023-11-14 08:32:47.0697:32AM INF received complete proposal block hash=48EA0DD7CFEB4EABD956B14B159D20170FCD4402583B2B31CBCBE45ACDC17B0A height=101331 module=consensus | 2023-11-14 08:32:47.069 | 7:32AM INF received complete proposal block hash=48EA0DD7CFEB4EABD956B14B159D20170FCD4402583B2B31CBCBE45ACDC17B0A height=101331 module=consensus |  
-- | -- | -- | --
2023-11-14 08:32:47.069 | 7:32AM INF received complete proposal block hash=48EA0DD7CFEB4EABD956B14B159D20170FCD4402583B2B31CBCBE45ACDC17B0A height=101331 module=consensus |  
2023-11-14 08:32:47.6027:32AM INF finalizing commit of block hash={} height=101331 module=consensus num_txs=1 root=B9CFF4FA62EB8B44DEE706C505A064C5056E02574D3DC06B0013F4981C67BAAF2023-11-14 08:32:47.6377:32AM INF minted coins from module account amount=29840334utia from=mint module=x/bank2023-11-14 08:32:47.6517:32AM INF executed block height=101331 module=state num_invalid_txs=0 num_valid_txs=12023-11-14 08:32:47.781fatal error: concurrent map iteration and map write2023-11-14 08:32:47.7852023-11-14 08:32:47.785goroutine 15580584 [running]:2023-11-14 08:32:47.785github.com/cosmos/iavl.NewUnsavedFastIterator({0xc0474ca4e0, 0x1e, 0x20}, {0xc0474ca800, 0x1e, 0x20}, 0x1, 0xc00098f600, 0xc0526eede0, 0xc0526eee70)2023-11-14 08:32:47.785   /go/pkg/mod/github.com/cosmos/iavl@v0.19.6/unsaved_fast_iterator.go:55 +0x2192023-11-14 08:32:47.785github.com/cosmos/iavl.(*MutableTree).Iterator(0xc0019bcf50, {0xc0474ca4e0, 0x1e, 0x20}, {0xc0474ca800, 0x1e, 0x20}, 0x60?)2023-11-14 08:32:47.785  /go/pkg/mod/github.com/cosmos/iavl@v0.19.6/mutable_tree.go:216 +0x1052023-11-14 08:32:47.785github.com/cosmos/cosmos-sdk/store/iavl.(*Store).Iterator(0xc111643f80?, {0xc0474ca4e0?, 0x20?, 0xc111643fc0?}, {0xc0474ca800?, 0x20?, 0xc03f3f4000?})2023-11-14 08:32:47.785   /go/pkg/mod/github.com/celestiaorg/cosmos-sdk@v1.18.3-sdk-v0.46.14/store/iavl/store.go:249 +0x2b2023-11-14 08:32:47.785github.com/cosmos/cosmos-sdk/store/cachekv.(*Store).iterator(0xc0c107a680, {0xc0474ca4e0, 0x1e, 0x20}, {0xc0474ca800, 0x1e, 0x20}, 0x1)2023-11-14 08:32:47.785   /go/pkg/mod/github.com/celestiaorg/cosmos-sdk@v1.18.3-sdk-v0.46.14/store/cachekv/store.go:185 +0x11d2023-11-14 08:32:47.785github.com/cosmos/cosmos-sdk/store/cachekv.(*Store).Iterator(0x3?, {0xc0474ca4e0?, 0x1e?, 0x1e?}, {0xc0474ca800?, 0x64?, 0x64?})2023-11-14 08:32:47.785  /go/pkg/mod/github.com/celestiaorg/cosmos-sdk@v1.18.3-sdk-v0.46.14/store/cachekv/store.go:170 +0x252023-11-14 08:32:47.785github.com/cosmos/cosmos-sdk/store/cachekv.(*Store).iterator(0xc03ffdb480, {0xc0474ca4e0, 0x1e, 0x20}, {0xc0474ca800, 0x1e, 0x20}, 0x1)2023-11-14 08:32:47.785    /go/pkg/mod/github.com/celestiaorg/cosmos-sdk@v1.18.3-sdk-v0.46.14/store/cachekv/store.go:185 +0x11d2023-11-14 08:32:47.785github.com/cosmos/cosmos-sdk/store/cachekv.(*Store).Iterator(0xc111643f20?, {0xc0474ca4e0?, 0x20?, 0xc111643f40?}, {0xc0474ca800?, 0x20?, 0xc111643f60?})2023-11-14 08:32:47.785 /go/pkg/mod/github.com/celestiaorg/cosmos-sdk@v1.18.3-sdk-v0.46.14/store/cachekv/store.go:170 +0x252023-11-14 08:32:47.785github.com/cosmos/cosmos-sdk/store/cachekv.(*Store).iterator(0xc03520e1c0, {0xc0474ca4e0, 0x1e, 0x20}, {0xc0474ca800, 0x1e, 0x20}, 0x1)2023-11-14 08:32:47.785    /go/pkg/mod/github.com/celestiaorg/cosmos-sdk@v1.18.3-sdk-v0.46.14/store/cachekv/store.go:185 +0x11d2023-11-14 08:32:47.785github.com/cosmos/cosmos-sdk/store/cachekv.(*Store).Iterator(0xc000581000?, {0xc0474ca4e0?, 0x0?, 0xc14afe90e0?}, {0xc0474ca800?, 0x20?, 0x0?})2023-11-14 08:32:47.785   /go/pkg/mod/github.com/celestiaorg/cosmos-sdk@v1.18.3-sdk-v0.46.14/store/cachekv/store.go:170 +0x252023-11-14 08:32:47.785github.com/cosmos/cosmos-sdk/store/gaskv.(*Store).iterator(0xc0d3f5dec0, {0xc0474ca4e0?, 0x64?, 0x15?}, {0xc0474ca800?, 0x20?, 0xd38b010000000000?}, 0x0?)2023-11-14 08:32:47.785 /go/pkg/mod/github.com/celestiaorg/cosmos-sdk@v1.18.3-sdk-v0.46.14/store/gaskv/store.go:98 +0x482023-11-14 08:32:47.785github.com/cosmos/cosmos-sdk/store/gaskv.(*Store).Iterator(0xc021cb62e8?, {0xc0474ca4e0?, 0x3045a50?, 0xc03520e240?}, {0xc0474ca800?, 0x1?, 0xc0776c68a0?})2023-11-14 08:32:47.785   /go/pkg/mod/github.com/celestiaorg/cosmos-sdk@v1.18.3-sdk-v0.46.14/store/gaskv/store.go:74 +0x252023-11-14 08:32:47.785github.com/cosmos/cosmos-sdk/x/distribution/keeper.Keeper.IterateValidatorSlashEventsBetween({{0x3019af8, 0xc0008b5400}, {0x3045b30, 0xc0008b5120}, {{0x3045b30, 0xc0008b5120}, 0xc0000b14c8, {0x3019af8, 0xc0008b5430}, {0x3019b20, ...}, ...}, ...}, ...)2023-11-14 08:32:47.785   /go/pkg/mod/github.com/celestiaorg/cosmos-sdk@v1.18.3-sdk-v0.46.14/x/distribution/keeper/store.go:337 +0x1472023-11-14 08:32:47.785github.com/cosmos/cosmos-sdk/x/distribution/keeper.Keeper.CalculateDelegationRewards({{0x3019af8, 0xc0008b5400}, {0x3045b30, 0xc0008b5120}, {{0x3045b30, 0xc0008b5120}, 0xc0000b14c8, {0x3019af8, 0xc0008b5430}, {0x3019b20, ...}, ...}, ...}, ...)2023-11-14 08:32:47.785   /go/pkg/mod/github.com/celestiaorg/cosmos-sdk@v1.18.3-sdk-v0.46.14/x/distribution/keeper/delegation.go:80 +0x2d92023-11-14 08:32:47.785github.com/cosmos/cosmos-sdk/x/distribution/keeper.Keeper.withdrawDelegationRewards({{0x3019af8, 0xc0008b5400}, {0x3045b30, 0xc0008b5120}, {{0x3045b30, 0xc0008b5120}, 0xc0000b14c8, {0x3019af8, 0xc0008b5430}, {0x3019b20, ...}, ...}, ...}, ...) |   |   | 2023-11-14 08:32:47.602 | 7:32AM INF finalizing commit of block hash={} height=101331 module=consensus num_txs=1 root=B9CFF4FA62EB8B44DEE706C505A064C5056E02574D3DC06B0013F4981C67BAAF |   |   |   | 2023-11-14 08:32:47.637 | 7:32AM INF minted coins from module account amount=29840334utia from=mint module=x/bank |   |   |   | 2023-11-14 08:32:47.651 | 7:32AM INF executed block height=101331 module=state num_invalid_txs=0 num_valid_txs=1 |   |   |   | 2023-11-14 08:32:47.781 | fatal error: concurrent map iteration and map write |   |   |   | 2023-11-14 08:32:47.785 |   |   |   |   | 2023-11-14 08:32:47.785 | goroutine 15580584 [running]: |   |   |   | 2023-11-14 08:32:47.785 | github.com/cosmos/iavl.NewUnsavedFastIterator({0xc0474ca4e0, 0x1e, 0x20}, {0xc0474ca800, 0x1e, 0x20}, 0x1, 0xc00098f600, 0xc0526eede0, 0xc0526eee70) |   |   |   | 2023-11-14 08:32:47.785 | /go/pkg/mod/github.com/cosmos/iavl@v0.19.6/unsaved_fast_iterator.go:55 +0x219 |   |   |   | 2023-11-14 08:32:47.785 | github.com/cosmos/iavl.(*MutableTree).Iterator(0xc0019bcf50, {0xc0474ca4e0, 0x1e, 0x20}, {0xc0474ca800, 0x1e, 0x20}, 0x60?) |   |   |   | 2023-11-14 08:32:47.785 | /go/pkg/mod/github.com/cosmos/iavl@v0.19.6/mutable_tree.go:216 +0x105 |   |   |   | 2023-11-14 08:32:47.785 | github.com/cosmos/cosmos-sdk/store/iavl.(*Store).Iterator(0xc111643f80?, {0xc0474ca4e0?, 0x20?, 0xc111643fc0?}, {0xc0474ca800?, 0x20?, 0xc03f3f4000?}) |   |   |   | 2023-11-14 08:32:47.785 | /go/pkg/mod/github.com/celestiaorg/cosmos-sdk@v1.18.3-sdk-v0.46.14/store/iavl/store.go:249 +0x2b |   |   |   | 2023-11-14 08:32:47.785 | github.com/cosmos/cosmos-sdk/store/cachekv.(*Store).iterator(0xc0c107a680, {0xc0474ca4e0, 0x1e, 0x20}, {0xc0474ca800, 0x1e, 0x20}, 0x1) |   |   |   | 2023-11-14 08:32:47.785 | /go/pkg/mod/github.com/celestiaorg/cosmos-sdk@v1.18.3-sdk-v0.46.14/store/cachekv/store.go:185 +0x11d |   |   |   | 2023-11-14 08:32:47.785 | github.com/cosmos/cosmos-sdk/store/cachekv.(*Store).Iterator(0x3?, {0xc0474ca4e0?, 0x1e?, 0x1e?}, {0xc0474ca800?, 0x64?, 0x64?}) |   |   |   | 2023-11-14 08:32:47.785 | /go/pkg/mod/github.com/celestiaorg/cosmos-sdk@v1.18.3-sdk-v0.46.14/store/cachekv/store.go:170 +0x25 |   |   |   | 2023-11-14 08:32:47.785 | github.com/cosmos/cosmos-sdk/store/cachekv.(*Store).iterator(0xc03ffdb480, {0xc0474ca4e0, 0x1e, 0x20}, {0xc0474ca800, 0x1e, 0x20}, 0x1) |   |   |   | 2023-11-14 08:32:47.785 | /go/pkg/mod/github.com/celestiaorg/cosmos-sdk@v1.18.3-sdk-v0.46.14/store/cachekv/store.go:185 +0x11d |   |   |   | 2023-11-14 08:32:47.785 | github.com/cosmos/cosmos-sdk/store/cachekv.(*Store).Iterator(0xc111643f20?, {0xc0474ca4e0?, 0x20?, 0xc111643f40?}, {0xc0474ca800?, 0x20?, 0xc111643f60?}) |   |   |   | 2023-11-14 08:32:47.785 | /go/pkg/mod/github.com/celestiaorg/cosmos-sdk@v1.18.3-sdk-v0.46.14/store/cachekv/store.go:170 +0x25 |   |   |   | 2023-11-14 08:32:47.785 | github.com/cosmos/cosmos-sdk/store/cachekv.(*Store).iterator(0xc03520e1c0, {0xc0474ca4e0, 0x1e, 0x20}, {0xc0474ca800, 0x1e, 0x20}, 0x1) |   |   |   | 2023-11-14 08:32:47.785 | /go/pkg/mod/github.com/celestiaorg/cosmos-sdk@v1.18.3-sdk-v0.46.14/store/cachekv/store.go:185 +0x11d |   |   |   | 2023-11-14 08:32:47.785 | github.com/cosmos/cosmos-sdk/store/cachekv.(*Store).Iterator(0xc000581000?, {0xc0474ca4e0?, 0x0?, 0xc14afe90e0?}, {0xc0474ca800?, 0x20?, 0x0?}) |   |   |   | 2023-11-14 08:32:47.785 | /go/pkg/mod/github.com/celestiaorg/cosmos-sdk@v1.18.3-sdk-v0.46.14/store/cachekv/store.go:170 +0x25 |   |   |   | 2023-11-14 08:32:47.785 | github.com/cosmos/cosmos-sdk/store/gaskv.(*Store).iterator(0xc0d3f5dec0, {0xc0474ca4e0?, 0x64?, 0x15?}, {0xc0474ca800?, 0x20?, 0xd38b010000000000?}, 0x0?) |   |   |   | 2023-11-14 08:32:47.785 | /go/pkg/mod/github.com/celestiaorg/cosmos-sdk@v1.18.3-sdk-v0.46.14/store/gaskv/store.go:98 +0x48 |   |   |   | 2023-11-14 08:32:47.785 | github.com/cosmos/cosmos-sdk/store/gaskv.(*Store).Iterator(0xc021cb62e8?, {0xc0474ca4e0?, 0x3045a50?, 0xc03520e240?}, {0xc0474ca800?, 0x1?, 0xc0776c68a0?}) |   |   |   | 2023-11-14 08:32:47.785 | /go/pkg/mod/github.com/celestiaorg/cosmos-sdk@v1.18.3-sdk-v0.46.14/store/gaskv/store.go:74 +0x25 |   |   |   | 2023-11-14 08:32:47.785 | github.com/cosmos/cosmos-sdk/x/distribution/keeper.Keeper.IterateValidatorSlashEventsBetween({{0x3019af8, 0xc0008b5400}, {0x3045b30, 0xc0008b5120}, {{0x3045b30, 0xc0008b5120}, 0xc0000b14c8, {0x3019af8, 0xc0008b5430}, {0x3019b20, ...}, ...}, ...}, ...) |   |   |   | 2023-11-14 08:32:47.785 | /go/pkg/mod/github.com/celestiaorg/cosmos-sdk@v1.18.3-sdk-v0.46.14/x/distribution/keeper/store.go:337 +0x147 |   |   |   | 2023-11-14 08:32:47.785 | github.com/cosmos/cosmos-sdk/x/distribution/keeper.Keeper.CalculateDelegationRewards({{0x3019af8, 0xc0008b5400}, {0x3045b30, 0xc0008b5120}, {{0x3045b30, 0xc0008b5120}, 0xc0000b14c8, {0x3019af8, 0xc0008b5430}, {0x3019b20, ...}, ...}, ...}, ...) |   |   |   | 2023-11-14 08:32:47.785 | /go/pkg/mod/github.com/celestiaorg/cosmos-sdk@v1.18.3-sdk-v0.46.14/x/distribution/keeper/delegation.go:80 +0x2d9 |   |   |   | 2023-11-14 08:32:47.785 | github.com/cosmos/cosmos-sdk/x/distribution/keeper.Keeper.withdrawDelegationRewards({{0x3019af8, 0xc0008b5400}, {0x3045b30, 0xc0008b5120}, {{0x3045b30, 0xc0008b5120}, 0xc0000b14c8, {0x3019af8, 0xc0008b5430}, {0x3019b20, ...}, ...}, ...}, ...)
  |   | 2023-11-14 08:32:47.602 | 7:32AM INF finalizing commit of block hash={} height=101331 module=consensus num_txs=1 root=B9CFF4FA62EB8B44DEE706C505A064C5056E02574D3DC06B0013F4981C67BAAF |  
  |   | 2023-11-14 08:32:47.637 | 7:32AM INF minted coins from module account amount=29840334utia from=mint module=x/bank |  
  |   | 2023-11-14 08:32:47.651 | 7:32AM INF executed block height=101331 module=state num_invalid_txs=0 num_valid_txs=1 |  
  |   | 2023-11-14 08:32:47.781 | fatal error: concurrent map iteration and map write |  
  |   | 2023-11-14 08:32:47.785 |   |  
  |   | 2023-11-14 08:32:47.785 | goroutine 15580584 [running]: |  
  |   | 2023-11-14 08:32:47.785 | github.com/cosmos/iavl.NewUnsavedFastIterator({0xc0474ca4e0, 0x1e, 0x20}, {0xc0474ca800, 0x1e, 0x20}, 0x1, 0xc00098f600, 0xc0526eede0, 0xc0526eee70) |  
  |   | 2023-11-14 08:32:47.785 | /go/pkg/mod/github.com/cosmos/iavl@v0.19.6/unsaved_fast_iterator.go:55 +0x219 |  
  |   | 2023-11-14 08:32:47.785 | github.com/cosmos/iavl.(*MutableTree).Iterator(0xc0019bcf50, {0xc0474ca4e0, 0x1e, 0x20}, {0xc0474ca800, 0x1e, 0x20}, 0x60?) |  
  |   | 2023-11-14 08:32:47.785 | /go/pkg/mod/github.com/cosmos/iavl@v0.19.6/mutable_tree.go:216 +0x105 |  
  |   | 2023-11-14 08:32:47.785 | github.com/cosmos/cosmos-sdk/store/iavl.(*Store).Iterator(0xc111643f80?, {0xc0474ca4e0?, 0x20?, 0xc111643fc0?}, {0xc0474ca800?, 0x20?, 0xc03f3f4000?}) |  
  |   | 2023-11-14 08:32:47.785 | /go/pkg/mod/github.com/celestiaorg/cosmos-sdk@v1.18.3-sdk-v0.46.14/store/iavl/store.go:249 +0x2b |  
  |   | 2023-11-14 08:32:47.785 | github.com/cosmos/cosmos-sdk/store/cachekv.(*Store).iterator(0xc0c107a680, {0xc0474ca4e0, 0x1e, 0x20}, {0xc0474ca800, 0x1e, 0x20}, 0x1) |  
  |   | 2023-11-14 08:32:47.785 | /go/pkg/mod/github.com/celestiaorg/cosmos-sdk@v1.18.3-sdk-v0.46.14/store/cachekv/store.go:185 +0x11d |  
  |   | 2023-11-14 08:32:47.785 | github.com/cosmos/cosmos-sdk/store/cachekv.(*Store).Iterator(0x3?, {0xc0474ca4e0?, 0x1e?, 0x1e?}, {0xc0474ca800?, 0x64?, 0x64?}) |  
  |   | 2023-11-14 08:32:47.785 | /go/pkg/mod/github.com/celestiaorg/cosmos-sdk@v1.18.3-sdk-v0.46.14/store/cachekv/store.go:170 +0x25 |  
  |   | 2023-11-14 08:32:47.785 | github.com/cosmos/cosmos-sdk/store/cachekv.(*Store).iterator(0xc03ffdb480, {0xc0474ca4e0, 0x1e, 0x20}, {0xc0474ca800, 0x1e, 0x20}, 0x1) |  
  |   | 2023-11-14 08:32:47.785 | /go/pkg/mod/github.com/celestiaorg/cosmos-sdk@v1.18.3-sdk-v0.46.14/store/cachekv/store.go:185 +0x11d |  
  |   | 2023-11-14 08:32:47.785 | github.com/cosmos/cosmos-sdk/store/cachekv.(*Store).Iterator(0xc111643f20?, {0xc0474ca4e0?, 0x20?, 0xc111643f40?}, {0xc0474ca800?, 0x20?, 0xc111643f60?}) |  
  |   | 2023-11-14 08:32:47.785 | /go/pkg/mod/github.com/celestiaorg/cosmos-sdk@v1.18.3-sdk-v0.46.14/store/cachekv/store.go:170 +0x25 |  
  |   | 2023-11-14 08:32:47.785 | github.com/cosmos/cosmos-sdk/store/cachekv.(*Store).iterator(0xc03520e1c0, {0xc0474ca4e0, 0x1e, 0x20}, {0xc0474ca800, 0x1e, 0x20}, 0x1) |  
  |   | 2023-11-14 08:32:47.785 | /go/pkg/mod/github.com/celestiaorg/cosmos-sdk@v1.18.3-sdk-v0.46.14/store/cachekv/store.go:185 +0x11d |  
  |   | 2023-11-14 08:32:47.785 | github.com/cosmos/cosmos-sdk/store/cachekv.(*Store).Iterator(0xc000581000?, {0xc0474ca4e0?, 0x0?, 0xc14afe90e0?}, {0xc0474ca800?, 0x20?, 0x0?}) |  
  |   | 2023-11-14 08:32:47.785 | /go/pkg/mod/github.com/celestiaorg/cosmos-sdk@v1.18.3-sdk-v0.46.14/store/cachekv/store.go:170 +0x25 |  
  |   | 2023-11-14 08:32:47.785 | github.com/cosmos/cosmos-sdk/store/gaskv.(*Store).iterator(0xc0d3f5dec0, {0xc0474ca4e0?, 0x64?, 0x15?}, {0xc0474ca800?, 0x20?, 0xd38b010000000000?}, 0x0?) |  
  |   | 2023-11-14 08:32:47.785 | /go/pkg/mod/github.com/celestiaorg/cosmos-sdk@v1.18.3-sdk-v0.46.14/store/gaskv/store.go:98 +0x48 |  
  |   | 2023-11-14 08:32:47.785 | github.com/cosmos/cosmos-sdk/store/gaskv.(*Store).Iterator(0xc021cb62e8?, {0xc0474ca4e0?, 0x3045a50?, 0xc03520e240?}, {0xc0474ca800?, 0x1?, 0xc0776c68a0?}) |  
  |   | 2023-11-14 08:32:47.785 | /go/pkg/mod/github.com/celestiaorg/cosmos-sdk@v1.18.3-sdk-v0.46.14/store/gaskv/store.go:74 +0x25 |  
  |   | 2023-11-14 08:32:47.785 | github.com/cosmos/cosmos-sdk/x/distribution/keeper.Keeper.IterateValidatorSlashEventsBetween({{0x3019af8, 0xc0008b5400}, {0x3045b30, 0xc0008b5120}, {{0x3045b30, 0xc0008b5120}, 0xc0000b14c8, {0x3019af8, 0xc0008b5430}, {0x3019b20, ...}, ...}, ...}, ...) |  
  |   | 2023-11-14 08:32:47.785 | /go/pkg/mod/github.com/celestiaorg/cosmos-sdk@v1.18.3-sdk-v0.46.14/x/distribution/keeper/store.go:337 +0x147 |  
  |   | 2023-11-14 08:32:47.785 | github.com/cosmos/cosmos-sdk/x/distribution/keeper.Keeper.CalculateDelegationRewards({{0x3019af8, 0xc0008b5400}, {0x3045b30, 0xc0008b5120}, {{0x3045b30, 0xc0008b5120}, 0xc0000b14c8, {0x3019af8, 0xc0008b5430}, {0x3019b20, ...}, ...}, ...}, ...) |  
  |   | 2023-11-14 08:32:47.785 | /go/pkg/mod/github.com/celestiaorg/cosmos-sdk@v1.18.3-sdk-v0.46.14/x/distribution/keeper/delegation.go:80 +0x2d9 |  
  |   | 2023-11-14 08:32:47.785 | github.com/cosmos/cosmos-sdk/x/distribution/keeper.Keeper.withdrawDelegationRewards({{0x3019af8, 0xc0008b5400}, {0x3045b30, 0xc0008b5120}, {{0x3045b30, 0xc0008b5120}, 0xc0000b14c8, {0x3019af8, 0xc0008b5430}, {0x3019b20, ...}, ...}, ...}, ...)

For Admin Use

evan-forbes commented 1 year ago

to add more context from a sync conversation, this seems to occur more frequently the more a node reads from state.

evan-forbes commented 1 year ago

seems related to https://github.com/cosmos/cosmos-sdk/issues/10568

we might actually need to go in and add a mutex somewhere in the read path

I'm curious if this error is always with the distribution keeper or do we see it elsewhere as well?

cmwaters commented 11 months ago

seems related to https://github.com/cosmos/cosmos-sdk/issues/10568

Are you sure? It looks like this a concurrency problem in the IAVL not the gRPC server

evan-forbes commented 11 months ago

Are you sure? It looks like this a concurrency problem in the IAVL not the gRPC server

yeah I think so, the title of that issue is misleading. its encountering a similar stack trace both from iavl. both scenario's are using the gprc server to read from iavl

to add more context from a sync conversation, this seems to occur more frequently the more a node reads from state.

evan-forbes commented 11 months ago

https://github.com/cosmos/cosmos-sdk/issues/10568#issuecomment-1452199752