search

vocdoni / vocdoni-node

A set of libraries and tools for the Vocdoni decentralized backend infrastructure, the main ground of our universally verifiable, privacy-centric and scalable digital voting protocol
GNU Affero General Public License v3.0
86 stars 17 forks source link

fatal error: concurrent map read and map write #940

Closed altergui closed 1 year ago

altergui commented 1 year ago

Disclaimer: i came across this while testing #925, so not on master. so there's a chance that it's a bug of that particular PR, but i want to copypaste the full stack trace here, just in case it's an actual bug from master that only happens very rarely

gateway0_1    | 2023-05-22T18:37:47.987Z DBG indexer/indexer.go:320 > updated process processID=c5d2460186f76d5cede4e1aab02636813cd05dec5b22253e2cac020800000000
gateway0_1    | 2023-05-22T18:37:48.019Z INF offchaindatahandler/offchaindatahandler.go:76 > importing data type="external census" uri=ipfs://bafybeihc6x4ebsrh3a47onutxk37qdh62zychyot6ustlxwi6pimlsh3ye
gateway0_1    | 2023-05-22T18:37:48.019Z INF offchaindatahandler/offchaindatahandler.go:80 > importing data type="election metadata" uri=ipfs://bafybeie4mp52apnuuqtxb7r6xxonclqp5tv7wjrpn7ub3gmxddcgpjmqpu
gateway0_1    | 2023-05-22T18:37:48.019Z INF offchaindatahandler/offchaindatahandler.go:80 > importing data type="election metadata" uri=ipfs://bafybeie4mp52apnuuqtxb7r6xxonclqp5tv7wjrpn7ub3gmxddcgpjmqpu
gateway0_1    | 2023-05-22T18:37:48.019Z INF vochain/app.go:379 > commit block height=970 txs=1
gateway0_1    | 2023-05-22T18:37:48.019Z DBG downloader/downloader.go:121 > pining remote file uri=ipfs://bafybeie4mp52apnuuqtxb7r6xxonclqp5tv7wjrpn7ub3gmxddcgpjmqpu
gateway0_1    | 2023-05-22T18:37:48.019Z DBG downloader/downloader.go:121 > pining remote file uri=ipfs://bafybeihc6x4ebsrh3a47onutxk37qdh62zychyot6ustlxwi6pimlsh3ye
gateway0_1    | 2023-05-22T18:37:48.019Z DBG ipfs/ipfs.go:267 > retrieved file /ipfs/bafybeie4mp52apnuuqtxb7r6xxonclqp5tv7wjrpn7ub3gmxddcgpjmqpu from cache
gateway0_1    | 2023-05-22T18:37:48.019Z INF offchaindatahandler/metadata.go:17 > metadata downloaded successfully from ipfs://bafybeie4mp52apnuuqtxb7r6xxonclqp5tv7wjrpn7ub3gmxddcgpjmqpu (139 bytes)
gateway0_1    | 2023-05-22T18:37:48.019Z DBG ipfs/ipfs.go:315 > rawdata received: {"type":1,"rootHash":"SnBMN5UY7Cow4X+KVbezeLJHYOZ2UdSVkltCgai4ytY=","data":"KLUv/QQAPQMAhAUUIADQB1EfIuoMV+L41GPCZcrxxZ69GwoAFCAAhI0vDr1XSyBPCCTt9XcBGH+s1L7DKZFv4wDb6HNjCH5EixgCXRchkLMWMvP7jRHT1r7fRm2xr42q018FBAAAztCKZGlFthDdGN8Ss4k=","maxLevels":160}
gateway0_1    | fatal error: concurrent map read and map write
gateway0_1    | 
gateway0_1    | goroutine 745741 [running]:
gateway0_1    | github.com/hashicorp/golang-lru/v2/simplelru.(*LRU[...]).Get(0x3e447e0, {0x0, 0x7?})
gateway0_1    |     /home/user/go/pkg/mod/github.com/hashicorp/golang-lru/v2@v2.0.2/simplelru/lru.go:70 +0x3d
gateway0_1    | go.vocdoni.io/dvote/data/ipfs.(*Handler).Retrieve(0xc000a4e3c0, {0x3e0fd58, 0xc0085cb110}, {0x0?, 0x50?}, 0x100000)
gateway0_1    |     /home/user/src/vocdoni/vocdoni-node/data/ipfs/ipfs.go:265 +0xdb
gateway0_1    | go.vocdoni.io/dvote/api.(*API).electionHandler(0xc000557b90, 0x0?, 0xc00655b960)
gateway0_1    |     /home/user/src/vocdoni/vocdoni-node/api/elections.go:220 +0x865
gateway0_1    | go.vocdoni.io/dvote/httprouter/apirest.(*API).RegisterMethod.func1({{0x30f1ae0, 0xc008dafaa0}, {0xc1130b7efa86b550, 0x32ee07e5da8, 0x55af7c0}, {0xc007142b50, 0x3, 0x3}, 0xc00655b960})
gateway0_1    |     /home/user/src/vocdoni/vocdoni-node/httprouter/apirest/apirest.go:209 +0x74
gateway0_1    | created by go.vocdoni.io/dvote/httprouter.(*HTTProuter).routerHandler.func1
gateway0_1    |     /home/user/src/vocdoni/vocdoni-node/httprouter/httprouter.go:261 +0x3f2
gateway0_1    | 

gateway0_1    | goroutine 179 [runnable]:
gateway0_1    | internal/poll.(*FD).RawRead(0xc000730f00, 0xc008248a50)
gateway0_1    |     /home/user/sdk/go1.20.2/src/internal/poll/fd_unix.go:763 +0x124
gateway0_1    | net.(*rawConn).Read(0xc0007a8478, 0x1c0864a?)
gateway0_1    |     /home/user/sdk/go1.20.2/src/net/rawconn.go:43 +0x45
gateway0_1    | golang.org/x/net/internal/socket.(*syscaller).recvmmsg(0xc009434708, {0x3e0d180?, 0xc0007a8478?}, {0xc0000ff200?, 0x3b06830?, 0x0?}, 0x3b06830?)
gateway0_1    |     /home/user/go/pkg/mod/golang.org/x/net@v0.10.0/internal/socket/mmsghdr_unix.go:121 +0x94
gateway0_1    | golang.org/x/net/internal/socket.(*Conn).recvMsgs(0xc000944440, {0xc0007ac580?, 0x8, 0x8}, 0x0?)
gateway0_1    |     /home/user/go/pkg/mod/golang.org/x/net@v0.10.0/internal/socket/rawconn_mmsg.go:25 +0x185
gateway0_1    | golang.org/x/net/internal/socket.(*Conn).RecvMsgs(...)
gateway0_1    |     /home/user/go/pkg/mod/golang.org/x/net@v0.10.0/internal/socket/socket.go:267
gateway0_1    | golang.org/x/net/ipv4.(*payloadHandler).ReadBatch(0xc000725a00, {0xc0007ac580?, 0x33f3f40?, 0x34fbe00?}, 0xc00ae0c4c0?)
gateway0_1    |     /home/user/go/pkg/mod/golang.org/x/net@v0.10.0/ipv4/batch.go:80 +0x57
gateway0_1    | github.com/quic-go/quic-go.(*oobConn).ReadPacket(0xc000730f80)
gateway0_1    |     /home/user/go/pkg/mod/github.com/quic-go/quic-go@v0.33.0/sys_conn_oob.go:154 +0x695
gateway0_1    | github.com/quic-go/quic-go.(*packetHandlerMap).listen(0xc0001e1810)
gateway0_1    |     /home/user/go/pkg/mod/github.com/quic-go/quic-go@v0.33.0/packet_handler_map.go:364 +0x70
gateway0_1    | created by github.com/quic-go/quic-go.newPacketHandlerMap
gateway0_1    |     /home/user/go/pkg/mod/github.com/quic-go/quic-go@v0.33.0/packet_handler_map.go:142 +0x35d
gateway0_1    | 

gateway0_1    | goroutine 1400 [runnable]:
gateway0_1    | github.com/hashicorp/golang-lru/v2/simplelru.(*LRU[...]).Add(0x3e447e0, {0xc0007c3bd0, 0x41}, {0xc005fd5e00, 0xfa, 0x200})
gateway0_1    |     /home/user/go/pkg/mod/github.com/hashicorp/golang-lru/v2@v2.0.2/simplelru/lru.go:58 +0x1ff
gateway0_1    | go.vocdoni.io/dvote/data/ipfs.(*Handler).Retrieve(0xc000a4e3c0, {0x3e0fd58, 0xc00808ed50}, {0xc0008d1e50?, 0x2?}, 0x6400000)
gateway0_1    |     /home/user/src/vocdoni/vocdoni-node/data/ipfs/ipfs.go:319 +0x434
gateway0_1    | go.vocdoni.io/dvote/data/downloader.(*Downloader).handleImport(0xc0008cc780, 0xc00655be40)
gateway0_1    |     /home/user/src/vocdoni/vocdoni-node/data/downloader/downloader.go:125 +0x17f
gateway0_1    | go.vocdoni.io/dvote/data/downloader.(*Downloader).importQueueDaemon(0xc0008cc780, {0x3e0fce8, 0xc001648190})
gateway0_1    |     /home/user/src/vocdoni/vocdoni-node/data/downloader/downloader.go:158 +0x52
gateway0_1    | created by go.vocdoni.io/dvote/data/downloader.(*Downloader).Start
gateway0_1    |     /home/user/src/vocdoni/vocdoni-node/data/downloader/downloader.go:71 +0x6d
gateway0_1    | 

gateway0_1    | goroutine 1402 [runnable]:
gateway0_1    | syscall.Syscall6(0x3dd7e80?, 0x1?, 0x7fb7dc14edd8?, 0x7fb80d6021d8?, 0xd0?, 0xc000680000?, 0xc00bcb5ba0?)
gateway0_1    |     /home/user/sdk/go1.20.2/src/syscall/syscall_linux.go:91 +0x36
gateway0_1    | syscall.fstatat(0xc?, {0x3dd7e80?, 0x2?}, 0xc00bcb5bd8, 0xc?)
gateway0_1    |     /home/user/sdk/go1.20.2/src/syscall/zsyscall_linux_amd64.go:1443 +0x9a
gateway0_1    | syscall.Stat(...)
gateway0_1    |     /home/user/sdk/go1.20.2/src/syscall/syscall_linux_amd64.go:62
gateway0_1    | os.statNolog.func1(...)
gateway0_1    |     /home/user/sdk/go1.20.2/src/os/stat_unix.go:32
gateway0_1    | os.ignoringEINTR(...)
gateway0_1    |     /home/user/sdk/go1.20.2/src/os/file_posix.go:245
gateway0_1    | os.statNolog({0x3dd7e80, 0x1})
gateway0_1    |     /home/user/sdk/go1.20.2/src/os/stat_unix.go:31 +0x58
gateway0_1    | os.Getwd()
gateway0_1    |     /home/user/sdk/go1.20.2/src/os/getwd.go:29 +0x29
gateway0_1    | github.com/rs/zerolog.consoleDefaultFormatCaller.func1({0x3165ee0?, 0xc0041a2e70?})
gateway0_1    |     /home/user/go/pkg/mod/github.com/rs/zerolog@v1.28.0/console.go:408 +0x5c
gateway0_1    | github.com/rs/zerolog.ConsoleWriter.writePart({{0x3df7200, 0xc000014018}, 0x0, {0x381ce52, 0x23}, {0xc008898c80, 0x4, 0x4}, {0x0, 0x0, ...}, ...}, ...)
gateway0_1    |     /home/user/go/pkg/mod/github.com/rs/zerolog@v1.28.0/console.go:293 +0x279
gateway0_1    | github.com/rs/zerolog.ConsoleWriter.Write({{0x3df7200, 0xc000014018}, 0x0, {0x381ce52, 0x23}, {0xc008898c80, 0x4, 0x4}, {0x0, 0x0, ...}, ...}, ...)
gateway0_1    |     /home/user/go/pkg/mod/github.com/rs/zerolog@v1.28.0/console.go:128 +0x4a9
gateway0_1    | github.com/rs/zerolog.levelWriterAdapter.WriteLevel(...)
gateway0_1    |     /home/user/go/pkg/mod/github.com/rs/zerolog@v1.28.0/writer.go:25
gateway0_1    | github.com/rs/zerolog.(*Event).write(0xc00c9170e0)
gateway0_1    |     /home/user/go/pkg/mod/github.com/rs/zerolog@v1.28.0/event.go:78 +0xff
gateway0_1    | github.com/rs/zerolog.(*Event).msg(0xc00c9170e0, {0x37886eb, 0x12})
gateway0_1    |     /home/user/go/pkg/mod/github.com/rs/zerolog@v1.28.0/event.go:149 +0x23c
gateway0_1    | github.com/rs/zerolog.(*Event).Msg(...)
gateway0_1    |     /home/user/go/pkg/mod/github.com/rs/zerolog@v1.28.0/event.go:108
gateway0_1    | go.vocdoni.io/dvote/log.Debugw({0x37886eb, 0x12}, {0xc003a04eb0, 0x2, 0x2})
gateway0_1    |     /home/user/src/vocdoni/vocdoni-node/log/log.go:251 +0xc6
gateway0_1    | go.vocdoni.io/dvote/data/downloader.(*Downloader).handleImport(0xc0008cc780, 0xc00c51d7e0)
gateway0_1    |     /home/user/src/vocdoni/vocdoni-node/data/downloader/downloader.go:121 +0xc7
gateway0_1    | go.vocdoni.io/dvote/data/downloader.(*Downloader).importQueueDaemon(0xc0008cc780, {0x3e0fce8, 0xc001648190})
gateway0_1    |     /home/user/src/vocdoni/vocdoni-node/data/downloader/downloader.go:158 +0x52
gateway0_1    | created by go.vocdoni.io/dvote/data/downloader.(*Downloader).Start
gateway0_1    |     /home/user/src/vocdoni/vocdoni-node/data/downloader/downloader.go:71 +0x6d
gateway0_1    | 

gateway0_1    | goroutine 2083 [runnable]:
gateway0_1    | syscall.Syscall(0x0?, 0x0?, 0x0?, 0x0?)
gateway0_1    |     /home/user/sdk/go1.20.2/src/syscall/syscall_linux.go:69 +0x27
gateway0_1    | syscall.Fsync(0xc008913fb8?)
gateway0_1    |     /home/user/sdk/go1.20.2/src/syscall/zsyscall_linux_amd64.go:438 +0x2a
gateway0_1    | internal/poll.(*FD).Fsync.func1(...)
gateway0_1    |     /home/user/sdk/go1.20.2/src/internal/poll/fd_fsync_posix.go:18
gateway0_1    | internal/poll.ignoringEINTR(...)
gateway0_1    |     /home/user/sdk/go1.20.2/src/internal/poll/fd_posix.go:74
gateway0_1    | internal/poll.(*FD).Fsync(0xc008914068?)
gateway0_1    |     /home/user/sdk/go1.20.2/src/internal/poll/fd_fsync_posix.go:17 +0xf4
gateway0_1    | os.(*File).Sync(0xc0003bc428)
gateway0_1    |     /home/user/sdk/go1.20.2/src/os/file_posix.go:168 +0x4e
gateway0_1    | github.com/syndtr/goleveldb/leveldb/storage.(*fileWrap).Sync(0xc000662180)
gateway0_1    |     /home/user/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.1-0.20220614013038-64ee5596c38a/leveldb/storage/file_storage.go:592 +0x28
gateway0_1    | github.com/syndtr/goleveldb/leveldb.(*DB).writeJournal(0xc001015180, {0xc008914238, 0x1, 0x1}, 0x4000?, 0x1)
gateway0_1    |     /home/user/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.1-0.20220614013038-64ee5596c38a/leveldb/db_write.go:30 +0x99
gateway0_1    | github.com/syndtr/goleveldb/leveldb.(*DB).writeLocked(0xc001015180, 0xc00ab28700, 0xc00ab28700, 0x1, 0x1)
gateway0_1    |     /home/user/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.1-0.20220614013038-64ee5596c38a/leveldb/db_write.go:231 +0x259
gateway0_1    | github.com/syndtr/goleveldb/leveldb.(*DB).putRec(0xc001015180, 0x1, {0x54b4750, 0x8, 0x8}, {0xc005eae500, 0x500, 0x500}, 0x3e06c70?)
gateway0_1    |     /home/user/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.1-0.20220614013038-64ee5596c38a/leveldb/db_write.go:365 +0x458
gateway0_1    | github.com/syndtr/goleveldb/leveldb.(*DB).Put(...)
gateway0_1    |     /home/user/go/pkg/mod/github.com/syndtr/goleveldb@v1.0.1-0.20220614013038-64ee5596c38a/leveldb/db_write.go:375
gateway0_1    | github.com/cometbft/cometbft-db.(*GoLevelDB).SetSync(0xb?, {0x54b4750?, 0xc001524658?, 0x6?}, {0xc005eae500?, 0xc?, 0x1?})
gateway0_1    |     /home/user/go/pkg/mod/github.com/cometbft/cometbft-db@v0.8.0/goleveldb.go:88 +0x65
gateway0_1    | github.com/cometbft/cometbft/state.dbStore.save({{_, _}, {_}}, {{{0xb, 0x0}, {0xc001524658, 0x6}}, {0xc001524670, 0xc}, 0x1, ...}, ...)
gateway0_1    |     /home/user/go/pkg/mod/github.com/cometbft/cometbft@v0.37.1/state/store.go:197 +0x1cf
gateway0_1    | github.com/cometbft/cometbft/state.dbStore.Save(...)
gateway0_1    |     /home/user/go/pkg/mod/github.com/cometbft/cometbft@v0.37.1/state/store.go:173
gateway0_1    | github.com/cometbft/cometbft/state.(*BlockExecutor).ApplyBlock(_, {{{0xb, 0x0}, {0xc001524658, 0x6}}, {0xc001524670, 0xc}, 0x1, 0x3ca, {{0xc006b50020, ...}, ...}, ...}, ...)
gateway0_1    |     /home/user/go/pkg/mod/github.com/cometbft/cometbft@v0.37.1/state/execution.go:253 +0x816
gateway0_1    | github.com/cometbft/cometbft/consensus.(*State).finalizeCommit(0xc000160e00, 0x3ca)
gateway0_1    |     /home/user/go/pkg/mod/github.com/cometbft/cometbft@v0.37.1/consensus/state.go:1700 +0xaa5
gateway0_1    | github.com/cometbft/cometbft/consensus.(*State).tryFinalizeCommit(0xc000160e00, 0x3ca)
gateway0_1    |     /home/user/go/pkg/mod/github.com/cometbft/cometbft@v0.37.1/consensus/state.go:1609 +0x2ff
gateway0_1    | github.com/cometbft/cometbft/consensus.(*State).enterCommit.func1()
gateway0_1    |     /home/user/go/pkg/mod/github.com/cometbft/cometbft@v0.37.1/consensus/state.go:1544 +0xaa
gateway0_1    | github.com/cometbft/cometbft/consensus.(*State).enterCommit(0xc000160e00, 0x3ca, 0x0)
gateway0_1    |     /home/user/go/pkg/mod/github.com/cometbft/cometbft@v0.37.1/consensus/state.go:1582 +0xccf
gateway0_1    | github.com/cometbft/cometbft/consensus.(*State).addVote(0xc000160e00, 0xc00c2e34a0, {0xc0009cc360, 0x28})
gateway0_1    |     /home/user/go/pkg/mod/github.com/cometbft/cometbft@v0.37.1/consensus/state.go:2212 +0x1a10
gateway0_1    | github.com/cometbft/cometbft/consensus.(*State).tryAddVote(0xc000160e00, 0xc00c2e34a0, {0xc0009cc360?, 0xc0061c3c08?})
gateway0_1    |     /home/user/go/pkg/mod/github.com/cometbft/cometbft@v0.37.1/consensus/state.go:2001 +0x2c
gateway0_1    | github.com/cometbft/cometbft/consensus.(*State).handleMsg(0xc000160e00, {{0x3df1b40, 0xc000ac3a70}, {0xc0009cc360, 0x28}})
gateway0_1    |     /home/user/go/pkg/mod/github.com/cometbft/cometbft@v0.37.1/consensus/state.go:861 +0x3ff
gateway0_1    | github.com/cometbft/cometbft/consensus.(*State).receiveRoutine(0xc000160e00, 0x0)
gateway0_1    |     /home/user/go/pkg/mod/github.com/cometbft/cometbft@v0.37.1/consensus/state.go:768 +0x3f0
gateway0_1    | created by github.com/cometbft/cometbft/consensus.(*State).OnStart
gateway0_1    |     /home/user/go/pkg/mod/github.com/cometbft/cometbft@v0.37.1/consensus/state.go:379 +0x12d
gateway0_1    | 

gateway0_1    | goroutine 745714 [runnable]:
gateway0_1    | internal/poll.runtime_pollWait(0x7fb7dc3c3510, 0x77)
gateway0_1    |     /home/user/sdk/go1.20.2/src/runtime/netpoll.go:306 +0x89
gateway0_1    | internal/poll.(*pollDesc).wait(0xc007b35f00?, 0x0?, 0x0)
gateway0_1    |     /home/user/sdk/go1.20.2/src/internal/poll/fd_poll_runtime.go:84 +0x32
gateway0_1    | internal/poll.(*pollDesc).waitWrite(...)
gateway0_1    |     /home/user/sdk/go1.20.2/src/internal/poll/fd_poll_runtime.go:93
gateway0_1    | internal/poll.(*FD).WaitWrite(...)
gateway0_1    |     /home/user/sdk/go1.20.2/src/internal/poll/fd_unix.go:741
gateway0_1    | net.(*netFD).connect(0xc007b35f00, {0x3e0fd58?, 0xc007a19560}, {0xa?, 0xc009360ea0?}, {0x3df7380?, 0xc007790cc0?})
gateway0_1    |     /home/user/sdk/go1.20.2/src/net/fd_unix.go:141 +0x799
gateway0_1    | net.(*netFD).dial(0xc007b35f00, {0x3e0fd58, 0xc007a19560}, {0x3e1b4f0?, 0xc0007cafc0?}, {0x3e1b4f0?, 0xc007a195f0}, 0xc00ddf10e8?)
gateway0_1    |     /home/user/sdk/go1.20.2/src/net/sock_posix.go:151 +0x3c5
gateway0_1    | net.socket({0x3e0fd58, 0xc007a19560}, {0x36633cc, 0x4}, 0x2, 0x1, 0x11c22d0?, 0x90?, {0x3e1b4f0, 0xc0007cafc0}, ...)
gateway0_1    |     /home/user/sdk/go1.20.2/src/net/sock_posix.go:70 +0x2d2
gateway0_1    | net.internetSocket({0x3e0fd58, 0xc007a19560}, {0x36633cc, 0x4}, {0x3e1b4f0, 0xc0007cafc0}, {0x3e1b4f0, 0xc007a195f0}, 0xc009e6a870?, 0x0, ...)
gateway0_1    |     /home/user/sdk/go1.20.2/src/net/ipsock_posix.go:142 +0xf8
gateway0_1    | net.(*sysDialer).doDialTCP(0xc006fbcfc0, {0x3e0fd58, 0xc007a19560}, 0xc0007cafc0, 0x0?)
gateway0_1    |     /home/user/sdk/go1.20.2/src/net/tcpsock_posix.go:74 +0xef
gateway0_1    | net.(*sysDialer).dialTCP(0xc00ddf1170?, {0x3e0fd58?, 0xc007a19560?}, 0x12d8206?, 0x130de85?)
gateway0_1    |     /home/user/sdk/go1.20.2/src/net/tcpsock_posix.go:64 +0x69
gateway0_1    | net.(*sysDialer).dialSingle(0xc006fbcfc0, {0x3e0fd58, 0xc007a19560}, {0x3e018b8?, 0xc007a195f0})
gateway0_1    |     /home/user/sdk/go1.20.2/src/net/dial.go:580 +0x1f2
gateway0_1    | net.(*sysDialer).dialSerial(0xc006fbcfc0, {0x3e0fd58, 0xc007a19560}, {0xc00a4877e0?, 0x1, 0xc00ddf1468?})
gateway0_1    |     /home/user/sdk/go1.20.2/src/net/dial.go:548 +0x245
gateway0_1    | net.(*sysDialer).dialParallel(0x3e018b8?, {0x3e0fd58?, 0xc007a19560?}, {0xc00a4877e0?, 0xc007a19560?, 0x3662eac?}, {0x0?, 0x36633cc?, 0x1?})
gateway0_1    |     /home/user/sdk/go1.20.2/src/net/dial.go:449 +0x413
gateway0_1    | net.(*Dialer).DialContext(0xc00ddf1730, {0x3e0fd58, 0xc007a19560}, {0x36633cc, 0x4}, {0xc009360e88, 0x14})
gateway0_1    |     /home/user/sdk/go1.20.2/src/net/dial.go:440 +0x70c
gateway0_1    | github.com/libp2p/go-libp2p/p2p/net/reuseport.reuseDial({0x3e0fd58, 0xc007a19560}, 0xc009360e98?, {0x36633cc, 0x4}, {0xc009360e88, 0x14})
gateway0_1    |     /home/user/go/pkg/mod/github.com/libp2p/go-libp2p@v0.27.3/p2p/net/reuseport/reuseport.go:23 +0xef
gateway0_1    | github.com/libp2p/go-libp2p/p2p/net/reuseport.(*dialer).DialContext(0xc0008e0c80, {0x3e0fd58, 0xc007a19560}, {0x36633cc, 0x4}, {0xc009360e88, 0x14})
gateway0_1    |     /home/user/go/pkg/mod/github.com/libp2p/go-libp2p@v0.27.3/p2p/net/reuseport/dialer.go:86 +0x26a
gateway0_1    | github.com/libp2p/go-libp2p/p2p/net/reuseport.(*Transport).DialContext(0xc000730730, {0x3e0fd58, 0xc007a19560}, {0x3e2da60?, 0xc0068fd2a8?})
gateway0_1    |     /home/user/go/pkg/mod/github.com/libp2p/go-libp2p@v0.27.3/p2p/net/reuseport/dial.go:36 +0xe8
gateway0_1    | github.com/libp2p/go-libp2p/p2p/transport/tcp.(*TcpTransport).maDial(0xc000730700?, {0x3e0fd58?, 0xc007a19500?}, {0x3e2da60?, 0xc0068fd2a8?})
gateway0_1    |     /home/user/go/pkg/mod/github.com/libp2p/go-libp2p@v0.27.3/p2p/transport/tcp/tcp.go:171 +0x186
gateway0_1    | github.com/libp2p/go-libp2p/p2p/transport/tcp.(*TcpTransport).dialWithScope(0xc000730700, {0x3e0fd58, 0xc007a19500}, {0x3e2da60?, 0xc0068fd2a8}, {0xc003c6b410, 0x26}, {0x3e1dcd0?, 0xc005059d80?})
gateway0_1    |     /home/user/go/pkg/mod/github.com/libp2p/go-libp2p@v0.27.3/p2p/transport/tcp/tcp.go:198 +0x255
gateway0_1    | github.com/libp2p/go-libp2p/p2p/transport/tcp.(*TcpTransport).Dial(0xc000730700, {0x3e0fd58, 0xc007a19500}, {0x3e2da60?, 0xc0068fd2a8}, {0xc003c6b410, 0x26})
gateway0_1    |     /home/user/go/pkg/mod/github.com/libp2p/go-libp2p@v0.27.3/p2p/transport/tcp/tcp.go:185 +0x25b
gateway0_1    | github.com/libp2p/go-libp2p/p2p/net/swarm.(*Swarm).dialAddr(0xc0001b8000, {0x3e0fd58, 0xc007a19500}, {0xc003c6b410, 0x26}, {0x3e2da60?, 0xc0068fd2a8})
gateway0_1    |     /home/user/go/pkg/mod/github.com/libp2p/go-libp2p@v0.27.3/p2p/net/swarm/swarm_dial.go:498 +0x389
gateway0_1    | github.com/libp2p/go-libp2p/p2p/net/swarm.(*dialLimiter).executeDial(0xc0001d2140, 0xc005059cc0)
gateway0_1    |     /home/user/go/pkg/mod/github.com/libp2p/go-libp2p@v0.27.3/p2p/net/swarm/limiter.go:219 +0xf0
gateway0_1    | created by github.com/libp2p/go-libp2p/p2p/net/swarm.(*dialLimiter).addCheckFdLimit
gateway0_1    |     /home/user/go/pkg/mod/github.com/libp2p/go-libp2p@v0.27.3/p2p/net/swarm/limiter.go:169 +0x4ad
gateway0_1    | 

gateway0_1    | goroutine 745756 [runnable]:
gateway0_1    | github.com/jbenet/goprocess.(*process).WaitFor(0xc007713e00, {0x3e24770, 0xc007713e60})
gateway0_1    |     /home/user/go/pkg/mod/github.com/jbenet/goprocess@v0.1.4/impl-mutex.go:58 +0x1bf
gateway0_1    | github.com/jbenet/goprocess.(*process).Go(0x0?, 0xc006db0b40)
gateway0_1    |     /home/user/go/pkg/mod/github.com/jbenet/goprocess@v0.1.4/impl-mutex.go:128 +0x1ab
gateway0_1    | github.com/ipfs/go-datastore/query.(*resultsIter).useLegacyResults(0xc005f99b90)
gateway0_1    |     /home/user/go/pkg/mod/github.com/ipfs/go-datastore@v0.6.0/query/query.go:408 +0x2b9
gateway0_1    | github.com/ipfs/go-datastore/query.(*resultsIter).Next(0xc005f99b90)
gateway0_1    |     /home/user/go/pkg/mod/github.com/ipfs/go-datastore@v0.6.0/query/query.go:352 +0x1e
gateway0_1    | github.com/ipfs/boxo/pinning/pinner/dsindex.(*indexer).ForEach(0xc000ccab90?, {0x3e0fd90, 0xc007859830}, {0xc0086da4b0?, 0xc0086da4b0?}, 0xc0043a5b10)
gateway0_1    |     /home/user/go/pkg/mod/github.com/ipfs/boxo@v0.8.1/pinning/pinner/dsindex/indexer.go:117 +0x142
gateway0_1    | github.com/ipfs/boxo/pinning/pinner/dsindex.(*indexer).HasAny(0x0?, {0x3e0fd90?, 0xc007859830?}, {0xc0086da4b0?, 0xc004ef2c30?})
gateway0_1    |     /home/user/go/pkg/mod/github.com/ipfs/boxo@v0.8.1/pinning/pinner/dsindex/indexer.go:153 +0x4e
gateway0_1    | github.com/ipfs/boxo/pinning/pinner/dspinner.(*pinner).doPinRecursive(0xc000a20580, {0x3e0fd90, 0xc007859830}, {{0xc0086da4b0?, 0xc007713c20?}}, 0x1)
gateway0_1    |     /home/user/go/pkg/mod/github.com/ipfs/boxo@v0.8.1/pinning/pinner/dspinner/pin.go:196 +0xf3
gateway0_1    | github.com/ipfs/boxo/pinning/pinner/dspinner.(*pinner).Pin(0xc000a20580, {0x3e0fd90, 0xc007859830}, {0x3e26f30?, 0xc007713c20?}, 0x1)
gateway0_1    |     /home/user/go/pkg/mod/github.com/ipfs/boxo@v0.8.1/pinning/pinner/dspinner/pin.go:184 +0x8f
gateway0_1    | github.com/ipfs/kubo/core/coreapi.(*PinAPI).Add(0xc000eb0d80, {0x3e0fd58, 0xc0078597d0}, {0x3e10570, 0xc00af40800}, {0x0, 0x0, 0x0})
gateway0_1    |     /home/user/go/pkg/mod/github.com/ipfs/kubo@v0.20.0/core/coreapi/pin.go:40 +0x603
gateway0_1    | go.vocdoni.io/dvote/data/ipfs.(*Handler).Pin(0xc000a4e3c0, {0x3e0fd58, 0xc0078597d0}, {0xc008f56190?, 0x2dc187f?})
gateway0_1    |     /home/user/src/vocdoni/vocdoni-node/data/ipfs/ipfs.go:174 +0xc2
gateway0_1    | go.vocdoni.io/dvote/data/downloader.(*Downloader).handleImport.func1()
gateway0_1    |     /home/user/src/vocdoni/vocdoni-node/data/downloader/downloader.go:143 +0xa7
gateway0_1    | created by go.vocdoni.io/dvote/data/downloader.(*Downloader).handleImport
gateway0_1    |     /home/user/src/vocdoni/vocdoni-node/data/downloader/downloader.go:140 +0x47e
gateway0_1    |

it's just a snip, there were thousands of goroutines in select or chan receive, or io wait. i just pasted here the only one running and the 6 ones runnable

altergui commented 1 year ago

full log attached fatal error: concurrent map read and map write.log.gz

altergui commented 1 year ago

it does look like a concurrency issue inside simplelru, wtf this is writing

gateway0_1    | goroutine 1400 [runnable]:
gateway0_1    | github.com/hashicorp/golang-lru/v2/simplelru.(*LRU[...]).Add(0x3e447e0, {0xc0007c3bd0, 0x41}, {0xc005fd5e00, 0xfa, 0x200})
gateway0_1    |     /home/user/go/pkg/mod/github.com/hashicorp/golang-lru/v2@v2.0.2/simplelru/lru.go:58 
    c.items[key] = ent

and this one is reading

gateway0_1    | goroutine 745741 [running]:
gateway0_1    | github.com/hashicorp/golang-lru/v2/simplelru.(*LRU[...]).Get(0x3e447e0, {0x0, 0x7?})
gateway0_1    |     /home/user/go/pkg/mod/github.com/hashicorp/golang-lru/v2@v2.0.2/simplelru/lru.go:70 +0x3d
gateway0_1    | go.vocdoni.io/dvote/data/ipfs.(*Handler).Retrieve(0xc000a4e3c0, {0x3e0fd58, 0xc0085cb110}, {0x0?, 0x50?}, 0x100000)
gateway0_1    |     /home/user/src/vocdoni/vocdoni-node/data/ipfs/ipfs.go:265 +0xdb
    if ent, ok := c.items[key]; ok {
altergui commented 1 year ago

oh. wait. 

// LRU implements a non-thread safe fixed size LRU cache

:facepalm:

altergui commented 1 year ago

https://pkg.go.dev/github.com/hashicorp/golang-lru/v2

All caches in this package take locks while operating, and are therefore thread-safe for consumers.

but that refers to caches (ARCCache, Cache or TwoQueueCache) not to the LRU

git says i should ping @mvdan or @p4u :heart_eyes: