search

filecoin-project / lassie

A minimal universal retrieval client library for IPFS and Filecoin
Other
109 stars 17 forks source link

Prefetching and total threads #375

Closed hannahhoward closed 1 year ago

hannahhoward commented 1 year ago

Capturing an issue from @magik6k --

It seems to be doing some kind of prefetching, to the point where occasionally I will see the process die with the "runtime: program exceeds 10000-thread limit" error
I have tuned down the
lassie.WithConcurrentSPRetrievals(50),
lassie.WithBitswapConcurrency(50),
settings from 1000, but I'm still seeing super aggressive prefetching when e.g. trying to open wikipedia in a browser
If you still have my instance around, opening http://127.0.0.1:8080/ipfs/bafybeicvwe7hv3uap425wgd57tda46wfpowqensytht55pmkchnz2xvoxa/wiki/ should do it.
Looking at the summary goroutine dumps I'm seeing:
goroutine profile: total 11722
6000 @ 0x11fd2d6 0x120ddfe 0x2b319db 0x1233441
#   0x2b319da   github.com/filecoin-project/lassie/pkg/retriever/bitswaphelpers.(*PreloadCachingStorage).run.func2+0xda /home/magik6k/.opt/go/pkg/mod/github.com/filecoin-project/lassie@v0.14.3/pkg/retriever/bitswaphelpers/preloadcachingstorage.go:414

794 @ 0x11fd2d6 0x120ddfe 0x20df9cc 0x1233441
#   0x20df9cb   github.com/ipfs/go-graphsync/taskqueue.(*WorkerTaskQueue).worker+0x18b  /home/magik6k/.opt/go/pkg/mod/github.com/ipfs/go-graphsync@v0.14.7/taskqueue/taskqueue.go:121
Another dump:
goroutine profile: total 11186
4257 @ 0x11c32dc 0x2db906c 0x2dc84e5 0x2dc8399 0x2dc6fd7 0x1d8b29a 0x1d7eb43 0x1d8137d 0x1d81215 0x1d7f967 0x1d8112c 0x2df8ce5 0x2df8c7c 0x2e04307 0x2b376c5 0x2b2ba7a 0x2b3713a 0x2b369a8 0x1233441
#   0x2db906b   github.com/mattn/go-sqlite3._Cfunc_sqlite3_exec+0x4b                                _cgo_gotypes.go:857
#   0x2dc84e4   github.com/mattn/go-sqlite3.(*SQLiteDriver).Open.func5.1+0x64                           /home/magik6k/.opt/go/pkg/mod/github.com/mattn/go-sqlite3@v1.14.16/sqlite3.go:1474
#   0x2dc8398   github.com/mattn/go-sqlite3.(*SQLiteDriver).Open.func5+0x38                         /home/magik6k/.opt/go/pkg/mod/github.com/mattn/go-sqlite3@v1.14.16/sqlite3.go:1474
#   0x2dc6fd6   github.com/mattn/go-sqlite3.(*SQLiteDriver).Open+0x3b16                             /home/magik6k/.opt/go/pkg/mod/github.com/mattn/go-sqlite3@v1.14.16/sqlite3.go:1738
#   0x1d8b299   database/sql.dsnConnector.Connect+0x39                                      /usr/lib/go/src/database/sql/sql.go:758
#   0x1d7eb42   database/sql.(*DB).conn+0x762                                           /usr/lib/go/src/database/sql/sql.go:1387
#   0x1d8137c   database/sql.(*DB).query+0x5c                                           /usr/lib/go/src/database/sql/sql.go:1721
#   0x1d81214   database/sql.(*DB).QueryContext.func1+0x54                                  /usr/lib/go/src/database/sql/sql.go:1704
#   0x1d7f966   database/sql.(*DB).retry+0x46                                           /usr/lib/go/src/database/sql/sql.go:1538
#   0x1d8112b   database/sql.(*DB).QueryContext+0xcb                                        /usr/lib/go/src/database/sql/sql.go:1703
#   0x2df8ce4   database/sql.(*DB).Query+0xc4                                           /usr/lib/go/src/database/sql/sql.go:1717
#   0x2df8c7b   github.com/lotus-web3/ribs/rbdeal.(*ribsDB).GetRetrievalCandidates+0x5b                     /home/magik6k/github.com/lotus_web3/ribs/rbdeal/deal_db.go:1191
#   0x2e04306   github.com/lotus-web3/ribs/rbdeal.(*retrievalProvider).FindCandidatesAsync+0x1e6                /home/magik6k/github.com/lotus_web3/ribs/rbdeal/retr_provider.go:87
#   0x2b376c4   github.com/filecoin-project/lassie/pkg/retriever.AssignableCandidateFinder.FindCandidates.func2+0x84        /home/magik6k/.opt/go/pkg/mod/github.com/filecoin-project/lassie@v0.14.3/pkg/retriever/assignablecandidatefinder.go:70
#   0x2b2ba79   github.com/filecoin-project/lassie/pkg/internal/candidatebuffer.(*CandidateBuffer).BufferStream+0x139       /home/magik6k/.opt/go/pkg/mod/github.com/filecoin-project/lassie@v0.14.3/pkg/internal/candidatebuffer/candidatebuffer.go:63
#   0x2b37139   github.com/filecoin-project/lassie/pkg/retriever.AssignableCandidateFinder.FindCandidates+0x3f9         /home/magik6k/.opt/go/pkg/mod/github.com/filecoin-project/lassie@v0.14.3/pkg/retriever/assignablecandidatefinder.go:66
#   0x2b369a7   github.com/filecoin-project/lassie/pkg/retriever/combinators.RetrieverWithCandidateFinder.Retrieve.func1+0x1c7  /home/magik6k/.opt/go/pkg/mod/github.com/filecoin-project/lassie@v0.14.3/pkg/retriever/combinators/retrieverwithcandidatefinder.go:31

1791 @ 0x11c32dc 0x2db758c 0x2dcd966 0x2dcc765 0x2dcc4f3 0x1d87ada 0x1d8798f 0x1d89eec 0x1d8790f 0x2df8de7 0x2e04307 0x2b376c5 0x2b2ba7a 0x2b3713a 0x2b369a8 0x1233441
#   0x2db758b   github.com/mattn/go-sqlite3._Cfunc__sqlite3_step_internal+0x4b                          _cgo_gotypes.go:367
#   0x2dcd965   github.com/mattn/go-sqlite3.(*SQLiteRows).nextSyncLocked.func1+0x45                     /home/magik6k/.opt/go/pkg/mod/github.com/mattn/go-sqlite3@v1.14.16/sqlite3.go:2186
#   0x2dcc764   github.com/mattn/go-sqlite3.(*SQLiteRows).nextSyncLocked+0x44                           /home/magik6k/.opt/go/pkg/mod/github.com/mattn/go-sqlite3@v1.14.16/sqlite3.go:2186
#   0x2dcc4f2   github.com/mattn/go-sqlite3.(*SQLiteRows).Next+0x2f2                                /home/magik6k/.opt/go/pkg/mod/github.com/mattn/go-sqlite3@v1.14.16/sqlite3.go:2163
#   0x1d87ad9   database/sql.(*Rows).nextLocked+0x119                                       /usr/lib/go/src/database/sql/sql.go:2974
#   0x1d8798e   database/sql.(*Rows).Next.func1+0x2e                                        /usr/lib/go/src/database/sql/sql.go:2952
#   0x1d89eeb   database/sql.withLock+0x8b                                          /usr/lib/go/src/database/sql/sql.go:3405
#   0x1d8790e   database/sql.(*Rows).Next+0x6e                                          /usr/lib/go/src/database/sql/sql.go:2951
#   0x2df8de6   github.com/lotus-web3/ribs/rbdeal.(*ribsDB).GetRetrievalCandidates+0x1c6                    /home/magik6k/github.com/lotus_web3/ribs/rbdeal/deal_db.go:1201
#   0x2e04306   github.com/lotus-web3/ribs/rbdeal.(*retrievalProvider).FindCandidatesAsync+0x1e6                /home/magik6k/github.com/lotus_web3/ribs/rbdeal/retr_provider.go:87
#   0x2b376c4   github.com/filecoin-project/lassie/pkg/retriever.AssignableCandidateFinder.FindCandidates.func2+0x84        /home/magik6k/.opt/go/pkg/mod/github.com/filecoin-project/lassie@v0.14.3/pkg/retriever/assignablecandidatefinder.go:70
#   0x2b2ba79   github.com/filecoin-project/lassie/pkg/internal/candidatebuffer.(*CandidateBuffer).BufferStream+0x139       /home/magik6k/.opt/go/pkg/mod/github.com/filecoin-project/lassie@v0.14.3/pkg/internal/candidatebuffer/candidatebuffer.go:63
#   0x2b37139   github.com/filecoin-project/lassie/pkg/retriever.AssignableCandidateFinder.FindCandidates+0x3f9         /home/magik6k/.opt/go/pkg/mod/github.com/filecoin-project/lassie@v0.14.3/pkg/retriever/assignablecandidatefinder.go:66
#   0x2b369a7   github.com/filecoin-project/lassie/pkg/retriever/combinators.RetrieverWithCandidateFinder.Retrieve.func1+0x1c7  /home/magik6k/.opt/go/pkg/mod/github.com/filecoin-project/lassie@v0.14.3/pkg/retriever/combinators/retrieverwithcandidatefinder.go:31

800 @ 0x11fd2d6 0x120ddfe 0x20df9cc 0x1233441
#   0x20df9cb   github.com/ipfs/go-graphsync/taskqueue.(*WorkerTaskQueue).worker+0x18b  /home/magik6k/.opt/go/pkg/mod/github.com/ipfs/go-graphsync@v0.14.7/taskqueue/taskqueue.go:121
Another:
goroutine profile: total 9077
1311 @ 0x11c32dc 0x2db906c 0x2dc84e5 0x2dc8399 0x2dc6fd7 0x1d8b29a 0x1d7eb43 0x1d8137d 0x1d81215 0x1d7f967 0x1d8112c 0x2df8ce5 0x2df8c7c 0x2e04307 0x2b376c5 0x2b2ba7a 0x2b3713a 0x2b369a8 0x1233441
#   0x2db906b   github.com/mattn/go-sqlite3._Cfunc_sqlite3_exec+0x4b                                _cgo_gotypes.go:857
#   0x2dc84e4   github.com/mattn/go-sqlite3.(*SQLiteDriver).Open.func5.1+0x64                           /home/magik6k/.opt/go/pkg/mod/github.com/mattn/go-sqlite3@v1.14.16/sqlite3.go:1474
#   0x2dc8398   github.com/mattn/go-sqlite3.(*SQLiteDriver).Open.func5+0x38                         /home/magik6k/.opt/go/pkg/mod/github.com/mattn/go-sqlite3@v1.14.16/sqlite3.go:1474
#   0x2dc6fd6   github.com/mattn/go-sqlite3.(*SQLiteDriver).Open+0x3b16                             /home/magik6k/.opt/go/pkg/mod/github.com/mattn/go-sqlite3@v1.14.16/sqlite3.go:1738
#   0x1d8b299   database/sql.dsnConnector.Connect+0x39                                      /usr/lib/go/src/database/sql/sql.go:758
#   0x1d7eb42   database/sql.(*DB).conn+0x762                                           /usr/lib/go/src/database/sql/sql.go:1387
#   0x1d8137c   database/sql.(*DB).query+0x5c                                           /usr/lib/go/src/database/sql/sql.go:1721
#   0x1d81214   database/sql.(*DB).QueryContext.func1+0x54                                  /usr/lib/go/src/database/sql/sql.go:1704
#   0x1d7f966   database/sql.(*DB).retry+0x46                                           /usr/lib/go/src/database/sql/sql.go:1538
#   0x1d8112b   database/sql.(*DB).QueryContext+0xcb                                        /usr/lib/go/src/database/sql/sql.go:1703
#   0x2df8ce4   database/sql.(*DB).Query+0xc4                                           /usr/lib/go/src/database/sql/sql.go:1717
#   0x2df8c7b   github.com/lotus-web3/ribs/rbdeal.(*ribsDB).GetRetrievalCandidates+0x5b                     /home/magik6k/github.com/lotus_web3/ribs/rbdeal/deal_db.go:1191
#   0x2e04306   github.com/lotus-web3/ribs/rbdeal.(*retrievalProvider).FindCandidatesAsync+0x1e6                /home/magik6k/github.com/lotus_web3/ribs/rbdeal/retr_provider.go:87
#   0x2b376c4   github.com/filecoin-project/lassie/pkg/retriever.AssignableCandidateFinder.FindCandidates.func2+0x84        /home/magik6k/.opt/go/pkg/mod/github.com/filecoin-project/lassie@v0.14.3/pkg/retriever/assignablecandidatefinder.go:70
#   0x2b2ba79   github.com/filecoin-project/lassie/pkg/internal/candidatebuffer.(*CandidateBuffer).BufferStream+0x139       /home/magik6k/.opt/go/pkg/mod/github.com/filecoin-project/lassie@v0.14.3/pkg/internal/candidatebuffer/candidatebuffer.go:63
#   0x2b37139   github.com/filecoin-project/lassie/pkg/retriever.AssignableCandidateFinder.FindCandidates+0x3f9         /home/magik6k/.opt/go/pkg/mod/github.com/filecoin-project/lassie@v0.14.3/pkg/retriever/assignablecandidatefinder.go:66
#   0x2b369a7   github.com/filecoin-project/lassie/pkg/retriever/combinators.RetrieverWithCandidateFinder.Retrieve.func1+0x1c7  /home/magik6k/.opt/go/pkg/mod/github.com/filecoin-project/lassie@v0.14.3/pkg/retriever/combinators/retrieverwithcandidatefinder.go:31

950 @ 0x11fd2d6 0x120ddfe 0x18fe8f6 0x1233441
#   0x18fe8f5   github.com/ipfs/boxo/bitswap/client/internal/messagequeue.(*MessageQueue).runQueue+0x215    /home/magik6k/.opt/go/pkg/mod/github.com/ipfs/boxo@v0.10.3-0.20230629151409-cfcbdbce297a/bitswap/client/internal/messagequeue/messagequeue.go:424

893 @ 0x11fd2d6 0x120ddfe 0x1bdecff 0x1bc160b 0x1233441
#   0x1bdecfe   github.com/quic-go/quic-go.(*sendQueue).Run+0xfe    /home/magik6k/.opt/go/pkg/mod/github.com/quic-go/quic-go@v0.33.0/send_queue.go:67
#   0x1bc160a   github.com/quic-go/quic-go.(*connection).run.func2+0x2a /home/magik6k/.opt/go/pkg/mod/github.com/quic-go/quic-go@v0.33.0/connection.go:533

Logs: https://gist.github.com/magik6k/a5b217a5d00d00a86cbcfffe063ad0c5

rvagg commented 1 year ago

Fixed (I hope) in https://github.com/filecoin-project/lassie/pull/406

are two options that can be tuned to make it work as desired; there's total concurrency and per-retrieval within that total.

rvagg commented 1 year ago

@magik6k, Lassie v0.18.0 is out and has this if you want to give it a try.