application-research / autoretrieve

A server to make GraphSync data accessible on IPFS
22 stars 7 forks source link

feat: update lassie to sync Retriever (before provider rewrite revert) #167

Open rvagg opened 1 year ago

rvagg commented 1 year ago

DRAFT for now because it's using https://github.com/filecoin-project/lassie/pull/41

@elijaharita I'd like your eyes on this for the synchronous call in the provider, provider.retriever.Retrieve() now blocks, holding up one of the goroutines while a retrieval attempt happens. Most of them still fail from indexer lookup fails but the ones that attempt may cause a backlog.

Is there any mechanism in here to prevent an excessive backlog of tasks on the retrievalQueue that we can't process fast enough to keep up with incoming? I don't see it but I might be missing something. I'm also not experiencing a huge queue while running this locally but that might change when on a faster open network.

codecov-commenter commented 1 year ago

Codecov Report

Base: 4.87% // Head: 5.24% // Increases project coverage by +0.36% :tada:

Coverage data is based on head (ebf5c4e) compared to base (4b4dcae). Patch coverage: 12.14% of modified lines in pull request are covered.

Additional details and impacted files ```diff @@ Coverage Diff @@ ## master #167 +/- ## ========================================= + Coverage 4.87% 5.24% +0.36% ========================================= Files 15 14 -1 Lines 1723 1697 -26 ========================================= + Hits 84 89 +5 + Misses 1634 1603 -31 Partials 5 5 ``` | [Impacted Files](https://codecov.io/gh/application-research/autoretrieve/pull/167?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=application-research) | Coverage Δ | | |---|---|---| | [autoretrieve.go](https://codecov.io/gh/application-research/autoretrieve/pull/167?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=application-research#diff-YXV0b3JldHJpZXZlLmdv) | `0.00% <0.00%> (ø)` | | | [bitswap/provider.go](https://codecov.io/gh/application-research/autoretrieve/pull/167?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=application-research#diff-Yml0c3dhcC9wcm92aWRlci5nbw==) | `0.00% <0.00%> (ø)` | | | [endpoint/estuary.go](https://codecov.io/gh/application-research/autoretrieve/pull/167?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=application-research#diff-ZW5kcG9pbnQvZXN0dWFyeS5nbw==) | `0.00% <0.00%> (ø)` | | | [blocks/manager.go](https://codecov.io/gh/application-research/autoretrieve/pull/167?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=application-research#diff-YmxvY2tzL21hbmFnZXIuZ28=) | `78.50% <92.85%> (+1.05%)` | :arrow_up: | Help us with your feedback. Take ten seconds to tell us [how you rate us](https://about.codecov.io/nps?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=application-research). Have a feature suggestion? [Share it here.](https://app.codecov.io/gh/feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=application-research)

:umbrella: View full report at Codecov.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.