Protocols in providers

[hannahhoward]

Goals

• Allow specifying providers with specific protocols
• Cleanup the FixedPeers on request concept that was previously inerted a bit haphazard
• Update to new boxo fork (on 0.15.0)

Implementation

• Removed retriever.CandidateFinder -- this was a confusing concept, since there is already a types.CandidateFinder that is different. I instead changed this to types.CandidateSource.
• retriever.CandidateFinder had two methods --> FindCandidates and FindCandidatesAsync. However, we only used FindCandidatesAsync. Moreover, this is a bit of a weird name, since while the method takes a callback, it is not actually an Async method in the sense that it is intended to be a blocking call. So on types.CandidateSource, we now have only one method FindCandidates, and it has the signature of the old FindCandidatesAsync. (note to keep in line with the interface name, I'm also find to rename this SourceCandidates if it feels important)
• created a new Provider type -- this is similar to a types.RetrievalCandidate, but without the cid (I considered going back and removing this rootCID entirely but I think maybe we use it in several places? Also does it always make sense for it to be the root cid?)
• modified the DirectCandidateFinder so it can be used with a fixed set of Providers with their own protocols, and you now specify an option if you want to add the protocol discovery features (or just return candidates immediately)
• added a new syntax for specifying a peer+protocol combo -- essentially at the end of your multiaddr string, you add a +. This does remove + as an option to use in your multiaddr, but I'm not immediately aware of any cases where this is an allowed character in any of the allowed libraries (other than maybe "httppath" which isn't merged? Not even sure usually + is used in query strings).
• updated ParseProviderStrings & GetDescriptorString to handle the parsing
• update feat/lassie-fork over in boxo to be rebased on v0.15.0 and update dependency here
• this is also supported at the CLI level

For Discussion

• I originally tried to put CandidateSource directly on types.RetrievalRequest but then that gets confusing with GetDescriptorString (we'd have to make candidateSource itself have a descriptorString)
• I then tried making CandidateSource a method on types.RetrievalRequest but that led to a lot of import cycling and I didn't get to the end of that :)
• not sure the best naming here open to input.

[codecov-commenter]

Codecov Report

Attention: 24 lines in your changes are missing coverage. Please review.

Comparison is base (f2e96a7) 75.63% compared to head (ab1e2b5) 75.79%.

Additional details and impacted files

[](https://app.codecov.io/gh/filecoin-project/lassie/pull/465?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=filecoin-project) ```diff @@ Coverage Diff @@ ## main #465 +/- ## ========================================== + Coverage 75.63% 75.79% +0.16% ========================================== Files 87 87 Lines 6657 6656 -1 ========================================== + Hits 5035 5045 +10 + Misses 1341 1335 -6 + Partials 281 276 -5 ``` | [Files](https://app.codecov.io/gh/filecoin-project/lassie/pull/465?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=filecoin-project) | Coverage Δ | | |---|---|---| | [cmd/lassie/flags.go](https://app.codecov.io/gh/filecoin-project/lassie/pull/465?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=filecoin-project#diff-Y21kL2xhc3NpZS9mbGFncy5nbw==) | `51.02% <100.00%> (ø)` | | | [cmd/lassie/main.go](https://app.codecov.io/gh/filecoin-project/lassie/pull/465?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=filecoin-project#diff-Y21kL2xhc3NpZS9tYWluLmdv) | `47.66% <100.00%> (ø)` | | | [pkg/indexerlookup/candidatesource.go](https://app.codecov.io/gh/filecoin-project/lassie/pull/465?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=filecoin-project#diff-cGtnL2luZGV4ZXJsb29rdXAvY2FuZGlkYXRlc291cmNlLmdv) | `66.29% <100.00%> (ø)` | | | [pkg/indexerlookup/options.go](https://app.codecov.io/gh/filecoin-project/lassie/pull/465?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=filecoin-project#diff-cGtnL2luZGV4ZXJsb29rdXAvb3B0aW9ucy5nbw==) | `31.66% <ø> (ø)` | | | [pkg/internal/itest/mocknet/mocknet.go](https://app.codecov.io/gh/filecoin-project/lassie/pull/465?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=filecoin-project#diff-cGtnL2ludGVybmFsL2l0ZXN0L21vY2tuZXQvbW9ja25ldC5nbw==) | `76.51% <100.00%> (ø)` | | | [pkg/internal/itest/testpeer/generator.go](https://app.codecov.io/gh/filecoin-project/lassie/pull/465?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=filecoin-project#diff-cGtnL2ludGVybmFsL2l0ZXN0L3Rlc3RwZWVyL2dlbmVyYXRvci5nbw==) | `75.00% <100.00%> (ø)` | | | [pkg/internal/testutil/gen.go](https://app.codecov.io/gh/filecoin-project/lassie/pull/465?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=filecoin-project#diff-cGtnL2ludGVybmFsL3Rlc3R1dGlsL2dlbi5nbw==) | `63.28% <100.00%> (+0.58%)` | :arrow_up: | | [pkg/internal/testutil/mockcandidatefinder.go](https://app.codecov.io/gh/filecoin-project/lassie/pull/465?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=filecoin-project#diff-cGtnL2ludGVybmFsL3Rlc3R1dGlsL21vY2tjYW5kaWRhdGVmaW5kZXIuZ28=) | `83.33% <100.00%> (ø)` | | | [pkg/internal/testutil/verifier.go](https://app.codecov.io/gh/filecoin-project/lassie/pull/465?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=filecoin-project#diff-cGtnL2ludGVybmFsL3Rlc3R1dGlsL3ZlcmlmaWVyLmdv) | `96.49% <100.00%> (ø)` | | | [pkg/retriever/assignablecandidatefinder.go](https://app.codecov.io/gh/filecoin-project/lassie/pull/465?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=filecoin-project#diff-cGtnL3JldHJpZXZlci9hc3NpZ25hYmxlY2FuZGlkYXRlZmluZGVyLmdv) | `100.00% <100.00%> (ø)` | | | ... and [7 more](https://app.codecov.io/gh/filecoin-project/lassie/pull/465?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=filecoin-project) | | ... and [4 files with indirect coverage changes](https://app.codecov.io/gh/filecoin-project/lassie/pull/465/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=filecoin-project)

[rvagg]

TestAssignableCandidateFinder race test failures in CI probably worth investigating!

[bajtos]

• Is there any documentation on the new provider format? (We are using the HTTP interface provided by the Lassie Daemon.)
• The change looks fully backwards compatible, could you please confirm?

[rvagg]

Not documented publicly (yet) but it impacts the "providers" query string and CLI argument, it's backward compatible and this just adds the ability to tack on protocols after the end of the provider ID string, just add a +http, +bitswap, +graphsync to limit the protocol(s) that will be attempted with the provider. They are additive, so:

• provider=/dns/beep.boop.com/tcp/3747/p2p/12D3KooWDXAVxjSTKbHKpNk8mFVQzHdBDvR4kybu582Xd4Zrvagg will do what it does now, query the peer and see what it supports and go from there
• provider=/dns/beep.boop.com/tcp/3747/p2p/12D3KooWDXAVxjSTKbHKpNk8mFVQzHdBDvR4kybu582Xd4Zrvagg+bitswap will skip the query and go straight to attempting an http retrieval to the address
• provider=/dns/beep.boop.com/tcp/3747/p2p/12D3KooWDXAVxjSTKbHKpNk8mFVQzHdBDvR4kybu582Xd4Zrvagg+bitswap+graphsync` will queue up the provider on both bitswap and graphsync protocols and end up attempting them both.