Spark: Sunset Graphsync retrievals

[bajtos]

There is no ecosystem interest in using Graphsync for retrievals; we want everyone to move to HTTP retrievals (using the protocol described in IPFS Trustless Gateway Specification). Based on what I've been told, Curio, the Boost successor, will operate completely over HTTP only, it won't support Graphsync at all.

For a brief period of time, Spark did support only deals advertising the availability of HTTP retrievals. However, there were only ~500 of such deals, so we quickly reintroduced Graphsync back.

Because Graphsync is the default used by Boost, and it's the fastest & easiest path for SPs to be compatible with Spark, we are seeing an increasing number of SPs that are advertising the availability of Graphsync retrievals only. (About 20% of retrievals happen over HTTP, 80% over Graphsync.) While this is great for the short term, as it drives the overall RSR up, it creates a future pain point for SPs - they will need to revisit their configuration when Spark drops support for Graphsync. Would it be better for them to configure IPNI & Retrievals for HTTP from the beginning instead of doing it for Graphsync first and HTTP later?

OTOH, maybe we don't need to drop support for Graphsync yet: when Boost becomes deprecated, and providers migrate to Curio, the Graphsync vs HTTP problem will go away organically.

Other aspects to consider:

• For deals managed by Boost, we need SPs to advertise Grapsync retrievals because that's the only way to allow Spark to link PieceCID to Payload CIDs (using the IPNI metadata specific to Graphsync protocol) when the DealProposal field Label does not provide the Payload CID.

• Spark checkers delegate Graphsync retrievals to Lassie.

  • This gives us very limited visibility into retrieval failures, e.g. we cannot tell when the SP did not return data from when they return invalid data not matching the requested CID.
  • We have to maintain rusty-lassie and zinnia<->rusty-lassie integration. This has been cumbersome because of Lassie's dependency on a fork of Go's TLS, which must be updated in lockstep with Go version upgrades.

[juliangruber]

While this is great for the short term, as it drives the overall RSR up, it creates a future pain point for SPs - they will need to revisit their configuration when Spark drops support for Graphsync.

I'm wondering here about the upgrade path from Boost to Curio - if all SPs upgrade to Curio before Spark switches to HTTP only, they won't need to change their configuration.

[bajtos]

While this is great for the short term, as it drives the overall RSR up, it creates a future pain point for SPs - they will need to revisit their configuration when Spark drops support for Graphsync.

I'm wondering here about the upgrade path from Boost to Curio - if all SPs upgrade to Curio before Spark switches to HTTP only, they won't need to change their configuration.

Exactly. That's why I am debating whether we should prioritise dropping Grapsync or whether things will sort out themselves.

Then there is Venus & Venus Droplet, the alternative implementation of SP software. I don't know the status of supporting Graphsync vs HTTP there.

[juliangruber]

Do you know expected timelines for the Curio migration?

[bajtos]

Do you know expected timelines for the Curio migration?

I've been told they want to release the first public version in 2-3 months.

[juliangruber]

I think the relevant timeline is by when they're expecting SPs to have migrated - since this is the point we could switch to HTTP only.