search

ipfs / go-delegated-routing

See https://github.com/ipfs/go-libipfs/tree/main/routing/http instead.
Other
5 stars 9 forks source link

Updated/new HTTP API Migration #63

Closed guseggert closed 1 year ago

guseggert commented 1 year ago

Context

Per the Delegated Routing HTTP API IPIP-337, we're expecting changes to ipfs/go-delegated-routing, either as a new HTTP Transport of Reframe or a "new" protocol. The specific path to be chosen is based on what's decided there. This tracking issue is for the resulting work and migrations for rolling it out in production.

Done Criteria

  1. Parties interested in the Delegated Routing HTTP API have a clean/sane client and server implementation.
  2. This has rolled out to Kubo, Hydras, and cid.contact
  3. There is followup action outlining the communication steps we need to like updating https://blog.ipfs.tech/2022-09-02-introducing-reframe/

Why Important

This will enable more/easier adoption of delegated routing across IPFS implementations. See the IPIP for more information.

User/Customer

Delegated routing servers and clients.

Notes

Known systems requiring migration:

Timeline

Cleanup is being tracked in https://github.com/ipfs/kubo/issues/9479. We can close this once the items above are completed.

BigLep commented 1 year ago

@guseggert : I provided some more info on what I think the done criteria is. Feel free to adjust.

Also, some other things I think we need notes on:

  1. Are we doing a new go major version?
  2. What happens to the existing "reframe" implementation?
  3. What is the migration sequence of events? For example, is cid.contact updating and going to break the hydras until Hydra updates?
willscott commented 1 year ago
  1. What is the migration sequence of events? For example, is cid.contact updating and going to break the hydras until Hydra updates?

i imagine cid.contact can implement the new endpoint in existing to the current reframe one. Once that's in place, then hydras / kubo can migrate at their leisure.

from looking at the proposed spec, initially cid.contact will only implement the /v1/providers/{cid} endpoint, since double hashing does not yet exist, and the indexer does not accept provide requests.

guseggert commented 1 year ago

What is the migration sequence of events? For example, is cid.contact updating and going to break the hydras until Hydra updates?

We can do it gracefully, I added details to the issue.

guseggert commented 1 year ago

Are we doing a new go major version?

I don't see a practical reason to do this, if anyone has one then we can consider it but mv bumps are painful and IMO should be avoided when possible. If this library had a huge amount of consumers then sure, but the number of consumers is tiny.

BigLep commented 1 year ago

Also, for anyone watching this issue, I added a callout up at the top that it's still an open question on how we do an HTTP delegated routing API. There good/important open questions in https://github.com/ipfs/specs/pull/337 and those should get resolved first. This issue is for accounting for making and rolling out whatever changes are decided on in IPIP-337.

guseggert commented 1 year ago

What happens to the existing "reframe" implementation?

It would exist in the Git history. If someone needs it they are free to fork it from the commit here and keep on using it, but I don't think it makes sense to keep it around when it is broken and we aren't using it. (By "broken" I'm referring to both implementation bugs such as this and the spec issues mentioned here.)

BigLep commented 1 year ago

@guseggert : thanks for the estimates on the work. Given this is actively being pursued and is important for any work involving potential adjustments to the hydras (https://pl-strflt.notion.site/2022Q4-Hydra-Dial-Down-c39f63d473e64f6fabcbfc38fb11d670 ), can you give please give an estimated timeline? It looks like we have these steps to track:

Those can be added to the description.

Thanks!

BigLep commented 1 year ago

Proposed timeline and tracking issues for the corresponding repos have been added to the description.

@willscott: if you can add in your planned dates to the description, that would be great - thanks.

willscott commented 1 year ago

can i assume the current draft branch (not yet in PR) of implementation in this repo, namely https://github.com/ipfs/go-delegated-routing/tree/guseggert/content-routing-poc - which is not yet reviewed - is the spec we're basing our changes off of?

i think we can do this pretty quickly, like ~ one week, but we'd like to do it once, rather than trying to anticipate where the evolving spec finalizes.

BigLep commented 1 year ago

Relevant PR: https://github.com/ipfs/go-delegated-routing/pull/65

BigLep commented 1 year ago

2022-11-22 standup conversation: need to handle STI being able to deploy old and new versions simultaneously.

BigLep commented 1 year ago

@guseggert : can you please update the timeline based on the latest understanding?

guseggert commented 1 year ago

Plans have changed due to ongoing work to scale down part of the Hydras. In order to not pollute the measurement effort involved in that (to ensure there is not a network-wide issue when they are scaled down), there is a code freeze on the Hydras until the end of next week (2022-12-08). So we will need to push Hydra integration back until then. But this should not block cid.contact from integrating the HTTP API, and we can test with Hydras before the final Kubo release, so I don't think this impacts the overall ETA.

guseggert commented 1 year ago

The delegated routing HTTP client works fine in Hydras with dev.cid.contact (it is not deployed to cid.contact yet). The only issue is that dev.cid.contact is returning an incorrect protocol id ("transport-bitswap" instead of "bitswap"), I have opened an issue for that here: https://github.com/ipni/indexstar/issues/50

Once cid.contact is deployed to with the correct protocol id, we should be good to roll out to all the Hydras. I will open a PR for this.