IPNS: should work with case-insensitive identifiers (Base32)

lidel commented 6 years ago

Version information:

0.4.16

Type:

bug

Description:

During the work related to https://github.com/ipfs/in-web-browsers/issues/89 and https://github.com/ipfs/ipfs/issues/337 we've identified open problem with IPNS. We are unable to use current case-sensitive IPNS identifiers in case-insensitive URL contexts, namely:

https://<keyId>.ipns.foo.tld/
ipns://<keyId>/

keyId – PeerID or ID of a key generated via ipfs key gen

Not sure what would be the best way to tackle this.
Adding only a multibase prefix? Switching all keyId to CIDv1 ?

Part of ipfs/ipfs#337 (cc @kyledrake + @alanshaw for parity in js-ipfs)

Additional References:

2018-Q4: Ethereum people started using IPFS+ENS for real, MetaMask supports resolver for hosting *.eth websites and they just noticed the Origin problem.
- The fix is cid-in-subdomain (https://github.com/ipfs/in-web-browsers/issues/89) but it only works for /ipfs/ and at some point they will want to point at /ipns/ instead of /ipfs/, which is blocked by this issue
2019-10: Tracking issue for peer IDs as cidv1b32: https://github.com/libp2p/specs/issues/216

ghost commented 6 years ago

I think we'll have to wait for ed25519 IPNS keys, or do the same hackery as with CIDs.

Stebalien commented 6 years ago

ed25519 IPNS keys are here. However, that doesn't help. We'll have to do the same CID hackery (or just switch to using CIDs).

ghost commented 6 years ago

Let's switch to CIDs?

@kyledrake this issue should potentially be part of the base32 task force

lidel commented 5 years ago

Related discussion about PeerID in libp2p continues in https://github.com/libp2p/specs/issues/138

lidel commented 5 years ago

I believe the MVP here is a scenario where:

people convert old Qm.. PeerID (base58btc - cidv0 - dag-pb - sha2-256) to bafy.. (base32 - cidv1 - dag-pb - sha2-256)
- (via http://cid.ipfs.io or ipfs cid base32 command)
use the base32 result in /ipns/bafy.. addresses
.. and it just works (tm)

@Stebalien What would be the smallest set of changes to support this type of interop in go-ipfs? Is it something someone not deeply familiar with go-ipfs codebase could PR? If so, can you give me some pointers where to start?

Stebalien commented 5 years ago

The current peer ID format is just base58btc(<sha2-256 multicodec><256><digest>) (just a multihash).
The new format would be <base32 multibase> ++ base32(<cidv1><libp2p key multicodec><the multihash>).

That is, we'd use an entirely new multicodec.

@Stebalien What would be the smallest set of changes to support this type of interop in go-ipfs?

That depends on how deeply we want to integrate this. If we only care about this for the subdomain feature, it would be pretty easy to redirect GATEWAY/ipns/Qm... to b...ipns.GATEWAY/.

lidel commented 5 years ago

There is an ongoing work to get IPNS fast (notes from today's meeting: https://github.com/ipfs/team-mgmt/pull/948). In the short term, fast resolve of IPNS in web browser context will most likely mean custom DNS server and resolving IPNS over DNS TXT records looking like this: dnslink=/ipns/<bafy>.ipns.dweb.link (where <bafy> is the new format in base32)

iiuc this means the immediate needs are:

Agree and add <libp2p key multicodec> to multicodec/table.csv and related libs, to unlock IPNS representation in subdomains (for actual hosting, and IPNS-over-DNS lookups)
- Q: will this be just a single multicodec, or a family of entries depending on key subtype? are there any prior proposals on this?
Add support to HTTP Gateway for requests for <bafy>.ipns.GATEWAY/ and /ipns/<bafy>
- Q: I imagine Gateway could just convert <bafy> to old school base58btc before doing the actual resolve, but it may get confusing if ipfs name resolve /ipns/<bafy> does not work. Would it be feasible to do the conversion there as well?

lidel commented 5 years ago

I see a small UX problem around having a dedicated multicodec: How will go-ipfs behave if codec in /ipns/{cidv1} is different than libp2p-key?

go-ipfs prints PeerID in Base58 QmNnooDu7bfjPFoTZYxMNLWUQJyrVwtbZg5gBMjTezGAJN
User wants to convert it to CIDv1, so they do ipfs cid base32 {cid} or use https://cid.ipfs.io/
cid base32 returns bafybeiagwnqiviaae5aet2zivwhhsorg75x2wka2pu55o7grr23ulx5kxm which has the dag-pb codec instead of libp2p-key (https://github.com/multiformats/multicodec/pull/131)

This issue will mostly go away when we switch to displaying PeerID in CIDv1 everywhere, but for now we need to account for the above edge case. I believe we should support support PeerID CIDv1 with dag-pb. If that is not feasible, then the gateway should do conversion and return HTTP 301 redirect from dag-pg PeerID to one with libp2p-key.

@Stebalien Thoughts?

Zolmeister commented 5 years ago

Please consider using Tor rend spec v3 address encoding for PeerIDs Ref. https://github.com/libp2p/specs/issues/139

lidel commented 5 years ago

@Zolmeister thank you for sharing. This issue is only about adding support for wrapping existing PeerIDs in CIDv1 to unlock web uses without huge refactoring (it does not change underlying way we construct PeerID). Discussion in https://github.com/libp2p/specs/issues/139 is not blocked by this, feel free to continue it there :+1:

lidel commented 5 years ago

Back to the issue at hand:

Adding IPLD codec for libp2p public keys (https://github.com/multiformats/multicodec/pull/131) got merged
I opened https://github.com/multiformats/js-multicodec/pull/45 and https://github.com/ipfs/go-cid/pull/86 to add support for it in GO and JS
- I assumed go-cid is the right place, as go-multicodec is deprecated
- When this lands, we will add support for CIDv1 in /ipns/ paths to go-ipfs/js-ipfs/is-ipfs and it should be it.

lidel commented 5 years ago

I've been looking at things we want to ship this in time for IPFS Camp. This one looks doable. What remains here is:

[x] add libp2p-key support to https://github.com/ipfs/go-cid/pull/86
[x] release new go-cid with https://github.com/ipfs/go-cid/pull/86
[x] update go-ipfs to use new go-cid

@Stebalien will this be enough for HTTP Gateway support /ipns/<cidv1-with-libp2p-key>? If not, can you point me at places that need tweaking?

Stebalien commented 5 years ago

https://github.com/ipfs/go-ipfs/issues/5287#issuecomment-492163929

Sorry, completely missed this conversation. That's a valid concern and we may want to add a "hack it" redirect for now, or maybe a warning?

@Stebalien will this be enough for HTTP Gateway support /ipns/? If not, can you point me at places that need tweaking?

Bit late but the latest status is https://github.com/libp2p/go-libp2p-core/pull/41

lidel commented 5 years ago

How will go-ipfs behave if codec in /ipns/{cidv1} is different than libp2p-key?

That's a valid concern and we may want to add a "hack it" redirect for now, or maybe a warning?

I am thinking the least amount of work/friction will be to

Do HTTP 301 redirect on the Gateway to fix the multicodec on-the-fly. This will ensure user ends up on a valid, canonical address in the browser.
"just work" on the CLI (ignore invalid multicodec on /ipns/{cid}, or display a warning with a valid path ready to copy&paste)

The issue will go away when we switch to CIDv1B32 everywhere, so we should not stress too much about beyond making it work.

Stebalien commented 5 years ago

Yeah, I guess. Just... grrr.

lidel commented 5 years ago

Tracking issue for peer IDs as cidv1b32: https://github.com/libp2p/specs/issues/216

lidel commented 4 years ago

We are in the process of adding support for PeerID encoded as CIDv1 with libp2p-key multihash (https://github.com/libp2p/specs/issues/216).

Seems that even tho we use go-libp2p-core v0.3.0 which includes https://github.com/libp2p/go-libp2p-core/pull/41 there are still places which assume Multihash instead of CID:

$ ipfs name resolve -r /ipns/bafzbeiedwhhfhtte23hocn4k2bi4awpqay5sdhzqju7htk3y6sk63lvuae
Error: not a valid proquint string

$ ipfs resolve -r /ipns/bafzbeiedwhhfhtte23hocn4k2bi4awpqay5sdhzqju7htk3y6sk63lvuae
Error: not a valid proquint string

I'll try to track them down and fix, as it is blocking me from writing tests for #6096 :grimacing:

Update: found, will fix in #6096

lidel commented 4 years ago

This was fixed in #6096 and will ship with go-ipfs 0.5.0 (https://github.com/ipfs/go-ipfs/issues/7109) :rocket:

When used in subdomain context, the HTTP Gateway will automatically convert case-sensitive identifier to base32 and redirect to URL with correct multicodec.

Command line and RPC API provide handy error message when an invalid multicodec is used in CIDv1:

# resolve Base58 PeerID
$ ipfs name resolve -r /ipns/QmY9DNm4UJxaTBB9FrhNkXQ75GaK43D8KhZtpJjmw8P7Sz
/ipfs/bafybeigdyrzt5sfp7udm7hu76uh7y26nf3efuylqabf3oclgtqy55fbzdi

# convert PeerID to CIDv1 (with invalid multicoded, to illustrate next step)
$ ipfs cid base32 QmY9DNm4UJxaTBB9FrhNkXQ75GaK43D8KhZtpJjmw8P7Sz
bafybeieru3k6wvdzsrsjcx33jund7ebawoqlyc7jpitmn7wd4cwveunolm

# resolve CIDv1 with dag-pb multicodec
$ ipfs name resolve -r /ipns/bafybeieru3k6wvdzsrsjcx33jund7ebawoqlyc7jpitmn7wd4cwveunolm
Error: peer ID represented as CIDv1 require libp2p-key multicodec: retry with /ipns/bafzbeieru3k6wvdzsrsjcx33jund7ebawoqlyc7jpitmn7wd4cwveunolm

# resolve CIDv1 with libp2p-key multicodec suggested in previous step
$ ipfs name resolve -r /ipns/bafzbeieru3k6wvdzsrsjcx33jund7ebawoqlyc7jpitmn7wd4cwveunolm
/ipfs/bafybeigdyrzt5sfp7udm7hu76uh7y26nf3efuylqabf3oclgtqy55fbzdi

ipfs / kubo