share.ipfs.io - pcp interoperability

[dennis-tra]

Hey there,

maintainer of pcp here. I had browser <-> CLI interoperability on the roadmap from the get-go. In the most recent IPFS newsletter I learned about this project and thought I may just hook into this existing web app. Currently, the protocols to exchange the identifier and file transfer differs significantly between both tools.

My question, would it be of interest to you if I integrated the word-sequence protocol into this web app? Or is it an unpromising thought that this would land here - which would be fine of course. I just want to set realistic expectations right from the start.

What I'll probably do anyways is adding the share.ipfs.io protocol to pcp which doesn't interfere with you - I'm interested in the other way around.

Best, Dennis

[lidel]

@dennis-tra I'm happy to brainstorm some sort of vendor-agnostic interop spec that could be used across sharing tools, but embedding pcp in this app as a drop-in does not seem to be feasible. There are some high level limitations that would have to be resolved first:

• the goal of this app is to dogfood js-ipfs for file sharing in browser context, which is severely limited
  • pcp protocol is not part of default js-ipfs distribution
  • js-ipfs in browser has no DHT atm (this app connets to /wss on bootstraps/preloads for DHT interop + webrtc-star for opportunistic p2p between browsers)
  • there is pubsub, which we dont use in this app atm, but not sure if pubsub would work for interop (late on my end but i think browser and native may end up in separate cohorts of the same room, unless there are nodes acting as a brigde)
• adding a word-sequence may be a stretch for the UI of this project, as it would introduce a confusing third-party alternative to CID for static sharing
  • don't think we can work around it in the current form of the share app
  • loose idea: if a bidirectional version for realtime group sharing is created at some point in the future (group.share.ipfs.io), then word-sequence could be used as a shareable pubsub room name in URL's #fragment, but this does not sound compatible with the way pcp works today

[dennis-tra]

Hi @lidel ,

sorry for the late reply and thanks for your thoughtful answer. Your points sound totally reasonable.

I'd need a more thorough look through the codebase to be able to discuss this further as the interop was just a thought that came to my mind.

I'll just ping you on this issue when I got the time 👍

[dennis-tra]

Hi @lidel,

I finally had a look through the code. I would like pcp to be interop with this web app. I'm imagining something like:

$ pcp fetch https://share.ipfs.io/#/bafybeie3i6irbahdr4ryzsxx6xoncza2cblq7duya67bimfraig3mq5kgi

This would fetch the data directly from the browser of the peer. Similarly, pcp could create a share link:

$ pcp share somefile.txt
Link: https://share.ipfs.io/#/bafybeie3i6irbahdr4ryzsxx6xoncza2cblq7duya67bimfraig3mq5kgi

Here the user would need to leave the tool running until the content was transferred - similar to the current UX.

I saw that share.ipfs.io uses the libp2p/js-libp2p-webrtc-star transport, so I jumped in and started my own implementation of a go-libp2p-webrtc-star transport based on mtojek/go-libp2p-webrtc-star (which is outdated) and libp2p/go-libp2p-webrtc-direct.

I still have trouble to completely grasp the peer and content discovery flow though. Please correct me if I'm wrong:

• js-libp2p-webrtc-star is not only used as a transport but also for peer discovery via its discovery property (next to the bootstrap discovery mechanism).
• When a user opens share.ipfs.io the web app opens a websocket connection to and registers itself at the signalling star (and simultaneously connects to the set of bootstrap nodes)
• Then the signalling star tells the app about all currently connected peers
• Since autoDial is enabled, the app optimistically connects to all peers via exchanging ICE candidates through the signalling star.

Now I'm wondering how content is discovered if the app is opened with the CID url fragment. If my understanding above is correct I could imagine that:

• the app connects to more and more peers and optimistically exchanges bitswap messages with newly connected peers until it finds the right peer.
• the app somehow leverages pubsub. I saw that pubsub with Gossipsub is enabled but I can't see any use of it.

Other questions:

• Is there a similar inter-module-communication mechanism for the double responsibility of the js-libp2p-webrtc-star transport that also does peer discovery in the Go module system?
• Is the Bootstrap discovery mechanism strictly necessary? Due to the autoDial and peers discovered through the signalling server.

Edit: Started a conversation here: https://discuss.libp2p.io/t/share-ipfs-io-peer-and-content-discovery/894

[lidel]

_(missed the notification for this :crying_catface: – apologies for delay in response)

Answered some of remaining questions about libp2p setup in https://discuss.libp2p.io/t/share-ipfs-io-peer-and-content-discovery/894, so just to comment on proposed UX, sounds good. Content routing to/from browser node remains the main challenge here.

Some thoughts if you want to maximize utility in the browser by leveraging existing preload infrastructure:

• as a client, when share.ipfs.io links is provided, you could proactively connect to preload nodes, as those most likely have a copy of shared CID, making an instant start of bitswap session possible.
  (In the future, we may add dnsaddr TXT records with preload/delegate node addrs, so client does not need to have hardcode them)
• as a provider, you could connect to preload node + execute /api/v0/refs call against one of them to proactively upload data to preload node as an ephemeral cache

Those are optional, but if enabled by default would improve time-to-first-byte.

[SgtPooki]

This issue was opened a long time ago.. I like the idea of updating PCP to output a link to share.ipfs.io, and I like the idea of share.ipfs.io outputting a CLI command for downloading the file(s).. this is still worth doing, but the landscape has changed since we started using Helia and js-libp2p in #138.

We may have better viability for interop now.