feat: create helia-remote-pinning example

SgtPooki commented 1 year ago

feat: start helia-remote-pinning-example
feat: add node & browser pinning example

Note: the version of @helia/remote-pinning being used here is currently in PR at https://github.com/ipfs/helia-remote-pinning/pull/13

Browser example

NOTE: the browser example does not succeed with web3.storage. There are two problems:

They do not seem to be dialing our origins, or we do not seem to be receiving their dials.
The returned delegates are not reachable from the browser

I've tested all the main pinning services I know to try, and they all have errors of some sort:

web3.storage
- does not seem to dial our p2p-circuit multiaddrs to pull the data
- Does not return browser-dialable delegates
estuary
- authentication is failing. our ipfs-webui CI is also failing to pin using estuary. The service may be down, or something changed and I need to update our tokens?
- Estuary is no longer supported/maintained.
Pinata
- Pinata doesn't like any origins we send in
- they only return 1 delegate: '/dns4/pinning-service-3.ipfs-swarm.use1.pinata.cloud/tcp/4001/p2p/12D3KooWQCtur8nTxKaDaZ9VaeaMJLB7d5UBcwHNBbyQhpagjLU8'
  - running https://gist.github.com/SgtPooki/588b9527bff706a7d2a32afbdb52708e results in a connection timeout
crust.io
- setup is a pain and authentication is failing.
4EVERLAND
- returns single delegate: '/dnsaddr/node-1.ipfs.4everland.net/p2p/12D3KooWQ85aSCFwFkByr5e3pUCQeuheVhobVxGSSs1DrRQHGv1t' that throws an error on unsupported multiaddr
  - running https://gist.github.com/SgtPooki/588b9527bff706a7d2a32afbdb52708e results in a connection timeout
- pinning moves from queued to pinning but doesn't seem to ever succeed.
scaleway
- pinning-service-client is CORS failure and also returning 405 for OPTIONS request that is sent as pre-flight request (pinged @vmscw on slack)
filebase
- TBD

Node example

The node example seems to work without issue (because tcp transports are available in nodejs)

Pinning Service	Success?	AgentVersion of delegates	Delegates returned
web3.storage	Yes	kubo/0.18.1/675f8bd/docker	'/ip4/54.68.186.137/tcp/4001/p2p/12D3KooWCw2pRmt1J8mLXRm7HQEVgqfWj3Njs7SkDimPLeyQyATK' '/ip4/54.68.186.137/udp/4001/quic-v1/p2p/12D3KooWCw2pRmt1J8mLXRm7HQEVgqfWj3Njs7SkDimPLeyQyATK' '/ip4/54.68.186.137/udp/4001/quic/p2p/12D3KooWCw2pRmt1J8mLXRm7HQEVgqfWj3Njs7SkDimPLeyQyATK'
pinata	No	kubo/0.18.1/675f8bd/docker	'/dns4/pinning-service-1.ipfs-swarm.use1.pinata.cloud/tcp/4001/p2p/12D3KooWFxvYLyabpcDtDLYd16nYygG3YS2FV6GYU9ut6M3rr5We'
crust.io	TBD	TBD	TBD
4everland	No	go-ipfs/0.11.0/67220ed	'/dnsaddr/node-1.ipfs.4everland.net/p2p/12D3KooWQ85aSCFwFkByr5e3pUCQeuheVhobVxGSSs1DrRQHGv1t'
scaleway	No	Error: connect QmcZf59bWwK5XFi76CZX8cbJ4BhTzzA3gU1ZjYZcYW3dwt failure: failed to dial: no good addresse [1m13.599s]	'/dnsaddr/delivery.ipfs.labs.scw.cloud/p2p/QmcZf59bWwK5XFi76CZX8cbJ4BhTzzA3gU1ZjYZcYW3dwt'
filebase	Sometimes (more than a few get stuck in pinning status for > 30m)	kubo/0.19.2/afb27ca/filebase	'/ip4/15.204.161.17/tcp/4001/p2p/12D3KooWErRwP21vdVD7rkaivnMfxXAnXs3djX4D5JNfzAYFmrCM' '/ip4/141.95.85.123/tcp/4001/p2p/12D3KooWNWgNz5QSL2D2YxfKcYo6HyEpFuRnW2fRYewrDmjWJECi' '/ip4/198.244.201.187/tcp/4001/p2p/12D3KooWC9L4RjPGgqpzBUBkcVpKjJYofCkC5i5QdQftg1LdsFb2'

NOTE: when running npm run start-node against filebase, retries needs to be bumped up so that the node is still running when filebase kicks off their task of pulling the content. Joshua Noble on slack "Pinning is not immediate, it goes into a queue."

Tests

Not started yet.. I think in order to test this properly, we will need to stand up more than a mock pinning provider that involves delegate and origin dials.

SgtPooki commented 1 year ago

@achingbrain can you help me figure out how to log incoming dial attempts for the browser example? I can't seem to get it to succeed unless the pinning provider is already aware of the CID (via npm run start-node -- "some string") and then running the browser example with the same string.

achingbrain commented 1 year ago

Something like this?


helia.libp2p.addEventListener('connection:open', (evt) => {
  const conn = evt.detail

  if (conn.direction === 'inbound') {
    // .. some code here
  }
})

SgtPooki commented 1 year ago

@achingbrain ok so that will listen for new connections, but I doubt we will get new inbound connections to the browser (currently) because only webrtc can provide that functionality and that doesn't exist in (m)any backend IPFS nodes yet.

Does any event fire when a connection comes through on an already open connection with a p2p-circuit relay? I imagine this is where we need to listen for connection attempts to helia browser nodes.

BigLep commented 1 year ago

@SgtPooki : Thanks for doing this. I'll look more a bit later as well, but some initial thoughts:

Thanks for summarizing the situation. Rough that none work...
We should remove Estuary. It is no longer supported/maintained.
Maybe try https://filebase.com/ipfs-pinning/ ?
I would reach out to these groups and ask how they test their service. You can also point to your example.

SgtPooki commented 1 year ago

@BigLep No problem.. happy to atleast be narrowing the focus for https://github.com/ipfs/helia/issues/256

We should remove Estuary. It is no longer supported/maintained.

ACK

Maybe try https://filebase.com/ipfs-pinning/ ?

Failing from nodejs :( -- I just created a table with all pinning services and only web3.storage is succeeding from nodejs currently.

Some note-able callouts:

web3.storage is the only successful pinning service from NodeJS in this example.
pinata can be connected to, but pinning doesn't succeed. Also, their handling of "origins" is flagging valid multiaddrs as invalid, so i have to explicitly filter out all multiaddrs containing quic
4everland can be connected to, but pinning doesn't succeed. They're running a very old version of go-ipfs
scaleway & filebase: pinning providers' returned delegates cannot be connected to from Helia nor the latest kubo version, nor ipfs-shipyard/vole
crust.io requires web3 wallet and i need new credentials to investigate further

I would reach out to these groups and ask how they test their service. You can also point to your example.

With most pinning services leaving something to be desired, I will reach out to them all, point them to this issue, and start work on other tasks. I will consider https://github.com/ipfs/helia/issues/256 blocked until we have some pinning services with webtransport.

BigLep commented 1 year ago

Thanks for the update here @SgtPooki. This is tough to see, but at least we're engaged with reality.

A couple of things:

I believe there are shared slack channels with most/all of these or that ecosystem/outercore can help us get in touch. It's probably worth requesting connections to these groups in #ipfs-ecosystem.
Is pinning service compliance (https://github.com/ipfs-shipyard/pinning-service-compliance ) showing similar issues that multiple providers don't work (either from the browser or nodejs)?

SgtPooki commented 1 year ago

I believe there are shared slack channels with most/all of these or that ecosystem/outercore can help us get in touch. It's probably worth requesting connections to these groups in #ipfs-ecosystem.

Will request in ipfs-ecosystem.

Is pinning service compliance (https://github.com/ipfs-shipyard/pinning-service-compliance ) showing similar issues that multiple providers don't work (either from the browser or nodejs)?

it's showing failures in estuary for pinning content, and crust.io authentication errors, but other services are showing successes primarily because we're using inlineCids. So data retrieval through the IPFS network is not required. IIRC, pinning-service-compliance was primarily intended to validate that services were implementing the pinning-services spec properly, not that the pinning services themselves worked well when doing IPFS network connectivity.

SgtPooki commented 1 year ago

slack threads or github issues with pinning providers & further discussions.

I will link any issues/tickets that come up, but for now, here are some slack threads where I'm trying to get in touch with the teams:

Service	Link
#ipfs-ecosystem help request	https://filecoinproject.slack.com/archives/C05PGBP697E/p1694802122998339
web3.storage	https://filecoinproject.slack.com/archives/C027XP5BTGB/p1694708735768809
Pinata	https://filecoinproject.slack.com/archives/C015WB84K4M/p1694800671205179
crust.io	https://github.com/crustio/ipfs-w3auth-pinning-service/issues/36
Scaleway	https://filecoinproject.slack.com/archives/C03KQ8MC62Y/p1694802383522219?thread_ts=1694512602.404439&cid=C03KQ8MC62Y
4EVERLAND	TBD
filebase	https://filecoinproject.slack.com/archives/C025RV6JLLB/p1694820067603299

SgtPooki commented 1 year ago

note: filebase did some updates and the returned delegates are now dialable:

╭─    ~/code/work/protocol.ai/ipfs-shipyard/vole 
╰─ ✔ ❯ ./vole libp2p identify /ip4/15.204.161.17/tcp/4001/p2p/12D3KooWErRwP21vdVD7rkaivnMfxXAnXs3djX4D5JNfzAYFmrCM
PeerID: "12D3KooWErRwP21vdVD7rkaivnMfxXAnXs3djX4D5JNfzAYFmrCM"
Protocol version: "ipfs/0.1.0"
Agent version: "kubo/0.19.2/afb27ca/filebase"
Listen addresses:
    - "/ip4/15.204.161.17/tcp/4001/p2p/12D3KooWErRwP21vdVD7rkaivnMfxXAnXs3djX4D5JNfzAYFmrCM"
Protocols:
    - "/ipfs/bitswap"
    - "/ipfs/bitswap/1.0.0"
    - "/ipfs/bitswap/1.1.0"
    - "/ipfs/bitswap/1.2.0"
    - "/ipfs/id/1.0.0"
    - "/ipfs/id/push/1.0.0"
    - "/ipfs/kad/1.0.0"
    - "/ipfs/lan/kad/1.0.0"
    - "/ipfs/ping/1.0.0"
    - "/libp2p/circuit/relay/0.2.0/stop"
    - "/x/"

╭─    ~/code/work/protocol.ai/ipfs-shipyard/vole
╰─ ✔ ❯ ./vole libp2p identify /ip4/141.95.85.123/tcp/4001/p2p/12D3KooWNWgNz5QSL2D2YxfKcYo6HyEpFuRnW2fRYewrDmjWJECi
PeerID: "12D3KooWNWgNz5QSL2D2YxfKcYo6HyEpFuRnW2fRYewrDmjWJECi"
Protocol version: "ipfs/0.1.0"
Agent version: "kubo/0.19.2/afb27ca/filebase"
Listen addresses:
    - "/ip4/141.95.85.123/tcp/4001/p2p/12D3KooWNWgNz5QSL2D2YxfKcYo6HyEpFuRnW2fRYewrDmjWJECi"
Protocols:
    - "/ipfs/bitswap"
    - "/ipfs/bitswap/1.0.0"
    - "/ipfs/bitswap/1.1.0"
    - "/ipfs/bitswap/1.2.0"
    - "/ipfs/id/1.0.0"
    - "/ipfs/id/push/1.0.0"
    - "/ipfs/kad/1.0.0"
    - "/ipfs/lan/kad/1.0.0"
    - "/ipfs/ping/1.0.0"
    - "/libp2p/circuit/relay/0.2.0/stop"
    - "/x/"

╭─    ~/code/work/protocol.ai/ipfs-shipyard/vole 
╰─ ✔ ❯ ./vole libp2p identify /ip4/198.244.201.187/tcp/4001/p2p/12D3KooWC9L4RjPGgqpzBUBkcVpKjJYofCkC5i5QdQftg1LdsFb2
PeerID: "12D3KooWC9L4RjPGgqpzBUBkcVpKjJYofCkC5i5QdQftg1LdsFb2"
Protocol version: "ipfs/0.1.0"
Agent version: "kubo/0.19.2/afb27ca/filebase"
Listen addresses:
    - "/ip4/198.244.201.187/tcp/4001/p2p/12D3KooWC9L4RjPGgqpzBUBkcVpKjJYofCkC5i5QdQftg1LdsFb2"
Protocols:
    - "/ipfs/bitswap"
    - "/ipfs/bitswap/1.0.0"
    - "/ipfs/bitswap/1.1.0"
    - "/ipfs/bitswap/1.2.0"
    - "/ipfs/id/1.0.0"
    - "/ipfs/id/push/1.0.0"
    - "/ipfs/kad/1.0.0"
    - "/ipfs/lan/kad/1.0.0"
    - "/ipfs/ping/1.0.0"
    - "/libp2p/circuit/relay/0.2.0/stop"
    - "/x/"

2color commented 10 months ago

Some updates:

web3.storage is discontinuing support for the Pinning API spec (source)
Estuary is discontinued
Unclear whether nft.storage will continue supporting the pinning API spec. I think they're running on the same infra as web3.storage, so it may also be discontinued.

vercel[bot] commented 3 weeks ago

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
verified-fetch	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Oct 10, 2024 11:17am

ipfs-examples / helia-examples