search

OriginProtocol / origin

Monorepo for our developer tools and decentralized marketplace application
https://www.originprotocol.com/developers
MIT License
652 stars 196 forks source link

Prod IPFS hashes should be available on other gateways #2511

Open nick opened 5 years ago

nick commented 5 years ago

It seems hashes pinned to https://ipfs.originprotocol.com are unavailable from other IPFS servers such as:

tomlinton commented 5 years ago

They do become available after requesting (eventually), e.g. https://gateway.pinata.cloud/ipfs/QmWP4wWSkZyymwFYMHNisEkVPD9gm7Y5ncq2ca4ZHAkwBH and https://gateway.ipfs.io/ipfs/QmWP4wWSkZyymwFYMHNisEkVPD9gm7Y5ncq2ca4ZHAkwBH.

We never request content through those gateways, so when you try and open a hash it can take a long time for it to find the hash amongst all the peers because we are the only peer holding it. That hash above that I retrieved will disappear from those gateways if they have GC enabled.

We could integrate more paid pinning services (e.g. pinata) which would improve things. We also need to repair our protocol labs integration. I think @DanielVF was going to take a look at it. If we have the hashes available on more nodes that just ours it'll improve the request time.

nick commented 5 years ago

In that case I wonder if we can get hashes propagating faster. Starting an IPFS daemon locally, pushing a hash and accessing it from gateway.ipfs.io is fairly immediate which makes me wonder if we don't have very good connectivity with the rest of the network for some reason. Are we using go-ipfs or js-ipfs? Is it started with the 'server' profile?

tomlinton commented 5 years ago

Hmm it shouldn't be immediate - as you are the only person that holds the file. We use go-ipfs and a manual config. There is probably things we can do to increase peering/discoverability and it'd be a good idea for us to double check our configs.

More importantly thought I think increasing the distribution of our content among IPFS nodes should be what we aim for. We aren't making regular requests via other nodes so there is no reason for them to keep our content around, and it'll just get garbage collected (ipfs.io is 12 hours at max I think, and it'll get shorter as IPFS gets more popular), so in the end we'll just be the single point of failure unless we keep up a constant request rate via other IPFS nodes.

I think a good approach would be to use an assigned IPFS gateway from a list rather than always using our own (maybe make it sticky in the clients local storage). That way every visitor to the DApp is ensuring our content is distributed around and its a good acid test to ensure our data is actually decentralised. I'm not sure how to handle the case for freshly uploaded hashes though (e.g. upload to our IPFS gateway, and then we immediately try and retrieve it from another), as we don't want a long delay there. I guess if we get our pinning integration back up and running and perhaps pay for a pinning service it should be reasonably quick.

nick commented 5 years ago

Yea totally agree that out hashes should be ore widely spread. Perhaps we could create a service that others can run on their machine that would listen for contract events, then download and pin those blobs to their own public IPFS server. We could then reward those network participants with OGN every week they keep their public service online. We would verify by pinging the participating server for a random IPFS hash every 15 mins and verify it served the correct result. Any bad responses or downtime would deduct from their overall reward of, say, 150 OGN/month.

cc @joshfraser

tomlinton commented 5 years ago

That is a cool idea and a good use for the token. I'm not sure how we can ensure that they are serving the content from a local drive rather than querying from another IPFS node. Perhaps response time?

Another idea I had is for every IPFS request that goes through ipfs-proxy, we can also fire off a request to another IPFS gateway for the same hash. We could maintain a list to ensure we are requesting every x hours. I think js-ipfs implemented something similar with preloading.

Edit: I think perhaps we could solve that local issue by hash listings with file sizes.

nick commented 5 years ago

I think we could use a similar concept for hosting other services such as Discovery, GraphQL and possibly Messaging. Growth and Bridge we'd probably want to keep central.

I like the idea on ipfs-proxy. I also found another pinning service we could use: eternum.io

nick commented 5 years ago

This is a useful article on how to speed up pinning...

https://medium.com/pinata/speeding-up-ipfs-pinning-through-swarm-connections-b509b1471986

Also a handy list of public gateways:

https://github.com/ipfs/public-gateway-checker/blob/master/gateways.json

franckc commented 5 years ago

FYI @DanielVF created #2543 to track the work for fixing the pinner.