search

probe-lab / roadmap

Activities and roadmap for ProbeLab
2 stars 0 forks source link

Milestone: Reprovide Sweep in Go DHT #35

Open yiannisbot opened 11 months ago

yiannisbot commented 11 months ago

ETA: 2023-11-30

Description: Tracking issue for the design, specification and implementation of Reprovide Sweep, as per the docs and discussions here:

Children:

j-adel commented 5 months ago

Hello. I was wondering if there are any updates on the timeline for the Reprovide Sweep. This feature is desperately needed for a large content providers.

yiannisbot commented 5 months ago

Hi @j-adel, unfortunately, we've had to deprioritise this work stream due to some reorganisation of the team and its targets. Can you give some more details about the use-case and how pressing Reprovide Sweep is for your organisation? Given the right resources/capacity/funding, we could reconsider and bring this back up the priority list.

If it's more convenient, feel free to reach out to me at: yiannis@probelab.io

Rashkae2 commented 5 months ago

I think it's already known, but if you have more than a few GB of content on an IPFS node, the default provide mechanism is unbelievable inefficient. It will generate lots of network traffic, keep open connections to hundreds of peers, and will not even be able to publish the content (to dht) fast enough to not fall off the network.

Accelerated DHT solves this, but is very problematic. The Hourly 10 minute sweep of the DHT will effectively DDoS your own network for a few minutes every hour. It is not something can can be left on a network connection used by other services/peers/home networks.

Personally, I have found the best way to work around both these limitations by implementing my own sweep via config files. Once every 24 hours or so, stop the deamon, swap in a config file with Accelerated dht, start the daemon for 45 minutes, (long enough to sweep the dht and republish all CID's).. stop deamon, swap in config file with Strategic providing, and restart. (config files need to be swapped because --config-file option doesn't seem to do anything whatsoever.)

This problem can't possibly he helping adoption.

j-adel commented 5 months ago

@yiannisbot That's unfortunate. I was commenting on behalf of one of the free access provider libraries. Several of them have tried to use IPFS as an alternative format for sharing books and articles as it lends itself quite nicely to many of the medium's demands. However, this issue in particular is causing issues in reproviding content to the network as there are hundreds of thousands of articles to reprovide every day. This issue is more apparent on lesser capable devices and for clients who want to help the service. I'm not sure what kind of resources can be offered but unfortunately financial doesn't seem likely. However, I do want to emphasize how useful IPFS could be for sharing open-access documents if it resolves these large scale content providers and efficiency in dealing with them.

pkieltyka commented 5 months ago

hi all, is this ticket related to the reprovide sweep mention in the https://blog.ipfs.tech/2023-09-amino-refactoring/ blog post, "Amino (the Public IPFS DHT) is getting a facelift" ?

The blog post was very encouraging to solve some key issues on IPFS/Amino, is the roadmap changing as outlined in the blog post..?

guillaumemichel commented 4 months ago

is this ticket related to the reprovide sweep mention in the https://blog.ipfs.tech/2023-09-amino-refactoring/ blog post

yes

The blog post was very encouraging to solve some key issues on IPFS/Amino, is the roadmap changing as outlined in the blog post..?

@pkieltyka, yes our priorities have shifted, we need funding in order to continue this workstream. See https://github.com/probe-lab/roadmap/issues/35#issuecomment-1908175106

j-adel commented 4 months ago

@guillaumemichel Is there the option to try to call for contributors to dig in and help out? Perhaps if the discussion of new features is more open and transparent more people would like to contribute to this and other features. I'd love to help out personally but I have zero experience with Go, unfortunately.

guillaumemichel commented 4 months ago

Yes, feel free to share it to potential contributors if you think they would be interested in working on the implementation 👍🏻

pkieltyka commented 3 months ago

I thought Protocol Labs was a well funded organization, and the blog post was published on the official IPFS blog, and seems this is a pretty critical component for IPFS to have a more robust network.. I'm confused whats going on over there.. IPFS has been in development for what, 15 years? .. why would the org behind IPFS post a roadmap on their official blog and then not follow through..? did they not consider the budget at the time of publishing the post?

j-adel commented 3 months ago

@pkieltyka 11 years in development, I think. And while this news is greatly disappointing, please remember that these are people doing hard work on an open-source service in a difficult economic environment.

guillaumemichel commented 3 months ago

@pkieltyka the team working on this development has nucleated out of Protocol Labs back in January 2024, and is now an independent entity. Hence, the team had to change its priorities to find funding.

Some background about Protocol Labs nucleations.

The blogpost was posted before the team was aware about nucleation timelines and conditions. I am sorry about the change of plans, we would have loved to pursue this workstream.