search

ChluNetwork / chlu-ipfs-support

Support libs to talk to IPFS
MIT License
1 stars 1 forks source link

Task management with queue, cache and concurrency #52

Open fazo96 opened 6 years ago

fazo96 commented 6 years ago

This issue comes from the multiple broadcasts attemps. Sometimes, many copies of the same message rebroadcasted arrive to a service node which schedules many pinning operations for the same exact content, which does not make sense at all.

In other words if two identical Chlu requests arrive I should only do the thing one time

kulpreet commented 6 years ago

I was thinking about decoupling here while reviewing an earlier PR. I recalled how google analytics solve this problem with a queue.

Here's a link describing what they do, if you are looking for some ideas. https://developers.google.com/analytics/devguides/collection/analyticsjs/how-analyticsjs-works#the_ga_command_queue

fazo96 commented 6 years ago

That is an excellent solution. If we use a queue like that for the requests that arrive from the pubsub topic, and before adding one to the queue we check if there is already an in progress identical one then we will solve this problem :)

We will need to be super sure that we don't leak memory by having entries in the queue that never get cleaned

fazo96 commented 6 years ago

A bug related to this has popped out:

If stop is called before an IPFS I/O operation is done, when the function that did the I/O resumes it will attempt to continue whatever it was doing not realizing that the whole thing has been shut down.

Maybe we can make a queue system with tasks that can be cancelled?