search

larsga / whazzup

Automatically exported from code.google.com/p/whazzup
0 stars 0 forks source link

Keep CheckFeed from blocking the queue #38

Closed GoogleCodeExporter closed 8 years ago

GoogleCodeExporter commented 8 years ago 
We can do this easily by doing it async. CheckFeed adds a feed-checking task to 
an internal in-memory queue. A sub-thread handles these tasks, and stores 
parsed feeds as in memory object structures. Then it sends a ProcessFeed 
message to the queue, which does the actual processing. Should be trivial to 
implement.

Original issue reported on code.google.com by lar...@gmail.com on 26 Jul 2011 at 6:18

GoogleCodeExporter commented 8 years ago 
Another consideration is that since CheckFeed takes 1.8 seconds on average we 
can only do 6000 feeds before it takes three hours simply to check them all. So 
beyond 6000 feeds the queue is just going to grow uncontrollably.

Original comment by lar...@gmail.com on 27 Jul 2011 at 12:28

GoogleCodeExporter commented 8 years ago 
It seems that most feeds don't actually have anything new. It may be that by 
keeping a timestamp or URL we can detect this, and get rid of most of the 
overhead from this case.

Original comment by lar...@gmail.com on 27 Jul 2011 at 1:08

GoogleCodeExporter commented 8 years ago

Original comment by lar...@gmail.com on 20 Aug 2011 at 6:04

GoogleCodeExporter commented 8 years ago 
This is done now.

Original comment by lar...@gmail.com on 1 Oct 2011 at 6:42