The current test for staleness doesn't seem to be smart enough. After running diffengine for over a month it is taking it 8 hours to check Breitbart, The Guardian and The Washington Post. I think it needs to be smarter about what to do with the backlog of sites. Perhaps randomly sampling from them?
The current test for staleness doesn't seem to be smart enough. After running diffengine for over a month it is taking it 8 hours to check Breitbart, The Guardian and The Washington Post. I think it needs to be smarter about what to do with the backlog of sites. Perhaps randomly sampling from them?