fetcher makes a lot of queries

mediacloud / rss-fetcher

Intelligently fetch lists of URLs from a large collection of RSS Feeds as part of the Media Cloud Directory.

https://search.mediacloud.org/directory

Apache License 2.0

5 stars 5 forks source link

Open philbudne opened 8 months ago

philbudne commented 8 months ago

The fetcher queries the feeds table for worker once a minute (headhunter.DB_READY_SEC)

One reason is so that "fetch soon" requests are noticed in a timely manner.

There are some efficiency improvement ideas at the top of fetcher/hunter.py and fetcher/scoreboard.py