(q)fetcher updates - Githubissues

mediacloud / story-indexer

The core pipeline used to ingest online news stories in the Media Cloud archive.

https://mediacloud.org

Apache License 2.0

2 stars 5 forks source link

(q)fetcher updates #259

Closed philbudne closed 8 months ago

philbudne commented 8 months ago

Update to mcmetadata 0.12.x
Update both batch and queue fetchers to use mcmetadata MEDIA_CLOUD_USER_AGENT constant

Queue based fetcher:

always used scrapy-esque connection timing to determine inter-request delay
fix rss-queuer.py to set "via" in RSSEntry
indexer/queuer.py: reenable logging
indexer/workers/fetcher/tqfetcher.py:
- add thoughts on how to improve (holding messages, less re-queuing)
- don't treat bad URLs as connection failures!!
- replace --issue-interval and --slot-requests with --target-concurrency
- add --busy-delay-seconds, --conn-retry-minutes, --no-second-try, --dump-slots
- try replacing SIGQUIT with SIGUSR1 (still kills process??)
- report "issue" timing for each attempt (don't include sleep and second try!)
- report "retire" timing
- disable flake8 complain about process_story method complexity (didn't see a nice way to streamline/break up)
indexer/workers/fetcher/sched.py:
- put second try under runtime control
- always use scrapy latency calculation for conn. rate: take target_concurrency argument to Scoreboard
- keep Slot.issue_interval
- add ConnStatus: passed to Slot.retire to report connection result
- add/use TS_IDLE constant
- pulled second try logic up to tqfetcher.py
- avoid logging while holding lock (may do DNS lookups!!!)