switch wayback machine from prefect to mutliprocess/scrapy

counterdata-network / story-processor

Story discovery engine for the Counterdata Network. Grabs relevant stories from various APIs, runs them against bespoke classifier models, post results to a central server.

Apache License 2.0

0 stars 2 forks source link

switch wayback machine from prefect to mutliprocess/scrapy #28

Closed rahulbot closed 1 year ago

rahulbot commented 1 year ago

This manually integrates changes from #26, making it invalid. A few changes:

switches to multiprocess and scrapy for fetching URLs and text
switches to higher-level mcproviders library so the chunking of domains is dome for us, and it has an easier API
fixes some paging and date picking issues that had been in the code for a while
removes some sources to try and get as much content as we can despite bugs in WM server

Still testing so I haven't merged this yet.

rahulbot commented 1 year ago

Worked on prod - merging to release