2023: Lisum changes & RSS Crawlers

openeduhub / oeh-search-etl

The Backend includes all data for the ETL process (Scrapy, Postgres, Elasticsearch)

7 stars 9 forks source link

2023: Lisum changes & RSS Crawlers #71

Closed Criamos closed 1 year ago

Criamos commented 1 year ago

This PR includes all "Lisum"-specific changes made to the crawlers

oeh_spider
sodix_spider
lehreronline_spider (small bugfix, v0.0.6)
as well as RSS-based crawlers (zdf_rss_spider & br_rss_spider) to make checking for duplicates easier in the future

Furthermore:

added long-overdue docs to items.py
updated requirements.txt according to Dependabot's recommendations from 2023-02-14

Criamos commented 1 year ago

Merging these changes into develop since oeh_spider v0.1.9 already crawled successfully against the Lisum Repo. (I also briefly tested the updated requirements.txt with 2 (unrelated to Lisum) crawlers).