change: use playwright for ZUM (MediaWikiBase) crawlers - Githubissues

openeduhub / oeh-search-etl

The Backend includes all data for the ETL process (Scrapy, Postgres, Elasticsearch)

7 stars 9 forks source link

change: use playwright for ZUM (MediaWikiBase) crawlers #69

Closed Criamos closed 1 year ago

Criamos commented 1 year ago

(see: SD_WLO-419)

during testing of the recent ZUM URL-encoding fixes (SD_WLO-419) it was noticed that crawlers waited for Splash containers (which never recovered from hanging) indefinitely in the ranger environment
- therefore added custom_settings to use Playwright instead
version bump zum_deutschlernen.py, zum_spider.py and zum_klexikon.py to refresh their metadata during the next crawl

zum_klexikon:

fix: changed license from CC-BY-SA 3.0 to 4.0
- Klexikon seems to have switched its license version sometime in the past, according to its DOM (footer)