Cache locations data with a scraper inside github actions

lsg551 / matricula-online-scraper

Scraper for Matricula Online

https://pypi.org/project/matricula-online-scraper/

MIT License

0 stars 0 forks source link

Cache locations data with a scraper inside github actions #30

Closed lsg551 closed 6 months ago

lsg551 commented 6 months ago

Description

Fetching all parishes (> 8000) usually takes scrapy a few minutes. Also, if many people do this regularly, it unnecessarily stresses Matricula's server. Especially since this data mostly remains untouched because it will only be regularly updated with new entries.

[x] A simple cron job inside a GitHub action could easily do this once a week, push the scraped data (as csv) to a dedicated branch. (#30)
[x] Compress the csv. (#30)
[x] Place a badge in the main's README that automatically downloads this file for the user. (#30)

lsg551 commented 6 months ago

Article on how to do this: https://github.com/swyxio/gh-action-data-scraping?tab=readme-ov-file

lsg551 commented 6 months ago

Maybe, add

[ ] a new option called --emit-tracking-file that creates or appends to a file for tracking changes in automated scraping processes

This adds another feature: tracking the number of newly added parishes. The file could look like

date       ,version ,files scraped
1714910198 ,v0.4.2  ,         8442

with

date being a unix timestamp
version being the used version of matricula-online-scraper
files scraped being the number of rows in the scraped data