search

freifunk-gluon / census-exporter

Scraping communities for census data
https://stats.darmstadt.freifunk.net/d/CA5PRFmMz/gluon-census?orgId=1
MIT License
10 stars 11 forks source link

drop nodes when json file's timestamp is too old #46

Open Djfe opened 3 weeks ago

Djfe commented 3 weeks ago

example: KBU has json files that haven't been updated in a year. (timestamp from may 2023) We should not track/crawl those i.e. ignore them.

AiyionPrime commented 3 weeks ago

Maybe start with a github action, which warns about too old files?

Djfe commented 3 weeks ago

probably. same goes for files that are not available anymore (404) or where the server doesn't respond, or where SSL errors. I sorted out quite a few of those with #43 and #44