search

lblod / app-lblod-harvester

Harvesting Self Service
MIT License
1 stars 4 forks source link

use lblod-scraper instead of download-url and collect service #42

Closed nvdk closed 1 year ago

nvdk commented 1 year ago

simplifies the flow by leveraging the scrapy framework.

Seems to be working well, mainly tested with Gent and Laarne. Automated runs (not via frontend) of other cities also worked.

Note: not 100% equal to previous services, lblod-scraper currently doesn't have a cron and doesn't create downloadEvent resources.

MPParsley commented 1 year ago

@nvdk, when running I get this error:

app-lblod-harvester-deltanotifier-1             | (node:33) UnhandledPromiseRejectionWarning: Error: getaddrinfo ENOTFOUND delta-producer-dump-file-publisher