[Backend] Migrate Scraper code to Geoharvester. - Githubissues

FHNW-IVGI / Geoharvester

NDGI Project Geoharvester

10 stars 1 forks source link

[Backend] Migrate Scraper code to Geoharvester. #16

Closed FStriewski closed 1 year ago

FStriewski commented 1 year ago

User story:

As developer / maintainer of the project, I would like to have all of the code running from the same repository

Acceptance criteria

[x] Scraper code is moved to this repository and integrated well in the file/project structure
[x] Scraper can run independently from the backend to perform its (daily) duties.
[x] Github actions are migrated so that scraper outputs data to a designated output folder
[x] Backend is tweaked so that in ingests from the new location or the file directly

Considerations

Keep csv output, so that it can be used by (potential) other projects as stand-alone data
Maybe tweak the scraper so that it outputs json in addition to csv
Maybe implement updating of redis based on the daily update routine (this is probably a ticket of its own and out of scope)
Evaluate if we can streamline ingestion - atm the data is retrieved via url (call), while the data will be in the same repo after migration.

FStriewski commented 1 year ago

Code copied to our repo: https://github.com/FHNW-IVGI/Geoharvester/pull/69

FStriewski commented 1 year ago

See https://github.com/FHNW-IVGI/Geoharvester/pull/74