search

neuland-ingolstadt / neuland.app

A free & open source, web-based replacement for the official app of the Technische Hochschule Ingolstadt built with React and Next.js.
https://neuland.app
GNU Affero General Public License v3.0
36 stars 33 forks source link

[#243] Rework scrape architecture #244

Closed shei99 closed 1 year ago

shei99 commented 1 year ago

This will implement the proposal of the Issue #243. To store the data, a Redis instance will be used.

shei99 commented 1 year ago

@alexhorn @M4GNV5 How often should the neuland.app be able to receive the latest data from the scrape jobs?

Context: If it is enough for all future scrape jobs, that the data gets baked into the docker image of each deployment of the neuland.app, then would be no need for an update strategy in the neuland.app. Otherwise, I would implement an API endpoint to update the necessary JSON documents.

alexhorn commented 1 year ago

I think about once a week should be appropriate. Maybe making it configurable via a cron string passed via an environment variable is an option?