Prototyping: Create a test environment for a minimal data collector with queues, logging and divided work phases

noi-techpark / bdp-core

Open Data Hub / Timeseries Core

https://opendatahub.com

Other

9 stars 4 forks source link

Open Piiit opened 4 years ago

Piiit commented 4 years ago

[ ] Search for a real-world data collector (or migrate an existing one, should be simple) @rcavaliere @ohnewein @bertolla @RudiThoeni
[ ] message queue
[ ] divide work into downloading --> queue#1 --> enrichment --> queue#2 --> one target (Postgres)
[ ] logging
[ ] scheduling
[ ] grafana

rcavaliere commented 4 years ago

@Piiit I would suggest to start with an easy and stable Data Collector, e.g. Parking Merano

alexlanz commented 4 years ago

Setting up one managed ActiveMQ cluster on AWS with two queues - 2-3d
Programming: 2-4d
- Write a specific data collector (downloading)
- Write a specific data enricher (enchriment)
- Write a general data writer (save into PostgreSQL)
Logging: out of the box with the new Docker setup - 0d
Scheduling: out of the box with using the Spring Boot scheduler - 0d
Grafana: dashboard for the queues - 2-3d