infra v2: migrate spreadsheets-google - Githubissues

noi-techpark / bdp-commons

GNU Affero General Public License v3.0

2 stars 12 forks source link

infra v2: migrate spreadsheets-google #648

Open clezag opened 10 months ago

clezag commented 10 months ago

Overview

Google spreadsheet integration has 4 components:

Register a notification URL using the google drive API
Listen for update triggers
Read spreadsheet and push content to ODH
Validate spreadsheet and mark invalid cells on the online sheet

Providers

This data collector handles 4 providers:

centro trevi
creative industries
sta-echarging
umadumm

but other collectors have spreadsheet functionality as well, which should be unified with this

Migration strategy

Single shared data collector that handles notification setup, listener and reading spreadsheet content
One instance per provider, so there will be separate ingress per collector (subject to change)
Transformers are split up per provider
Possibly drop the validation / marking feature

clezag commented 8 months ago

the spreadsheet data collector is working in principle (no transformers yet), but we're currently stuck due to mongodb only supporting max 16MB file size, and some spreadsheets are already bigger