choose ingest solution - Githubissues

subugoe / leine

Data Pipelines for @subugoe/wag

https://subugoe.github.io/leine

MIT License

1 stars 0 forks source link

choose ingest solution #8

Open maxheld83 opened 3 years ago

maxheld83 commented 3 years ago

alternatives:

GWDG HPC (batch with a full VM) -- bad for reproducibility and maybe slow (not important).
GCP Cloud Run (would just replace above)
~~BigQuery AWS S3 ingest~~: snapshots are on S3, but this isn't for the public http interface of S3.
GCP Dataflow (?!)
GCP BigQuery Data Transfer Service
GCP Cloud Storage Transfer Service

There's gotta be something serverless-ish. Too much hassle to maintain a vm/container just for this.