Feature request: For GCSOnly = true, remove BQ dependency for tracking bundle progress

fullstorydev / hauser

Service for moving your Fullstory export files to a data warehouse

MIT License

49 stars 23 forks source link

Feature request: For GCSOnly = true, remove BQ dependency for tracking bundle progress #74

Closed glarchev closed 3 years ago

glarchev commented 4 years ago

Currently when GCSOnly flag is set to true, hauser still uses a SyncTable in BQ to track bundle load progress. It would be nice to eliminate any dependency on BQ for GCSOnly uploads.

jameremo commented 4 years ago

We should also consider this for the S3 only case, which I think also tracks bundle progress in Redshift, similar to GCS + BigQuery on GCP.