Google Cloud Storage (GCS) ingestion source

[Samrose-Ahmed]

Overview

Currently we support ingesting from S3, with an SQS ingestion integration in progress.

We should also support ingesting directly from GCS buckets, will be useful for e.g. GCP audit and other logs.

Goal

Be able to ingest data for a log source from a GCS bucket via Pub/Sub notifications (similar to S3 ingestion via SQS notifications).

Will allow easily ingesting data from GCP.

Notes

Note this issue is not about deploying the entirety of Matano to GCP (which is a separate issue further on our roadmap) but rather supporting ingesting GCS data into a Matano deployment in AWS

• Set up a mechanism for deploying infra into a GCP account
• Set up GCS Pub/Sub notifications for buckets
• Forward Pub/sub notifications to SQS
• Modify transformer to handle both SQS and GCP Pub/Sub notifications
• Modify transformer to be able to read objects from GCS (S3 compatible)
• Add a new config: ingest.gcs_source similar to ingest.s3_source

[timoguin]

We should definitely add GCP managed log sources prior to this. This one is pretty big. Prior to this being finished, I'd like to wire things up on the GCP side to replicate GCS to S3, but that can be done outside the scope of Matano for now.