search

gfilicetti / gke-github-deployment

This repository is a full example of a full CI/CD pipeline using Github actions, terraform and other tech.
Apache License 2.0
3 stars 2 forks source link

Move Log Sink -> BigQuery exports to streaming pipeline #85

Closed alanpoole closed 1 month ago

alanpoole commented 2 months ago

The configured log sinks are batch exports to BigQuery. The table isn't there until that log event happens. And the log sink export table should be configured with a data retention policy.

To solve this, move to a streaming pipeline. Log Sink -> Pub/Sub -> (streaming pipeline) -> log-events-gke.

Then deprecate the scheduled queries that run every 60 minutes.

alanpoole commented 1 month ago

Closing this issue, as it is not important in the short-term.