search

robertsahlin / comments

comments for robertsahlin.com
0 stars 0 forks source link

Schema evolution in streaming Dataflow jobs and BigQuery tables, part 3 · robertsahlin.com #7

Open robertsahlin opened 3 years ago

robertsahlin commented 3 years ago

https://robertsahlin.com/schema-evolution-in-streaming-dataflow-jobs-and-bigquery-tables-part-3/

In the previous post, I covered how we create or patch BigQuery tables without interrupting the real-time ingestion. This post will focus on how we update the d

xmlking commented 2 years ago

This is an enlightening article for me. I was debating myself, whether to use avro or protobuf for my dataflow pipeline.

I decided to follow your option of protobuf. I am also planing to implement generic field-level encryption/description ParDo using proto annotations.

Did you explore newly added GCP schema registry? https://cloud.google.com/pubsub/docs/schemas

xmlking commented 2 years ago

Since protobuf release 3.15, proto3 supports using the optional keyword (just as in proto2) for presence information