storage: new source: GCP PubSub

[hden]

I saw you already have AWS Kinesis streams and Azure Event Hub in mind. It might be an interesting strategy to be cloud and stream agnostic.

FYI: Currently we are extracting CDC logs by Debezium, but instead of maintaining a Kafka cluster, we decided to stream the data through a PubSub topic, which ended up in a BigQuery table as an append-only data store.

[hden]

In case you you need strict messaging ordering, I found an interesting option in the Official Java client. Maybe it's available in the rust client as well?

https://javadoc.io/doc/com.google.api.grpc/proto-google-cloud-pubsub-v1/1.73.0/com/google/pubsub/v1/Subscription.Builder.html#setEnableMessageOrdering-boolean-

[rjnn]

Hi @hden! Thanks for the suggestion. PubSub is on our radar - to give you context, Kinesis was prioritized due to a surprisingly large number of user asks. Azure Event Hub has Kafka client bindings which makes it fairly easy to support (although I'm sure things won't be 100% perfect). You are the first asker for PubSub, so we'll definitely watch this issue to see what the user appetite is!

Part of the challenge is that right now, adding additional connectors is difficult (see #2059), and we need to refactor the input side to make that easier. So this should become a lot easier in about the 2 months timeframe.

[ajroetker]

:+1: it'd be cool to see pubsub connectors

[hden]

FYI message ordering just landed in beta.

https://cloud.google.com/pubsub/docs/ordering

[jameshartig]

Would love to see Pub/Sub support!

[bodschut]

On my side, the lack of pubsub support is also a showstopper to test out materialize at veepee :-)

[hui-zheng]

I think that two connectors GCS (Google Cloud Storage) and Google PubSub would open up a world of opportunities for Materialized that allows practitioners to bring Materialized into the systems build around GCP.

[jakthom]

+1

[cheparukhin]

Any updates on this?

[jineshshah36]

We would also love to see support for pubsub