document: google pubsub source

[neverchanje]

Related code PR

No response

Which part(s) of the docs might be affected or should be updated? And how?

Although barely requested, we still maintain the code of pubsub source. Thus, we need documentation as well.

Reference

No response

[hengm3467]

https://github.com/risingwavelabs/risingwave/blob/main/e2e_test/source/basic/pubsub.slt

I think we can use the examples as reference about the syntax.

[dermasmid]

the test there does not show an example with creds, as its using an emulator

[dermasmid]

here's where the structure is https://github.com/risingwavelabs/risingwave/blob/7b6bbf3831aba05d8ce7f68080534a13b032317a/src/connector/src/source/google_pubsub/mod.rs#L33

[hengm3467]

@dermasmid Thanks for pointing that out! I wasn't taking a close look at the parameters when I pointed people to the e2e test case. @WanYixian Please use the above mod.rs file as reference for the parameters, and confirm with engineers about the list of parameters and their descriptions.

[neverchanje]

@dermasmid A more doc-team-facing reference is maintained in the with_options_source.yaml:

https://github.com/risingwavelabs/risingwave/blob/028f0aea915b966e0bbdd555f7670331fa76aa18/src/connector/with_options_source.yaml#L400-L424

PubsubProperties:
  fields:
  - name: pubsub.split_count
    field_type: u32
    required: true
  - name: pubsub.subscription
    field_type: String
    comments: pubsub subscription to consume messages from  The subscription should be configured with the `retain-on-ack` property to enable  message recovery within risingwave.
    required: true
  - name: pubsub.emulator_host
    field_type: String
    comments: use the connector with a pubsub emulator  <https://cloud.google.com/pubsub/docs/emulator>
    required: false
  - name: pubsub.credentials
    field_type: String
    comments: credentials JSON object encoded with base64  See the [service-account credentials guide](https://developers.google.com/workspace/guides/create-credentials#create_credentials_for_a_service_account).  The service account must have the `pubsub.subscriber` [role](https://cloud.google.com/pubsub/docs/access-control#roles).
    required: false
  - name: pubsub.start_offset
    field_type: String
    comments: '`start_offset` is a numeric timestamp, ideallly the publish timestamp of a message  in the subscription. If present, the connector will attempt to seek the subscription  to the timestamp and start consuming from there. Note that the seek operation is  subject to limitations around the message retention policy of the subscription. See  [Seeking to a timestamp](https://cloud.google.com/pubsub/docs/replay-overview#seeking_to_a_timestamp) for  more details.'
    required: false
  - name: pubsub.start_snapshot
    field_type: String
    comments: '`start_snapshot` is a named pub/sub snapshot. If present, the connector will first seek  to the snapshot before starting consumption. Snapshots are the preferred seeking mechanism  in pub/sub because they guarantee retention of:  - All unacknowledged messages at the time of their creation.  - All messages created after their creation.  Besides retention guarantees, timestamps are also more precise than timestamp-based seeks.  See [Seeking to a snapshot](https://cloud.google.com/pubsub/docs/replay-overview#seeking_to_a_timestamp) for  more details.'
    required: false

[tabVersion]

Hi @neverchanje, I'd propose a refactor about functional validation and arg re-naming before releasing because it has been poorly maintained since v0.15.

[neverchanje]

Thanks for clarification. Let's put this issue on hold until the refactoring merged. However, it should be noted that GCP pubsub is currently in the low priority.

[WanYixian]

This issue tracks the doc change for https://github.com/risingwavelabs/risingwave/pull/14531