search

googleapis / nodejs-pubsub

Node.js client for Google Cloud Pub/Sub: Ingest event streams from anywhere, at any scale, for simple, reliable, real-time stream analytics.
https://cloud.google.com/pubsub/
Apache License 2.0
516 stars 227 forks source link

What's the optimal way to deploy and run an efficient pull subscriber? #1939

Closed theabhinavdas closed 2 weeks ago

theabhinavdas commented 3 weeks ago

What I'm trying to do:

I am using pub/sub to move processing of some tasks to the background. The use-case has known transient errors that may be retried. I believe this calls for the use of a pull subscription vs. a push subscription as it gives me more control on flow. Looking for some architecture that ensures max throughput while being resilient to failure.

Questions:

  1. What is the recommended way to create a pull subscriber that has some "fault tolerance" where transient errors are retried and new messages are pulled only when the previous message is "processed" and ack'd?
  2. What is the purpose of timeout in this example? I expect the subscriber to run always and continuously process incoming messages.
  3. What is the recommended way to deploy such a pull subscriber? Assuming it would make sense to deploy it to cloud run or kubernetes as some service that runs continuously?
feywind commented 3 weeks ago

@theabhinavdas You might get some better answers for conceptual questions by posting on Stack Overflow or requesting a design consult. (I can get the link if you want.)

Edit: re your second question about timeout, as you've said, it's meant to run indefinitely and listening to messages, but since these are just samples, they need a terminating condition. (We run them as part of CI.) So your normal client code wouldn't do that.

theabhinavdas commented 2 weeks ago

@feywind Understood, thank you. I'll cross-post to Stack Overflow. Happy to get a design consult so please do share the link! Marking as closed.