googleapis / google-cloud-go

Google Cloud Client Libraries for Go.
https://cloud.google.com/go/docs/reference
Apache License 2.0
3.72k stars 1.28k forks source link

pubsub: batch processing instead of 1 goroutine per message #10912

Closed kokizzu closed 5 days ago

kokizzu commented 5 days ago

Is your feature request related to a problem? Please describe.

currently I'm trying to do batch fetch from pubsub, transform, then insert to bigquery. but the pubsub library spawns a goroutine for each message, this causes bunch of memory consumption and inefficient processing

func (s *Subscription) Receive(ctx context.Context, f func(context.Context, *Message)) error {

Describe the solution you'd like

I need batch processing something like this

func (s *Subscription) ReceiveN(ctx context.Context, n items, f func(context.Context, []*Message)) error {

Describe alternatives you've considered

taking out the pubsub library and write own proper batch processing

Additional context

by doing this, it would be super efficient to do batch processing

hongalex commented 5 days ago

Thanks for filing an issue.

This is a duplicate of https://issuetracker.google.com/288595422, please +1 that issue if this is affecting you.