Make threadpool size configurable in publisher/consumer builder

[pax95]

Currently it is very low level to override the default behaviour on how many threads are assigned to the publisher/consumer.

We are seeing too many GAX threads created for our usecase causing an overhead in memory consumption on k8s for low traffic situations. There is some talk about it her

The documentation for override the pool size might be a bit to low leven for many situations. It would be good if the pool would be configurable from outside especially since many users of pubsub are using a framework like spring-boot pubsub, or in our case apache camel google pubsub.

[lqiu96]

Thanks for raising this issue! I have a few questions to get a bit more clarity about the issue you're seeing.

Currently it is very low level to override the default behaviour on how many threads are assigned to the publisher/consumer.

Can you explain this part a bit more? I'm a bit confused what you mean by very low level. I took a look at the pubsub docs and I think there is a setting inside pubsub to configure the executor provider:

     ExecutorProvider executorProvider =
          InstantiatingExecutorProvider.newBuilder().setExecutorThreadCount(4).build();

      // `setExecutorProvider` configures an executor for the publisher.
      publisher = Publisher.newBuilder(topicName).setExecutorProvider(executorProvider).build();

It would be good if the pool would be configurable from outside

I'm also a bit confused by this part. What do you mean by configurable from outside?

[pax95]

I think the internal threading model in pubsub might benefit from a better abstraction as eg. a configuration class, where users can override the default behaviour of pubsub (number of threads in pool asf.) , and then inject an instance of that when instantiating pubsub, or a method on the Publisher/Consumer to set no of threads. In this way users do not have to deal with executors if they want to override the defaults.

[lqiu96]

For an abstraction in pubsub, I think this would probably better suited as a feature request for the pubsub folks. I'll transfer the issue there and CC the some of the devs for their thoughts

[lqiu96]

CC @michaelpri10 @hannahrogers-google if you two have any thoughts on this