The pubsub Source from common-streams is a wrapper around Subscriber provided by the 3rd-party pubsub sdk. That Subscriber is a wrapper around a lower-level GRPC stub.
This commit adds an alternative Source which directly wraps the GRPC stub, not the higher-level Subscriber.
Compared with the previous Source implementation it has these differences in behaviour:
In the V1 source, ack extension periods were adjusted dynamically according to runtime heuristics of message processing times. In the V2 source, the ack extension period is a fixed configurable period.
The V1 source made a modack request (extending ack deadline) immediately after receiving any message. Whereas the V2 source does not modack a message unless its deadline is about the expire.
The V1 source periodically modacks all unacked messages currently held in memory. This is a problem for e.g. the Lake Loader which can have a very large number of unacked messages at any one time. The V2 source only modacks messages when they are approaching their ack deadline.
The V2 source uses a smaller thread pool for GRPC callbacks. The V1 source needed a very large thread pool to avoid deadlocks in setups that opened a large number of streaming pulls.
If this experimental V2 Source is successful, it is likely to be the replacement of the V1 Source in a future release of common-streams.
The pubsub Source from common-streams is a wrapper around
Subscriberprovided by the 3rd-party pubsub sdk. ThatSubscriberis a wrapper around a lower-level GRPC stub.This commit adds an alternative Source which directly wraps the GRPC stub, not the higher-level Subscriber.
Compared with the previous Source implementation it has these differences in behaviour:
If this experimental V2 Source is successful, it is likely to be the replacement of the V1 Source in a future release of common-streams.