The pubsub Source from common-streams is a wrapper around Subscriber provided by the 3rd-party pubsub sdk. That Subscriber is a wrapper around a lower-level GRPC stub.
This commit adds an alternative Source which directly wraps the GRPC stub, not the higher-level Subscriber.
Compared with the previous Source implementation it has these differences in behaviour:
In the V1 source, ack extension periods were adjusted dynamically according to runtime heuristics of message processing times. In the V2 source, the ack extension period is a fixed configurable period.
The V1 source made a modack request (extending ack deadline) immediately after receiving any message. Whereas the V2 source does not modack a message unless its deadline is about the expire.
The V1 source periodically modacks all unacked messages currently held in memory. This is a problem for e.g. the Lake Loader which can have a very large number of unacked messages at any one time. The V2 source only modacks messages when they are approaching their ack deadline.
The V2 source uses a smaller thread pool for GRPC callbacks. The V1 source needed a very large thread pool to avoid deadlocks in setups that opened a large number of streaming pulls.
V2 opens PullRequests with pubsub, whereas V1 opened StreamingPullRequests
If this experimental V2 Source is successful, it is likely to be the replacement of the V1 Source in a future release of common-streams.
The pubsub Source from common-streams is a wrapper around
Subscriberprovided by the 3rd-party pubsub sdk. ThatSubscriberis a wrapper around a lower-level GRPC stub.This commit adds an alternative Source which directly wraps the GRPC stub, not the higher-level Subscriber.
Compared with the previous Source implementation it has these differences in behaviour:
If this experimental V2 Source is successful, it is likely to be the replacement of the V1 Source in a future release of common-streams.