search

spotify / async-google-pubsub-client

[SUNSET] Async Google Pubsub Client
Apache License 2.0
157 stars 40 forks source link

Future plans for async-google-pubsub-client? #13

Closed mattnworb closed 4 years ago

mattnworb commented 8 years ago

Hey @danielnorberg or anyone else maintaining this library, do you have plans to continue to maintain it in the future?

Recently Google has made a pubsub client with async operations available in com.google.cloud:gcloud-java-pubsub.

This client has a Puller-like class in it's MessageConsumer concept, which will continuously pull messages from a subscription and update their ack timeouts until your code is finished processing the message, with some options to tune how many messages to queue up and how many threads to use in processing messages concurrently.

gcloud-java-pubsub uses the gRPC API for Cloud PubSub and also offers some async methods for each operation, for instance Future<List<String>> publishAsync(String topic, Iterable<Message> messages) or Future<Iterator<ReceivedMessage>> pullAsync(String subscription, int maxMessages).

(One annoyance is that the library returns naked Futures rather than ListenableFutures, so they have to be adapted if you want to use ListenableFuture or CompletableFuture)

One thing that seems to be lacking from gcloud-java-pubsub so far is an analogue to the Publisher class here that offers batching and internal queuing for publishing messages.

Given that Google has been advancing their client for pubsub, I am curious to see how much of a use case there is still for this custom client.

danielnorberg commented 8 years ago

Cool, they finally released a library using grpc. Unfortunate that it uses old-style blocking futures.

I remember claims that grpc would eliminate any need for batching, which would seem to obsolete the Publisher class in this client but I guess that should be verified for the specific use-case.

If the new official pubsub client performs well, then I guess there shouldn't be any need to still use and maintain this custom client.

mattnworb commented 8 years ago

I made an issue in gcloud-java requesting that they could at least return ListenableFuture instances in their client (in the implementation, if not in the interface signature) to help improve that part of it.

I know it's been a little while but do you remember if there were any other use cases that spawned this library not solved in the official client, besides blocking futures and the batching publisher?

Hope all is well over there in the far east.

danielnorberg commented 7 years ago

@mattnworb https://github.com/spotify/async-google-pubsub-client#why =)

danielnorberg commented 7 years ago

I think there was also other inefficiencies in the official client like gzipping payloads twice etc, causing it to bottleneck on CPU. But that's a long time ago now. I'm sure the grpc client is much better, although I wonder how/if they use all cpu cores for encryption or only one due to the single tcp connection nature of grpc. But that's only an issue if you're pushing a lot of messages.

danielnorberg commented 7 years ago

I think @i-maravic told me something about the grpc client just not working very well and that they are using this lib instead, but ¯\(ツ)/¯

mattnworb commented 7 years ago

I think there have been a lot of changes in google-cloud-pubsub library recently so it might be worth reexamining some of these things for anyone using this library.

btw the naked Future thing was addressed a while back in https://github.com/GoogleCloudPlatform/google-cloud-java/issues/1148

danielnorberg commented 6 years ago

Latest status I believe is that we're now moving to using the official client in gabo. This client is on life support.

renannprado commented 5 years ago

I'm planning to start exploring a little bit with google's pubsub and vertx. Is there any example where these two are used together and well integrated? I saw an example where a vertx application exposes an endpoint to be called by google's pubsub, but I would like to see an example using google's pubsub official java API to pull messages instead (gRPC) as you are no longer recommending to use this library.