permutive-engineering / fs2-pubsub

Google Cloud Pub/Sub stream-based client built on top of cats-effect, fs2 and http4s.
Apache License 2.0
48 stars 16 forks source link

Emulator support for Google based client #348

Closed mmatloka closed 5 months ago

mmatloka commented 2 years ago

Hello! Readme says :

Doesn't work with the official PubSub emulator (is in feature backlog)

Link to that backlog is no longer working. I've looked though google cloud docs and they show here usage of java client with the emulator.

String hostport = System.getenv("PUBSUB_EMULATOR_HOST");
ManagedChannel channel = ManagedChannelBuilder.forTarget(hostport).usePlaintext().build();

Does it mean that this library could be adjusted for emulator support?

bastewart commented 2 years ago

Ah, thanks! Yes this almost certainly does mean we could update this to support the emulator. (FWIW the HTTP variant does support it already.)

Daenyth commented 5 months ago

https://cloud.google.com/pubsub/docs/emulator#accessing_environment_variables

It's a little more complicated; one issue is that ManagedChannel is a lifecycle'd resource with a shutdown method - you ideally wouldn't create one inside of the customizePublisher etc method, since it wants to return Publisher and not a Resource

I think the cleanest resolution would be to expose an additional emulatorHost: Option[String] field on the PubsubProducerConfig which gets used in GooglePubsubProducer.of (and similarly for subscriber). It might also be nice to read the PUBSUB_EMULATOR_HOST variable automatically, since the non-java gcp sdks do that for you

alejandrohdezma commented 5 months ago

This should no longer be an issue on the new revamped library (fs2-pubsub). Feel free to re-open it in case it is though.

Daenyth commented 5 months ago

@alejandrohdezma How do you do it in the new library? The readme doesn't mention "emulator" at all

alejandrohdezma commented 5 months ago

Hey @Daenyth, the tests are using an emulator and as you can see there, you just need to point the URI to use to one of the emulator and use a non-authenticated client.