search

logstash-plugins / logstash-input-google_pubsub

Logstash input for pulling events from Google Pub/Sub service
Apache License 2.0
19 stars 33 forks source link

Feature request to pull host from PUBSUB_EMULATOR_HOST #30

Open SebScoFr opened 6 years ago

SebScoFr commented 6 years ago

This plugin should have the ability to pull the PubSub host from the PUBSUB_EMULATOR_HOST environment variable, so that it can be used in dev mode with the PubSub emulator.

josephlewis42 commented 6 years ago

@SebScoFr update on this, I have the fix in this branch. I pulled the code that was doing the connections out of the Java lib and into the project so we can modify the host. The plugin seems to be working fine, but it looks like it's trying to send the emulator commands that it doesn't support.

Tagging @rosbo to see if he's seen something like this before because I think he's used the emulator.

SebScoFr commented 6 years ago

Thanks @josephlewis42

rosbo commented 6 years ago

@josephlewis42 I have used it a little while ago and didn't see this issue, do you mind sharing the logs of the messages that are failing?

josephlewis42 commented 6 years ago

@rosbo after a little more digging it looks like it might be an encryption issue. The local Pub/Sub emulator does not use encryption and the library seemingly removed support for setting the transports to using plaintext with the gRPC client. It's a known issue https://github.com/GoogleCloudPlatform/google-cloud-java/issues/2248 and is on their work backlog.

Here are my logs from the Pub/Sub emulator with LogStash running:

[pubsub] May 09, 2018 4:56:23 PM io.gapi.emulators.grpc.GrpcServer$3 operationComplete
[pubsub] INFO: Adding handler(s) to newly registered Channel.
[pubsub] May 09, 2018 4:56:23 PM io.gapi.emulators.netty.HttpVersionRoutingHandler channelRead
[pubsub] INFO: Detected non-HTTP/2 connection.
[pubsub] May 09, 2018 4:56:23 PM io.gapi.emulators.netty.NotFoundHandler handleRequest
[pubsub] INFO: Unknown request URI: /bad-request

Here they are with the Python client:

[pubsub] May 09, 2018 4:57:58 PM io.gapi.emulators.grpc.GrpcServer$3 operationComplete
[pubsub] INFO: Adding handler(s) to newly registered Channel.
[pubsub] May 09, 2018 4:57:58 PM io.gapi.emulators.netty.HttpVersionRoutingHandler channelRead
[pubsub] INFO: Detected HTTP/2 connection.

I could switch us over to using the HTTP2 client, but it's going to be a fair bit of work on our end and will probably decrease efficiency.