search

rakutentech / kafka-firehose-nozzle

Forward logs from the Cloud Foundry Firehose to Apache Kafka
MIT License
13 stars 8 forks source link

Deploying as a cloud foundry app: out of available brokers #24

Open mrodriguezio opened 5 years ago

mrodriguezio commented 5 years ago

The README file mentions this project can be installed as a cloud foundry app. So, I wanted to do exactly that, I already have the kafka-boshrelease and kafka-service-broker-boshrelease deployed, and the error I'm getting is:

[ERROR] Failed to construct kafka producer: kafka: client has run out of available brokers to talk to (Is your cluster reachable?)

Can anyone route me to a working example :)?

webbbret commented 5 years ago

Kafka Version: 2.1.0 CF Version: 6.41.0+dd4c76cdd.2018-11-28 Go Version: go1.10.3 linux/amd64 OS: RHEL 7

I too am seeing this happen after what appears to be a successful app startup. The app runs for about 5 minutes and then the following:

2019-01-02T14:42:29.046-08:00 [APP/PROC/WEB/2] [OUT] 2019/01/02 22:42:29 [ERROR] Failed to construct kafka producer: kafka: client has run out of available brokers to talk to (Is your cluster reachable?) 2019-01-02T14:42:29.056-08:00 [APP/PROC/WEB/2] [OUT] Exit status 2 2019-01-02T14:42:29.058-08:00 [CELL/SSHD/2] [OUT] Exit status 0 I have confirmed that the cluster is available by using the kafka command bin/kafka-console-producer.sh --broker-list myservername:9092 --topic test to add messages to the topic. The messages show up using Kafdrop to view the topic test.

There are no firewalls in front of the kafka instance.

In addition, none of the topics in the config.toml get created. I am assuming this is do to the broker never getting reached. Any help here would be greatly appreciated.

webbbret commented 5 years ago

Maybe found the answer here. I modified the cli.go script to log the username and subscription id and got the following:

2019-01-02T15:19:46.269-08:00 [APP/PROC/WEB/2] [OUT] 2019/01/02 23:19:46 [INFO] Subscription ID: example-kafka-firehose-nozzle 2019-01-02T15:19:46.269-08:00 [APP/PROC/WEB/2] [OUT] 2019/01/02 23:19:46 [INFO] Username: tcnksm

Which tells me that the app is not recognizing my config.toml file, or is not using it.

webbbret commented 5 years ago

DefaultCfgPath, in the cli.go script, points at the .toml file used by the app. Turns out that config.toml is not the default, but the example toml file is. I changed this to the location of my config.toml file and now get the correct settings in the log. Hope this helps.