Unable to build due to missing dependencies

[webbbret]

When I run "godep go test -bench" I get a series of errors complaining about not finding packages. For example: cannot find package "github.com/Shopify/sarama" in any of: /usr/local/go/src/github.com/Shopify/sarama (from $GOROOT) This is after I have run "make updatedeps". There are not many, but enough that building fails. In addition, running "go get github.com/rakutentech/kafka-firehose-nozzle" works, but when I try and push to Cloud Foundry, I get the following error:

**ERROR** To use go native vendoring set the $GOPACKAGENAME environment variable to your app's package name

The go buildpack version we are using is 1.8.28

I also want to use a custom .toml file but do not know where it goes. Does it need to be in the cloned source before building, or is there a way to pass the file location to the go get command?

Thanks in advance for your help.

[giner]

Hi @webbbret, are you trying to use native vendoring? The following should be enough to push:

applications:
- name: kafka-firehose-nozzle
  memory: 1G
  instances: 3
  buildpack: https://github.com/cloudfoundry/go-buildpack
  env:
    UAA_PASSWORD: ...

[webbbret]

Thanks for the response @giner.. I am assuming that these are entries for a manifest file? Thought the UAA stuff was in the .toml file... Is there a way in include the .toml file in the snippet above? Can you please provide a sample cf push command to get the snippet and .toml to deploy? Many thanks in advance.

[webbbret]

CF Version: 6.41.0+dd4c76cdd.2018-11-28 Go Version: go1.10.3 linux/amd64 RHEL 7

I think I have progressed further after reinstalling go and setting the env variables correctly. However, now when I try and do a push from the root of the project (bin and src visible) I get other errors. I have posted a ticket with go-buildpack here: https://github.com/cloudfoundry/go-buildpack/issues/57

For your information, when I try and do a cf push I get the following now: -----> Go Buildpack version 1.8.30 **ERROR** Cloud Foundry does not support the GB package manager. We currently only support the Godep, Glide and dep package managers for go apps. For support please file an issue: https://github.com/cloudfoundry/go-buildpack/issues **ERROR** Unable to select Go vendor tool: gb unsupported

I am thinking this is not your guys issue. If I try and rename the src folder, I get a different error similar to the one I got before:

**WARNING** Installing package '.' (default) -----> Running: go install -tags cloudfoundry -buildmode pie . can't load package: package kafka-firehose-nozzle: no Go files in /tmp/gobuildpack.gopath121497756/.go/src/kafka-firehose-nozzle **ERROR** Unable to compile application: exit status 1 Failed to compile droplet: Failed to run finalize script: exit status 12 Please confirm if anyone is able to push this nozzle to cloud foundry, or has a way of getting around this. Your help would be greatly appreciated.

[giner]

@webbbret, I have just tried to push with the latest buildpack and it worked for me. There is no need in building it before pushing.

git clone https://github.com/rakutentech/kafka-firehose-nozzle
cd kafka-firehose-nozzle
# Add config.toml and manifest.yml
cf push

[webbbret]

@giner Perfect! The app deploys now. However, I think something may be wrong with my config.toml file, as I cannot seem to get the app started. The error I am getting is:

2018-12-21T12:53:33.690-08:00 [CELL/2] [ERR] Timed out after 1m0s: health check never passed. 2018-12-21T12:53:33.690-08:00 [HEALTH/2] [ERR] Failed to make TCP connection to port 8080: connection refused

Is there anything in the .toml file here that would make this happen? I have removed the user and pwd for display purposes, but there are present when I push the app.

`subscription_id = "kafka-firehose-nozzle" doppler_address = "wss://doppler.system.cloud.dev.enforma.io:443" uaa_address = "https://uaa.system.cloud.dev.enforma.io" username = "blahblah" password = "lalalal"

[kafka] brokers = ["kafka-single-node.enforma.io:9092"] retry_max = 10 retry_backoff_ms = 500 compression = "none" # or "snappy", "gzip"

[kafka.topic] log_message = "pcf_metrics" log_message_fmt = "log-%s" value_metric = "metric"`

[webbbret]

For others sake... Turns out that the following was needed in my manifest.yml. This enabled the application to start. If this is not the correct setting, please indicate what it should be:

health-check-type: none

[webbbret]

@giner Can you elaborate more on the following: [kafka.topic] log_message = "log" log_message_fmt = "log-%s" value_metric = "metric" My understanding is that for each event type, you would need to have an entry? For example:

HttpStartStop LogMessage ValueMetric CounterEvent Error ContainerMetric These are the event types I believe are available. If I wanted to send lets say ValueMetric to kafka, I would need to add the following to the config.toml?

value_metric = "value" value_metric_fmt = value-%s"

or is it:

`value-metric = "value" value-metric-fmt = value-%s"

Like the readme points out? Thanks in advance.

[giner]

Here is a working example we use:

subscription_id = "kfn-201612"

[cf]
doppler_address= "wss://doppler.domain.com"
uaa_address ="https://uaa.domain.com"
username = "kafka-nozzle"
idle_timeout = 15 # seconds

[kafka]
brokers = ["100.0.0.1:9092","100.0.0.2:9092"]
retry_max = 3
retry_backoff_ms = 100
compression = "gzip"

  [kafka.topic]
  log_message = "cf-app-log"
  value_metric = "cf-metrics"
  container_metric = "cf-app-metrics-containermetrics"
  http_start_stop = "cf-app-metrics-httpstartstop"

[webbbret]

@giner The application deploys without error, however, I am not seeing the topics using Kafdrop and do not see any messages. My toml looks very similar with exception to the subscription _id. I am assuming the app should be creating the topics within kafka.topic when the app launches? Many thanks in advance. Also, looks like the default for the subscription id is "example-kafka-firehose-nozzle"? and not kafka-firehose-nozzle as the readme states? The following is the output when not setting in the .toml file.

2018-12-26T10:17:18.759-08:00 [APP/PROC/WEB/0] [OUT] 2018/12/26 18:17:18 [INFO] Start consuming firehose events from Doppler (wss://doppler...) with subscription ID "example-kafka-firehose-nozzle"

[giner]

Any errors in logs? Which version of Kafka are you using? It was tested with Kafka 0.8.x.

[webbbret]

I found the following in the logs:

2018-12-27T08:31:01.991-08:00 [APP/PROC/WEB/1] [OUT] 2018/12/27 16:31:01 [ERROR] Failed to construct kafka producer: kafka: client has run out of available brokers to talk to (Is your cluster reachable?) 2018-12-27T08:31:02.001-08:00 [APP/PROC/WEB/1] [OUT] Exit status 2 2018-12-27T08:31:02.004-08:00 [CELL/SSHD/1] [OUT] Exit status 0 2018-12-27T08:31:06.393-08:00 [API/1] [OUT] Process has crashed with type: "web" 2018-12-27T08:31:06.399-08:00 [CELL/0] [OUT] Cell 63eee8ae-6215-48de-93eb-e70d6f43b622 destroying container for instance 777c6afe-122a-4c3b-7b9d-d3ba 2018-12-27T08:31:06.399-08:00 [CELL/0] [OUT] Cell 63eee8ae-6215-48de-93eb-e70d6f43b622 stopping instance 777c6afe-122a-4c3b-7b9d-d3ba 2018-12-27T08:31:06.415-08:00 [API/1] [OUT] App instance exited with guid 5b208896-c5a7-4b54-96e7-7d88ca4d7104 payload: {"instance"=>"777c6afe-122a-4c3b-7b9d-d3ba", "index"=>0, "cell_id"=>"63eee8ae-6215-48de-93eb-e70d6f43b622", "reason"=>"CRASHED", "exit_description"=>"APP/PROC/WEB: Exited with status 2", "crash_count"=>1, "crash_timestamp"=>1545928266362105623, "version"=>"9bf5bc38-9b6f-4762-aaae-d95cb66c6b87"} 2018-12-27T08:31:06.459-08:00 [CELL/0] [OUT] Cell f25a268e-fe93-421f-bb69-31f971c4afc8 creating container for instance e02f935d-86fc-42f6-4d3f-6054 2018-12-27T08:31:06.556-08:00 [PROXY/0] [OUT] Exit status 137 2018-12-27T08:31:06.613-08:00 [API/1] [OUT] Process has crashed with type: "web"

Looks like the nozzle may be having trouble communicating with the broker. Not sure if this is a certificate issue, TSL issue or what. Will look into the version of kafka as well.

[webbbret]

@giner Putting this with the other ticket that is related..

https://github.com/rakutentech/kafka-firehose-nozzle/issues/24

[giner]

Closing this one. Continue in #24.