search

cloudfoundry-community / firehose-to-syslog

Send firehose events from Cloud Foundry to syslog.
MIT License
44 stars 58 forks source link

How can GoVersion be updated in nozzle v4.1.1? #197

Open r2d2c3p0 opened 6 years ago

r2d2c3p0 commented 6 years ago

Hi,

We are currently running version 2.3 of the nozzle on PCF 1.8. In order to get it installed we had to update Godeps.json to "go1.6". Is there a way to change that value in version 4.1.1 of the nozzle?

We also would like to increase the keep-alive timeout using an environment variable, is that possible.

r2d2c3p0

shinji62 commented 6 years ago

Just use the binary no ? https://github.com/cloudfoundry-community/firehose-to-syslog/releases/tag/5.0.0

shinji62 commented 6 years ago

And 4.1.1 do not use godeps anymore but dep tool from golang.

r2d2c3p0 commented 6 years ago

Apologize for the confusion we realized we do not need the binary. We are trying to push into PCF v1.8 from behind a firewall that can not directly download dependencies from github. The question is what version of the nozzle that can be pushed to PCF v1.8 using go_buildpack-cached-v1.8.2? We previously installed nozzle version 2.3 as mentioned in original post.

r2d2c3p0 commented 6 years ago

This is the error we are getting when trying to push the nozzle into PCF:

Downloading go_buildpack... Downloaded go_buildpack (415.1M) Creating container Successfully created container Downloading app package... Downloaded app package (14.2M) Staging... -----> Go Buildpack version 1.8.2 -----> Installing godep v79 Copy [/tmp/buildpacks/a65c3a517f59654e7330a2db45496728/dependencies/https_buildpacks.cloudfoundry.org_dependencies_godepgodep-v79-linux-x64-9e37ce0f.tgz] -----> Installing glide v0.12.3 Copy [/tmp/buildpacks/a65c3a517f59654e7330a2db45496728/dependencies/httpsbuildpacks.cloudfoundry.org_dependencies_glide_glide-v0.12.3-linux-x64-aa256363.tgz] -----> Installing go 1.7.5 Copy [/tmp/buildpacks/a65c3a517f59654e7330a2db45496728/dependencies/https___buildpacks.cloudfoundry.org_dependencies_go_go1.7.5.linux-amd64-c9de5bb9.tar.gz] WARNING Installing package '.' (default) -----> Running: go install -tags cloudfoundry -buildmode pie . cli.go:11:2: cannot find package "github.com/cloudfoundry-community/firehose-to-syslog/caching" in any of: /tmp/gobuildpack.gopath279386110/.go/src/firehose-to-syslog/vendor/github.com/cloudfoundry-community/firehose-to-syslog/caching (vendor tree) /tmp/app/.cloudfoundry/0/go1.7.5/go/src/github.com/cloudfoundry-community/firehose-to-syslog/caching (from $GOROOT) /tmp/gobuildpack.gopath279386110/.go/src/github.com/cloudfoundry-community/firehose-to-syslog/caching (from $GOPATH) cli.go:12:2: cannot find package "github.com/cloudfoundry-community/firehose-to-syslog/eventRouting" in any of: /tmp/gobuildpack.gopath279386110/.go/src/firehose-to-syslog/vendor/github.com/cloudfoundry-community/firehose-to-syslog/eventRouting (vendor tree) /tmp/app/.cloudfoundry/0/go1.7.5/go/src/github.com/cloudfoundry-community/firehose-to-syslog/eventRouting (from $GOROOT) /tmp/gobuildpack.gopath279386110/.go/src/github.com/cloudfoundry-community/firehose-to-syslog/eventRouting (from $GOPATH) cli.go:13:2: cannot find package "github.com/cloudfoundry-community/firehose-to-syslog/firehoseclient" in any of: /tmp/gobuildpack.gopath279386110/.go/src/firehose-to-syslog/vendor/github.com/cloudfoundry-community/firehose-to-syslog/firehoseclient (vendor tree) /tmp/app/.cloudfoundry/0/go1.7.5/go/src/github.com/cloudfoundry-community/firehose-to-syslog/firehoseclient (from $GOROOT) /tmp/gobuildpack.gopath279386110/.go/src/github.com/cloudfoundry-community/firehose-to-syslog/firehoseclient (from $GOPATH) cli.go:14:2: cannot find package "github.com/cloudfoundry-community/firehose-to-syslog/logging" in any of: /tmp/gobuildpack.gopath279386110/.go/src/firehose-to-syslog/vendor/github.com/cloudfoundry-community/firehose-to-syslog/logging (vendor tree) /tmp/app/.cloudfoundry/0/go1.7.5/go/src/github.com/cloudfoundry-community/firehose-to-syslog/logging (from $GOROOT) /tmp/gobuildpack.gopath279386110/.go/src/github.com/cloudfoundry-community/firehose-to-syslog/logging (from $GOPATH) cli.go:15:2: cannot find package "github.com/cloudfoundry-community/firehose-to-syslog/stats" in any of: /tmp/gobuildpack.gopath279386110/.go/src/firehose-to-syslog/vendor/github.com/cloudfoundry-community/firehose-to-syslog/stats (vendor tree) /tmp/app/.cloudfoundry/0/go1.7.5/go/src/github.com/cloudfoundry-community/firehose-to-syslog/stats (from $GOROOT) /tmp/gobuildpack.gopath279386110/.go/src/github.com/cloudfoundry-community/firehose-to-syslog/stats (from $GOPATH) cli.go:16:2: cannot find package "github.com/cloudfoundry-community/firehose-to-syslog/uaatokenrefresher" in any of: /tmp/gobuildpack.gopath279386110/.go/src/firehose-to-syslog/vendor/github.com/cloudfoundry-community/firehose-to-syslog/uaatokenrefresher (vendor tree) /tmp/app/.cloudfoundry/0/go1.7.5/go/src/github.com/cloudfoundry-community/firehose-to-syslog/uaatokenrefresher (from $GOROOT) /tmp/gobuildpack.gopath279386110/.go/src/github.com/cloudfoundry-community/firehose-to-syslog/uaatokenrefresher (from $GOPATH) ERROR Unable to compile application: exit status 1 Failed to compile droplet Exit status 223 Staging failed: Exited with status 223

shinji62 commented 6 years ago

@r2d2c3p0 dep have been introduce in buildpack v1.8.6 I will advise you to use the binary provide in the Githut release you should not be block to use the new feature and fix because of your old buildpack.

It's compile against last alpine and last patched version of Golang.

So just run using the binary buildpack.

r2d2c3p0 commented 6 years ago

ok will try that and get back to you. thank you for your help.

r2d2c3p0 commented 6 years ago

When running the binary we get the following error: panic: runtime error: invalid memory address or nil pointer dereference

Here are the flags we are using: ./firehose-to-syslog_linux_386 \ --debug \ --api-endpoint="https://api.our-cfsys-domain.net" \ --doppler-endpoint=wss://doppler.api.our-cfsys-domain.net:443 \ --syslog-server=our.splunk:20000 \ --syslog-protocol="tcp" \ --skip-ssl-validation-syslog \ --subscription-id="firehose" \ --client-id="fh-client" \ --client-secret="MySecret" \ --fh-keep-alive=60s \ --events="LogMessage" \ --cc-pull-time=3600s \ --cert-pem-syslog="cert.pem" \ 2>~/F2S-Error.log

shinji62 commented 6 years ago

Can you give me the full trace please ?

And please use the firehose-to-syslog_linux_amd64 lib.

Thanks

r2d2c3p0 commented 6 years ago

We were able to deploy firehose-to-syslog_linux_amd64 5.0.0 and we see some logs in Splunk but we are getting these errors:

"[2018-08-28 18:10:38.993902536 +0000 UTC m=+343817.709723692] Exception occurred! Message: Missed Logs  Details: 110000"

"[2018-08-28 18:10:41.181384421 +0000 UTC m=+343819.897205567] Exception occurred! Message: Missed Logs  Details: 10000"

"[2018-08-28 18:10:44.324543501 +0000 UTC m=+343823.110589349] Exception occurred! Message: Missed Logs  Details: 10000"

What does "Details" 110000 & 10000 means in above exceptions?

r2d2

shinji62 commented 5 years ago

Means you are losing logs..

Your buffer seems full, you can increase the number of nozzle and make splunk ingesting faster.