Open daichi703n opened 5 years ago
@daichi703n Thanks for the report. I guess will be better with the v2 implementation.
Can you test the develop branch please ?
Believe I'm noticing the same thing. We've got dual datacenter deploys, one 2.3, and one 2.5 (mid upgrades for everything). Will test out the older version and validate. If I can confirm, will send more details.
@shinji62 I tested new one, but publish_per_sec
is still about under 10...
Do I need some properties?
@daichi703n Did you get any further with this issue?
@sba30 No, I don't get any further.
I re-tested with PAS2.6.3, but the result is same as before. A little things that I figured out, is the throughput depends on EVENT types as following.
EVENT | Throughput |
---|---|
LogMessage | under 10 |
ContainerMetric | under 10 |
CounterEvent | over 100 |
@daichi703n can you try increasing this value to say 600s
cloudfoundry.firehose_cc_pull_interval: description: "full app metadata update interval; defaults to 60s" default: "60s"
and setting this true
cloudfoundry.ignore_missing_apps: description: "Enable throttling on cache lookup for missing apps" default: false
and see if that makes a difference? https://github.com/cloudfoundry-community/logsearch-for-cloudfoundry/blob/develop/jobs/ingestor_cloudfoundry-firehose/spec
@sba30
Thanks!!
cloudfoundry.ignore_missing_apps
: NO improvements
cloudfoundry.firehose_cc_pull_interval
: Improved
I had deployed firehose-to-syslog app to CF with CF_PULL_TIME: 0
(env var same with cloudfoundry.firehose_cc_pull_interval
).
applications:
- name: firehose-to-syslog
buildpack: binary_buildpack
command: ./firehose-to-syslog_linux_amd64
...
env:
GOPACKAGENAME: github.com/cloudfoundry-community/firehose-to-syslog
API_ENDPOINT: https://api.sys.${DOMAIN}
...
CF_PULL_TIME: 0 # -> 600s
After set CF_PULL_TIME: 600s
(larger than 0) , performance improved!!
Maybe cachingClient
(changed in 5.x) causes low performance because it is using tickerTime (cc-pull-time)
.
@lrstanley
Do your environment has same setting?
firehose-to-syslog/v5.0.0+ with PCF/PAS 2.5 is too slow logging.
On my env,
publish_per_sec
is about under 10. Switch to firehose-to-syslog/v4.1.1,publish_per_sec
is over 1,000.Which update cause this issue? How to fix this with tuning parameters?
Environment
I'm using firehose-to-syslog as App on PCF with this manifest.
Result of tests
Version matrix
Matrix of this issue occurrence is as below;
Logs
I checked the performance at https://firehose-to-syslog.sys.${DOMAIN}/stats/app .
NG case:
publish_per_sec
is about 10, thereforesubinupt_buffer
keep increasing.OK case:
publish_per_sec
is enough large to logging at realtime.CF Loggregator Components
I also checked CF loggregator component in PAS2.5.2 works properly. Each test, I got logs at realtime.
log-cache Test Log Cache with
log-cache-cli
(cf tail
). https://github.com/cloudfoundry/log-cache-clifirehose Test Traffic Controller/Firehose with
firehose-plugin
(cf nozzle
,cf app-nozzle
). https://github.com/cloudfoundry-attic/firehose-pluginCF Components Version
https://docs.pivotal.io/pivotalcf/2-5/pcf-release-notes/runtime-rn.html