search

cloudfoundry-community / splunk-firehose-nozzle

Send CF component metrics, CF app logs, and CF app metrics to Splunk
Apache License 2.0
29 stars 30 forks source link

excessive response times for CC queries due to inefficient IRD=2 calls #143

Closed metskem closed 5 years ago

metskem commented 6 years ago

We are seeing flapping GoRouter alerts because of high response times. Further investigation reveals that these high response times come from queries to the CloudController done by the splunk-nozzle. We run the nozzle with envvar APP_CACHE_INVALIDATE_TTL on 30s.

Example request:

[2018-09-14T13:52:02.341+0000] "GET /v2/apps?inline-relations-depth=2&order-direction=asc&page=3&results-per-page=50 HTTP/1.1" 200 0 3285743 "-" "Go-CF-client/1.1" "10.253.4.60:59650" "10.253.4.29:9022" x_forwarded_for:"10.253.4.60" x_forwarded_proto:"https" vcap_request_id:"9e80ec7d-e909-4f81-78d0-f8bb5c297bf4" response_time:10.2663151 app_id:"-" app_index:"-" x_b3_traceid:"f68850e15f557804" x_b3_spanid:"f68850e15f557804" x_b3_parentspanid:"-"

Which is a response time over 3 seconds.

This type of call is also deprecated, see https://github.com/cloudfoundry-community/go-cfclient/pull/169

Can this be fixed?

cheers, Harry

luckyj5 commented 5 years ago

@metskem This issue is addressed in https://github.com/cloudfoundry-community/splunk-firehose-nozzle/pull/172 and will be a part of upcoming release 1.1 Thanks!

metskem commented 5 years ago

Thanks @luckyj5 , looking forward for the 1.1 release.

cheers, Harry