search

cloudfoundry-community / splunk-firehose-nozzle

Send CF component metrics, CF app logs, and CF app metrics to Splunk
Apache License 2.0
29 stars 29 forks source link

Forward envelope tags to logs #278

Closed vchrisb closed 3 years ago

vchrisb commented 3 years ago

What would you like to be added Forward the tags from envelope to splunk. e.g.:

origin:"gorouter" 
eventType:HttpStartStop 
timestamp:1625813158139554538 
deployment:"cf-33e2ae07e971738ddbe2" 
job:"router" 
index:"e7dbde0a-97b2-42f2-8ed2-4ab85b9c4764" 
ip:"10.0.4.14" 
tags:<key:"app_id" value:"eec4cd74-ab25-4eb1-9d14-244bb8f6bf95" > 
tags:<key:"app_name" value:"backend" > 
tags:<key:"component" value:"route-emitter" > 
tags:<key:"instance_id" value:"0" > 
tags:<key:"organization_id" value:"8835f19d-fb52-4429-b105-bd5fa856ac63" > 
tags:<key:"organization_name" value:"test-org" > 
tags:<key:"process_id" value:"eec4cd74-ab25-4eb1-9d14-244bb8f6bf95" > 
tags:<key:"process_instance_id" value:"c6e77d62-a69b-4bfe-4575-d724" > 
tags:<key:"process_type" value:"web" > 
tags:<key:"product" value:"VMware Tanzu Application Service" > 
tags:<key:"source_id" value:"b6c3b06c-7dd9-420a-a796-ba77c865f3b2" > 
tags:<key:"space_id" value:"da10ee5a-19f6-4628-abe2-37a7436a571f" > 
tags:<key:"space_name" value:"test-space" > 
tags:<key:"system_domain" value:"sys.newark.cf-app.com" > 
httpStartStop:<
startTimestamp:1625813158138344512 
stopTimestamp:1625813158139540230 
requestId:83bbd901-ec7f-4723-45e6-b0cdf46c7b8e 
peerType:Server 
method:GET 
uri:"http://backend.apps.newark.cf-app.com/" 
remoteAddress:"34.136.235.194:48008" 
userAgent:"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.114 Safari/537.36" 
statusCode:304 
contentLength:0 
applicationId:b6c3b06c-7dd9-420a-a796-ba77c865f3b2
instanceIndex:0 
instanceId:"5c68d529-4653-42b1-4e92-c73c" 
forwarded:"91.6.45.61" 
>

Why is this needed: This helps to further filter the logs. Especially for a HttpStartStop event, there is a difference between source_id and app_id. The HttpStartStop envelope does use source_id to fill the applicationId. For requests being forwarded, with the X-Vcap-Request-Id header in the platform, the source_id will be the application id of the app that forwarded the request. Hence without the tags being present, it is hard to correlate the requests.

Additional context In an earlier commit this functionality was already introduced and merged into develop but some changes dropped this commit at some point. https://github.com/cloudfoundry-community/splunk-firehose-nozzle/pull/241/files The commit was done by @luckyj5

luckyj5 commented 3 years ago

@vchrisb Thanks for reaching out. Are tags a part of firehose v1 envelope? or v2 envelope? or both?

Also, including tags might help correlating in specific scenarios, but can it create confusion as well? As an event can have 2 diff app_ids in the same event?

Regarding previous PR - it was added as part of v1-v2 update, but due to v2 perf impact, we reverted the commits and continued to use v1 instead. Thanks!

jak74 commented 3 years ago

I'm up voting this issue. The new RMQ 2.x emits metrics heavily relying on tags for "per object metrics." Example of the metric for how many consumers a queue has:

origin:"" eventType:ValueMetric timestamp:1627591850128628985 deployment:"service-instance_xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx" job:"rabbitmq-server" index:"xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx" ip:"x.x.x.x" tags:<key:"instance_id" value:"rabbit@xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx.rabbitmq-server.pcfsvcs.service-instance-xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx.bosh" > tags:<key:"queue" value:"my-queue-name.consumer" > tags:<key:"source_id" value:"rabbit@localhost" > tags:<key:"vhost" value:"xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx" > valueMetric:<name:"rabbitmq_queue_consumers" value:3 unit:"" >

The queue name is included as a tag, which isn't being passed in to Splunk which makes it impossible to look at the value for any specific queue and know which queue it is you're looking at. Just a single example, appears all the metrics for the new RMQ 2.x are similar so dashboards for monitoring individual queues are non-functional.

luckyj5 commented 3 years ago

This will be added as an optional configuration option with the next release of the nozzle.

jak74 commented 3 years ago

Thanks @luckyj5 ! Is there an estimate on when this release will go out?

luckyj5 commented 3 years ago

Thanks @luckyj5 ! Is there an estimate on when this release will go out?

We are working on the release (with few more updates) and it will be out soon. (few weeks)