cityindex-attic / logsearch

[unmaintained] A development environment for ELK
Apache License 2.0
24 stars 8 forks source link

Add relp endpoint and parse cloudfoundry messages #289

Closed mrdavidlaing closed 9 years ago

mrdavidlaing commented 10 years ago

All CloudFoundry jobs include a syslog_forwarder, which forwards using the following format:

$template LongTagForwardFormat, "<%%PRI%>%TIMESTAMP:::date-rfc3339% %HOSTNAME% %syslogtag%%msg:::sp-if-no-1st-sp%%msg%" #Eliminate 32 character tag limit
:programname, startswith, "vcap." :omrelp:<%= address %>:<%= port %>;LongTagForwardFormat

We need to support the RELP logstash input - http://logstash.net/docs/1.3.2/inputs/relp

Sample messages are just JSON:

{"timestamp":1388405347.8615224,"message":"hm.actual-state.process-heartbeat","log_level":"debug","source":"hm","data":{"dea":"0-3527a277bb8846f190c18c32586d6982"},"thread_id":20748720,"fiber_id":16471520,"process_id":21205,"file":"/var/vcap/packages/health_manager_next/health_manager_next/lib/health_manager/actual_state.rb","lineno":62,"method":"process_heartbeat"}
{"timestamp":1388405314.6058202,"message":"Token received from the UAA nil","log_level":"info","source":"cc.api","data":{"request_guid":"9d0da25e-cbd8-4091-a934-c53201ef6605"},"thread_id":35440980,"fiber_id":39853800,"process_id":21074,"file":"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/lib/cloud_controller.rb","lineno":75,"method":"decode_token"}

Some further sample messages shipped over regular syslog are at http://logsearch-labs.cityindextest5.co.uk/index.html#dashboard/temp/HSekWHw7Td2uK8YvLJzvAQ

sopel commented 9 years ago

Closed as Won't Fix due to project being retired to the CityIndex Attic.