search

itamarst / eliot

Eliot: the logging system that tells you *why* it happened
https://eliot.readthedocs.io
Apache License 2.0
1.1k stars 66 forks source link

Best way to send messages to logstash in realtime? #305

Open suryabahubalendruni opened 6 years ago

suryabahubalendruni commented 6 years ago

The example in the documentation for using logstash involves reading a json log file into logstash in an ad-hoc manner. How would one go about sending messages to logstash as they are produced, say via TCP or syslog?

To explain a bit about my use case: I am running a few different python services in docker that communicate using rabbitmq.

itamarst commented 6 years ago

I believe logstash can follow a file (like tail -f does). So you can have eliot log to a file and then have logstash read from that. This also has that benefit that if logstash crashes or centralized log server goes down, you still have the local logs.

Does that seem reasonable in your particular use case?

suryabahubalendruni commented 6 years ago

My system is an event handler for a trading app different stages (streaming market data, trade signal generation, trade execution, data persistence), so I am going to be generating a great number of logs continuously from all the processes. The system should have an indefinite runtime (ideally), so I don’t want to bloat my docker containers with large log files. Tailing the log file should be good for me initially, but if you have a different approach I would greatly appreciate it.