Open marksteele opened 11 years ago
An AMQP ingest would be cool.
Re CollectD, we're using it to feed OpenTSDB. One of our guys wrote code to publish directly to OpenTSDB over the Telnet interface and it works nicely. I'll see if he put it up in a public repo somewhere. I also wrote code for the OpenTSDB HTTP API that accepts CollectD's HTTP JSON output. It just has to be converted to a proper serializer plugin for the official 2.0. It's in my "scratch" branch if you wanna take a look.
well the reason I wanted an amqp ingestor is for the semantics you can get with amqp. my end goal is to feed evrything in via amqp and process in several separate processing streams (eg: trending via opentsdb, monitoring via esper, etc...)
ping!
:+1:
I recommend writing "robots" to do that rather than implementing it in OTSDB itself. My $0.02USD.
Don't think that's the job of OpenTSDB, to ingest data like that, we should keep HTTP and TCP as input, and use scripts/software like Nifi or a script in python, to send data over protocols as I said before..
What do you think @manolama ??
A feature request for a plug-in is reasonable probably, could go either way. We have a few built in plugins for other things, including the telnet and http protocols.
Yeah sure, I didn't saw that there's already Real Time Publishing part on the resources, so yeah it is reasonable. Thanks
It would be great if we could get a plugin that connects to a queue on an AMQP broker and binds to an exchange+topic to receive JSON formatted inputs.
Something like: {"metric": "cpu","timestamp":12355623,"value":123,"tags":["host":"1.2.3.4","plugin":"foo"]"}\n {"metric": "mem","timestamp":12355623,"value":123,"tags":["host":"1.2.3.4","plugin":"foo"]"}\n {"metric": "disk","timestamp":12355623,"value":123,"tags":["host":"1.2.3.4","plugin":"foo"]"}\n
(newlines added for clarity, would be nice to be able to push multiple metrics per AMQP message for efficiency)
I've built something similar to integrate with opentsdb (https://github.com/marksteele/collectd-amqp-graphite-opentsdb-gateway), however as there's already publisher to amqp, an ingestor would be a nice touch.
Bonus points if there's an option for toggling snappy compression on the AMQP messages.