search

OpenTSDB / opentsdb

A scalable, distributed Time Series Database.
http://opentsdb.net
GNU Lesser General Public License v2.1
4.99k stars 1.25k forks source link

amqp ingestion plugin #218

Open marksteele opened 11 years ago

marksteele commented 11 years ago

It would be great if we could get a plugin that connects to a queue on an AMQP broker and binds to an exchange+topic to receive JSON formatted inputs.

Something like: {"metric": "cpu","timestamp":12355623,"value":123,"tags":["host":"1.2.3.4","plugin":"foo"]"}\n {"metric": "mem","timestamp":12355623,"value":123,"tags":["host":"1.2.3.4","plugin":"foo"]"}\n {"metric": "disk","timestamp":12355623,"value":123,"tags":["host":"1.2.3.4","plugin":"foo"]"}\n

(newlines added for clarity, would be nice to be able to push multiple metrics per AMQP message for efficiency)

I've built something similar to integrate with opentsdb (https://github.com/marksteele/collectd-amqp-graphite-opentsdb-gateway), however as there's already publisher to amqp, an ingestor would be a nice touch.

Bonus points if there's an option for toggling snappy compression on the AMQP messages.

manolama commented 11 years ago

An AMQP ingest would be cool.

Re CollectD, we're using it to feed OpenTSDB. One of our guys wrote code to publish directly to OpenTSDB over the Telnet interface and it works nicely. I'll see if he put it up in a public repo somewhere. I also wrote code for the OpenTSDB HTTP API that accepts CollectD's HTTP JSON output. It just has to be converted to a proper serializer plugin for the official 2.0. It's in my "scratch" branch if you wanna take a look.

manolama commented 11 years ago

https://github.com/bpiraeus/collectd

marksteele commented 11 years ago

well the reason I wanted an amqp ingestor is for the semantics you can get with amqp. my end goal is to feed evrything in via amqp and process in several separate processing streams (eg: trending via opentsdb, monitoring via esper, etc...)

marksteele commented 11 years ago

ping!

guilhermeblanco commented 10 years ago

:+1:

zstyblik commented 9 years ago

I recommend writing "robots" to do that rather than implementing it in OTSDB itself. My $0.02USD.

IDerr commented 7 years ago

Don't think that's the job of OpenTSDB, to ingest data like that, we should keep HTTP and TCP as input, and use scripts/software like Nifi or a script in python, to send data over protocols as I said before..

What do you think @manolama ??

johann8384 commented 7 years ago

A feature request for a plug-in is reasonable probably, could go either way. We have a few built in plugins for other things, including the telnet and http protocols.

IDerr commented 7 years ago

Yeah sure, I didn't saw that there's already Real Time Publishing part on the resources, so yeah it is reasonable. Thanks