search

hynd / heka-tsutils-plugins

Some work-in-progress Heka plugins
Mozilla Public License 2.0
10 stars 4 forks source link

batch encoder for OpenTSDB #3

Open timurb opened 8 years ago

timurb commented 8 years ago

Hi, I'm going to create a batch encoder for OpenTSDB based on your opentsdb_http so that a single Heka message could hold in it several metrics at once (vs a single metric as processed at the moment).

Do you think I should create a new encoder or use the same one but try to maintain backward compatibility (or may be drop compatibility)?

BTW, do you have any plans on contributing encoders to Heka core? I can do that for this new encoder if you are not going to do that soon (referring to you as an original author of course). What do you think?

hynd commented 8 years ago

Go for it - if it starts getting messy, i'd create a separate one. And by all means contribute yours to Heka core!

I have a few others that we use internally, but they'd need some tweaking to make them reusable by others. Because of the opinionated way (eg; single messages and some fields) we run things through our pipeline, i'm not sure Heka would want them in the core package - i was content just leaving it linked from https://github.com/mozilla-services/heka/wiki/Community-Plugins.

There were a couple of changes i was going to make to this one (switching to decode_message(read_message("raw")) instead of read_next_field(), and bumping the JSON number precision up now that https://github.com/trink/lua-cjson/pull/1 made its way into Heka, but both would tie this encoder to a pretty recent Heka release.

timurb commented 8 years ago

Ok, I'll try doing a contribution to Heka core then. If they don't accept it I'll send a pull request to you. There is also a distant possibility that my design suites nor Heka upstream nor your vision -- in that case I'll just leave a comment here for you.

I'm not planning to support Heka versions lower than 0.10 so I was going to use read_message instead of read_next_field too. What is the story about JSON number precision? I'm afraid I don't understand what is that ticket about.

timurb commented 8 years ago

https://github.com/mozilla-services/heka/pull/1751