search

spinscale / elasticsearch-graphite-plugin

[UNMAINTAINED] Elasticsearch plugin which pushes data to a graphite server
Do What The F*ck You Want To Public License
66 stars 32 forks source link

Is it possible to disable certain metrics? #3

Closed cjs226 closed 11 years ago

cjs226 commented 11 years ago

We're new to ES and are testing a use case where we'll have tens of thousands of indexes. When using this plugin we generated over 500GB of data in Graphite. In a few hours. :-) I know I can setup a blacklist within Graphite, but would prefer to tailor the data that's transferred over the wire and curtail the overhead of generating it.

Thanks, Clif

blakesmith commented 11 years ago

Wow, that sounds like a tough challenge. Can you explain your use case for having so many indexes a bit more?

cjs226 commented 11 years ago

Our use case involves a small amount of data per user, with a very large amount of users. By using a 1 to 1 relationship of user to index, it allows us to open/close/destroy indexes with a smaller memory footprint.

On Aug 29, 2013, at 1:42 PM, Blake Smith notifications@github.com wrote:

Wow, that sounds like a tough challenge. Can you explain your use case for having so many indexes a bit more?

— Reply to this email directly or view it on GitHub.

blakesmith commented 11 years ago

For user sharded multi-tenant setups, I've had great success using the 'routing' search param. There's a good blog post about it here: http://www.elasticsearch.org/blog/customizing-your-document-routing/

Basically, it allows you to create 'virtual' indexes based on a custom identifier (in your case, user_id). We've used this graphite plugin quite successfully for several months with routing, and have also gained efficiency by not having to query all ES shards.

cjs226 commented 11 years ago

Thanks for the link. I've forwarded it on to our devs.

spinscale commented 11 years ago

@blakesmith Thx for jumping in.. was on holiday without internet :-)

@cjs226 I have an idea, lets see what you think about it in your current usecase

What about having a metrics.graphite.include and a metrics.graphite.exclude configuration field, which allows you to configure a regular expression, which needs to (not) match against the metric name.

Would this be scalable to your usecase? Or do you have any other idea from your use-case point of view?

cjs226 commented 11 years ago

@spinscale That'd be great! Thx for being open to other use cases!

spinscale commented 11 years ago

@cjs226 that'll not too hard to implement, I'll see what the weekend brings in terms of hacking time :-)

cjs226 commented 11 years ago

@spinscale Thank you very much!

spinscale commented 11 years ago

I just pushed a new version. Would be awesome if you could test it!

cjs226 commented 11 years ago

I'll do so, thanks for helping us out!

cjs226 commented 11 years ago

Works like a charm! Thanks again for helping us out!