search

elastic / beats-dashboards

DEPRECATED. Moved to https://github.com/elastic/beats. Please use the new repository to add new issues.
Other
196 stars 73 forks source link

Configurable Index Pattern #69

Closed megastef closed 8 years ago

megastef commented 8 years ago

Hi,

Would it be possible to make the index-pattern configurable?

I store topbeat data with an index pattern TOKEN_YYYY-DD-MM - while token is a prefix for each client/customer/system. In my case topbeat logs are shipped with another log shipper, which can insert any json logs to ES.

What needs to be changed to import the dashboards? Just change title in dashboards/topbeat.json?

Thanks Stefan

ruflin commented 8 years ago

Is your topbeat data shipped by an other "shipper" having the exact same document structure?

megastef commented 8 years ago

Yes, just another index pattern. I could use direct elasticsearch inserts as well (if index-pattern would be configurable in libbeat ..). But the index-pattern problem with the dashboard would remain. So I assume only this line would need to be changed by the load.sh script? https://github.com/elastic/beats-dashboards/blob/master/dashboards/index-pattern/topbeat.json#L5

megastef commented 8 years ago

Wow there is a lot of hard-coded stuff in libbeat and this dashboards. I see the index-pattern is in every "search" with the prefixes topbeat, filebeat,... like in here: https://github.com/elastic/beats-dashboards/blob/master/dashboards/search/Proc-stats.json#L11 So I'm thinking of a dashboard generator now.

damm commented 8 years ago

That would be great; I always found the idea of loading a bunch of json dashboards and blobs into ES to be a bit scary.

I Didn't used to; until I ran load.sh on this repo once and broke my default search pattern.

monicasarbu commented 8 years ago

Indeed the index pattern is hardcoded everywhere in the dashboards. An option would be to replace "topbeat-" with "token-" in all files under beats-dashboards/dashboards: search/System-wide.json, search/Processes.json, search/Proc-stats.json, search/Filesystem-stats.json and index-pattern/topbeat.json.

As you can configure in the beats the index pattern, it would be nice to add an option to the load.sh to pass the index pattern name. So, I am setting this issue to a feature request. Thanks for raising this issue!

megastef commented 8 years ago

Thanks for your quick reply. Well the pattern is not completly configurable https://github.com/elastic/libbeat/issues/318 - or did this change?

damm commented 8 years ago

You can also ship to Logstash which can alternate the pattern...

radoondas commented 8 years ago

Hi, I have created my solution for this issue. radoondas/beats-dashboards commit: 38f60b0 I suppose, it should be discussed. It's more as a draft for now. It is universal for 'unlimited' number of beats. You can also add your beats and it will work fine after new line in .beatconfig file. Also script doesn't touch original file.

ruflin commented 8 years ago

@radoondas I would suggest to split up this commit into two parts: 1 is cleanup with variables. Best would be to these in. Then I think on line 100 the change we should discuss start. Can you open a separate PR for the first part so the discussion gets more focused on the second part? Probably it is also worth mentioning this Idea here https://github.com/elastic/beats-dashboards/pull/84 Have a look.

radoondas commented 8 years ago

@ruflin you are right. I was planing it but then I did just one commit. I'll split changes in to 2 different commits which can be then separately discussed. PR will be then very simple request.

ruflin commented 8 years ago

@radoondas I would split it even into two PR's. Like this one can already be merged even if the discussion for the second should take longer.

radoondas commented 8 years ago

@ruflin this is just the commit for configurable index patterns: 929d684 I had to 'revert' one big commit and I did split it in to two. I'll check how will PR look like.

megastef commented 8 years ago

Hi, I used https://github.com/elastic/beats-dashboards/commit/929d684163f37fa9db0e48729cd34b73763f333a for some tests and wonder why it is not merged. It did work so far (beside the fact that dashboards did not use .raw fields for strings, and had not the right mapping for topbeat in place, but I assume this was my fault). It would be helpful if the dashboard loader could place the right mapping for topbeat (and other beats), when specifying the data index like --data-index=INDEX-NAME.

A little issue is the creation of index-patterns, we have those in place, and I removed then the topbeat, packetbeat, etc. index patterns manually. I think if index-pattern creation runs, it should use the values from .beatconfig, like it is done for searches (run sed first).

megastef commented 8 years ago

Any news on this? I've seen this merge: https://github.com/elastic/beats/pull/2119/files which closes https://github.com/elastic/beats/issues/921

monicasarbu commented 8 years ago

@megastef Yes, https://github.com/elastic/beats/pull/2119 adds support to configure the entire index pattern in the Beat configuration file, not only the index base name.

We improved the way we import the Beats dashboards in 5.0. Each Beat package comes with the scripts/import_dashboards app that imports the index-pattern, Kibana dashboards, and its dependencies (visualizations and searches) for a single Beat. You can change the index name that is used in the dashboards, its dependencies and the index pattern, by passing it in the -i option. Here you can find more details about app.

You can try these changes, here. We would love to hear your feedback about it before releasing this solution.