ekutner commented 9 years ago

In Kibana 3 it was possible to define multiple indexes in the index pattern box. That capability is required when using a tribe node to consolidate data from multiple locations.

markmcdowell commented 9 years ago

This would be really useful for us, at the moment we're having to do the same search multiple times.

iozozturk commented 9 years ago

i could manage to define multiple indices with comma, but i got error when i try to discover:

pbayley commented 9 years ago

Hi Guys, any idea which milestone this will make its way into as this is a regression / breaking change compared to Kibana 3 and means that Kibana 4 won't work well within a tribe owing to the need to use unique indices at each member of the tribe.

sfritz commented 9 years ago

+1, I structured my indices under the assumption that I could create dashboards with multiple indices. My current index naming scheme also makes it easier to work with curator, so I'd rather not have to change them.

agirbal commented 9 years ago

+1 on this, much needed thx

rore commented 9 years ago

+1 !

cfeio commented 9 years ago

+1

charleseads commented 9 years ago

+1 This would be appreciated!

zaakiy commented 9 years ago

+1.

AlexRRR commented 9 years ago

I was not aware this was no longer possible. This is a no-go for us as we use tribe nodes as well. From the ES docs: _The merged view cannot handle indices with the same name in multiple clusters. By default it will pick one of them, see later for onconflict options.

dev-shubh commented 9 years ago

+1 This would be appreciated!

dev-shubh commented 9 years ago

Any idea when this enhancement will be released ???

TinLe commented 9 years ago

We depend on tribe nodes. This is a stop everything proof of concept killer.

Need this working. +1

jlintz commented 9 years ago

We really depend on this as well

cfeio commented 9 years ago

Any idea when this enhancement will be added? It appears there is a lot of interest since this is functionality that has been removed since Kibana 3.

This issue has existed for 6 months without resolve, is this functionality going to be supported?

wenshin commented 9 years ago

+1 This would be appreciated!

sakalajuraj commented 9 years ago

+1 I am not able to migrate to Kibana 4 because of this issue. We are dependent on this functionality.

jerr0328 commented 9 years ago

I'm not a Kibana/ES developer, but I took a look at the code to see if it's something I could fix.

It looks like there's two files that could be affected by this. Mainly, the _mapper.js file: https://github.com/elastic/kibana/blob/master/src/kibana/components/index_patterns/_mapper.js

The mapper has a function called getIndicesForIndexPattern which takes the pattern provided, strips out the dates (more on that in a second) and then checks the date pattern against that.

For the pattern, it gets converted to something elasticsearch can parse over here: https://github.com/elastic/kibana/blob/master/src/kibana/components/index_patterns/_pattern_to_wildcard.js

Basically, this strips out the date by converting it to a wildcard search when it exits the escaped section in square brackets. So it takes something like [logstash-]YYYY.MM.DD and sends elasticsearch a query with logstash-*. Anything outside of the brackets, with that function, is returned as a wildcard.

Back to the getIndicesForIndexPattern function, we can see it looks for matches by filtering the "all" list on the index names returned and parsing it with the format string passed to this function. The problem, then, is that this will fail because it only uses moment to check the formatting, and doesn't also factor in a regular expression.

I think adding comma-separated values could probably be supported, with a format such as [logstash-one-]YYYY.MM.DD,[logstash-two-]YYYY.MM by splitting the string into an array at the comma, and passing the array to moment as the format string when parsing. For matching it back, one would have to change the following line to check if the returned index name matches either format.

I'm sure there's much more work to this than I can tell. This is just a quick stab at what I'd do. I may fork and try it out myself if I'm on the right track and it's as "low fruit" as the tag makes it seem.

pbayley commented 9 years ago

Hi Guys,

Seriously what is happening with this issue, I see it has been marked as 'low fruit' meaning that effectively you are saying that it isn't much of an issue. Given that Tribe is a significant feature of ES and is now effectively no longer supported by Kibana I would hope this would get a much better focus.

@rashidkpc - can you indicate why this issue has been moved in and out of milestones and no one seems to wish to resolve this anytime soon?

rashidkpc commented 9 years ago

Tribe nodes should still work, you should be able to name your indices with a common prefix and use wildcards. We don't plan to bring back multiple timestamped patterns since we're deprecating them: https://github.com/elastic/kibana/issues/4342

cfeio commented 9 years ago

There are sitll use cases where we would want multiple patterns (timestamped, or not) for a dashboard. This is particularly useful for multitenant clusters, where we would want a dashboard that looks at a subset of tenant index patterns.

Using a common prefix will not work in this case since a tenant could be in multiple subsets. Also, we may want to change which indexes this dashboard looks at without having to change our naming convention.

Regardless of whether or not you will support timestamps, can you please consider adding back support for multiple index patterns?

leonger commented 9 years ago

+1 for the feature. Luck of it really invalidates our pilot.

bendpx commented 9 years ago

+1

Can someone from the contributors team update if this feature request is under development?

spalger commented 9 years ago

@bendpx it's not actively being worked on, but is planned for the next major release

cinhtau commented 9 years ago

In Kibana 3 is was possible to take log data from application index and log data from metrics index. Thanks for opening this issue.

:+1:

and thanks for the update, could you elaborate a bit what means major release? Kibana 5 or Kibana 4.3?

sbiffi commented 9 years ago

Any news about when this will be available?

simianhacker commented 9 years ago

This should be handled by #4342

charleseads commented 9 years ago

4342 does not satisfy our interest in this ticket. Our need for multiple indexes is related to combining indexes from multiple sources, and is not related to index timestamp patterns. For example, consider a case where there are separate nginx and logstash indexes. It's often useful to consider these separately to maintain data clarity. However, it's also useful to correlate the data in a common view. Currently, we're making do with index definitions in Kibana like g to match nginx and logstash.

tkuther commented 8 years ago

+1

Separated and timestamped indices are required in our setup for several reasons:

a) data retention vs. cluster performance

using curator to close indices after x days, archive them after x+n days
e.g. Apache logs are kept open for 6 months, closed for another 6 months, and then tarball'ed and moved to an archive where they are kept for another year

b) lowering storage overhead

we use separate indices for production and test-environment log data
searches need to be possible from one dashboard (or saved search in Kibana4) using "env" field filter

c) correlating data from different indices (e.g. Apache and Wildfly logs)

All this works nicely with curator and Kibana 3 in our current setup. Migration to Kibana 4 is not possible due to lack of proper support for b) and c). b) is possible only when searching through all indexes using wildcard patterns causing massive performance issues, c) isn't possible at all. Only way is using aliases, but then again only across all indices

I'm not sure if #4342 is helpful when elasticsearch still has to go through 180 indices, because I don't see a more sane way to manage a) and b) currently.

Being able to just specify a timestamped wildcard pattern like [logs-*-apache-access-]YYYY.MM.DD would be so simple.

Another way around it would be support for date patterns in the aliases section of index template, as requested in https://github.com/elastic/elasticsearch/issues/5359. That would be a valid solution for b), still quite ugly for c).

epixa commented 8 years ago

It seems clear that while #4342 will solve a lot of time based problems for folks, there is still a need to define multiple indices as a single index pattern, in particular when dealing with non-time-based data. At this point, it is unlikely that this will happen for 4.3, but I'm going to shoot to work on this after #4342 is finished.

faxm0dem commented 8 years ago

we also need the feature (back) thanks for taking this into account

bradvido commented 8 years ago

Tribe clusters need this feature. The only way to accomplish it currently is to use some fancy aliasing and a wildcard index pattern. This means that pre #4342 all of the time-series indices are (inefficiently) searched instead of just the ones that the selected time range applies to.

rashidkpc commented 8 years ago

This is solved in #4342 by simply adding a prefix, eg logs-apache-2015, logs-nginx-2015 and pointing Kibana at logs-*. If something more complex is required, make an elasticsearch alias, kibana will take care of efficiently resolving it.

pjcard commented 7 years ago

Anyone stumbling upon this should note that, for the example given, an alias which combined logs-apache- and logs-nginx- would need to be rebuilt each time an index was added, as its wildcard form only includes indices available at the time of creation.

https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-aliases.html

In this case, the alias is a point-in-time alias that will group all current indices that match, it will not automatically update as new indices that match this pattern are added/removed.

elastic / kibana

Multiple indices in pattern #2017