search

StamusNetworks / KTS6

Kibana 6 Templates for Suricata IDPS Threat Hunting
GNU General Public License v3.0
25 stars 6 forks source link

Could not locate that index-pattern (id: logstash-*) #4

Open alphaDev23 opened 5 years ago

alphaDev23 commented 5 years ago

Received the following error in Kibana: "Could not locate that index-pattern (id: logstash-*), click here to re-create it"

Note, there is nothing to click and the index does exist. There are events in Discover filtered on that index.

pevma commented 5 years ago

How did you receive the error ? A bit of background info would help troubleshooting :)

-- Regards, Peter Manev

On 17 Nov 2018, at 23:13, alphaDev23 notifications@github.com wrote:

Received the following error in Kibana: "Could not locate that index-pattern (id: logstash-*), click here to re-create it"

Note, there is nothing to click and the index does exist. There are events in Discover filtered on that index.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or mute the thread.

alphaDev23 commented 5 years ago

There is not much of a background. Selected a KTS6 dashboard, e.g. SN-ALL, and the error is visible in several panels. Same result in other dashboards.

Again, the 'logstash-*' index shows results in the the Discover tab so it is unclear why these dashboards are producing an error stating that they could not locate the same index pattern that is selected in the Discover tab.

pevma commented 5 years ago

On 19 Nov 2018, at 02:13, alphaDev23 notifications@github.com wrote:

There is not much of a background. Selected a KTS6 dashboard, e.g. SN-ALL, and the error is visible in several panels. Same result in other dashboards.

Again, the 'logstash-*' index shows results in the the Discover tab so it is unclear why these dashboards are producing an error stating that they could not locate the same index pattern that is selected in the Discover tab.

Is this a fresh import ? Then you need to select default index in Kibana first.(logstash-*)

— You are receiving this because you commented. Reply to this email directly, view it on GitHub, or mute the thread.

alphaDev23 commented 5 years ago

The default index, logstash-, was created and selected prior to opening the dashboard. I re-loaded the templates (./load.sh) to reinitialize and was required to reset the default index, which I did, to the same, logstash-

The same error, 'Could not locate that index-pattern (id: logstash-*), click here to re-create it ' is still reported after the above.

I then recreated the logstash-* index (it says "click here to recreate" in the dashboard even though there is no place to click here) and that did not work.

pevma commented 5 years ago

What is your default index in Kibana?

alphaDev23 commented 5 years ago

As noted above, it is:
logstash-*

pevma commented 5 years ago

Which ELK stack are you using ? I am not sure i understand - you mention the default index is already set logstash-* , but Kibana can not find it? sounds strange.

alphaDev23 commented 5 years ago

6.3.2. As noted above, the Discover tab in Kibana shows documents indexed under the 'logstash-*' index. It is the dashboards that are producing the error.

pevma commented 5 years ago

I maybe missing some information - I don’t see 6.3.2 version mentioned in any of your previous messages?

And you are tuning the latest KTS6 revision ?(just double checking ).

Do you have a similar problem with Kibana 6.4.x or 6.5 ?

-- Regards, Peter Manev

On 20 Nov 2018, at 17:31, alphaDev23 notifications@github.com wrote:

6.3.2. As noted above, the Discover tab in Kibana shows documents indexed under the 'logstash-*' index. It is the dashboards that are producing the error.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub, or mute the thread.

alphaDev23 commented 5 years ago

Given that the KTS6 templates are being loaded, and there is only a master branch and no tags in this repository, does it matter which version of Kibana 6 I'm running? Also, it is actually Kibana version 6.2.3 as there is a typo in my previous reply.

I'm using the following commit of KTS6: commit a8c8ff890739c8e7de95dda0b3a8a7e64c302c3f Author: Peter Manev pmanev@stamus-networks.com Date: Fri Nov 9 04:13:38 2018 -0800

dashboards: Adjust time span for SN-TLS to the default "now-24hr"

Were there changes between that commit and the latest which may have affected the issue?

I would prefer not to upgrade to 6.4 and 6.5 because that then requires upgrades to shippers such as filebeat, Is there a change between 6.2 and 6.4/5 that would affect the issue?

pevma commented 5 years ago

Thank you for confirming - i wanted to make sure you are on the latest commit.

I tried to reproduce your issue on Kibana 6.5 - and could not. I have not tested import on every single Kibana version from 6.x.x to the current 6.5 but have not experienced or am aware of similar err like you are getting on 6.3/4.x - hence suspecting it may be related to the Kibana version or something with the set up.

Is there anything specific to your set up? (or is it similar to the one in SELKS - ELK stack on the same machine etc...)

pevma commented 5 years ago

Adding onto that - KTS6 would most likely need some logstash template like that here - https://github.com/StamusNetworks/SELKS/blob/SELKS5/staging/etc/logstash/conf.d/logstash.conf

alphaDev23 commented 5 years ago

Upgrading to 6.4.2 resolved the issue. Thank you. Your suggestions were helpful in resolving the issue.

alphaDev23 commented 5 years ago

The root cause of the issue was not the Kibana version (although it may be related but I did not retest on the previous version) but rather that the 'Custom Index Pattern' under advanced options when creating the 'logstash-' index also needs to be set as 'logstash-.' Otherwise a UUID will be created for the index resulting in the dashboards not recognizing the index.

Please update the README file because the documentation only states, "You would need to select logstash-* as a default index once you open any dashboard for the first time after initial load/import.", and does not state that it also needs to be set in the advanced options during index creation.

pevma commented 5 years ago

Can you please list the exact steps you followed to make it work in your set up ?

alphaDev23 commented 5 years ago

While adding the 'logstash-' index I selected 'Advanced Options' (in the 2nd step where the time filter is added) and entered 'logstash-' into the Custom Index Pattern.

Marshal27 commented 5 years ago

I just ran into this issue, not sure I fully understand what @alphaDev23 did to resolve the issue.

I used the load.sh per the instructions, I see the list in kibana, when I attempt to select logstash-* per the installation instructions, I receive the following message in a toast lower right corner.

Saved object is missing

Could not locate that index-pattern (id: index-patternlogstash-), click here to re-create it

I click re-create it, and nothing happens.

pevma commented 5 years ago

Which Kibana / ELK stack version is that ? Can you share a screenshot ?

-- Regards, Peter Manev

On 17 Jan 2019, at 05:40, Marshal27 notifications@github.com wrote:

I just ran into this issue, not sure I fully understand what @alphaDev23 did to resolve the issue.

I used the load.sh per the instructions, I see the list in kibana, when I attempt to select logstash-* per the installation instructions, I receive the following message in a toast lower right corner.

Saved object is missing

Could not locate that index-pattern (id: index-patternlogstash-), click here to re-create it

I click re-create it, and nothing happens.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub, or mute the thread.

Marshal27 commented 5 years ago

This was user error on my part... I am new to the ELK stack, the sincedb piece is what was causing me issues... I imported your templates after ensuring logstash parsed correctly and created the indexes in elasticsearch... after this, I was able to select the default index per your instructions.... in my scenario, I did not have the underlying indexes/data correct and is what caused my issue.