search

Graylog2 / graylog2-server

Free and open log management
https://www.graylog.org
Other
7.37k stars 1.06k forks source link

Regress in queries in Graylog 3.0 #5694

Open Nklya opened 5 years ago

Nklya commented 5 years ago

I think there might be regress in queries in Graylog 3.0. I updated my Graylog setup from Graylog 2.5.1/Elasticsearch 5.6 to Graylog 3.0/Elasticsearch 6.6 and discovered a problem with queries.

Expected Behavior

According to help, I can use parentheses in queries to find any of word http://docs.graylog.org/en/3.0/pages/queries.html#syntax

For example NOT user:(gitlab-runner jenkins git). This worked as expected on Graylog 2.5.1 and doesn't on 3.0. For now I had to change query to NOT (user:gitlab-runner OR user:git OR user:jenkins)

Current Behavior

Query NOT user:(gitlab-runner jenkins git) return all items.

Possible Solution

Change query to NOT (user:gitlab-runner OR user:git OR user:jenkins)

Steps to Reproduce (for bugs)

  1. Run query with any_field: (value1 value2)
  2. It doesn't work in ver 3.0
  3. Change query to any_field:value1 OR any_field:value2 4.It works

Context

I used these queries to filter some events from alerts. I had to change them to more complicated like NOT (user:gitlab-runner OR user:git OR user:jenkins)

Your Environment

P.S. I haven't found related issues

edmundoa commented 5 years ago

I could reproduce the issue in Elasticsearch 6.5.1, but not on 5.6.13. I guess there was some change in behaviour we weren't aware of.

edmundoa commented 5 years ago

We managed to find out why this behaviour changed:

The split_on_whitespace parameter for the query_string query has been removed. If provided, it will be ignored and issue a deprecation warning. The query_string query now splits on operator only.

Extract from the ES 6.0 breaking changes: https://www.elastic.co/guide/en/elasticsearch/reference/6.6/breaking-changes-6.0.html

This was confirmed in https://github.com/elastic/elasticsearch/issues/31297#issuecomment-401238569, which also links to a new way of getting the same result that got introduced after the change. Unfortunately it needs to be set in index templates, so we would need to update Graylog's templates to be able to workaround the problem.

To summarize:

Nklya commented 5 years ago

Thanks! I already use OR to workaround this.

dennisoelkers commented 5 years ago

@edmundoa: Are there any follow up tasks for this issue or can it be closed?

edmundoa commented 5 years ago

@dennisoelkers as far as I know only the last point remains:

We should consider modifying our ES templates to include the new split_queries_on_whitespace option, or try to find another way to make old queries keep their old behaviour.

dennisoelkers commented 5 years ago

Should we create a new issue for that or modify the scope of this one?