Elasticsearch exception: Failed to parse query

[frantz45]

Sometimes an exception is throwed as you can see below. Graylog version : 4.1.6 Plugin version : 4.0.0

2021-11-04T17:19:23.249+01:00 ERROR [JobExecutionEngine] Unhandled job execution error - trigger=6184080b2f904c5cd9dc731e job=6183fea41c042859f953bd0b
org.graylog.shaded.elasticsearch7.org.elasticsearch.ElasticsearchException: Unable to perform search query
    at org.graylog.storage.elasticsearch7.ElasticsearchClient.exceptionFrom(ElasticsearchClient.java:136) ~[?:?]
    at org.graylog.storage.elasticsearch7.ElasticsearchClient.firstResponseFrom(ElasticsearchClient.java:85) ~[?:?]
    at org.graylog.storage.elasticsearch7.ElasticsearchClient.search(ElasticsearchClient.java:61) ~[?:?]
    at org.graylog.storage.elasticsearch7.SearchesAdapterES7.search(SearchesAdapterES7.java:122) ~[?:?]
    at org.graylog2.indexer.searches.Searches.search(Searches.java:156) ~[graylog.jar:?]
    at org.graylog2.indexer.searches.Searches.search(Searches.java:149) ~[graylog.jar:?]
    at com.airbus_cyber_security.graylog.events.notifications.types.LoggingAlertUtils.getAggregationAlertID(LoggingAlertUtils.java:108) ~[?:?]
    at com.airbus_cyber_security.graylog.events.notifications.types.LoggingAlertUtils.getListOfLoggingAlertField(LoggingAlertUtils.java:254) ~[?:?]
    at com.airbus_cyber_security.graylog.events.notifications.types.LoggingAlert.execute(LoggingAlert.java:112) ~[?:?]
    at org.graylog.events.notifications.EventNotificationExecutionJob.execute(EventNotificationExecutionJob.java:135) ~[graylog.jar:?]
    at org.graylog.scheduler.JobExecutionEngine.executeJob(JobExecutionEngine.java:166) ~[graylog.jar:?]
    at org.graylog.scheduler.JobExecutionEngine.lambda$handleTrigger$2(JobExecutionEngine.java:144) ~[graylog.jar:?]
    at com.codahale.metrics.Timer.time(Timer.java:151) ~[graylog.jar:?]
    at org.graylog.scheduler.JobExecutionEngine.handleTrigger(JobExecutionEngine.java:144) ~[graylog.jar:?]
    at org.graylog.scheduler.JobExecutionEngine.lambda$execute$0(JobExecutionEngine.java:119) ~[graylog.jar:?]
    at org.graylog.scheduler.worker.JobWorkerPool.lambda$execute$0(JobWorkerPool.java:110) ~[graylog.jar:?]
    at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:180) [graylog.jar:?]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_262]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_262]
    at com.codahale.metrics.InstrumentedThreadFactory$InstrumentedRunnable.run(InstrumentedThreadFactory.java:66) [graylog.jar:?]
    at java.lang.Thread.run(Thread.java:748) [?:1.8.0_262]
Caused by: org.graylog.shaded.elasticsearch7.org.elasticsearch.ElasticsearchException: Elasticsearch exception [type=search_phase_execution_exception, reason=all shards failed]
    at org.graylog.shaded.elasticsearch7.org.elasticsearch.ElasticsearchException.innerFromXContent(ElasticsearchException.java:496) ~[?:?]
    at org.graylog.shaded.elasticsearch7.org.elasticsearch.ElasticsearchException.failureFromXContent(ElasticsearchException.java:603) ~[?:?]
    at org.graylog.shaded.elasticsearch7.org.elasticsearch.action.search.MultiSearchResponse.itemFromXContent(MultiSearchResponse.java:215) ~[?:?]
    at org.graylog.shaded.elasticsearch7.org.elasticsearch.action.search.MultiSearchResponse.lambda$static$1(MultiSearchResponse.java:56) ~[?:?]
    at org.graylog.shaded.elasticsearch7.org.elasticsearch.common.xcontent.AbstractObjectParser.lambda$declareObjectArray$13(AbstractObjectParser.java:254) ~[?:?]
    at org.graylog.shaded.elasticsearch7.org.elasticsearch.common.xcontent.AbstractObjectParser.lambda$declareFieldArray$22(AbstractObjectParser.java:300) ~[?:?]
    at org.graylog.shaded.elasticsearch7.org.elasticsearch.common.xcontent.AbstractObjectParser.parseArray(AbstractObjectParser.java:382) ~[?:?]
    at org.graylog.shaded.elasticsearch7.org.elasticsearch.common.xcontent.AbstractObjectParser.lambda$declareFieldArray$23(AbstractObjectParser.java:300) ~[?:?]
    at org.graylog.shaded.elasticsearch7.org.elasticsearch.common.xcontent.ObjectParser.lambda$declareField$9(ObjectParser.java:386) ~[?:?]
    at org.graylog.shaded.elasticsearch7.org.elasticsearch.common.xcontent.ObjectParser.parseValue(ObjectParser.java:529) ~[?:?]
    at org.graylog.shaded.elasticsearch7.org.elasticsearch.common.xcontent.ObjectParser.parseArray(ObjectParser.java:523) ~[?:?]
    at org.graylog.shaded.elasticsearch7.org.elasticsearch.common.xcontent.ObjectParser.parseSub(ObjectParser.java:555) ~[?:?]
    at org.graylog.shaded.elasticsearch7.org.elasticsearch.common.xcontent.ObjectParser.parse(ObjectParser.java:324) ~[?:?]
    at org.graylog.shaded.elasticsearch7.org.elasticsearch.common.xcontent.ConstructingObjectParser.parse(ConstructingObjectParser.java:171) ~[?:?]
    at org.graylog.shaded.elasticsearch7.org.elasticsearch.common.xcontent.ConstructingObjectParser.apply(ConstructingObjectParser.java:163) ~[?:?]
    at org.graylog.shaded.elasticsearch7.org.elasticsearch.action.search.MultiSearchResponse.fromXContext(MultiSearchResponse.java:194) ~[?:?]
    at org.graylog.shaded.elasticsearch7.org.elasticsearch.client.RestHighLevelClient.parseEntity(RestHighLevelClient.java:1892) ~[?:?]
    at org.graylog.shaded.elasticsearch7.org.elasticsearch.client.RestHighLevelClient.lambda$performRequestAndParseEntity$8(RestHighLevelClient.java:1554) ~[?:?]
    at org.graylog.shaded.elasticsearch7.org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1630) ~[?:?]
    at org.graylog.shaded.elasticsearch7.org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:1583) ~[?:?]
    at org.graylog.shaded.elasticsearch7.org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:1553) ~[?:?]
    at org.graylog.shaded.elasticsearch7.org.elasticsearch.client.RestHighLevelClient.msearch(RestHighLevelClient.java:1118) ~[?:?]
    at org.graylog.storage.elasticsearch7.ElasticsearchClient.lambda$search$0(ElasticsearchClient.java:59) ~[?:?]
    at org.graylog.storage.elasticsearch7.ElasticsearchClient.execute(ElasticsearchClient.java:97) ~[?:?]
    at org.graylog.storage.elasticsearch7.ElasticsearchClient.search(ElasticsearchClient.java:59) ~[?:?]
    ... 18 more
    Suppressed: org.graylog.shaded.elasticsearch7.org.elasticsearch.ElasticsearchException: Elasticsearch exception [type=query_shard_exception, reason=Failed to parse query [streams:6183fe751c042859f953b84aalert_id:(01FKNVCGP742626JSEFX4Q8K79-838939282 OR 01FKNTT6ZT60CAGFW27RSQDW1G-838939282 OR 01FKNT7X5HW1ZJVPJE0WD6A7X6-838939282)]]
        at org.graylog.shaded.elasticsearch7.org.elasticsearch.ElasticsearchException.innerFromXContent(ElasticsearchException.java:496) ~[?:?]
        at org.graylog.shaded.elasticsearch7.org.elasticsearch.ElasticsearchException.fromXContent(ElasticsearchException.java:407) ~[?:?]
        at org.graylog.shaded.elasticsearch7.org.elasticsearch.ElasticsearchException.innerFromXContent(ElasticsearchException.java:469) ~[?:?]
        at org.graylog.shaded.elasticsearch7.org.elasticsearch.ElasticsearchException.failureFromXContent(ElasticsearchException.java:603) ~[?:?]
        at org.graylog.shaded.elasticsearch7.org.elasticsearch.action.search.MultiSearchResponse.itemFromXContent(MultiSearchResponse.java:215) ~[?:?]
        at org.graylog.shaded.elasticsearch7.org.elasticsearch.action.search.MultiSearchResponse.lambda$static$1(MultiSearchResponse.java:56) ~[?:?]
        at org.graylog.shaded.elasticsearch7.org.elasticsearch.common.xcontent.AbstractObjectParser.lambda$declareObjectArray$13(AbstractObjectParser.java:254) ~[?:?]
        at org.graylog.shaded.elasticsearch7.org.elasticsearch.common.xcontent.AbstractObjectParser.lambda$declareFieldArray$22(AbstractObjectParser.java:300) ~[?:?]
        at org.graylog.shaded.elasticsearch7.org.elasticsearch.common.xcontent.AbstractObjectParser.parseArray(AbstractObjectParser.java:382) ~[?:?]
        at org.graylog.shaded.elasticsearch7.org.elasticsearch.common.xcontent.AbstractObjectParser.lambda$declareFieldArray$23(AbstractObjectParser.java:300) ~[?:?]
        at org.graylog.shaded.elasticsearch7.org.elasticsearch.common.xcontent.ObjectParser.lambda$declareField$9(ObjectParser.java:386) ~[?:?]
        at org.graylog.shaded.elasticsearch7.org.elasticsearch.common.xcontent.ObjectParser.parseValue(ObjectParser.java:529) ~[?:?]
        at org.graylog.shaded.elasticsearch7.org.elasticsearch.common.xcontent.ObjectParser.parseArray(ObjectParser.java:523) ~[?:?]
        at org.graylog.shaded.elasticsearch7.org.elasticsearch.common.xcontent.ObjectParser.parseSub(ObjectParser.java:555) ~[?:?]
        at org.graylog.shaded.elasticsearch7.org.elasticsearch.common.xcontent.ObjectParser.parse(ObjectParser.java:324) ~[?:?]
        at org.graylog.shaded.elasticsearch7.org.elasticsearch.common.xcontent.ConstructingObjectParser.parse(ConstructingObjectParser.java:171) ~[?:?]
        at org.graylog.shaded.elasticsearch7.org.elasticsearch.common.xcontent.ConstructingObjectParser.apply(ConstructingObjectParser.java:163) ~[?:?]
        at org.graylog.shaded.elasticsearch7.org.elasticsearch.action.search.MultiSearchResponse.fromXContext(MultiSearchResponse.java:194) ~[?:?]
        at org.graylog.shaded.elasticsearch7.org.elasticsearch.client.RestHighLevelClient.parseEntity(RestHighLevelClient.java:1892) ~[?:?]
        at org.graylog.shaded.elasticsearch7.org.elasticsearch.client.RestHighLevelClient.lambda$performRequestAndParseEntity$8(RestHighLevelClient.java:1554) ~[?:?]
        at org.graylog.shaded.elasticsearch7.org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1630) ~[?:?]
        at org.graylog.shaded.elasticsearch7.org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:1583) ~[?:?]
        at org.graylog.shaded.elasticsearch7.org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:1553) ~[?:?]
        at org.graylog.shaded.elasticsearch7.org.elasticsearch.client.RestHighLevelClient.msearch(RestHighLevelClient.java:1118) ~[?:?]
        at org.graylog.storage.elasticsearch7.ElasticsearchClient.lambda$search$0(ElasticsearchClient.java:59) ~[?:?]
        at org.graylog.storage.elasticsearch7.ElasticsearchClient.execute(ElasticsearchClient.java:97) ~[?:?]
        at org.graylog.storage.elasticsearch7.ElasticsearchClient.search(ElasticsearchClient.java:59) ~[?:?]
        at org.graylog.storage.elasticsearch7.SearchesAdapterES7.search(SearchesAdapterES7.java:122) ~[?:?]
        at org.graylog2.indexer.searches.Searches.search(Searches.java:156) ~[graylog.jar:?]
        at org.graylog2.indexer.searches.Searches.search(Searches.java:149) ~[graylog.jar:?]
        at com.airbus_cyber_security.graylog.events.notifications.types.LoggingAlertUtils.getAggregationAlertID(LoggingAlertUtils.java:108) ~[?:?]
        at com.airbus_cyber_security.graylog.events.notifications.types.LoggingAlertUtils.getListOfLoggingAlertField(LoggingAlertUtils.java:254) ~[?:?]
        at com.airbus_cyber_security.graylog.events.notifications.types.LoggingAlert.execute(LoggingAlert.java:112) ~[?:?]
        at org.graylog.events.notifications.EventNotificationExecutionJob.execute(EventNotificationExecutionJob.java:135) ~[graylog.jar:?]
        at org.graylog.scheduler.JobExecutionEngine.executeJob(JobExecutionEngine.java:166) ~[graylog.jar:?]
        at org.graylog.scheduler.JobExecutionEngine.lambda$handleTrigger$2(JobExecutionEngine.java:144) ~[graylog.jar:?]
        at com.codahale.metrics.Timer.time(Timer.java:151) ~[graylog.jar:?]
        at org.graylog.scheduler.JobExecutionEngine.handleTrigger(JobExecutionEngine.java:144) ~[graylog.jar:?]
        at org.graylog.scheduler.JobExecutionEngine.lambda$execute$0(JobExecutionEngine.java:119) ~[graylog.jar:?]
        at org.graylog.scheduler.worker.JobWorkerPool.lambda$execute$0(JobWorkerPool.java:110) ~[graylog.jar:?]
        at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:180) [graylog.jar:?]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_262]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_262]
        at com.codahale.metrics.InstrumentedThreadFactory$InstrumentedRunnable.run(InstrumentedThreadFactory.java:66) [graylog.jar:?]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_262]

Indeed the ES query is wrong: streams:6183fe751c042859f953b84aalert_id:(01FKNVCGP742626JSEFX4Q8K79-838939282 OR 01FKNTT6ZT60CAGFW27RSQDW1G-838939282 OR 01FKNT7X5HW1ZJVPJE0WD6A7X6-838939282)

You need to add some spaces and the AND keyword: streams:6183fe751c042859f953b84a AND alert_id:(01FKNVCGP742626JSEFX4Q8K79-838939282 OR 01FKNTT6ZT60CAGFW27RSQDW1G-838939282 OR 01FKNT7X5HW1ZJVPJE0WD6A7X6-838939282)

[c8y3]

Scripted scenario to reproduce the issue in python

scenario_issue30.py

[c8y3]

TODO

• convert the scenario into a backend test (simplify even further if possible)
• in the workflow add a job to execute backend tests and set the release job after them
• bump 4.0.1 version and enrich changelog

[frantz45]

I confirm it's fixed in v4.0.0. I let you close the issue after you did the "TODO"