Closed frantz45 closed 1 year ago
Reproduced once on Graylog v4.2.5 with Wizard v4.1.0.
2022-02-28T11:04:28.628Z INFO [AggregationField] It seems there are two results with the same key. Listing all results...
2022-02-28T11:04:28.628Z INFO [AggregationField] key: 192.168.1.198 - 192.168.1.254 ->
2022-02-28T11:04:28.628Z INFO [AggregationField] value: 2
2022-02-28T11:04:28.628Z INFO [AggregationField] key: 192.168.1.190 - 192.168.1.254 ->
2022-02-28T11:04:28.628Z INFO [AggregationField] value: 1
2022-02-28T11:04:28.628Z INFO [AggregationField] key: 192.168.1.191 - 192.168.1.254 ->
2022-02-28T11:04:28.628Z INFO [AggregationField] value: 1
2022-02-28T11:04:28.628Z INFO [AggregationField] key: 192.168.1.192 - 192.168.1.254 ->
2022-02-28T11:04:28.628Z INFO [AggregationField] value: 1
2022-02-28T11:04:28.628Z INFO [AggregationField] key: 192.168.1.197 - 192.168.1.254 ->
2022-02-28T11:04:28.628Z INFO [AggregationField] value: 1
2022-02-28T11:04:28.628Z INFO [AggregationField] key: 192.168.1.198 - 192.168.1.254 ->
2022-02-28T11:04:28.628Z INFO [AggregationField] value: 2
2022-02-28T11:04:28.628Z INFO [AggregationField] key: 192.168.1.190 - 192.168.1.254 ->
2022-02-28T11:04:28.628Z INFO [AggregationField] value: 1
2022-02-28T11:04:28.628Z INFO [AggregationField] key: 192.168.1.191 - 192.168.1.254 ->
2022-02-28T11:04:28.628Z INFO [AggregationField] value: 1
2022-02-28T11:04:28.628Z INFO [AggregationField] key: 192.168.1.192 - 192.168.1.254 ->
2022-02-28T11:04:28.628Z INFO [AggregationField] value: 1
2022-02-28T11:04:28.628Z INFO [AggregationField] key: 192.168.1.197 - 192.168.1.254 ->
2022-02-28T11:04:28.628Z INFO [AggregationField] value: 1
2022-02-28T11:04:28.628Z INFO [AggregationField] key: 192.168.1.198 - 192.168.1.254 ->
2022-02-28T11:04:28.628Z INFO [AggregationField] value: 2
2022-02-28T11:04:28.628Z INFO [AggregationField] key: 192.168.1.190 - 192.168.1.254 ->
2022-02-28T11:04:28.628Z INFO [AggregationField] value: 1
2022-02-28T11:04:28.628Z INFO [AggregationField] key: 192.168.1.191 - 192.168.1.254 ->
2022-02-28T11:04:28.628Z INFO [AggregationField] value: 1
2022-02-28T11:04:28.628Z INFO [AggregationField] key: 192.168.1.192 - 192.168.1.254 ->
2022-02-28T11:04:28.628Z INFO [AggregationField] value: 1
2022-02-28T11:04:28.628Z INFO [AggregationField] key: 192.168.1.197 - 192.168.1.254 ->
2022-02-28T11:04:28.628Z INFO [AggregationField] value: 1
2022-02-28T11:04:28.628Z INFO [AggregationField] key: 192.168.1.198 - 192.168.1.254 ->
2022-02-28T11:04:28.628Z INFO [AggregationField] value: 2
2022-02-28T11:04:28.628Z INFO [AggregationField] key: 192.168.1.190 - 192.168.1.254 ->
2022-02-28T11:04:28.628Z INFO [AggregationField] value: 1
2022-02-28T11:04:28.628Z INFO [AggregationField] key: 192.168.1.191 - 192.168.1.254 ->
2022-02-28T11:04:28.628Z INFO [AggregationField] value: 1
2022-02-28T11:04:28.628Z INFO [AggregationField] key: 192.168.1.192 - 192.168.1.254 ->
2022-02-28T11:04:28.628Z INFO [AggregationField] value: 1
2022-02-28T11:04:28.628Z INFO [AggregationField] key: 192.168.1.197 - 192.168.1.254 ->
2022-02-28T11:04:28.628Z INFO [AggregationField] value: 1
2022-02-28T11:04:28.628Z INFO [AggregationField] key: 192.168.1.198 - 192.168.1.254 ->
2022-02-28T11:04:28.628Z INFO [AggregationField] value: 2
2022-02-28T11:04:28.628Z INFO [AggregationField] key: 192.168.1.190 - 192.168.1.254 ->
2022-02-28T11:04:28.628Z INFO [AggregationField] value: 1
2022-02-28T11:04:28.628Z INFO [AggregationField] key: 192.168.1.191 - 192.168.1.254 ->
2022-02-28T11:04:28.628Z INFO [AggregationField] value: 1
2022-02-28T11:04:28.628Z INFO [AggregationField] key: 192.168.1.192 - 192.168.1.254 ->
2022-02-28T11:04:28.628Z INFO [AggregationField] value: 1
2022-02-28T11:04:28.628Z INFO [AggregationField] key: 192.168.1.197 - 192.168.1.254 ->
2022-02-28T11:04:28.628Z INFO [AggregationField] value: 1
2022-02-28T11:04:28.628Z INFO [AggregationField] key: 192.168.1.198 - 192.168.1.254 ->
2022-02-28T11:04:28.628Z INFO [AggregationField] value: 2
2022-02-28T11:04:28.628Z INFO [AggregationField] key: 192.168.1.190 - 192.168.1.254 ->
2022-02-28T11:04:28.628Z INFO [AggregationField] value: 1
2022-02-28T11:04:28.628Z INFO [AggregationField] key: 192.168.1.191 - 192.168.1.254 ->
2022-02-28T11:04:28.628Z INFO [AggregationField] value: 1
2022-02-28T11:04:28.628Z INFO [AggregationField] key: 192.168.1.192 - 192.168.1.254 ->
2022-02-28T11:04:28.628Z INFO [AggregationField] value: 1
2022-02-28T11:04:28.628Z INFO [AggregationField] key: 192.168.1.197 - 192.168.1.254 ->
2022-02-28T11:04:28.628Z INFO [AggregationField] value: 1
2022-02-28T11:04:28.628Z INFO [AggregationField] key: 192.168.1.198 - 192.168.1.254 ->
2022-02-28T11:04:28.628Z INFO [AggregationField] value: 2
2022-02-28T11:04:28.628Z INFO [AggregationField] key: 192.168.1.190 - 192.168.1.254 ->
2022-02-28T11:04:28.628Z INFO [AggregationField] value: 1
2022-02-28T11:04:28.628Z INFO [AggregationField] key: 192.168.1.191 - 192.168.1.254 ->
2022-02-28T11:04:28.628Z INFO [AggregationField] value: 1
2022-02-28T11:04:28.628Z INFO [AggregationField] key: 192.168.1.192 - 192.168.1.254 ->
2022-02-28T11:04:28.628Z INFO [AggregationField] value: 1
2022-02-28T11:04:28.628Z INFO [AggregationField] key: 192.168.1.197 - 192.168.1.254 ->
2022-02-28T11:04:28.628Z INFO [AggregationField] value: 1
2022-02-28T11:04:28.628Z INFO [AggregationField] key: 192.168.1.190 - 192.168.1.254 ->
2022-02-28T11:04:28.628Z INFO [AggregationField] value: 1
2022-02-28T11:04:28.628Z INFO [AggregationField] key: 192.168.1.191 - 192.168.1.254 ->
2022-02-28T11:04:28.628Z INFO [AggregationField] value: 1
2022-02-28T11:04:28.628Z INFO [AggregationField] key: 192.168.1.192 - 192.168.1.254 ->
2022-02-28T11:04:28.628Z INFO [AggregationField] value: 1
2022-02-28T11:04:28.628Z INFO [AggregationField] key: 192.168.1.198 - 192.168.1.254 ->
2022-02-28T11:04:28.628Z INFO [AggregationField] value: 1
2022-02-28T11:04:28.628Z INFO [AggregationField] key: 192.168.1.190 - 192.168.1.254 ->
2022-02-28T11:04:28.628Z INFO [AggregationField] value: 1
2022-02-28T11:04:28.628Z INFO [AggregationField] key: 192.168.1.191 - 192.168.1.254 ->
2022-02-28T11:04:28.628Z INFO [AggregationField] value: 1
2022-02-28T11:04:28.628Z INFO [AggregationField] key: 192.168.1.192 - 192.168.1.254 ->
2022-02-28T11:04:28.628Z INFO [AggregationField] value: 1
2022-02-28T11:04:28.628Z INFO [AggregationField] key: 192.168.1.198 - 192.168.1.254 ->
2022-02-28T11:04:28.628Z INFO [AggregationField] value: 1
2022-02-28T11:04:28.628Z INFO [AggregationField] key: 192.168.1.190 - 192.168.1.254 ->
2022-02-28T11:04:28.628Z INFO [AggregationField] value: 1
2022-02-28T11:04:28.628Z INFO [AggregationField] key: 192.168.1.191 - 192.168.1.254 ->
2022-02-28T11:04:28.628Z INFO [AggregationField] value: 1
2022-02-28T11:04:28.628Z INFO [AggregationField] key: 192.168.1.192 - 192.168.1.254 ->
2022-02-28T11:04:28.628Z INFO [AggregationField] value: 1
2022-02-28T11:04:28.628Z INFO [AggregationField] key: 192.168.1.198 - 192.168.1.254 ->
2022-02-28T11:04:28.628Z INFO [AggregationField] value: 1
2022-02-28T11:04:28.628Z INFO [AggregationField] key: 192.168.1.190 - 192.168.1.254 ->
2022-02-28T11:04:28.628Z INFO [AggregationField] value: 1
2022-02-28T11:04:28.628Z INFO [AggregationField] key: 192.168.1.191 - 192.168.1.254 ->
2022-02-28T11:04:28.628Z INFO [AggregationField] value: 1
2022-02-28T11:04:28.628Z INFO [AggregationField] key: 192.168.1.192 - 192.168.1.254 ->
2022-02-28T11:04:28.628Z INFO [AggregationField] value: 1
2022-02-28T11:04:28.628Z INFO [AggregationField] key: 192.168.1.198 - 192.168.1.254 ->
2022-02-28T11:04:28.628Z INFO [AggregationField] value: 1
2022-02-28T11:04:28.628Z INFO [AggregationField] key: 192.168.1.190 - 192.168.1.254 ->
2022-02-28T11:04:28.628Z INFO [AggregationField] value: 1
2022-02-28T11:04:28.628Z INFO [AggregationField] key: 192.168.1.191 - 192.168.1.254 ->
2022-02-28T11:04:28.628Z INFO [AggregationField] value: 1
2022-02-28T11:04:28.628Z INFO [AggregationField] key: 192.168.1.192 - 192.168.1.254 ->
2022-02-28T11:04:28.628Z INFO [AggregationField] value: 1
2022-02-28T11:04:28.628Z INFO [AggregationField] key: 192.168.1.190 - 192.168.1.254 ->
2022-02-28T11:04:28.628Z INFO [AggregationField] value: 1
2022-02-28T11:04:28.628Z INFO [AggregationField] key: 192.168.1.191 - 192.168.1.254 ->
2022-02-28T11:04:28.628Z INFO [AggregationField] value: 1
2022-02-28T11:04:28.628Z INFO [AggregationField] key: 192.168.1.192 - 192.168.1.254 ->
2022-02-28T11:04:28.628Z INFO [AggregationField] value: 1
2022-02-28T11:04:28.628Z INFO [AggregationField] key: 192.168.1.190 - 192.168.1.254 ->
2022-02-28T11:04:28.628Z INFO [AggregationField] value: 1
2022-02-28T11:04:28.628Z INFO [AggregationField] key: 192.168.1.191 - 192.168.1.254 ->
2022-02-28T11:04:28.628Z INFO [AggregationField] value: 1
2022-02-28T11:04:28.628Z INFO [AggregationField] key: 192.168.1.192 - 192.168.1.254 ->
2022-02-28T11:04:28.629Z INFO [AggregationField] value: 1
2022-02-28T11:04:28.629Z ERROR [EventProcessorEngine] Caught an unhandled exception while executing event processor <aggregation-count/Multiple ping attacks from the same host on multiple destinations/620cead637bdee594691295c> - Make sure to modify the event processor to throw only EventProcessorExecutionException so we get more context!
java.lang.IllegalArgumentException: Multiple entries with same key: 192.168.1.198 - 192.168.1.254=2 and 192.168.1.198 - 192.168.1.254=2
at com.google.common.collect.ImmutableMap.conflictException(ImmutableMap.java:216) ~[graylog.jar:?]
at com.google.common.collect.ImmutableMap.checkNoConflict(ImmutableMap.java:210) ~[graylog.jar:?]
at com.google.common.collect.RegularImmutableMap.checkNoConflictInKeyBucket(RegularImmutableMap.java:146) ~[graylog.jar:?]
at com.google.common.collect.RegularImmutableMap.fromEntryArray(RegularImmutableMap.java:109) ~[graylog.jar:?]
at com.google.common.collect.ImmutableMap$Builder.build(ImmutableMap.java:395) ~[graylog.jar:?]
at com.airbus_cyber_security.graylog.events.processor.aggregation.checks.AggregationField.convertResult(AggregationField.java:220) ~[?:?]
at com.airbus_cyber_security.graylog.events.processor.aggregation.checks.AggregationField.getTermsResult(AggregationField.java:200) ~[?:?]
at com.airbus_cyber_security.graylog.events.processor.aggregation.checks.AggregationField.run(AggregationField.java:266) ~[?:?]
at com.airbus_cyber_security.graylog.events.processor.aggregation.checks.AggregationCount.runCheck(AggregationCount.java:55) ~[?:?]
at com.airbus_cyber_security.graylog.events.processor.aggregation.AggregationCountProcessor.createEvents(AggregationCountProcessor.java:79) ~[?:?]
at org.graylog.events.processor.EventProcessorEngine.execute(EventProcessorEngine.java:92) ~[graylog.jar:?]
at org.graylog.events.processor.EventProcessorExecutionJob.execute(EventProcessorExecutionJob.java:115) ~[graylog.jar:?]
at org.graylog.scheduler.JobExecutionEngine.executeJob(JobExecutionEngine.java:166) ~[graylog.jar:?]
at org.graylog.scheduler.JobExecutionEngine.lambda$handleTrigger$2(JobExecutionEngine.java:144) ~[graylog.jar:?]
at com.codahale.metrics.Timer.time(Timer.java:151) ~[graylog.jar:?]
at org.graylog.scheduler.JobExecutionEngine.handleTrigger(JobExecutionEngine.java:144) ~[graylog.jar:?]
at org.graylog.scheduler.JobExecutionEngine.lambda$execute$0(JobExecutionEngine.java:119) ~[graylog.jar:?]
at org.graylog.scheduler.worker.JobWorkerPool.lambda$execute$0(JobWorkerPool.java:110) ~[graylog.jar:?]
at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:180) [graylog.jar:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_312]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_312]
at com.codahale.metrics.InstrumentedThreadFactory$InstrumentedRunnable.run(InstrumentedThreadFactory.java:66) [graylog.jar:?]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_312]
2022-02-28T11:04:28.629Z ERROR [EventProcessorExecutionJob] Event processor <aggregation-count/620cead637bdee594691295c> failed to execute: Couldn't create events for: EventDefinitionDto{id=620cead637bdee594691295c, title=Multiple ping attacks from the same host on multiple destinations, description=Generated by the alert wizard, priority=2, alert=true, config=AggregationCountProcessorConfig{type=aggregation-count, stream=620cead637bdee5946912956, thresholdType=MORE, threshold=4, groupingFields=[src_ip], distinctionFields=[dest_ip], comment=Generated by the alert wizard, searchQuery=*, searchWithinMs=900000, executeEveryMs=60000}, fieldSpec={}, keySpec=[], notificationSettings=EventNotificationSettings{gracePeriodMs=0, backlogSize=500}, notifications=[Config{notificationId=620cead637bdee594691295a, notificationParameters=Optional.empty}], storage=[Config{type=persist-to-streams-v1, streams=[000000000000000000000002]}]} (retry in 5000 ms)
org.graylog.events.processor.EventProcessorException: Couldn't create events for: EventDefinitionDto{id=620cead637bdee594691295c, title=Multiple ping attacks from the same host on multiple destinations, description=Generated by the alert wizard, priority=2, alert=true, config=AggregationCountProcessorConfig{type=aggregation-count, stream=620cead637bdee5946912956, thresholdType=MORE, threshold=4, groupingFields=[src_ip], distinctionFields=[dest_ip], comment=Generated by the alert wizard, searchQuery=*, searchWithinMs=900000, executeEveryMs=60000}, fieldSpec={}, keySpec=[], notificationSettings=EventNotificationSettings{gracePeriodMs=0, backlogSize=500}, notifications=[Config{notificationId=620cead637bdee594691295a, notificationParameters=Optional.empty}], storage=[Config{type=persist-to-streams-v1, streams=[000000000000000000000002]}]}
at org.graylog.events.processor.EventProcessorEngine.execute(EventProcessorEngine.java:106) ~[graylog.jar:?]
at org.graylog.events.processor.EventProcessorExecutionJob.execute(EventProcessorExecutionJob.java:115) ~[graylog.jar:?]
at org.graylog.scheduler.JobExecutionEngine.executeJob(JobExecutionEngine.java:166) ~[graylog.jar:?]
at org.graylog.scheduler.JobExecutionEngine.lambda$handleTrigger$2(JobExecutionEngine.java:144) ~[graylog.jar:?]
at com.codahale.metrics.Timer.time(Timer.java:151) ~[graylog.jar:?]
at org.graylog.scheduler.JobExecutionEngine.handleTrigger(JobExecutionEngine.java:144) ~[graylog.jar:?]
at org.graylog.scheduler.JobExecutionEngine.lambda$execute$0(JobExecutionEngine.java:119) ~[graylog.jar:?]
at org.graylog.scheduler.worker.JobWorkerPool.lambda$execute$0(JobWorkerPool.java:110) ~[graylog.jar:?]
at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:180) [graylog.jar:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_312]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_312]
at com.codahale.metrics.InstrumentedThreadFactory$InstrumentedRunnable.run(InstrumentedThreadFactory.java:66) [graylog.jar:?]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_312]
Caused by: java.lang.IllegalArgumentException: Multiple entries with same key: 192.168.1.198 - 192.168.1.254=2 and 192.168.1.198 - 192.168.1.254=2
at com.google.common.collect.ImmutableMap.conflictException(ImmutableMap.java:216) ~[graylog.jar:?]
at com.google.common.collect.ImmutableMap.checkNoConflict(ImmutableMap.java:210) ~[graylog.jar:?]
at com.google.common.collect.RegularImmutableMap.checkNoConflictInKeyBucket(RegularImmutableMap.java:146) ~[graylog.jar:?]
at com.google.common.collect.RegularImmutableMap.fromEntryArray(RegularImmutableMap.java:109) ~[graylog.jar:?]
at com.google.common.collect.ImmutableMap$Builder.build(ImmutableMap.java:395) ~[graylog.jar:?]
at com.airbus_cyber_security.graylog.events.processor.aggregation.checks.AggregationField.convertResult(AggregationField.java:220) ~[?:?]
at com.airbus_cyber_security.graylog.events.processor.aggregation.checks.AggregationField.getTermsResult(AggregationField.java:200) ~[?:?]
at com.airbus_cyber_security.graylog.events.processor.aggregation.checks.AggregationField.run(AggregationField.java:266) ~[?:?]
at com.airbus_cyber_security.graylog.events.processor.aggregation.checks.AggregationCount.runCheck(AggregationCount.java:55) ~[?:?]
at com.airbus_cyber_security.graylog.events.processor.aggregation.AggregationCountProcessor.createEvents(AggregationCountProcessor.java:79) ~[?:?]
at org.graylog.events.processor.EventProcessorEngine.execute(EventProcessorEngine.java:92) ~[graylog.jar:?]
... 12 more
It seems this problem occurs when graylog is catching up with old messages. It processes multiple windows during one run. To increase the probability of it occuring, play with the "Catch Up Window" configuration (which is set to 1 hour by default)
Fixed
My Graylog throws an Event Processor exception as you can see below. I don't know how to reproduce it yet. Graylog version : 4.1.6 Plugin version : 4.0.0