search

Graylog2 / graylog2-server

Free and open log management
https://www.graylog.org
Other
7.33k stars 1.06k forks source link

(1.1.3) key=value tokenizer extractor fails #1356

Closed jskrivseth closed 9 years ago

jskrivseth commented 9 years ago

Graylog 1.1.3 throws this exception when handling a key=value tokenizer

2015-08-14_18:30:25.29992 ERROR [Extractor] Could not apply converter [tokenizer] of extractor [80a48fa0-41e6-11e5-8cd8-0ace55ff8537].
2015-08-14_18:30:25.29994 java.lang.UnsupportedOperationException
2015-08-14_18:30:25.29994       at com.google.common.collect.ImmutableMap.remove(ImmutableMap.java:338)
2015-08-14_18:30:25.29994       at org.graylog2.plugin.inputs.Extractor.runConverters(Extractor.java:260)
2015-08-14_18:30:25.29995       at org.graylog2.plugin.inputs.Extractor.runExtractor(Extractor.java:228)
2015-08-14_18:30:25.29995       at org.graylog2.filters.ExtractorFilter.filter(ExtractorFilter.java:62)
2015-08-14_18:30:25.29997       at org.graylog2.buffers.processors.ServerProcessBufferProcessor.handleMessage(ServerProcessBufferProcessor.java:97)
2015-08-14_18:30:25.29997       at org.graylog2.shared.buffers.processors.ProcessBufferProcessor.onEvent(ProcessBufferProcessor.java:71)
2015-08-14_18:30:25.29998       at org.graylog2.shared.buffers.processors.ProcessBufferProcessor.onEvent(ProcessBufferProcessor.java:33)
2015-08-14_18:30:25.29998       at com.lmax.disruptor.WorkProcessor.run(WorkProcessor.java:138)
2015-08-14_18:30:25.29998       at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176)
2015-08-14_18:30:25.29998       at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
2015-08-14_18:30:25.29999       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
2015-08-14_18:30:25.29999       at java.lang.Thread.run(Thread.java:745)

Extractor config

  {
      "condition_type": "regex",
      "condition_value": "(reason=|cb\\-notifications)",
      "converters": [
        {
          "type": "tokenizer",
          "config": {}
        }
      ],
      "cursor_strategy": "copy",
      "extractor_config": {},
      "extractor_type": "copy_input",
      "order": 0,
      "source_field": "full_message",
      "target_field": "full_message",
      "title": "Key=Value Tokenizer"
    }
  ],
  "version": "1.1.3 (611c8b2)"
}
jskrivseth commented 9 years ago

Thanks Joschi! I back-ported this to 1.1.6 and ran a build for our environment. It appears to work great. I noticed that the tokenizer doesn't handle quoted strings - it seems to delimit only on spaces. I'll open a new issue for that.