search

Graylog2 / graylog2-server

Free and open log management
https://www.graylog.org
Other
7.31k stars 1.05k forks source link

Cannot edit extractors anymore #4615

Closed berndleinfelder closed 6 years ago

berndleinfelder commented 6 years ago

I’m running graylog-2.4.3 and some time recently have started to notice that editing extractors is failing. eg if I choose an existing extractor and click “Edit”, the “loading…” thingy appears, but it never actually loads. Listing the extractors works and so does adding a new one - but I cannot edit existing ones

I get the same issue under Firefox and Chrome, on all inputs and with different users. All users have admin role.

Expected Behavior

Clicking on "Edit" lets me edit the existing extractor.

Current Behavior

"Loading" forever.

Possible Solution

No idea.

Steps to Reproduce (for bugs)

  1. Click on Inputs - Manage extractors - Edit
  2. Loading forever

Context

Your Environment

berndleinfelder commented 6 years ago

Same was reported by Jason Haar in the forum, but obviously no bug was reported. https://community.graylog.org/t/can-no-longer-edit-extractors/2954

joschi commented 6 years ago

@berndleinfelder What's in the logs of your Graylog node? ➡️ http://docs.graylog.org/en/2.4/pages/configuration/file_location.html Are there any warnings or errors in the JavaScript console of the developer tools of your browser? Are you able to delete extractors using the Graylog REST API?

berndleinfelder commented 6 years ago

Deleting an extractor from the WebUI works without issues. Javacsript console without errors. In network console I have an timeout on the following URL:

https://graylog.intern.ti8m.ch/api/search/universal/relative?query=gl2_source_input%3A581344a7a0474e1cf763f7f4%20OR%20gl2_source_radio_input%3A581344a7a0474e1cf763f7f4&limit=1

In JS console I have a corresponding code 500 and a stacktrace from ThreatIntelPlugin:

bluebird.js:1545 Unhandled rejection Error: cannot GET https://graylog.intern.ti8m.ch/api/search/universal/relative?query=gl2_source_input%3A581344a7a0474e1cf763f7f4%OR%20gl2_source_radio_input%3A581344a7a0474e1cf763f7f4&limit=1 (500)
    at new e (https://graylog.intern.ti8m.ch/assets/plugin/org.graylog.plugins.threatintel.ThreatIntelPlugin/plugin.org.graylog.plugins.threatintel.ThreatIntelPlugin.f39eb5f4d05eb44f4da5.js:1:2346)
    at https://graylog.intern.ti8m.ch/assets/plugin/org.graylog.plugins.threatintel.ThreatIntelPlugin/plugin.org.graylog.plugins.threatintel.ThreatIntelPlugin.f39eb5f4d05eb44f4da5.js:1:3747
    at i (https://graylog.intern.ti8m.ch/assets/plugin/org.graylog.plugins.threatintel.ThreatIntelPlugin/plugin.org.graylog.plugins.threatintel.ThreatIntelPlugin.f39eb5f4d05eb44f4da5.js:37:73003)

Maybe the timeout is related to the following message in server.log (timestamp does not match exactly):

2018-02-26T08:48:14.283+01:00 ERROR [IndexerClusterCheckerThread] Uncaught exception in periodical
org.graylog2.indexer.ElasticsearchException: Unable to read Elasticsearch node information
        at org.graylog2.indexer.cluster.jest.JestUtils.execute(JestUtils.java:51) ~[graylog.jar:?]
        at org.graylog2.indexer.cluster.jest.JestUtils.execute(JestUtils.java:62) ~[graylog.jar:?]
        at org.graylog2.indexer.cluster.Cluster.catNodes(Cluster.java:121) ~[graylog.jar:?]
        at org.graylog2.indexer.cluster.Cluster.getFileDescriptorStats(Cluster.java:126) ~[graylog.jar:?]
        at org.graylog2.periodical.IndexerClusterCheckerThread.doRun(IndexerClusterCheckerThread.java:58) ~[graylog.jar:?]
        at org.graylog2.plugin.periodical.Periodical.run(Periodical.java:77) [graylog.jar:?]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_161]
        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [?:1.8.0_161]
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_161]
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [?:1.8.0_161]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_161]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_161]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_161]
Caused by: java.net.SocketTimeoutException: Read timed out
        at java.net.SocketInputStream.socketRead0(Native Method) ~[?:1.8.0_161]
berndleinfelder commented 6 years ago

I disabled (removed) the the ThreatIntel Plugin and now the JS console shows the following:

Unhandled rejection Error: cannot GET https://graylog.intern.ti8m.ch/api/search/universal/relative?query=gl2_source_input%3A581344a7a0474e1cf763f7f4%OR%20gl2_source_radio_input%3A581344a7a0474e1cf763f7f4&limit=1 (500) at new t (https://graylog.intern.ti8m.ch/assets/plugin/org.graylog.plugins.pipelineprocessor.ProcessorPlugin/plugin.org.graylog.plugins.pipelineprocessor.PipelineProcessorPlugin.4d62a989321ebfb895e4.js:1:3610) at https://graylog.intern.ti8m.ch/assets/plugin/org.graylog.plugins.pipelineprocessor.ProcessorPlugin/plugin.org.graylog.plugins.pipelineprocessor.PipelineProcessorPlugin.4d62a989321ebfb895e4.js:1:5011

berndleinfelder commented 6 years ago

Tried to remove the PipelineProcessorPlugin, but then graylog could'nt start anymore.

joschi commented 6 years ago

@berndleinfelder Please attach the complete logs of your Graylog node(s).

berndleinfelder commented 6 years ago

attached this mornings server.log

Thank you for your support!

server.log.gz

joschi commented 6 years ago

@berndleinfelder There are quite a few things broken in your Graylog cluster according to these logs.

I'd recommend solving each of these problems, then check if the problem with your extractors still exists.

We are using GitHub issues for tracking bugs in Graylog itself, but this doesn't look like one. Please post this issue to our discussion forum or join the #graylog channel on freenode IRC.

Thank you!

dmayan-ss commented 4 years ago

Hi,

Just solved the same problem deleting the index_ranges collection and recalculating them. It was an index that doesn't exist anymore and Graylog tried to access.