search

graylog-labs / graylog2-web-interface

[DEPRECATED]
https://www.graylog.org/
611 stars 174 forks source link

Indices page showing API error since 1.3 #1719

Closed pschichtel closed 8 years ago

pschichtel commented 8 years ago

The /system/indices page is showing an API error since I upgraded to 1.3 from 1.2. Everything else seems to be working as expected.

(You caused a org.graylog2.restclient.lib.APIException. API call failed GET http://@127.0.0.1:12900/system/indexer/indices returned 500 Internal Server Error body: {"type":"ApiError","message":""})

Reason: Could not get indices. We expected HTTP 200, but got a HTTP 500.

    org.graylog2.restclient.lib.ApiClientImpl$ApiRequestBuilder#handleResponse (ApiClientImpl.java:511)
    org.graylog2.restclient.lib.ApiClientImpl$ApiRequestBuilder#execute (ApiClientImpl.java:441)
    org.graylog2.restclient.models.IndexService#allIndicesInfo (IndexService.java:69)
    controllers.IndicesController#index (IndicesController.java:79)
    Routes$$anonfun$routes$1$$anonfun$applyOrElse$61$$anonfun$apply$631#apply (routes_routing.scala:2126)
    Routes$$anonfun$routes$1$$anonfun$applyOrElse$61$$anonfun$apply$631#apply (routes_routing.scala:2126)
    play.core.Router$HandlerInvokerFactory$$anon$4#resultCall (Router.scala:264)
    play.core.Router$HandlerInvokerFactory$JavaActionInvokerFactory$$anon$15$$anon$1#invocation (Router.scala:255)
    play.core.j.JavaAction$$anon$1#call (JavaAction.scala:55)
    play.GlobalSettings$1#call (GlobalSettings.java:67)
    play.mvc.Security$AuthenticatedAction#call (Security.java:44)
    play.core.j.JavaAction$$anonfun$11#apply (JavaAction.scala:82)
    play.core.j.JavaAction$$anonfun$11#apply (JavaAction.scala:82)
    scala.concurrent.impl.Future$PromiseCompletingRunnable#liftedTree1$1 (Future.scala:24)
    scala.concurrent.impl.Future$PromiseCompletingRunnable#run (Future.scala:24)
    play.core.j.HttpExecutionContext$$anon$2#run (HttpExecutionContext.scala:40)
    play.api.libs.iteratee.Execution$trampoline$#execute (Execution.scala:46)
    play.core.j.HttpExecutionContext#execute (HttpExecutionContext.scala:32)
    scala.concurrent.impl.Future$#apply (Future.scala:31)
    scala.concurrent.Future$#apply (Future.scala:485)
    play.core.j.JavaAction$class#apply (JavaAction.scala:82)
    play.core.Router$HandlerInvokerFactory$JavaActionInvokerFactory$$anon$15$$anon$1#apply (Router.scala:252)
    play.api.mvc.Action$$anonfun$apply$1$$anonfun$apply$4$$anonfun$apply$5#apply (Action.scala:130)
    play.api.mvc.Action$$anonfun$apply$1$$anonfun$apply$4$$anonfun$apply$5#apply (Action.scala:130)
    play.utils.Threads$#withContextClassLoader (Threads.scala:21)
    play.api.mvc.Action$$anonfun$apply$1$$anonfun$apply$4#apply (Action.scala:129)
    play.api.mvc.Action$$anonfun$apply$1$$anonfun$apply$4#apply (Action.scala:128)
    scala.Option#map (Option.scala:145)
    play.api.mvc.Action$$anonfun$apply$1#apply (Action.scala:128)
    play.api.mvc.Action$$anonfun$apply$1#apply (Action.scala:121)
    play.api.libs.iteratee.Iteratee$$anonfun$mapM$1#apply (Iteratee.scala:483)
    play.api.libs.iteratee.Iteratee$$anonfun$mapM$1#apply (Iteratee.scala:483)
    play.api.libs.iteratee.Iteratee$$anonfun$flatMapM$1#apply (Iteratee.scala:519)
    play.api.libs.iteratee.Iteratee$$anonfun$flatMapM$1#apply (Iteratee.scala:519)
    play.api.libs.iteratee.Iteratee$$anonfun$flatMap$1$$anonfun$apply$14#apply (Iteratee.scala:496)
    play.api.libs.iteratee.Iteratee$$anonfun$flatMap$1$$anonfun$apply$14#apply (Iteratee.scala:496)
    scala.concurrent.impl.Future$PromiseCompletingRunnable#liftedTree1$1 (Future.scala:24)
    scala.concurrent.impl.Future$PromiseCompletingRunnable#run (Future.scala:24)
    akka.dispatch.TaskInvocation#run (AbstractDispatcher.scala:41)
    akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask#exec (AbstractDispatcher.scala:393)
    scala.concurrent.forkjoin.ForkJoinTask#doExec (ForkJoinTask.java:260)
    scala.concurrent.forkjoin.ForkJoinPool$WorkQueue#runTask (ForkJoinPool.java:1339)
    scala.concurrent.forkjoin.ForkJoinPool#runWorker (ForkJoinPool.java:1979)
    scala.concurrent.forkjoin.ForkJoinWorkerThread#run (ForkJoinWorkerThread.java:107)
Method
    GET
Query

Referer
    http://hostname:9000/search?rangetype=relative&fields=message%2Csource&width=1399&relative=300&q=source%3Avir123
Connection
    keep-alive
Accept
    text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language
    de-DE,de;q=0.8,en-US;q=0.5,en;q=0.3
Accept-Encoding
    gzip, deflate
User-Agent
    Mozilla/5.0 (X11; Linux x86_64; rv:43.0) Gecko/20100101 Firefox/43.0
Host
    hostname:9000
pschichtel commented 8 years ago

elasticsearch version is 2.1.1 (but also happened with 1.x) and the system is Debian Jessy

edmundoa commented 8 years ago

Hi,

Graylog 1.3 does not support Elasticsearch 2.x, so you should go back to 1.7 before trying anything else. Unfortunately, the error you posted is not saying much, could you please check for any errors in your Graylog server logs when you try to access that page?

joschi commented 8 years ago

Graylog 1.x isn't compatible with ES 2.x at all, so the list of indices also can't be retrieved. Closing this issue.

pschichtel commented 8 years ago

Didn't you read my comment? it also happened with 1.x ! I upgraded Elasticsearch after I saw the error as I thought the ES version might be too old

joschi commented 8 years ago

@pschichtel Please check the logs of your Graylog server node(s) for error messages regarding Elasticsearch and post them in this issue.

pschichtel commented 8 years ago 
2016-01-11T11:36:05.160+01:00 ERROR [AnyExceptionClassMapper] Unhandled exception in REST resource
java.lang.NullPointerException
    at org.graylog2.rest.resources.system.indexer.IndicesResource.indexStats(IndicesResource.java:262)
    at org.graylog2.rest.resources.system.indexer.IndicesResource.all(IndicesResource.java:120)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory$1.invoke(ResourceMethodInvocationHandlerFactory.java:81)
    at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:144)
    at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:161)
    at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$TypeOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:205)
    at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:99)
    at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:389)
    at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:347)
    at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:102)
    at org.glassfish.jersey.server.ServerRuntime$2.run(ServerRuntime.java:326)
    at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271)
    at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267)
    at org.glassfish.jersey.internal.Errors.process(Errors.java:315)
    at org.glassfish.jersey.internal.Errors.process(Errors.java:297)
    at org.glassfish.jersey.internal.Errors.process(Errors.java:267)
    at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:317)
    at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:305)
    at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:1154)
    at org.graylog2.jersey.container.netty.NettyContainer.messageReceived(NettyContainer.java:356)
    at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
    at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
    at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
    at org.jboss.netty.handler.execution.ChannelUpstreamEventRunnable.doRun(ChannelUpstreamEventRunnable.java:43)
    at org.jboss.netty.handler.execution.ChannelEventRunnable.run(ChannelEventRunnable.java:67)
    at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176)
    at org.jboss.netty.handler.execution.MemoryAwareThreadPoolExecutor$MemoryAwareRunnable.run(MemoryAwareThreadPoolExecutor.java:606)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:745)
pschichtel commented 8 years ago

elasticsearch version is now 1.7.4 (installed using the puppet module)

joschi commented 8 years ago

@pschichtel Please post the output of the following curl commands (replace "localhost" with the host name or IP address of one of your Elasticsearch nodes):

# curl -i -X GET http://localhost:9200/?pretty
# curl -i -X GET http://localhost:9200/_stats?pretty

Are there any error messages in your Elasticsearch logs?

wagnst commented 8 years ago

Hi, i'm just sending the outputs, because @pschichtel is n/a today anymore...

# curl -i -X GET http://<hostname>:9200/?pretty

HTTP/1.1 200 OK
Content-Type: application/json; charset=UTF-8
Content-Length: 247

{
  "status" : 200,
  "name" : "<hostname>-graylog",
  "version" : {
    "number" : "1.0.3",
    "build_hash" : "NA",
    "build_timestamp" : "NA",
    "build_snapshot" : false,
    "lucene_version" : "4.6"
  },
  "tagline" : "You Know, for Search"
}

# curl -i -X GET http://localhost:9200/_stats?pretty see https://gist.github.com/wagnst/bbdb822f91d608500643

@joschi wasn't able to find any errors in the elasticsearch logs. There are only "INFO" entries...

Inside application log of graylog-web i find ERRORS like below...


 63
 64 2016-01-11T11:35:12.691+01:00 - [ERROR] - from org.graylog2.restclient.lib.ApiClient in pool-11-thread-1
 65 Connection refused: /127.0.0.1:12900
 66
 67 2016-01-11T11:35:14.622+01:00 - [ERROR] - from org.graylog2.restclient.lib.ApiClient in servernodes-refresh-0
 68 Connection refused: /127.0.0.1:12900
 69
 70 2016-01-11T11:35:19.625+01:00 - [ERROR] - from org.graylog2.restclient.lib.ApiClient in servernodes-refresh-0
 71 Connection refused: /127.0.0.1:12900
 72
 73 2016-01-11T11:35:24.627+01:00 - [ERROR] - from org.graylog2.restclient.lib.ApiClient in servernodes-refresh-0
 74 Connection refused: /127.0.0.1:12900

Appreciate your help, Steffen

pschichtel commented 8 years ago

btw I guess this should be reopened

joschi commented 8 years ago

@pschichtel @wagnst Thanks for posting those outputs. You're using Elasticsearch 1.0.3 which isn't supported by Graylog 1.1.x, 1.2.x, or 1.3.x. Please upgrade to Elasticsearch 1.7.4 and try again.

For now I don't see a bug in Graylog, so I won't reopen the issue.

pschichtel commented 8 years ago

@joschi that is interesting, because APT tells me that 1.7.4 is installed

pschichtel commented 8 years ago

ok seems like the puppet module failed to properly restart the instance, so it was still the 1.x instance running.

The exception is gone now

wagnst commented 8 years ago

@joschi thanks for your help!