search

Graylog2 / graylog2-server

Free and open log management
https://www.graylog.org
Other
7.33k stars 1.06k forks source link

Nullpointer Error on Alerts & Events AND Events Definition Page #8410

Open thesubmitter opened 4 years ago

thesubmitter commented 4 years ago

While I was running 3.1.3 I had a a problem which caused massive mail flooding with alerts. I issued the following commands to mongo

db.event_notification_status.remove({}) db.alerts.remove({}) db.event_notification_status.remove({}) db.scheduler_triggers.remove({}) db.event_processor_state.remove({})

The Alerts pages in 3.1 still loaded...

I later upgraded to 3.3 one machine at a time (I didn't make any changes to config files, issue any graylog or elasticsearch api commands (I think they are optional) and now the pages

Expected Behavior

Pages should load

Current Behavior

Alerts & Events Events Definitions

Never Load and I have the following errors in the logs

2020-06-23T12:53:49.265-04:00 ERROR [AnyExceptionClassMapper] Unhandled exception in REST resource java.lang.NullPointerException: null at org.graylog.events.context.EventDefinitionContextService.schedulerContext(EventDefinitionContextService.java:104) ~[graylog.jar:?] at org.graylog.events.context.EventDefinitionContextService.contextFor(EventDefinitionContextService.java:57) ~[graylog.jar:?] at org.graylog.events.rest.EventDefinitionsResource.list(EventDefinitionsResource.java:114) ~[graylog.jar:?] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_252] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_252] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_252] at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_252] at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory$1.invoke(ResourceMethodInvocationHandlerFactory.java:81) ~[graylog.jar:?] at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:144) ~[graylog.jar:?] at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:161) ~[graylog.jar:?] at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$TypeOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:205) ~[graylog.jar:?] at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:99) ~[graylog.jar:?] at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:389) ~[graylog.jar:?] at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:347) ~[graylog.jar:?] at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:102) ~[graylog.jar:?] at org.glassfish.jersey.server.ServerRuntime$2.run(ServerRuntime.java:326) [graylog.jar:?] at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271) [graylog.jar:?] at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267) [graylog.jar:?] at org.glassfish.jersey.internal.Errors.process(Errors.java:315) [graylog.jar:?] at org.glassfish.jersey.internal.Errors.process(Errors.java:297) [graylog.jar:?] at org.glassfish.jersey.internal.Errors.process(Errors.java:267) [graylog.jar:?] at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:317) [graylog.jar:?] at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:305) [graylog.jar:?] at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:1154) [graylog.jar:?] at org.glassfish.jersey.grizzly2.httpserver.GrizzlyHttpContainer.service(GrizzlyHttpContainer.java:384) [graylog.jar:?] at org.glassfish.grizzly.http.server.HttpHandler$1.run(HttpHandler.java:224) [graylog.jar:?] at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:181) [graylog.jar:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_252] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_252] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_252]

Possible Solution

Maybe repair mongo indexes?

Steps to Reproduce (for bugs)

  1. Upgrade from 3.1 to 3.3 directly?
  2. Erase mongod records

Context

I can't acces alerts

Your Environment

houstonheat commented 4 years ago

Same here: Graylog 3.3.2+ade4779 MongoDB: 3.6.14 Elasticsearch : 6.8.5

Empty MongoDB collections:

alarmcallbackconfigurations
alarmcallbackhistory
alerts
event_definitions
event_notification_status
event_notifications
event_processor_state
scheduler_triggers

Request (browser network tab):

req: http://graylog.xxx.xx/api/events/definitions?query=&page=1&per_page=10
resp: HTTP code 500; body: {"type":"ApiError","message":""}

Graylog log entry:

2020-07-30 13:48:12,757 ERROR: org.graylog2.shared.rest.exceptionmappers.AnyExceptionClassMapper - Unhandled exception in REST resource
java.lang.NullPointerException: null
    at org.graylog.events.context.EventDefinitionContextService.schedulerContext(EventDefinitionContextService.java:104) ~[graylog.jar:?]
    at org.graylog.events.context.EventDefinitionContextService.contextFor(EventDefinitionContextService.java:57) ~[graylog.jar:?]
    at org.graylog.events.rest.EventDefinitionsResource.list(EventDefinitionsResource.java:114) ~[graylog.jar:?]
    at sun.reflect.GeneratedMethodAccessor967.invoke(Unknown Source) ~[?:?]
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_252]
    at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_252]
    at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory$1.invoke(ResourceMethodInvocationHandlerFactory.java:81) ~[graylog.jar:?]
    at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:144) ~[graylog.jar:?]
    at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:161) ~[graylog.jar:?]
    at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$TypeOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:205) ~[graylog.jar:?]
    at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:99) ~[graylog.jar:?]
    at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:389) ~[graylog.jar:?]
    at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:347) ~[graylog.jar:?]
    at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:102) ~[graylog.jar:?]
    at org.glassfish.jersey.server.ServerRuntime$2.run(ServerRuntime.java:326) [graylog.jar:?]
    at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271) [graylog.jar:?]
    at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267) [graylog.jar:?]
    at org.glassfish.jersey.internal.Errors.process(Errors.java:315) [graylog.jar:?]
    at org.glassfish.jersey.internal.Errors.process(Errors.java:297) [graylog.jar:?]
    at org.glassfish.jersey.internal.Errors.process(Errors.java:267) [graylog.jar:?]
    at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:317) [graylog.jar:?]
    at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:305) [graylog.jar:?]
    at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:1154) [graylog.jar:?]
    at org.glassfish.jersey.grizzly2.httpserver.GrizzlyHttpContainer.service(GrizzlyHttpContainer.java:384) [graylog.jar:?]
    at org.glassfish.grizzly.http.server.HttpHandler$1.run(HttpHandler.java:224) [graylog.jar:?]
    at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:181) [graylog.jar:?]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_252]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_252]
    at java.lang.Thread.run(Thread.java:748) [?:1.8.0_252]

@thesubmitter As soon as i remove documents in scheduler_job_definitions collection (look, exception message contains org.graylog.events.context.EventDefinitionContextService.schedulerContext), pages /alerts, /alerts/definitions are load successfully and migrated definitions are loaded again. 

> db.getCollection('scheduler_job_definitions').count()
73
> db.getCollection('scheduler_job_definitions').find()
...
{ "_id" : ObjectId("5e58b64467210f001204b842"), "title" : "Playlist: stderr", "description" : "Migrated message count alert condition", "config" : { "type" : "event-processor-execution-v1", "event_definition_id" : "5e58b64467210f001204b840", "parameters" : { "type" : "aggregation-v1", "timerange" : { "type" : "absolute", "from" : ISODate("2020-02-28T06:37:12.276Z"), "to" : ISODate("2020-02-28T06:42:12.276Z") }, "streams" : [ ], "batch_size" : 500 }, "processing_window_size" : NumberLong(300000), "processing_hop_size" : NumberLong(60000) } }
...
> db.getCollection('scheduler_job_definitions').remove({})
> db.getCollection('event_definitions').count({})
73

hope it'll helps you

Carlos-ZRM commented 2 years ago

@houstonheat Tank you . I had the same problem and your solutions works for me.

ducna96 commented 2 years ago

Same here: Graylog 3.3.2+ade4779 MongoDB: 3.6.14 Elasticsearch : 6.8.5

Empty MongoDB collections:

alarmcallbackconfigurations
alarmcallbackhistory
alerts
event_definitions
event_notification_status
event_notifications
event_processor_state
scheduler_triggers

Request (browser network tab):

req: http://graylog.xxx.xx/api/events/definitions?query=&page=1&per_page=10
resp: HTTP code 500; body: {"type":"ApiError","message":""}

Graylog log entry:

2020-07-30 13:48:12,757 ERROR: org.graylog2.shared.rest.exceptionmappers.AnyExceptionClassMapper - Unhandled exception in REST resource
java.lang.NullPointerException: null
  at org.graylog.events.context.EventDefinitionContextService.schedulerContext(EventDefinitionContextService.java:104) ~[graylog.jar:?]
  at org.graylog.events.context.EventDefinitionContextService.contextFor(EventDefinitionContextService.java:57) ~[graylog.jar:?]
  at org.graylog.events.rest.EventDefinitionsResource.list(EventDefinitionsResource.java:114) ~[graylog.jar:?]
  at sun.reflect.GeneratedMethodAccessor967.invoke(Unknown Source) ~[?:?]
  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_252]
  at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_252]
  at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory$1.invoke(ResourceMethodInvocationHandlerFactory.java:81) ~[graylog.jar:?]
  at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:144) ~[graylog.jar:?]
  at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:161) ~[graylog.jar:?]
  at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$TypeOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:205) ~[graylog.jar:?]
  at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:99) ~[graylog.jar:?]
  at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:389) ~[graylog.jar:?]
  at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:347) ~[graylog.jar:?]
  at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:102) ~[graylog.jar:?]
  at org.glassfish.jersey.server.ServerRuntime$2.run(ServerRuntime.java:326) [graylog.jar:?]
  at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271) [graylog.jar:?]
  at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267) [graylog.jar:?]
  at org.glassfish.jersey.internal.Errors.process(Errors.java:315) [graylog.jar:?]
  at org.glassfish.jersey.internal.Errors.process(Errors.java:297) [graylog.jar:?]
  at org.glassfish.jersey.internal.Errors.process(Errors.java:267) [graylog.jar:?]
  at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:317) [graylog.jar:?]
  at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:305) [graylog.jar:?]
  at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:1154) [graylog.jar:?]
  at org.glassfish.jersey.grizzly2.httpserver.GrizzlyHttpContainer.service(GrizzlyHttpContainer.java:384) [graylog.jar:?]
  at org.glassfish.grizzly.http.server.HttpHandler$1.run(HttpHandler.java:224) [graylog.jar:?]
  at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:181) [graylog.jar:?]
  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_252]
  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_252]
  at java.lang.Thread.run(Thread.java:748) [?:1.8.0_252]

@thesubmitter As soon as i remove documents in scheduler_job_definitions collection (look, exception message contains org.graylog.events.context.EventDefinitionContextService.schedulerContext), pages /alerts, /alerts/definitions are load successfully and migrated definitions are loaded again.

> db.getCollection('scheduler_job_definitions').count()
73
> db.getCollection('scheduler_job_definitions').find()
...
{ "_id" : ObjectId("5e58b64467210f001204b842"), "title" : "Playlist: stderr", "description" : "Migrated message count alert condition", "config" : { "type" : "event-processor-execution-v1", "event_definition_id" : "5e58b64467210f001204b840", "parameters" : { "type" : "aggregation-v1", "timerange" : { "type" : "absolute", "from" : ISODate("2020-02-28T06:37:12.276Z"), "to" : ISODate("2020-02-28T06:42:12.276Z") }, "streams" : [ ], "batch_size" : 500 }, "processing_window_size" : NumberLong(300000), "processing_hop_size" : NumberLong(60000) } }
...
> db.getCollection('scheduler_job_definitions').remove({})
> db.getCollection('event_definitions').count({})
73

hope it'll helps you

hello bro, i just run db.getCollection('db.scheduler_triggers').remove({}) , but it not work for me, page alerts can not load, i just drop collection to re gen again ,but no work for me... please help me...

Slapper commented 1 year ago

Hello,

Late to the party here, but we had a similar issue in our environment. Our solution was to delete every event_definitions through the terraform code and deploy every event definition from the beginning. Unfortunately the error bellow cannot give you more information about the problematic schedule job definition 

ERROR: org.graylog2.shared.rest.exceptionmappers.AnyExceptionClassMapper - Unhandled exception in REST resource
java.lang.NullPointerException: null

P.S : also running newer version of graylog 4.2.13