search

Graylog2 / graylog2-server

Free and open log management
https://www.graylog.org
Other
7.33k stars 1.05k forks source link

Unhandled exception in REST resource #16474

Closed tanji closed 12 months ago

tanji commented 1 year ago

After migration from Graylog 5.0 to 5.1, the /streams page fails to load with the following exception:

Loading streams failed with status: FetchError: There was an error fetching a resource: Internal Server Error. Additional information: Null matchingType

This is the error log from Graylog server

2023-09-14 14:55:40,719 ERROR: org.graylog2.shared.rest.exceptionmappers.AnyExceptionClassMapper - Unhandled exception in REST resource
java.lang.NullPointerException: Null matchingType
    at org.graylog2.streams.$AutoValue_StreamDTO$Builder.matchingType($AutoValue_StreamDTO.java:367) ~[graylog.jar:?]
    at org.graylog2.streams.StreamDTO.fromDocument(StreamDTO.java:198) ~[graylog.jar:?]
    at java.util.stream.ReferencePipeline$3$1.accept(Unknown Source) ~[?:?]
    at java.util.Iterator.forEachRemaining(Unknown Source) ~[?:?]
    at java.util.Spliterators$IteratorSpliterator.forEachRemaining(Unknown Source) ~[?:?]
    at java.util.stream.AbstractPipeline.copyInto(Unknown Source) ~[?:?]
    at java.util.stream.AbstractPipeline.wrapAndCopyInto(Unknown Source) ~[?:?]
    at java.util.stream.AbstractPipeline.evaluate(Unknown Source) ~[?:?]
    at java.util.stream.AbstractPipeline.evaluateToArrayNode(Unknown Source) ~[?:?]
    at java.util.stream.ReferencePipeline.toArray(Unknown Source) ~[?:?]
    at java.util.stream.ReferencePipeline.toArray(Unknown Source) ~[?:?]
    at java.util.stream.ReferencePipeline.toList(Unknown Source) ~[?:?]
    at org.graylog2.streams.PaginatedStreamService.findPaginated(PaginatedStreamService.java:90) ~[graylog.jar:?]
    at org.graylog2.rest.resources.streams.StreamResource.getPage(StreamResource.java:234) ~[graylog.jar:?]
    at jdk.internal.reflect.GeneratedMethodAccessor820.invoke(Unknown Source) ~[?:?]
    at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) ~[?:?]
    at java.lang.reflect.Method.invoke(Unknown Source) ~[?:?]
    at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:52) ~[graylog.jar:?]
    at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:134) ~[graylog.jar:?]
    at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:177) ~[graylog.jar:?]
    at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$TypeOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:219) ~[graylog.jar:?]
    at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:81) ~[graylog.jar:?]
    at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:478) ~[graylog.jar:?]
    at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:400) ~[graylog.jar:?]
    at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:81) ~[graylog.jar:?]
    at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:255) [graylog.jar:?]
    at org.glassfish.jersey.internal.Errors$1.call(Errors.java:248) [graylog.jar:?]
    at org.glassfish.jersey.internal.Errors$1.call(Errors.java:244) [graylog.jar:?]
    at org.glassfish.jersey.internal.Errors.process(Errors.java:292) [graylog.jar:?]
    at org.glassfish.jersey.internal.Errors.process(Errors.java:274) [graylog.jar:?]
    at org.glassfish.jersey.internal.Errors.process(Errors.java:244) [graylog.jar:?]
    at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:265) [graylog.jar:?]
    at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:234) [graylog.jar:?]
    at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:684) [graylog.jar:?]
    at org.glassfish.jersey.grizzly2.httpserver.GrizzlyHttpContainer.service(GrizzlyHttpContainer.java:356) [graylog.jar:?]
    at org.glassfish.grizzly.http.server.HttpHandler$1.run(HttpHandler.java:200) [graylog.jar:?]
    at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:180) [graylog.jar:?]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) [?:?]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [?:?]
    at java.lang.Thread.run(Unknown Source) [?:?]

We do not encounter this failure elsewhere, so I suspect it is caused by a particular stream.

Your Environment

dennisoelkers commented 1 year ago

Hey @tanji,

it looks like there is at least one stream which is missing a mandatory field (matching_type). You could find the stream(s) using this MongoDB-query run in your Graylog database: db.streams.find({ $or: [{ matching_type: { $exists: false } }, { matching_type: null }] }).

When you have found it, you can either set that field for the stream(s) to AND or OR (depending on what kind of matching you want) or delete the document(s). It would be interesting to hear if you have any ideas how the stream(s) ended up in that state?

tanji commented 12 months ago

@dennisoelkers Thank you very much, that does return quite a bit of documents. Here is an example:

  {
    _id: ObjectId("584e9cf3e4b045166233b911"),
    creator_user_id: 'yt',
    index_set_id: '592d92bd28c7eb00014907b0',
    description: 'Errors or notices from the ImageCDN',
    created_at: ISODate("2016-12-12T12:49:55.568Z"),
    disabled: false,
    title: 'Image CDN',
    content_pack: null
  }

As you can see we've had this Graylog installation since 2015 so it used to run Graylog 2.x and underwent several upgrades since. Maybe the matching type wasn't necessary by then, or there was some kind of default value that allowed it to work. It is interesting to note that all the matching streams exactly had 1 rule. So probably that's the cause since no logic is needed in that case.

dennisoelkers commented 12 months ago

@tanji: I was looking a little bit deeper. We do have a default value (which is AND) in place, which is used when the matching_type field is not present. In your case, it looks like some documents have "matching_type": null instead, which leads to null being used as value instead of the default. The query I gave you returns documents which do either miss the matching_type field or do have a null value in it. Can you try to execute this slightly modified query which will only return the latter:

db.streams.find({ matching_type: null })

It would be interesting to see which streams show up then.

tanji commented 12 months ago

@dennisoelkers , this query is technically equivalent to the preceding one because in MongoDB, the { item : null } query matches documents that either contain the item field whose value is null or that do not contain the item field. I used the following query:

db.streams.find( { matching_type : { $type: 10 } } )

This checks for BSON type null, and does not match anything. To be fair I also looked at an old backup of the MongoDB database from the Graylog 3.0 times and it did not contain any matching_type fields.

dennisoelkers commented 12 months ago

Hey @tanji,

@dennisoelkers , this query is technically equivalent to the preceding one because in MongoDB, the { item : null } query matches documents that either contain the item field whose value is null or that do not contain the item field.

thanks a lot for the learning! :+1:

I had some insight today when I realized that the streams page is using the paginated endpoint and a slightly different code path, where the matching type default is not applied. So it looks like this is a bug. A fix for this is trivial and should be in shortly, the next 5.1.x release will be in the first week of October. Does this work for you?