search

Graylog2 / graylog2-server

Free and open log management
https://www.graylog.org
Other
7.41k stars 1.07k forks source link

notification polling #2489

Closed colmaengus closed 8 years ago

colmaengus commented 8 years ago

Problem description

XHR request to system/cluster/nodes, system/notifications, cluster/metrics/multiple are occurring every second or so. I would have assumed that these kind of notifications would be done via websocket.

Steps to reproduce the problem

  1. Connect with browser

    Environment

    • Graylog Version: 2.1.0-alpha.2
    • Elasticsearch Version:
    • MongoDB Version:
    • Operating System:
    • Browser version:
jalogisch commented 8 years ago

If you are using the OVA you will get the following error when connect to system / indices

system_indices_call

and when you click on "update the configuration" this happens - nothing is loaded (no error give)

update_configuration

The Logfile looks like the following when you open system / indices

``` 2016-07-15_10:23:18.97672 2016-07-15 10:23:18,957 ERROR: org.graylog2.shared.rest.exceptionmappers.AnyExceptionClassMapper - Unhandled exception in REST resource 2016-07-15_10:23:18.97742 com.google.inject.ProvisionException: Unable to provision, see the following errors: 2016-07-15_10:23:18.97757 2016-07-15_10:23:18.99202 1) Error injecting constructor, java.lang.NoSuchMethodError: org.graylog2.indexer.retention.strategies.AbstractIndexCountBasedRetentionStrategy.(Lorg/graylog2/indexer/Deflector;Lorg/graylog2/indexer/indices/Indices;Lorg/graylog2/shared/system/activities/ActivityWriter;)V 2016-07-15_10:23:18.99939 at org.graylog.plugins.archive.indexer.retention.strategies.ArchiveRetentionStrategy.(ArchiveRetentionStrategy.java:45) 2016-07-15_10:23:19.01902 while locating org.graylog.plugins.archive.indexer.retention.strategies.ArchiveRetentionStrategy 2016-07-15_10:23:19.01915 while locating org.graylog2.plugin.indexer.retention.RetentionStrategy annotated with @com.google.inject.multibindings.Element(setName=,uniqueId=286, type=MAPBINDER, keyType=java.lang.String) 2016-07-15_10:23:19.01918 2016-07-15_10:23:19.01921 1 error 2016-07-15_10:23:19.01922 at com.google.inject.internal.InjectorImpl$2.get(InjectorImpl.java:1028) ~[graylog.jar:?] 2016-07-15_10:23:19.01924 at com.google.inject.spi.ProviderLookup$1.get(ProviderLookup.java:104) ~[graylog.jar:?] 2016-07-15_10:23:19.02027 at com.google.inject.multibindings.MapBinder$RealMapBinder$ValueProvider.get(MapBinder.java:821) ~[graylog.jar:?] 2016-07-15_10:23:19.02892 at org.graylog2.rest.resources.system.indices.RetentionStrategyResource.config(RetentionStrategyResource.java:96) ~[graylog.jar:?] 2016-07-15_10:23:19.03463 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_77] 2016-07-15_10:23:19.03965 at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_77] 2016-07-15_10:23:19.04234 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_77] 2016-07-15_10:23:19.04617 at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_77] 2016-07-15_10:23:19.05772 at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory$1.invoke(ResourceMethodInvocationHandlerFactory.java:81) ~[graylog.jar:?] 2016-07-15_10:23:19.06194 at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:144) ~[graylog.jar:?] 2016-07-15_10:23:19.07075 at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:161) ~[graylog.jar:?] 2016-07-15_10:23:19.08122 at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$TypeOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:205) ~[graylog.jar:?] 2016-07-15_10:23:19.08751 at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:99) ~[graylog.jar:?] 2016-07-15_10:23:19.08883 at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:389) ~[graylog.jar:?] 2016-07-15_10:23:19.09092 at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:347) ~[graylog.jar:?] 2016-07-15_10:23:19.09574 at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:102) ~[graylog.jar:?] 2016-07-15_10:23:19.09706 at org.glassfish.jersey.server.ServerRuntime$2.run(ServerRuntime.java:326) [graylog.jar:?] 2016-07-15_10:23:19.09863 at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271) [graylog.jar:?] 2016-07-15_10:23:19.10037 at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267) [graylog.jar:?] 2016-07-15_10:23:19.10870 at org.glassfish.jersey.internal.Errors.process(Errors.java:315) [graylog.jar:?] 2016-07-15_10:23:19.11463 at org.glassfish.jersey.internal.Errors.process(Errors.java:297) [graylog.jar:?] 2016-07-15_10:23:19.11737 at org.glassfish.jersey.internal.Errors.process(Errors.java:267) [graylog.jar:?] 2016-07-15_10:23:19.11967 at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:317) [graylog.jar:?] 2016-07-15_10:23:19.12130 at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:305) [graylog.jar:?] 2016-07-15_10:23:19.12434 at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:1154) [graylog.jar:?] 2016-07-15_10:23:19.12530 at org.glassfish.jersey.grizzly2.httpserver.GrizzlyHttpContainer.service(GrizzlyHttpContainer.java:384) [graylog.jar:?] 2016-07-15_10:23:19.12632 at org.glassfish.grizzly.http.server.HttpHandler$1.run(HttpHandler.java:224) [graylog.jar:?] 2016-07-15_10:23:19.13050 at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176) [graylog.jar:?] 2016-07-15_10:23:19.13778 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_77] 2016-07-15_10:23:19.13941 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_77] 2016-07-15_10:23:19.14203 at java.lang.Thread.run(Thread.java:745) [?:1.8.0_77] 2016-07-15_10:23:19.14678 Caused by: java.lang.NoSuchMethodError: org.graylog2.indexer.retention.strategies.AbstractIndexCountBasedRetentionStrategy.(Lorg/graylog2/indexer/Deflector;Lorg/graylog2/indexer/indices/Indices;Lorg/graylog2/shared/system/activities/ActivityWriter;)V 2016-07-15_10:23:19.15320 at org.graylog.plugins.archive.indexer.retention.strategies.ArchiveRetentionStrategy.(ArchiveRetentionStrategy.java:45) ~[?:?] 2016-07-15_10:23:19.15575 at org.graylog.plugins.archive.indexer.retention.strategies.ArchiveRetentionStrategy$$FastClassByGuice$$9d47f4fc.newInstance() ~[?:?] 2016-07-15_10:23:19.15685 at com.google.inject.internal.DefaultConstructionProxyFactory$FastClassProxy.newInstance(DefaultConstructionProxyFactory.java:89) ~[graylog.jar:?] 2016-07-15_10:23:19.15825 at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:111) ~[graylog.jar:?] 2016-07-15_10:23:19.16074 at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:90) ~[graylog.jar:?] 2016-07-15_10:23:19.17125 at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:268) ~[graylog.jar:?] 2016-07-15_10:23:19.17287 at com.google.inject.internal.FactoryProxy.get(FactoryProxy.java:56) ~[graylog.jar:?] 2016-07-15_10:23:19.17755 at com.google.inject.internal.InjectorImpl$2$1.call(InjectorImpl.java:1019) ~[graylog.jar:?] 2016-07-15_10:23:19.17881 at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1085) ~[graylog.jar:?] 2016-07-15_10:23:19.18432 at com.google.inject.internal.InjectorImpl$2.get(InjectorImpl.java:1015) ~[graylog.jar:?] 2016-07-15_10:23:19.18516 ... 30 more 2016-07-15_10:23:19.18700 2016-07-15 10:23:18,958 ERROR: org.graylog2.shared.rest.exceptionmappers.AnyExceptionClassMapper - Unhandled exception in REST resource 2016-07-15_10:23:19.18787 com.google.inject.ProvisionException: Unable to provision, see the following errors: 2016-07-15_10:23:19.19073 2016-07-15_10:23:19.19141 1) Error injecting constructor, java.lang.NoSuchMethodError: org.graylog2.indexer.retention.strategies.AbstractIndexCountBasedRetentionStrategy.(Lorg/graylog2/indexer/Deflector;Lorg/graylog2/indexer/indices/Indices;Lorg/graylog2/shared/system/activities/ActivityWriter;)V 2016-07-15_10:23:19.19345 at org.graylog.plugins.archive.indexer.retention.strategies.ArchiveRetentionStrategy.(ArchiveRetentionStrategy.java:45) 2016-07-15_10:23:19.19420 while locating org.graylog.plugins.archive.indexer.retention.strategies.ArchiveRetentionStrategy 2016-07-15_10:23:19.19608 while locating org.graylog2.plugin.indexer.retention.RetentionStrategy annotated with @com.google.inject.multibindings.Element(setName=,uniqueId=286, type=MAPBINDER, keyType=java.lang.String) 2016-07-15_10:23:19.19714 2016-07-15_10:23:19.20653 1 error 2016-07-15_10:23:19.20740 at com.google.inject.internal.InjectorImpl$2.get(InjectorImpl.java:1028) ~[graylog.jar:?] 2016-07-15_10:23:19.20945 at com.google.inject.spi.ProviderLookup$1.get(ProviderLookup.java:104) ~[graylog.jar:?] 2016-07-15_10:23:19.21059 at com.google.inject.multibindings.MapBinder$RealMapBinder$ValueProvider.get(MapBinder.java:821) ~[graylog.jar:?] 2016-07-15_10:23:19.21129 at org.graylog2.rest.resources.system.indices.RetentionStrategyResource.getRetentionStrategyDescription(RetentionStrategyResource.java:163) ~[graylog.jar:?] 2016-07-15_10:23:19.21211 at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193) ~[?:1.8.0_77] 2016-07-15_10:23:19.21697 at java.util.Iterator.forEachRemaining(Iterator.java:116) ~[?:1.8.0_77] 2016-07-15_10:23:19.21826 at java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801) ~[?:1.8.0_77] 2016-07-15_10:23:19.21973 at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481) ~[?:1.8.0_77] 2016-07-15_10:23:19.22182 at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471) ~[?:1.8.0_77] 2016-07-15_10:23:19.22235 at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708) ~[?:1.8.0_77] 2016-07-15_10:23:19.22586 at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[?:1.8.0_77] 2016-07-15_10:23:19.22706 at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499) ~[?:1.8.0_77] 2016-07-15_10:23:19.23585 at org.graylog2.rest.resources.system.indices.RetentionStrategyResource.list(RetentionStrategyResource.java:142) ~[graylog.jar:?] 2016-07-15_10:23:19.23977 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_77] 2016-07-15_10:23:19.24077 at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_77] 2016-07-15_10:23:19.24448 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_77] 2016-07-15_10:23:19.24549 at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_77] 2016-07-15_10:23:19.24729 at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory$1.invoke(ResourceMethodInvocationHandlerFactory.java:81) ~[graylog.jar:?] 2016-07-15_10:23:19.25320 at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:144) ~[graylog.jar:?] 2016-07-15_10:23:19.25490 at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:161) ~[graylog.jar:?] 2016-07-15_10:23:19.26439 at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$TypeOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:205) ~[graylog.jar:?] 2016-07-15_10:23:19.26665 at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:99) ~[graylog.jar:?] 2016-07-15_10:23:19.26881 at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:389) ~[graylog.jar:?] 2016-07-15_10:23:19.27107 at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:347) ~[graylog.jar:?] 2016-07-15_10:23:19.27111 at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:102) ~[graylog.jar:?] 2016-07-15_10:23:19.27225 at org.glassfish.jersey.server.ServerRuntime$2.run(ServerRuntime.java:326) [graylog.jar:?] 2016-07-15_10:23:19.28057 at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271) [graylog.jar:?] 2016-07-15_10:23:19.28063 at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267) [graylog.jar:?] 2016-07-15_10:23:19.28277 at org.glassfish.jersey.internal.Errors.process(Errors.java:315) [graylog.jar:?] 2016-07-15_10:23:19.29308 at org.glassfish.jersey.internal.Errors.process(Errors.java:297) [graylog.jar:?] 2016-07-15_10:23:19.29359 at org.glassfish.jersey.internal.Errors.process(Errors.java:267) [graylog.jar:?] 2016-07-15_10:23:19.29594 at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:317) [graylog.jar:?] 2016-07-15_10:23:19.30318 at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:305) [graylog.jar:?] 2016-07-15_10:23:19.31086 at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:1154) [graylog.jar:?] 2016-07-15_10:23:19.31140 at org.glassfish.jersey.grizzly2.httpserver.GrizzlyHttpContainer.service(GrizzlyHttpContainer.java:384) [graylog.jar:?] 2016-07-15_10:23:19.31192 at org.glassfish.grizzly.http.server.HttpHandler$1.run(HttpHandler.java:224) [graylog.jar:?] 2016-07-15_10:23:19.31276 at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176) [graylog.jar:?] 2016-07-15_10:23:19.31760 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_77] 2016-07-15_10:23:19.32281 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_77] 2016-07-15_10:23:19.32409 at java.lang.Thread.run(Thread.java:745) [?:1.8.0_77] 2016-07-15_10:23:19.32525 Caused by: java.lang.NoSuchMethodError: org.graylog2.indexer.retention.strategies.AbstractIndexCountBasedRetentionStrategy.(Lorg/graylog2/indexer/Deflector;Lorg/graylog2/indexer/indices/Indices;Lorg/graylog2/shared/system/activities/ActivityWriter;)V 2016-07-15_10:23:19.32647 at org.graylog.plugins.archive.indexer.retention.strategies.ArchiveRetentionStrategy.(ArchiveRetentionStrategy.java:45) ~[?:?] 2016-07-15_10:23:19.32742 at org.graylog.plugins.archive.indexer.retention.strategies.ArchiveRetentionStrategy$$FastClassByGuice$$9d47f4fc.newInstance() ~[?:?] 2016-07-15_10:23:19.32875 at com.google.inject.internal.DefaultConstructionProxyFactory$FastClassProxy.newInstance(DefaultConstructionProxyFactory.java:89) ~[graylog.jar:?] 2016-07-15_10:23:19.33246 at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:111) ~[graylog.jar:?] 2016-07-15_10:23:19.34213 at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:90) ~[graylog.jar:?] 2016-07-15_10:23:19.34459 at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:268) ~[graylog.jar:?] 2016-07-15_10:23:19.34531 at com.google.inject.internal.FactoryProxy.get(FactoryProxy.java:56) ~[graylog.jar:?] 2016-07-15_10:23:19.34659 at com.google.inject.internal.InjectorImpl$2$1.call(InjectorImpl.java:1019) ~[graylog.jar:?] 2016-07-15_10:23:19.35412 at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1085) ~[graylog.jar:?] 2016-07-15_10:23:19.35481 at com.google.inject.internal.InjectorImpl$2.get(InjectorImpl.java:1015) ~[graylog.jar:?] 2016-07-15_10:23:19.35641 ... 39 more ```

My Setup was updated from 2.0.3 (OVA) via Omnibus to 2.1.Alpha.2 (only for the records)

colmaengus commented 8 years ago

@jalogisch: I'm not sure how your comment is related. My query wasn't regarding an error but just wondering why polling was being used instead of websockets for notification. In our setup we are using graylog to monitor a kubernetes cluster which includes the reverse proxy fronting graylog. So a lot of the logs we are seeing are from graylog polling.

kroepke commented 8 years ago

@colmaengus We used websockets at some point but feedback overwhelmingly was that it breaks in many of our users environments due to heavy use of proxies which simply do not support them.

Ideally I'd like us to use long polling, too, but from a systems point of view this should not make a difference.

Did you observe a performance problem or is this simply a question?

jalogisch commented 8 years ago

@colmaengus - wrong end from me but let the comment be present for reference

colmaengus commented 8 years ago

@kroepke: no particular performance issue (apart from filling up logs). it would be nice though to have an option to enable websocket as was done in your previous front end arch.

On a side note we've just started using graylog and its really nice to use. We use elasticsearch/kibana for some of our products and its nice to see an opinionated use of elasticsearch for logs compared to what your get with vanilla ELK.

kroepke commented 8 years ago

Glad you like it :)

Regarding websockets, I don't think we are likely to look at it again any time soon. Previously we used sockjs which was supposed to take care of the incompatibilities, but in the end it didn't reliably for our users. Most ended up disabling it so we didn't use it when implementing 2.0.

If you feel strongly about it we are happy to look at pull requests to make it an option but quite honestly I feel it's mostly a waste of effort (even though the reloading annoys me greatly during development 😉 )

colmaengus commented 8 years ago

we use socket.io and haven't had issues with it yet but its early days in the product so maybe we've not hit enough proxy issues.

bernd commented 8 years ago

Closing this because we are not going to implement this in the near future. Websockets and proxy setups are still very hard to get right.

ernani commented 7 years ago

Can the time of the polling or every second or so be reduced to a more fine tuned time?