Graylog2 / graylog2-server

Free and open log management
https://www.graylog.org
Other
7.41k stars 1.07k forks source link

ClusterSystemJobResource - Unknown HK2 failure detected ( Graylog v5.0.0-beta.4+2d0b0a6 ) #13948

Open B3K7 opened 2 years ago

B3K7 commented 2 years ago

Graylog is stuck in a loop.

Elasticsearch cluster Could not retrieve Elasticsearch cluster health. Fetching Elasticsearch cluster health failed: There was an error fetching a resource: Internal Server Error. Additional information: Couldn't read Elasticsearch cluster health

Expected Behavior

Current Behavior

2022-10-29T14:40:54.806-04:00 INFO  [ServerBootstrap] Graylog server up and running.
2022-10-29T14:40:58.961-04:00 INFO  [connection] Opened connection [connectionId{localValue:9, serverValue:705}] to localhost:27017
2022-10-29T14:40:58.962-04:00 INFO  [connection] Opened connection [connectionId{localValue:10, serverValue:706}] to localhost:27017
2022-10-29T14:41:51.536-04:00 WARN  [Errors] The following warnings have been detected: WARNING: Unknown HK2 failure detected:
MultiException stack 1 of 3
org.glassfish.hk2.api.UnsatisfiedDependencyException: There was no object available for injection at SystemInjecteeImpl(requiredType=JobResourceHandlerService,parent=ClusterSystemJobResource,qualifiers={},position=4,optional=false,self=f
alse,unqualified=null,1667087060)
        at org.jvnet.hk2.internal.ThreeThirtyResolver.resolve(ThreeThirtyResolver.java:51)
        at org.jvnet.hk2.internal.ClazzCreator.resolve(ClazzCreator.java:188)
        at org.jvnet.hk2.internal.ClazzCreator.resolveAllDependencies(ClazzCreator.java:205)
        at org.jvnet.hk2.internal.ClazzCreator.create(ClazzCreator.java:334)
        at org.jvnet.hk2.internal.SystemDescriptor.create(SystemDescriptor.java:463)
        at org.glassfish.jersey.inject.hk2.RequestContext.findOrCreate(RequestContext.java:59)
        at org.jvnet.hk2.internal.Utilities.createService(Utilities.java:2102)
        at org.jvnet.hk2.internal.ServiceLocatorImpl.internalGetService(ServiceLocatorImpl.java:758)
        at org.jvnet.hk2.internal.ServiceLocatorImpl.internalGetService(ServiceLocatorImpl.java:721)
        at org.jvnet.hk2.internal.ServiceLocatorImpl.getService(ServiceLocatorImpl.java:691)
        at org.glassfish.jersey.inject.hk2.AbstractHk2InjectionManager.getInstance(AbstractHk2InjectionManager.java:160)
        at org.glassfish.jersey.inject.hk2.ImmediateHk2InjectionManager.getInstance(ImmediateHk2InjectionManager.java:30)
        at org.glassfish.jersey.internal.inject.Injections.getOrCreate(Injections.java:105)
        at org.glassfish.jersey.server.model.MethodHandler$ClassBasedMethodHandler.getInstance(MethodHandler.java:260)
        at org.glassfish.jersey.server.internal.routing.PushMethodHandlerRouter.apply(PushMethodHandlerRouter.java:51)
        at org.glassfish.jersey.server.internal.routing.RoutingStage._apply(RoutingStage.java:86)
        at org.glassfish.jersey.server.internal.routing.RoutingStage._apply(RoutingStage.java:89)
        at org.glassfish.jersey.server.internal.routing.RoutingStage._apply(RoutingStage.java:89)
        at org.glassfish.jersey.server.internal.routing.RoutingStage._apply(RoutingStage.java:89)
        at org.glassfish.jersey.server.internal.routing.RoutingStage.apply(RoutingStage.java:69)
        at org.glassfish.jersey.server.internal.routing.RoutingStage.apply(RoutingStage.java:38)
        at org.glassfish.jersey.process.internal.Stages.process(Stages.java:173)
        at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:247)
        at org.glassfish.jersey.internal.Errors$1.call(Errors.java:248)
        at org.glassfish.jersey.internal.Errors$1.call(Errors.java:244)
        at org.glassfish.jersey.internal.Errors.process(Errors.java:292)
        at org.glassfish.jersey.internal.Errors.process(Errors.java:274)
        at org.glassfish.jersey.internal.Errors.process(Errors.java:244)
        at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:265)
        at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:234)
        at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:684)
        at org.glassfish.jersey.grizzly2.httpserver.GrizzlyHttpContainer.service(GrizzlyHttpContainer.java:356)
        at org.glassfish.grizzly.http.server.HttpHandler$1.run(HttpHandler.java:200)
        at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:180)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
        at java.base/java.lang.Thread.run(Thread.java:833)
MultiException stack 2 of 3
java.lang.IllegalArgumentException: While attempting to resolve the dependencies of org.graylog2.rest.resources.cluster.ClusterSystemJobResource errors were found
        at org.jvnet.hk2.internal.ClazzCreator.resolveAllDependencies(ClazzCreator.java:224)
        at org.jvnet.hk2.internal.ClazzCreator.create(ClazzCreator.java:334)
        at org.jvnet.hk2.internal.SystemDescriptor.create(SystemDescriptor.java:463)
        at org.glassfish.jersey.inject.hk2.RequestContext.findOrCreate(RequestContext.java:59)
        at org.jvnet.hk2.internal.Utilities.createService(Utilities.java:2102)
        at org.jvnet.hk2.internal.ServiceLocatorImpl.internalGetService(ServiceLocatorImpl.java:758)
        at org.jvnet.hk2.internal.ServiceLocatorImpl.internalGetService(ServiceLocatorImpl.java:721)
        at org.jvnet.hk2.internal.ServiceLocatorImpl.getService(ServiceLocatorImpl.java:691)
        at org.glassfish.jersey.inject.hk2.AbstractHk2InjectionManager.getInstance(AbstractHk2InjectionManager.java:160)
        at org.glassfish.jersey.inject.hk2.ImmediateHk2InjectionManager.getInstance(ImmediateHk2InjectionManager.java:30)
        at org.glassfish.jersey.internal.inject.Injections.getOrCreate(Injections.java:105)
        at org.glassfish.jersey.server.model.MethodHandler$ClassBasedMethodHandler.getInstance(MethodHandler.java:260)
        at org.glassfish.jersey.server.internal.routing.PushMethodHandlerRouter.apply(PushMethodHandlerRouter.java:51)
        at org.glassfish.jersey.server.internal.routing.RoutingStage._apply(RoutingStage.java:86)
        at org.glassfish.jersey.server.internal.routing.RoutingStage._apply(RoutingStage.java:89)
        at org.glassfish.jersey.server.internal.routing.RoutingStage._apply(RoutingStage.java:89)
        at org.glassfish.jersey.server.internal.routing.RoutingStage._apply(RoutingStage.java:89)
        at org.glassfish.jersey.server.internal.routing.RoutingStage.apply(RoutingStage.java:69)
        at org.glassfish.jersey.server.internal.routing.RoutingStage.apply(RoutingStage.java:38)
        at org.glassfish.jersey.process.internal.Stages.process(Stages.java:173)
        at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:247)
        at org.glassfish.jersey.internal.Errors$1.call(Errors.java:248)
        at org.glassfish.jersey.internal.Errors$1.call(Errors.java:244)
        at org.glassfish.jersey.internal.Errors.process(Errors.java:292)
        at org.glassfish.jersey.internal.Errors.process(Errors.java:274)
        at org.glassfish.jersey.internal.Errors.process(Errors.java:244)
        at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:265)
        at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:234)
        at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:684)
        at org.glassfish.jersey.grizzly2.httpserver.GrizzlyHttpContainer.service(GrizzlyHttpContainer.java:356)
        at org.glassfish.grizzly.http.server.HttpHandler$1.run(HttpHandler.java:200)
        at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:180)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
        at java.base/java.lang.Thread.run(Thread.java:833)
MultiException stack 3 of 3
java.lang.IllegalStateException: Unable to perform operation: resolve on org.graylog2.rest.resources.cluster.ClusterSystemJobResource
        at org.jvnet.hk2.internal.ClazzCreator.create(ClazzCreator.java:363)
        at org.jvnet.hk2.internal.SystemDescriptor.create(SystemDescriptor.java:463)
        at org.glassfish.jersey.inject.hk2.RequestContext.findOrCreate(RequestContext.java:59)
        at org.jvnet.hk2.internal.Utilities.createService(Utilities.java:2102)
        at org.jvnet.hk2.internal.ServiceLocatorImpl.internalGetService(ServiceLocatorImpl.java:758)
        at org.jvnet.hk2.internal.ServiceLocatorImpl.internalGetService(ServiceLocatorImpl.java:721)
        at org.jvnet.hk2.internal.ServiceLocatorImpl.getService(ServiceLocatorImpl.java:691)
        at org.glassfish.jersey.inject.hk2.AbstractHk2InjectionManager.getInstance(AbstractHk2InjectionManager.java:160)
        at org.glassfish.jersey.inject.hk2.ImmediateHk2InjectionManager.getInstance(ImmediateHk2InjectionManager.java:30)
        at org.glassfish.jersey.internal.inject.Injections.getOrCreate(Injections.java:105)
        at org.glassfish.jersey.server.model.MethodHandler$ClassBasedMethodHandler.getInstance(MethodHandler.java:260)
        at org.glassfish.jersey.server.internal.routing.PushMethodHandlerRouter.apply(PushMethodHandlerRouter.java:51)
        at org.glassfish.jersey.server.internal.routing.RoutingStage._apply(RoutingStage.java:86)
        at org.glassfish.jersey.server.internal.routing.RoutingStage._apply(RoutingStage.java:89)
        at org.glassfish.jersey.server.internal.routing.RoutingStage._apply(RoutingStage.java:89)
        at org.glassfish.jersey.server.internal.routing.RoutingStage._apply(RoutingStage.java:89)
        at org.glassfish.jersey.server.internal.routing.RoutingStage.apply(RoutingStage.java:69)
        at org.glassfish.jersey.server.internal.routing.RoutingStage.apply(RoutingStage.java:38)
        at org.glassfish.jersey.process.internal.Stages.process(Stages.java:173)
        at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:247)
        at org.glassfish.jersey.internal.Errors$1.call(Errors.java:248)
        at org.glassfish.jersey.internal.Errors$1.call(Errors.java:244)
        at org.glassfish.jersey.internal.Errors.process(Errors.java:292)
        at org.glassfish.jersey.internal.Errors.process(Errors.java:274)
        at org.glassfish.jersey.internal.Errors.process(Errors.java:244)
        at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:265)
        at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:234)
        at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:684)
        at org.glassfish.jersey.grizzly2.httpserver.GrizzlyHttpContainer.service(GrizzlyHttpContainer.java:356)
        at org.glassfish.grizzly.http.server.HttpHandler$1.run(HttpHandler.java:200)
        at com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:180)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
        at java.base/java.lang.Thread.run(Thread.java:833)

2022-10-29T14:41:53.534-04:00 WARN  [Errors] The following warnings have been detected: WARNING: Unknown HK2 failure detected:
...

Possible Solution

Steps to Reproduce (for bugs)

1. 2. 3. 4.

Context

Your Environment

B3K7 commented 2 years ago

image

B3K7 commented 2 years ago

user@host # curl -X GET "localhost:9200/_cluster/health?pretty" { "cluster_name" : "graylog", "status" : "green", "timed_out" : false, "number_of_nodes" : 1, "number_of_data_nodes" : 1, "active_primary_shards" : 12, "active_shards" : 12, "relocating_shards" : 0, "initializing_shards" : 0, "unassigned_shards" : 0, "delayed_unassigned_shards" : 0, "number_of_pending_tasks" : 0, "number_of_in_flight_fetch" : 0, "task_max_waiting_in_queue_millis" : 0, "active_shards_percent_as_number" : 100.0 }

B3K7 commented 2 years ago

user@host # curl -XGET 'http://localhost:9200/_cat/health?pretty' 1668178556 14:55:56 graylog green 1 1 12 12 0 0 0 0 - 100.0%

B3K7 commented 2 years ago

user@host # curl -XGET 'http://127.0.0.1:9200/_cluster/health?level=indices' {"cluster_name":"graylog","status":"green","timed_out":false,"number_of_nodes":1,"number_of_data_nodes":1,"active_primary_shards":12,"active_shards":12,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0,"delayed_unassigned_shards":0,"number_of_pending_tasks":0,"number_of_in_flight_fetch":0,"task_max_waiting_in_queue_millis":0,"active_shards_percent_as_number":100.0,"indices":{"gl-events_0":{"status":"green","number_of_shards":4,"number_of_replicas":0,"active_primary_shards":4,"active_shards":4,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0},"graylog_0":{"status":"green","number_of_shards":4,"number_of_replicas":0,"active_primary_shards":4,"active_shards":4,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0},"gl-system-events_0":{"status":"green","number_of_shards":4,"number_of_replicas":0,"active_primary_shards":4,"active_shards":4,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0}}}

B3K7 commented 1 year ago

Uninstalled graylog-server Installed graylog-enterprise

Issue resolved.

B3K7 commented 1 year ago

Issue resolved

B3K7 commented 1 year ago

It is recommended that Graylog improve the quality of licensing related errors.

B3K7 commented 1 year ago

Dennis Oelkers Recommend changing this case from a bug to an issue. (Or closing - your call.)

bernd commented 1 year ago

@B3K7 Can you provide us with some details? Was this a new installation of Graylog 5.0.0-beta, or did you update from an earlier version? (Notice: 5.0 is not stable yet) If you updated from an earlier version, which version did you update from, and did you use Enterprise and a valid license in your old setup?

Thank you!

B3K7 commented 1 year ago

Hi Bernd

To the best of my knowledge, my machine was running 4.3 Graylog Open with the Enterprise Plugin and a valid 2GB free License Key - prior to upgrade. (The website seems to have changed so the terminology may not be 100% correct.) Brent

On Mon, Nov 14, 2022, 5:54 a.m. Bernd Ahlers @.***> wrote:

@B3K7 https://github.com/B3K7 Can you provide us with some details? Was this a new installation of Graylog 5.0.0-beta, or did you update from an earlier version? (Notice: 5.0 is not stable yet) If you updated from an earlier version, which version did you update from, and did you use Enterprise and a valid license in your old setup?

Thank you!

— Reply to this email directly, view it on GitHub https://github.com/Graylog2/graylog2-server/issues/13948#issuecomment-1313473918, or unsubscribe https://github.com/notifications/unsubscribe-auth/AS36C3IBIKG23LZAED4EKJDWIIK7VANCNFSM6AAAAAAR5U6YEQ . You are receiving this because you were mentioned.Message ID: @.***>