search

Graylog2 / graylog2-server

Free and open log management
https://www.graylog.org
Other
7.37k stars 1.06k forks source link

Graylog Server Extremely slow and taking whole cpus and RAM capacity #4359

Closed pgnaleen closed 6 years ago

pgnaleen commented 6 years ago

I am using Graylog server 2.3.2 with elastic search 5.6. It worked few days. suddenly i have noticed that graylog server java deamon taking almost all the cpu and ram. loging into graylog is extremely slow. sometimes can't connect to rest web api of graylog.

Expected Behavior

graylog server work fast with all the funcionality

Current Behavior

can't log into the server most of the times. it show following error

We are experiencing problems connecting to the Graylog server running on http://172.26.76.125:8889/. Please verify that the server is healthy and working correctly.

You will be automatically redirected to the previous page once we can connect to the server.

Do you need a hand? We can help you.

Possible Solution

Steps to Reproduce (for bugs)

  1. just install elastic search and gralog 2.3.2 and wait few days.

Context

Your Environment

graylog logs

2017-11-17T08:55:17.646+05:30 INFO  [PeriodicalsService] Shutdown of periodical [org.graylog.plugins.collector.periodical.PurgeExpiredCollectorsThread] complete, took <0ms>.
2017-11-17T08:55:17.650+05:30 INFO  [GracefulShutdown] Goodbye.
2017-11-17T08:55:17.699+05:30 INFO  [LogManager] Shutting down.
2017-11-17T08:55:17.730+05:30 INFO  [LookupDataAdapterRefreshService] Stopping 0 jobs
2017-11-17T08:55:17.771+05:30 INFO  [JerseyService] Shutting down HTTP listener at <http://172.26.76.125:8889/>
2017-11-17T08:55:17.839+05:30 INFO  [LogManager] Shutdown complete.
2017-11-17T08:55:31.644+05:30 INFO  [NetworkListener] Stopped listener bound to [172.26.76.125:8889]
2017-11-17T08:55:31.646+05:30 INFO  [JerseyService] Shutting down HTTP listener at <http://172.26.76.125:8888/>
2017-11-17T08:55:31.656+05:30 INFO  [NetworkListener] Stopped listener bound to [172.26.76.125:8888]
2017-11-17T08:55:31.657+05:30 INFO  [ServiceManagerListener] Services are now stopped.
2017-11-17T08:55:39.937+05:30 INFO  [CmdLineTool] Loaded plugin: Elastic Beats Input 2.3.2 [org.graylog.plugins.beats.BeatsInputPlugin]
2017-11-17T08:55:39.944+05:30 INFO  [CmdLineTool] Loaded plugin: Collector 2.3.2 [org.graylog.plugins.collector.CollectorPlugin]
2017-11-17T08:55:39.947+05:30 INFO  [CmdLineTool] Loaded plugin: Enterprise Integration Plugin 2.3.2 [org.graylog.plugins.enterprise_integration.EnterpriseIntegrationPlugin]
2017-11-17T08:55:39.949+05:30 INFO  [CmdLineTool] Loaded plugin: MapWidgetPlugin 2.3.2 [org.graylog.plugins.map.MapWidgetPlugin]
2017-11-17T08:55:39.979+05:30 INFO  [CmdLineTool] Loaded plugin: Pipeline Processor Plugin 2.3.2 [org.graylog.plugins.pipelineprocessor.ProcessorPlugin]
2017-11-17T08:55:39.982+05:30 INFO  [CmdLineTool] Loaded plugin: Anonymous Usage Statistics 2.3.2 [org.graylog.plugins.usagestatistics.UsageStatsPlugin]
2017-11-17T08:55:40.574+05:30 INFO  [CmdLineTool] Running with JVM arguments: -Xms2g -Xmx2g -XX:NewRatio=1 -XX:+ResizeTLAB -XX:+UseConcMarkSweepGC -XX:+CMSConcurrentMTEnabled -XX:+CMSClassUnloadingEnabled -XX:+UseParNewGC -XX:-OmitStackTraceInFastThrow -Dlog4j.configurationFile=file:///etc/graylog/server/log4j2.xml -Djava.library.path=/usr/share/graylog-server/lib/sigar -Dgraylog2.installation_source=rpm
2017-11-17T08:55:41.256+05:30 INFO  [Version] HV000001: Hibernate Validator null
2017-11-17T08:55:47.232+05:30 INFO  [InputBufferImpl] Message journal is enabled.
2017-11-17T08:55:47.300+05:30 INFO  [NodeId] Node ID: ff758132-62e1-450c-b0dd-4ef795ce871d
2017-11-17T08:55:47.989+05:30 INFO  [LogManager] Loading logs.
2017-11-17T08:55:48.132+05:30 INFO  [LogManager] Logs loading complete.
2017-11-17T08:55:48.134+05:30 INFO  [KafkaJournal] Initialized Kafka based journal at /var/lib/graylog-server/journal
2017-11-17T08:55:48.170+05:30 INFO  [InputBufferImpl] Initialized InputBufferImpl with ring size <65536> and wait strategy <BlockingWaitStrategy>, running 2 parallel message handlers.
2017-11-17T08:55:48.218+05:30 INFO  [cluster] Cluster created with settings {hosts=[localhost:27017], mode=SINGLE, requiredClusterType=UNKNOWN, serverSelectionTimeout='30000 ms', maxWaitQueueSize=5000}
2017-11-17T08:55:48.335+05:30 INFO  [cluster] No server chosen by ReadPreferenceServerSelector{readPreference=primary} from cluster description ClusterDescription{type=UNKNOWN, connectionMode=SINGLE, serverDescriptions=[ServerDescription{address=localhost:27017, type=UNKNOWN, state=CONNECTING}]}. Waiting for 30000 ms before timing out
2017-11-17T08:55:48.392+05:30 INFO  [connection] Opened connection [connectionId{localValue:1, serverValue:953}] to localhost:27017
2017-11-17T08:55:48.399+05:30 INFO  [cluster] Monitor thread successfully connected to server with description ServerDescription{address=localhost:27017, type=STANDALONE, state=CONNECTED, ok=true, version=ServerVersion{versionList=[3, 2, 17]}, minWireVersion=0, maxWireVersion=4, maxDocumentSize=16777216, roundTripTimeNanos=1216636}
2017-11-17T08:55:48.416+05:30 INFO  [connection] Opened connection [connectionId{localValue:2, serverValue:954}] to localhost:27017
2017-11-17T08:55:49.124+05:30 INFO  [AbstractJestClient] Setting server pool to a list of 1 servers: [http://127.0.0.1:9200]
2017-11-17T08:55:49.128+05:30 INFO  [JestClientFactory] Using multi thread/connection supporting pooling connection manager
2017-11-17T08:55:49.229+05:30 INFO  [JestClientFactory] Using custom ObjectMapper instance
2017-11-17T08:55:49.229+05:30 INFO  [JestClientFactory] Node Discovery disabled...
2017-11-17T08:55:49.230+05:30 INFO  [JestClientFactory] Idle connection reaping disabled...
2017-11-17T08:55:49.617+05:30 INFO  [ProcessBuffer] Initialized ProcessBuffer with ring size <65536> and wait strategy <BlockingWaitStrategy>.
2017-11-17T08:55:53.269+05:30 INFO  [RulesEngineProvider] No static rules file loaded.
2017-11-17T08:55:53.504+05:30 INFO  [connection] Opened connection [connectionId{localValue:3, serverValue:955}] to localhost:27017
2017-11-17T08:55:54.244+05:30 WARN  [GeoIpResolverEngine] GeoIP database file does not exist: /etc/graylog/server/GeoLite2-City.mmdb
2017-11-17T08:55:54.261+05:30 INFO  [OutputBuffer] Initialized OutputBuffer with ring size <65536> and wait strategy <BlockingWaitStrategy>.
2017-11-17T08:55:54.305+05:30 WARN  [GeoIpResolverEngine] GeoIP database file does not exist: /etc/graylog/server/GeoLite2-City.mmdb
2017-11-17T08:55:54.339+05:30 WARN  [GeoIpResolverEngine] GeoIP database file does not exist: /etc/graylog/server/GeoLite2-City.mmdb
2017-11-17T08:55:54.379+05:30 WARN  [GeoIpResolverEngine] GeoIP database file does not exist: /etc/graylog/server/GeoLite2-City.mmdb
2017-11-17T08:55:54.415+05:30 WARN  [GeoIpResolverEngine] GeoIP database file does not exist: /etc/graylog/server/GeoLite2-City.mmdb
2017-11-17T08:55:55.075+05:30 INFO  [ServerBootstrap] Graylog server 2.3.2+3df951e starting up
2017-11-17T08:55:55.077+05:30 INFO  [ServerBootstrap] JRE: Oracle Corporation 1.8.0_111 on Linux 3.10.0-693.5.2.el7.x86_64
2017-11-17T08:55:55.077+05:30 INFO  [ServerBootstrap] Deployment: rpm
2017-11-17T08:55:55.078+05:30 INFO  [ServerBootstrap] OS: Red Hat Enterprise Linux Server 7.4 (Maipo) (rhel)
2017-11-17T08:55:55.078+05:30 INFO  [ServerBootstrap] Arch: amd64
2017-11-17T08:55:55.085+05:30 WARN  [DeadEventLoggingListener] Received unhandled event of type <org.graylog2.plugin.lifecycles.Lifecycle> from event bus <AsyncEventBus{graylog-eventbus}>
2017-11-17T08:55:55.163+05:30 INFO  [PeriodicalsService] Starting 26 periodicals ...
2017-11-17T08:55:55.165+05:30 INFO  [Periodicals] Starting [org.graylog2.periodical.ThroughputCalculator] periodical in [0s], polling every [1s].
2017-11-17T08:55:55.196+05:30 INFO  [Periodicals] Starting [org.graylog2.periodical.AlertScannerThread] periodical in [10s], polling every [60s].
2017-11-17T08:55:55.198+05:30 INFO  [Periodicals] Starting [org.graylog2.periodical.BatchedElasticSearchOutputFlushThread] periodical in [0s], polling every [1s].
2017-11-17T08:55:55.201+05:30 INFO  [Periodicals] Starting [org.graylog2.periodical.ClusterHealthCheckThread] periodical in [120s], polling every [20s].
2017-11-17T08:55:55.204+05:30 INFO  [Periodicals] Starting [org.graylog2.periodical.ContentPackLoaderPeriodical] periodical, running forever.
2017-11-17T08:55:55.363+05:30 INFO  [Periodicals] Starting [org.graylog2.periodical.GarbageCollectionWarningThread] periodical, running forever.
2017-11-17T08:55:55.365+05:30 INFO  [Periodicals] Starting [org.graylog2.periodical.IndexerClusterCheckerThread] periodical in [0s], polling every [30s].
2017-11-17T08:55:55.368+05:30 INFO  [Periodicals] Starting [org.graylog2.periodical.IndexRetentionThread] periodical in [0s], polling every [300s].
2017-11-17T08:55:55.512+05:30 INFO  [connection] Opened connection [connectionId{localValue:4, serverValue:956}] to localhost:27017
2017-11-17T08:55:55.533+05:30 INFO  [Periodicals] Starting [org.graylog2.periodical.IndexRotationThread] periodical in [0s], polling every [10s].
2017-11-17T08:55:55.542+05:30 INFO  [connection] Opened connection [connectionId{localValue:5, serverValue:957}] to localhost:27017
2017-11-17T08:55:55.544+05:30 INFO  [Periodicals] Starting [org.graylog2.periodical.NodePingThread] periodical in [0s], polling every [1s].
2017-11-17T08:55:55.549+05:30 INFO  [Periodicals] Starting [org.graylog2.periodical.VersionCheckThread] periodical in [300s], polling every [1800s].
2017-11-17T08:55:55.554+05:30 INFO  [Periodicals] Starting [org.graylog2.periodical.ThrottleStateUpdaterThread] periodical in [1s], polling every [1s].
2017-11-17T08:55:55.556+05:30 INFO  [Periodicals] Starting [org.graylog2.events.ClusterEventPeriodical] periodical in [0s], polling every [1s].
2017-11-17T08:55:55.562+05:30 INFO  [Periodicals] Starting [org.graylog2.events.ClusterEventCleanupPeriodical] periodical in [0s], polling every [86400s].
2017-11-17T08:55:55.578+05:30 INFO  [Periodicals] Starting [org.graylog2.periodical.ClusterIdGeneratorPeriodical] periodical, running forever.
2017-11-17T08:55:55.579+05:30 INFO  [Periodicals] Starting [org.graylog2.periodical.IndexRangesMigrationPeriodical] periodical, running forever.
2017-11-17T08:55:55.593+05:30 INFO  [Periodicals] Starting [org.graylog2.periodical.IndexRangesCleanupPeriodical] periodical in [15s], polling every [3600s].
2017-11-17T08:55:55.611+05:30 INFO  [connection] Opened connection [connectionId{localValue:7, serverValue:959}] to localhost:27017
2017-11-17T08:55:55.629+05:30 INFO  [connection] Opened connection [connectionId{localValue:6, serverValue:958}] to localhost:27017
2017-11-17T08:55:55.629+05:30 INFO  [connection] Opened connection [connectionId{localValue:10, serverValue:962}] to localhost:27017
2017-11-17T08:55:55.641+05:30 INFO  [connection] Opened connection [connectionId{localValue:8, serverValue:960}] to localhost:27017
2017-11-17T08:55:55.685+05:30 INFO  [connection] Opened connection [connectionId{localValue:9, serverValue:961}] to localhost:27017
2017-11-17T08:55:55.736+05:30 INFO  [PeriodicalsService] Not starting [org.graylog2.periodical.UserPermissionMigrationPeriodical] periodical. Not configured to run on this node.
2017-11-17T08:55:55.736+05:30 INFO  [Periodicals] Starting [org.graylog2.periodical.AlarmCallbacksMigrationPeriodical] periodical, running forever.
2017-11-17T08:55:55.743+05:30 INFO  [Periodicals] Starting [org.graylog2.periodical.ConfigurationManagementPeriodical] periodical, running forever.
2017-11-17T08:55:55.769+05:30 INFO  [Periodicals] Starting [org.graylog2.periodical.LdapGroupMappingMigration] periodical, running forever.
2017-11-17T08:55:55.777+05:30 INFO  [Periodicals] Starting [org.graylog2.periodical.IndexFailuresPeriodical] periodical, running forever.
2017-11-17T08:55:55.795+05:30 INFO  [Periodicals] Starting [org.graylog.plugins.usagestatistics.UsageStatsNodePeriodical] periodical in [300s], polling every [21600s].
2017-11-17T08:55:55.802+05:30 INFO  [Periodicals] Starting [org.graylog.plugins.usagestatistics.UsageStatsClusterPeriodical] periodical in [300s], polling every [21600s].
2017-11-17T08:55:55.974+05:30 INFO  [Periodicals] Starting [org.graylog.plugins.pipelineprocessor.periodical.LegacyDefaultStreamMigration] periodical, running forever.
2017-11-17T08:55:55.977+05:30 INFO  [Periodicals] Starting [org.graylog.plugins.collector.periodical.PurgeExpiredCollectorsThread] periodical in [0s], polling every [3600s].
2017-11-17T08:55:56.052+05:30 INFO  [LegacyDefaultStreamMigration] Legacy default stream has no connections, no migration needed.
2017-11-17T08:55:56.892+05:30 INFO  [JerseyService] Enabling CORS for HTTP endpoint
2017-11-17T08:56:20.812+05:30 INFO  [NetworkListener] Started listener bound to [172.26.76.125:8889]
2017-11-17T08:56:20.818+05:30 INFO  [HttpServer] [HttpServer] Started.
2017-11-17T08:56:20.818+05:30 INFO  [JerseyService] Started REST API at <http://172.26.76.125:8889/>
2017-11-17T08:56:27.012+05:30 INFO  [NetworkListener] Started listener bound to [172.26.76.125:8888]
2017-11-17T08:56:27.013+05:30 INFO  [HttpServer] [HttpServer-1] Started.
2017-11-17T08:56:27.014+05:30 INFO  [JerseyService] Started Web Interface at <http://172.26.76.125:8888/>
2017-11-17T08:56:27.025+05:30 INFO  [ServiceManagerListener] Services are healthy
2017-11-17T08:56:27.033+05:30 INFO  [InputSetupService] Triggering launching persisted inputs, node transitioned from Uninitialized [LB:DEAD] to Running [LB:ALIVE]
2017-11-17T08:56:27.037+05:30 INFO  [ServerBootstrap] Services started, startup times in ms: {OutputSetupService [RUNNING]=193, BufferSynchronizerService [RUNNING]=202, KafkaJournal [RUNNING]=334, InputSetupService [RUNNING]=492, StreamCacheService [RUNNING]=591, LookupTableService [RUNNING]=604, JournalReader [RUNNING]=613, ConfigurationEtagService [RUNNING]=627, PeriodicalsService [RUNNING]=856, JerseyService [RUNNING]=31865}
2017-11-17T08:56:27.048+05:30 INFO  [ServerBootstrap] Graylog server up and running.
2017-11-17T08:56:27.071+05:30 INFO  [KafkaJournal] Read offset 780932 before start of log at 793060, starting to read from the beginning of the journal.
2017-11-17T08:56:27.119+05:30 INFO  [InputStateListener] Input [Syslog UDP/5a0c212c4780d83c1f41f2ed] is now STARTING
2017-11-17T08:56:27.317+05:30 WARN  [NettyTransport] receiveBufferSize (SO_RCVBUF) for input SyslogUDPInput{title=service loan logs, type=org.graylog2.inputs.syslog.udp.SyslogUDPInput, nodeId=null} should be 262144 but is 212992.
2017-11-17T08:56:27.370+05:30 INFO  [InputStateListener] Input [Syslog UDP/5a0c212c4780d83c1f41f2ed] is now RUNNING
2017-11-17T09:09:10.907+05:30 WARN  [NodePingThread] Did not find meta info of this node. Re-registering.
2017-11-17T09:17:24.514+05:30 WARN  [NodePingThread] Did not find meta info of this node. Re-registering.
2017-11-17T09:18:48.662+05:30 WARN  [NodePingThread] Did not find meta info of this node. Re-registering.
2017-11-17T09:18:56.520+05:30 WARN  [NodePingThread] Did not find meta info of this node. Re-registering.
2017-11-17T09:19:11.441+05:30 WARN  [NodePingThread] Did not find meta info of this node. Re-registering.
2017-11-17T09:19:46.889+05:30 WARN  [ProxiedResource] Unable to call http://172.26.76.125:8889/system/metrics/multiple on node <ff758132-62e1-450c-b0dd-4ef795ce871d>
java.net.SocketTimeoutException: Read timed out
    at java.net.SocketInputStream.socketRead0(Native Method) ~[?:1.8.0_111]
    at java.net.SocketInputStream.socketRead(SocketInputStream.java:116) ~[?:1.8.0_111]
    at java.net.SocketInputStream.read(SocketInputStream.java:170) ~[?:1.8.0_111]
    at java.net.SocketInputStream.read(SocketInputStream.java:141) ~[?:1.8.0_111]
    at okio.Okio$2.read(Okio.java:139) ~[graylog.jar:?]
    at okio.AsyncTimeout$2.read(AsyncTimeout.java:237) ~[graylog.jar:?]
    at okio.RealBufferedSource.indexOf(RealBufferedSource.java:345) ~[graylog.jar:?]
    at okio.RealBufferedSource.readUtf8LineStrict(RealBufferedSource.java:217) ~[graylog.jar:?]
    at okio.RealBufferedSource.readUtf8LineStrict(RealBufferedSource.java:211) ~[graylog.jar:?]
    at okhttp3.internal.http1.Http1Codec.readResponseHeaders(Http1Codec.java:189) ~[graylog.jar:?]
    at okhttp3.internal.http.CallServerInterceptor.intercept(CallServerInterceptor.java:75) ~[graylog.jar:?]
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92) ~[graylog.jar:?]
    at okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.java:45) ~[graylog.jar:?]
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92) ~[graylog.jar:?]
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67) ~[graylog.jar:?]
    at okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.java:93) ~[graylog.jar:?]
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92) ~[graylog.jar:?]
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67) ~[graylog.jar:?]
    at okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.java:93) ~[graylog.jar:?]
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92) ~[graylog.jar:?]
    at okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.java:120) ~[graylog.jar:?]
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92) ~[graylog.jar:?]
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67) ~[graylog.jar:?]
    at org.graylog2.rest.RemoteInterfaceProvider.lambda$get$0(RemoteInterfaceProvider.java:59) ~[graylog.jar:?]
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92) ~[graylog.jar:?]
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67) ~[graylog.jar:?]
    at okhttp3.RealCall.getResponseWithInterceptorChain(RealCall.java:185) ~[graylog.jar:?]
    at okhttp3.RealCall.execute(RealCall.java:69) ~[graylog.jar:?]
    at retrofit2.OkHttpCall.execute(OkHttpCall.java:180) ~[graylog.jar:?]
    at org.graylog2.shared.rest.resources.ProxiedResource.lambda$getForAllNodes$0(ProxiedResource.java:76) ~[graylog.jar:?]
    at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_111]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_111]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_111]
    at java.lang.Thread.run(Thread.java:745) [?:1.8.0_111]
2017-11-17T09:20:11.571+05:30 WARN  [NodePingThread] Did not find meta info of this node. Re-registering.
2017-11-17T09:21:49.414+05:30 WARN  [ProxiedResource] Unable to call http://172.26.76.125:8889/system/inputstates on node <ff758132-62e1-450c-b0dd-4ef795ce871d>
java.net.SocketTimeoutException: Read timed out
    at java.net.SocketInputStream.socketRead0(Native Method) ~[?:1.8.0_111]
    at java.net.SocketInputStream.socketRead(SocketInputStream.java:116) ~[?:1.8.0_111]
    at java.net.SocketInputStream.read(SocketInputStream.java:170) ~[?:1.8.0_111]
    at java.net.SocketInputStream.read(SocketInputStream.java:141) ~[?:1.8.0_111]
    at okio.Okio$2.read(Okio.java:139) ~[graylog.jar:?]
    at okio.AsyncTimeout$2.read(AsyncTimeout.java:237) ~[graylog.jar:?]
    at okio.RealBufferedSource.indexOf(RealBufferedSource.java:345) ~[graylog.jar:?]
    at okio.RealBufferedSource.readUtf8LineStrict(RealBufferedSource.java:217) ~[graylog.jar:?]
    at okio.RealBufferedSource.readUtf8LineStrict(RealBufferedSource.java:211) ~[graylog.jar:?]
    at okhttp3.internal.http1.Http1Codec.readResponseHeaders(Http1Codec.java:189) ~[graylog.jar:?]
    at okhttp3.internal.http.CallServerInterceptor.intercept(CallServerInterceptor.java:75) ~[graylog.jar:?]
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92) ~[graylog.jar:?]
    at okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.java:45) ~[graylog.jar:?]
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92) ~[graylog.jar:?]
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67) ~[graylog.jar:?]
    at okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.java:93) ~[graylog.jar:?]
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92) ~[graylog.jar:?]
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67) ~[graylog.jar:?]
    at okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.java:93) ~[graylog.jar:?]
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92) ~[graylog.jar:?]
    at okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.java:120) ~[graylog.jar:?]
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92) ~[graylog.jar:?]
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67) ~[graylog.jar:?]
    at org.graylog2.rest.RemoteInterfaceProvider.lambda$get$0(RemoteInterfaceProvider.java:59) ~[graylog.jar:?]
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92) ~[graylog.jar:?]
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67) ~[graylog.jar:?]
    at okhttp3.RealCall.getResponseWithInterceptorChain(RealCall.java:185) ~[graylog.jar:?]
    at okhttp3.RealCall.execute(RealCall.java:69) ~[graylog.jar:?]
    at retrofit2.OkHttpCall.execute(OkHttpCall.java:180) ~[graylog.jar:?]
    at org.graylog2.shared.rest.resources.ProxiedResource.lambda$getForAllNodes$0(ProxiedResource.java:76) ~[graylog.jar:?]
    at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_111]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_111]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_111]
    at java.lang.Thread.run(Thread.java:745) [?:1.8.0_111]
2017-11-17T09:22:12.613+05:30 WARN  [ProxiedResource] Unable to call http://172.26.76.125:8889/system/inputstates on node <ff758132-62e1-450c-b0dd-4ef795ce871d>
java.net.SocketTimeoutException: Read timed out
    at java.net.SocketInputStream.socketRead0(Native Method) ~[?:1.8.0_111]
    at java.net.SocketInputStream.socketRead(SocketInputStream.java:116) ~[?:1.8.0_111]
    at java.net.SocketInputStream.read(SocketInputStream.java:170) ~[?:1.8.0_111]
    at java.net.SocketInputStream.read(SocketInputStream.java:141) ~[?:1.8.0_111]
    at okio.Okio$2.read(Okio.java:139) ~[graylog.jar:?]
    at okio.AsyncTimeout$2.read(AsyncTimeout.java:237) ~[graylog.jar:?]
    at okio.RealBufferedSource.indexOf(RealBufferedSource.java:345) ~[graylog.jar:?]
    at okio.RealBufferedSource.readUtf8LineStrict(RealBufferedSource.java:217) ~[graylog.jar:?]
    at okio.RealBufferedSource.readUtf8LineStrict(RealBufferedSource.java:211) ~[graylog.jar:?]
    at okhttp3.internal.http1.Http1Codec.readResponseHeaders(Http1Codec.java:189) ~[graylog.jar:?]
    at okhttp3.internal.http.CallServerInterceptor.intercept(CallServerInterceptor.java:75) ~[graylog.jar:?]
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92) ~[graylog.jar:?]
    at okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.java:45) ~[graylog.jar:?]
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92) ~[graylog.jar:?]
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67) ~[graylog.jar:?]
    at okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.java:93) ~[graylog.jar:?]
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92) ~[graylog.jar:?]
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67) ~[graylog.jar:?]
    at okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.java:93) ~[graylog.jar:?]
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92) ~[graylog.jar:?]
    at okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.java:120) ~[graylog.jar:?]
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92) ~[graylog.jar:?]
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67) ~[graylog.jar:?]
    at org.graylog2.rest.RemoteInterfaceProvider.lambda$get$0(RemoteInterfaceProvider.java:59) ~[graylog.jar:?]
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92) ~[graylog.jar:?]
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67) ~[graylog.jar:?]
    at okhttp3.RealCall.getResponseWithInterceptorChain(RealCall.java:185) ~[graylog.jar:?]
    at okhttp3.RealCall.execute(RealCall.java:69) ~[graylog.jar:?]
    at retrofit2.OkHttpCall.execute(OkHttpCall.java:180) ~[graylog.jar:?]
    at org.graylog2.shared.rest.resources.ProxiedResource.lambda$getForAllNodes$0(ProxiedResource.java:76) ~[graylog.jar:?]
    at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_111]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_111]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_111]
    at java.lang.Thread.run(Thread.java:745) [?:1.8.0_111]
2017-11-17T09:25:57.513+05:30 WARN  [NodePingThread] Did not find meta info of this node. Re-registering.
2017-11-17T09:26:16.718+05:30 WARN  [NodePingThread] Did not find meta info of this node. Re-registering.
2017-11-17T09:31:31.818+05:30 WARN  [NodePingThread] Did not find meta info of this node. Re-registering.
2017-11-17T09:33:36.825+05:30 WARN  [NodePingThread] Did not find meta info of this node. Re-registering.
2017-11-17T09:37:38.965+05:30 WARN  [NodePingThread] Did not find meta info of this node. Re-registering.

top result

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND                                                                                                                   
31804 graylog   20   0 5375288 2.273g  20720 S 100.0 39.0  58:54.38 java                                                                                                                      
12641 mongod    20   0  874952  68628  52496 S   1.7  1.1  25:39.09 mongod                                                                                                                    
 9195 elastic+  20   0 5339432 427500   5600 S   0.7  7.0   7:30.67 java                                                                                                                      
12361 root      20   0  176268   1836   1140 S   0.3  0.0   3:12.82 vmtoolsd                                                                                                                  
12443 root      20   0  200652    428    124 S   0.3  0.0   1:23.14 ManagementAgent                                                                                                           
32126 root      20   0  157712   2184   1508 R   0.3  0.0   0:00.06 top                                                                                                                       
    1 root      20   0  128168   4092   2452 S   0.0  0.1   0:35.88 systemd                                                                                                                   
    2 root      20   0       0      0      0 S   0.0  0.0   0:00.03 kthreadd
iliaselmatani commented 6 years ago

It’s probably because Graylog is unable to connect to one services :

WARN [ProxiedResource] Unable to call http://172.26.76.125:8889/system/metrics/multiple on node java.net.SocketTimeoutException: Read timed out

Are all services up and running and reachable?

Regards, Ilias

pgnaleen commented 6 years ago

[root@testmaster ~]# netstat -nlp | grep 8889 tcp6 0 0 172.26.76.125:8889 ::: LISTEN 32345/java
[root@testmaster ~]# netstat -nlp | grep 8888 tcp6 0 0 172.26.76.125:8888 ::: LISTEN 32345/java
[root@testmaster ~]#

pgnaleen commented 6 years ago

both working fine

pgnaleen commented 6 years ago

image

iliaselmatani commented 6 years ago

Can you share your configuration lines, especially the listen URI lines.

iliaselmatani commented 6 years ago

Please edit your first part of the comment and remove your password (hash - root_password_sha2) and email address. Can you try using the 'Insert a quote' for pasting your configuration? That's making a lot of sense.

pgnaleen commented 6 years ago

server.log

pgnaleen commented 6 years ago

this is teh config file i have changed the extention in order to upload to github

edmundoa commented 6 years ago

Hi,

We use GitHub issues for tracking bugs in Graylog itself, but this looks like a configuration issue. Would you please be so kind and move the conversation to our discussion forum or the #graylog channel on freenode IRC?

Thank you!

jalogisch commented 6 years ago

For reference: https://community.graylog.org/t/graylog-server-extremely-slow-cant-get-system-inputs/3186/12

pgnaleen commented 6 years ago

no one replying there. please reply here. i will reopen ticket

joschi commented 6 years ago

@pgnaleen No. The discussion will stay in the community forums.