Graylog Web Interface randomly disconnects

[demodav]

Server: Ubuntu Server 14.04.3-LTS (amd64 20150908-) for Microsoft Azure. Installation: https://packages.graylog2.org/releases/graylog2-omnibus/ubuntu/graylog_latest.deb Data: graylog-web-interface v1.2.0 (2136f0c) (Oracle Corporation 1.8.0_60 / Linux 3.19.0-28-generic) on SSI-GrayLog

Issue: Fresh installation, No Data inputted simply just clicking the top links [Search, Streams, Dashboard, Sources, Systems] back and forth not doing anything else. Did it a few times when received error message attached.

[mariussturm]

Hi, could you please take into log files under /var/log/graylog/{server|elasticsearch} and provide error messages you find there?

On 17 September 2015 at 19:04, demodav notifications@github.com wrote:

Server: Ubuntu Server 14.04.3-LTS (amd64 20150908-) for Microsoft Azure. Installation: https://packages.graylog2.org/releases/graylog2-omnibus/ubuntu/graylog_latest.deb Data: graylog-web-interface v1.2.0 (2136f0c) (Oracle Corporation 1.8.0_60 / Linux 3.19.0-28-generic) on SSI-GrayLog

Issue: Fresh installation, No Data inputted simply just clicking the top links [Search, Streams, Dashboard, Sources, Systems] back and forth not doing anything else. Did it a few times when received error message attached.

[image: image] https://cloud.githubusercontent.com/assets/651633/9940231/24f1919e-5d34-11e5-8096-cfed3ada3c29.png

— Reply to this email directly or view it on GitHub https://github.com/Graylog2/graylog2-web-interface/issues/1610.

Developer

Tel.: +49 (0)40 609 452 077 Fax.: +49 (0)40 609 452 078

TORCH GmbH - A Graylog Company Steckelhörn 11 20457 Hamburg Germany

https://www.graylog.com https://www.torch.sh/

Commercial Reg. (Registergericht): Amtsgericht Hamburg, HRB 125175 Geschäftsführer: Lennart Koopmann (CEO)

[demodav]

Elasticsearch:

[2015-09-17 21:24:54,718][INFO ][node                     ] [Paladin] version[1.7.1], pid[1935], build[b88f43f/2015-07-29T09:54:16Z]
[2015-09-17 21:24:54,725][INFO ][node                     ] [Paladin] initializing ...
[2015-09-17 21:24:55,035][INFO ][plugins                  ] [Paladin] loaded [], sites []
[2015-09-17 21:24:55,141][INFO ][env                      ] [Paladin] using [1] data paths, mounts [[/ (/dev/sda1)]], net usable_space [25.8gb], net total_space [28.8gb], types [ext4]
[2015-09-17 21:25:04,309][INFO ][node                     ] [Paladin] initialized
[2015-09-17 21:25:04,318][INFO ][node                     ] [Paladin] starting ...
[2015-09-17 21:25:04,655][INFO ][transport                ] [Paladin] bound_address {inet[/100.74.62.62:9300]}, publish_address {inet[/100.74.62.62:9300]}
[2015-09-17 21:25:04,760][INFO ][discovery                ] [Paladin] graylog2/lSiWXX1DQsaGuYG-UEo5Mw
[2015-09-17 21:25:14,955][INFO ][cluster.service          ] [Paladin] new_master [Paladin][lSiWXX1DQsaGuYG-UEo5Mw][SSI-GrayLog][inet[/100.74.62.62:9300]], reason: zen-disco-join (elected_as_master)
[2015-09-17 21:25:15,310][INFO ][http                     ] [Paladin] bound_address {inet[/100.74.62.62:9200]}, publish_address {inet[/100.74.62.62:9200]}
[2015-09-17 21:25:15,311][INFO ][node                     ] [Paladin] started
[2015-09-17 21:25:15,721][INFO ][gateway                  ] [Paladin] recovered [0] indices into cluster_state
[2015-09-17 21:26:04,247][INFO ][node                     ] [Paladin] stopping ...
[2015-09-17 21:26:04,559][INFO ][node                     ] [Paladin] stopped
[2015-09-17 21:26:04,559][INFO ][node                     ] [Paladin] closing ...
[2015-09-17 21:26:04,721][INFO ][node                     ] [Paladin] closed
[2015-09-17 21:26:17,714][INFO ][node                     ] [Arcade] version[1.7.1], pid[2892], build[b88f43f/2015-07-29T09:54:16Z]
[2015-09-17 21:26:17,723][INFO ][node                     ] [Arcade] initializing ...
[2015-09-17 21:26:18,374][INFO ][plugins                  ] [Arcade] loaded [], sites []
[2015-09-17 21:26:18,660][INFO ][env                      ] [Arcade] using [1] data paths, mounts [[/ (/dev/sda1)]], net usable_space [25.4gb], net total_space [28.8gb], types [ext4]
[2015-09-17 21:26:43,606][INFO ][node                     ] [Arcade] initialized
[2015-09-17 21:26:43,625][INFO ][node                     ] [Arcade] starting ...
[2015-09-17 21:26:44,569][INFO ][transport                ] [Arcade] bound_address {inet[/100.74.62.62:9300]}, publish_address {inet[/100.74.62.62:9300]}
[2015-09-17 21:26:44,756][INFO ][discovery                ] [Arcade] graylog2/1zh0StORSayjFSHihOgKhA
[2015-09-17 21:26:54,961][INFO ][cluster.service          ] [Arcade] new_master [Arcade][1zh0StORSayjFSHihOgKhA][SSI-GrayLog][inet[/100.74.62.62:9300]], reason: zen-disco-join (elected_as_master)
[2015-09-17 21:26:55,040][INFO ][http                     ] [Arcade] bound_address {inet[/100.74.62.62:9200]}, publish_address {inet[/100.74.62.62:9200]}
[2015-09-17 21:26:55,048][INFO ][node                     ] [Arcade] started
[2015-09-17 21:26:55,148][INFO ][gateway                  ] [Arcade] recovered [0] indices into cluster_state

Server:

2015-09-18_13:04:49.19815 It looks like you are trying to access MongoDB over HTTP on the native driver port.
2015-09-18_13:04:49.22830 Java HotSpot(TM) 64-Bit Server VM warning: ignoring option PermSize=128m; support was removed in 8.0
2015-09-18_13:04:49.23278 Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=256m; support was removed in 8.0
2015-09-18_13:04:55.80925 INFO  [CmdLineTool] Loaded plugins: [Anonymous Usage Statistics 1.1.1 [org.graylog.plugins.usagestatistics.UsageStatsPlugin]]
2015-09-18_13:04:56.47551 INFO  [CmdLineTool] Running with JVM arguments: -Xms1g -Xmx1500m -XX:NewRatio=1 -XX:PermSize=128m -XX:MaxPermSize=256m -XX:+ResizeTLAB -XX:+UseConcMarkSweepGC -XX:+CMSConcurrentMTEnabled -XX:+CMSClassUnloadingEnabled -XX:+UseParNewGC -XX:-OmitStackTraceInFastThrow -Dlog4j.configuration=file:///opt/graylog/conf/log4j.xml -Djava.library.path=/opt/graylog/server/lib/sigar/
2015-09-18_13:05:23.24889 INFO  [InputBufferImpl] Message journal is enabled.
2015-09-18_13:05:25.02753 INFO  [LogManager] Loading logs.
2015-09-18_13:05:25.81013 INFO  [LogManager] Logs loading complete.
2015-09-18_13:05:25.81033 INFO  [KafkaJournal] Initialized Kafka based journal at /var/opt/graylog/data/journal
2015-09-18_13:05:25.85174 INFO  [InputBufferImpl] Initialized InputBufferImpl with ring size <65536> and wait strategy <BlockingWaitStrategy>, running 2 parallel message handlers.
2015-09-18_13:05:27.10634 INFO  [NodeId] Node ID: e087ded4-ca0d-4ac5-8b35-709e1f9b61c1
2015-09-18_13:05:28.03465 INFO  [node] [graylog2-server] version[1.7.1], pid[1081], build[b88f43f/2015-07-29T09:54:16Z]
2015-09-18_13:05:28.03474 INFO  [node] [graylog2-server] initializing ...
2015-09-18_13:05:28.60598 INFO  [plugins] [graylog2-server] loaded [graylog-monitor], sites []
2015-09-18_13:05:38.78349 INFO  [node] [graylog2-server] initialized
2015-09-18_13:05:39.09068 INFO  [Version] HV000001: Hibernate Validator 5.1.3.Final
2015-09-18_13:05:39.77937 INFO  [ProcessBuffer] Initialized ProcessBuffer with ring size <65536> and wait strategy <BlockingWaitStrategy>.
2015-09-18_13:05:46.88935 INFO  [RulesEngineProvider] No static rules file loaded.
2015-09-18_13:05:47.23310 INFO  [OutputBuffer] Initialized OutputBuffer with ring size <65536> and wait strategy <BlockingWaitStrategy>.
2015-09-18_13:05:49.56411 INFO  [ServerBootstrap] Graylog server 1.2.0 (2136f0c) starting up. (JRE: Oracle Corporation 1.8.0_60 on Linux 3.19.0-28-generic)
2015-09-18_13:05:49.68358 INFO  [PeriodicalsService] Starting 22 periodicals ...
2015-09-18_13:05:49.68433 INFO  [Periodicals] Starting [org.graylog2.periodical.ThroughputCalculator] periodical in [0s], polling every [1s].
2015-09-18_13:05:49.68477 INFO  [node] [graylog2-server] starting ...
2015-09-18_13:05:49.68609 INFO  [Periodicals] Starting [org.graylog2.periodical.AlertScannerThread] periodical in [10s], polling every [60s].
2015-09-18_13:05:49.68640 INFO  [Periodicals] Starting [org.graylog2.periodical.BatchedElasticSearchOutputFlushThread] periodical in [0s], polling every [1s].
2015-09-18_13:05:49.69452 INFO  [Periodicals] Starting [org.graylog2.periodical.ClusterHealthCheckThread] periodical in [0s], polling every [20s].
2015-09-18_13:05:49.73031 INFO  [Periodicals] Starting [org.graylog2.periodical.ContentPackLoaderPeriodical] periodical, running forever.
2015-09-18_13:05:49.76528 INFO  [Periodicals] Starting [org.graylog2.periodical.DeadLetterThread] periodical, running forever.
2015-09-18_13:05:49.78057 INFO  [Periodicals] Starting [org.graylog2.periodical.GarbageCollectionWarningThread] periodical, running forever.
2015-09-18_13:05:49.78472 INFO  [Periodicals] Starting [org.graylog2.periodical.IndexerClusterCheckerThread] periodical in [0s], polling every [30s].
2015-09-18_13:05:49.79442 INFO  [Periodicals] Starting [org.graylog2.periodical.IndexRetentionThread] periodical in [0s], polling every [300s].
2015-09-18_13:05:49.82224 INFO  [Periodicals] Starting [org.graylog2.periodical.IndexRotationThread] periodical in [0s], polling every [10s].
2015-09-18_13:05:49.82928 INFO  [Periodicals] Starting [org.graylog2.periodical.NodePingThread] periodical in [0s], polling every [1s].
2015-09-18_13:05:49.82975 INFO  [Periodicals] Starting [org.graylog2.periodical.StreamThroughputCounterManagerThread] periodical in [0s], polling every [1s].
2015-09-18_13:05:49.85931 INFO  [Periodicals] Starting [org.graylog2.periodical.VersionCheckThread] periodical in [300s], polling every [1800s].
2015-09-18_13:05:49.85954 INFO  [Periodicals] Starting [org.graylog2.periodical.ThrottleStateUpdaterThread] periodical in [1s], polling every [1s].
2015-09-18_13:05:49.85966 INFO  [Periodicals] Starting [org.graylog2.events.ClusterEventPeriodical] periodical in [0s], polling every [1s].
2015-09-18_13:05:49.86171 INFO  [IndexRetentionThread] Elasticsearch cluster not available, skipping index retention checks.
2015-09-18_13:05:49.86437 INFO  [Periodicals] Starting [org.graylog2.events.ClusterEventCleanupPeriodical] periodical in [0s], polling every [300s].
2015-09-18_13:05:49.87417 INFO  [Periodicals] Starting [org.graylog2.periodical.ClusterIdGeneratorPeriodical] periodical, running forever.
2015-09-18_13:05:49.87870 INFO  [Periodicals] Starting [org.graylog2.periodical.PurgeExpiredCollectorsThread] periodical in [0s], polling every [3600s].
2015-09-18_13:05:49.88443 INFO  [Periodicals] Starting [org.graylog2.periodical.IndexRangesMigrationPeriodical] periodical, running forever.
2015-09-18_13:05:49.93337 INFO  [IndexerClusterCheckerThread] Indexer not fully initialized yet. Skipping periodic cluster check.
2015-09-18_13:05:50.34891 INFO  [PeriodicalsService] Not starting [org.graylog2.periodical.UserPermissionMigrationPeriodical] periodical. Not configured to run on this node.
2015-09-18_13:05:50.36343 INFO  [Periodicals] Starting [org.graylog.plugins.usagestatistics.UsageStatsNodePeriodical] periodical in [300s], polling every [21600s].
2015-09-18_13:05:50.36365 INFO  [Periodicals] Starting [org.graylog.plugins.usagestatistics.UsageStatsClusterPeriodical] periodical in [300s], polling every [21600s].
2015-09-18_13:05:50.83631 INFO  [transport] [graylog2-server] bound_address {inet[/0:0:0:0:0:0:0:0:9350]}, publish_address {inet[/100.74.62.62:9350]}
2015-09-18_13:05:51.14837 INFO  [discovery] [graylog2-server] graylog2/URdfM8NkRl-HKPMnyVe9nw
2015-09-18_13:05:51.61600 INFO  [Reflections] Reflections took 1413 ms to scan 1 urls, producing 2 keys and 2 values
2015-09-18_13:05:54.14572 WARN  [discovery] [graylog2-server] waited for 3s and no initial state was set by the discovery
2015-09-18_13:05:54.14606 INFO  [node] [graylog2-server] started
2015-09-18_13:05:55.42269 INFO  [service] [graylog2-server] detected_master [Anvil][sjO44jRQQVGDAr8Y3FSFHA][SSI-GrayLog][inet[/100.74.62.62:9300]], added {[Anvil][sjO44jRQQVGDAr8Y3FSFHA][SSI-GrayLog][inet[/100.74.62.62:9300]],}, reason: zen-disco-receive(from master [[Anvil][sjO44jRQQVGDAr8Y3FSFHA][SSI-GrayLog][inet[/100.74.62.62:9300]]])
2015-09-18_13:06:11.48463 INFO  [RestApiService] Adding security context factory: <org.graylog2.security.ShiroSecurityContextFactory@1294b364>
2015-09-18_13:06:11.56083 INFO  [RestApiService] Started REST API at <http://0.0.0.0:12900/>
2015-09-18_13:06:11.56280 INFO  [ServerBootstrap] Services started, startup times in ms: {OutputSetupService [RUNNING]=26, BufferSynchronizerService [RUNNING]=29, MetricsReporterService [RUNNING]=30, KafkaJournal [RUNNING]=31, JournalReader [RUNNING]=35, DashboardRegistryService [RUNNING]=43, InputSetupService [RUNNING]=56, PeriodicalsService [RUNNING]=739, IndexerSetupService [RUNNING]=5917, RestApiService [RUNNING]=21925}
2015-09-18_13:06:11.56829 INFO  [ServiceManagerListener] Services are healthy
2015-09-18_13:06:11.56927 INFO  [InputSetupService] Triggering launching persisted inputs, node transitioned from Uninitialized?[LB:DEAD] to Running?[LB:ALIVE]
2015-09-18_13:06:11.58255 INFO  [ServerBootstrap] Graylog server up and running.
2015-09-18_13:06:11.78674 INFO  [InputStateListener] Input [GELF TCP/55fb309fe4b0de71abe1302e] is now STARTING

Also, The graylog server periodically shutsdown & The web-interface has to be restarted to view.

I'm using azure I don't know if this matters or not.

[edmundoa]

Hi,

I don't think that using Azure is making any difference, but the sizing of your environment definitely does. Could you please share more information about your environment? At least we need to know how many VMs you use to run Graylog server, web interface, and Elasticsearch, and the resources you gave to each one of them.

[joschi]

@demodav Additionally to what @edmundoa wrote, please also check the logs of your Graylog web interface.

[demodav]

@edmundoa the size of my environment is only 1 VM. Standard A1 (1 core, 1.75 GB memory) I also had the same issue using 2 core.

[edmundoa]

That is definitely not enough for running everything in one box. I would start with 4GB of memory and 2 cores, that should at least work for a small amount of messages.

I am closing the issue now, as the problem is most likely related to not having enough memory. Please reopen if you see the problem still happens with a bigger setup.