search

Graylog2 / graylog2-server

Free and open log management
https://www.graylog.org
Other
7.37k stars 1.06k forks source link

Aws elasticsearch issues #4330

Closed jean-francois-labbe closed 6 years ago

jean-francois-labbe commented 6 years ago

I could get Graylog work with an elasticsearch using docker compose https://hub.docker.com/r/graylog/graylog

Now I need to use Aws elasticsearch, but there is no clear configuration to use with https://hub.docker.com/r/graylog/graylog as the default options in the docker container seems to be different than those in the documentation (http://docs.graylog.org/en/2.3/pages/configuration/server.conf.html#elasticsearch)

for example: elasticsearch_discovery_enabled = false but it's set to true in the container. I had to manually set it to false using: GRAYLOG_ELASTICSEARCH_DISCOVERY_ENABLED

I managed to get Graylog to discover the elastic cluster and send all it's journal to Aws Elasticsearch but on the search page I get the error: Unable to perform search query. In the system overview it never displays the elasticsearch cluster state even though it sent all the messages from the journal.

What should I do to make it work?

Here are the server start logs:

2017-11-08 20:06:01,903 INFO : org.graylog2.bootstrap.CmdLineTool - Loaded plugin: Elastic Beats Input 2.3.2 [org.graylog.plugins.beats.BeatsInputPlugin]
2017-11-08 20:06:01,908 INFO : org.graylog2.bootstrap.CmdLineTool - Loaded plugin: Collector 2.3.2 [org.graylog.plugins.collector.CollectorPlugin]
2017-11-08 20:06:01,909 INFO : org.graylog2.bootstrap.CmdLineTool - Loaded plugin: Enterprise Integration Plugin 2.3.2 [org.graylog.plugins.enterprise_integration.EnterpriseIntegrationPlugin]
2017-11-08 20:06:01,909 INFO : org.graylog2.bootstrap.CmdLineTool - Loaded plugin: MapWidgetPlugin 2.3.2 [org.graylog.plugins.map.MapWidgetPlugin]
2017-11-08 20:06:01,917 INFO : org.graylog2.bootstrap.CmdLineTool - Loaded plugin: Pipeline Processor Plugin 2.3.2 [org.graylog.plugins.pipelineprocessor.ProcessorPlugin]
2017-11-08 20:06:01,918 INFO : org.graylog2.bootstrap.CmdLineTool - Loaded plugin: QuickValuesPlusWidget 3.0.1 [org.graylog.plugins.quickvaluesplus.QuickValuesPlusWidgetPlugin]
2017-11-08 20:06:01,919 INFO : org.graylog2.bootstrap.CmdLineTool - Loaded plugin: Anonymous Usage Statistics 2.3.2 [org.graylog.plugins.usagestatistics.UsageStatsPlugin]
2017-11-08 20:06:01,919 INFO : org.graylog2.bootstrap.CmdLineTool - Loaded plugin: Slack 2.4.0 [org.graylog2.plugins.slack.callback.SlackAlarmCallback]
2017-11-08 20:06:02,157 INFO : org.graylog2.bootstrap.CmdLineTool - Running with JVM arguments: -XX:+UnlockExperimentalVMOptions -XX:+UseCGroupMemoryLimitForHeap -XX:NewRatio=1 -XX:MaxMetaspaceSize=256m -XX:+ResizeTLAB -XX:+UseConcMarkSweepGC -XX:+CMSConcurrentMTEnabled -XX:+CMSClassUnloadingEnabled -XX:+UseParNewGC -XX:-OmitStackTraceInFastThrow -Dlog4j.configurationFile=/usr/share/graylog/data/config/log4j2.xml -Djava.library.path=/usr/share/graylog/lib/sigar/ -Dgraylog2.installation_source=docker
2017-11-08 20:06:02,370 INFO : org.hibernate.validator.internal.util.Version - HV000001: Hibernate Validator null
2017-11-08 20:06:05,354 INFO : org.graylog2.shared.buffers.InputBufferImpl - Message journal is enabled.
2017-11-08 20:06:05,393 INFO : org.graylog2.plugin.system.NodeId - Node ID: 2f00df0f-166d-43e2-9abd-97b115d618cb
2017-11-08 20:06:05,608 INFO : kafka.log.LogManager - Loading logs.
2017-11-08 20:06:05,747 INFO : kafka.log.LogManager - Logs loading complete.
2017-11-08 20:06:05,747 INFO : org.graylog2.shared.journal.KafkaJournal - Initialized Kafka based journal at /usr/share/graylog/data/journal
2017-11-08 20:06:05,759 INFO : org.graylog2.shared.buffers.InputBufferImpl - Initialized InputBufferImpl with ring size <65536> and wait strategy <BlockingWaitStrategy>, running 2 parallel message handlers.
2017-11-08 20:06:05,776 INFO : org.mongodb.driver.cluster - Cluster created with settings {hosts=[mongo:27017], mode=SINGLE, requiredClusterType=UNKNOWN, serverSelectionTimeout='30000 ms', maxWaitQueueSize=500}
2017-11-08 20:06:05,814 INFO : org.mongodb.driver.cluster - No server chosen by ReadPreferenceServerSelector{readPreference=primary} from cluster description ClusterDescription{type=UNKNOWN, connectionMode=SINGLE, serverDescriptions=[ServerDescription{address=mongo:27017, type=UNKNOWN, state=CONNECTING}]}. Waiting for 30000 ms before timing out
2017-11-08 20:06:05,837 INFO : org.mongodb.driver.connection - Opened connection [connectionId{localValue:1, serverValue:181}] to mongo:27017
2017-11-08 20:06:05,838 INFO : org.mongodb.driver.cluster - Monitor thread successfully connected to server with description ServerDescription{address=mongo:27017, type=STANDALONE, state=CONNECTED, ok=true, version=ServerVersion{versionList=[3, 2, 17]}, minWireVersion=0, maxWireVersion=4, maxDocumentSize=16777216, roundTripTimeNanos=640927}
2017-11-08 20:06:05,845 INFO : org.mongodb.driver.connection - Opened connection [connectionId{localValue:2, serverValue:182}] to mongo:27017
2017-11-08 20:06:06,211 INFO : io.searchbox.client.AbstractJestClient - Setting server pool to a list of 1 servers: [https://aws-vpc-graylog.eu-west-1.es.amazonaws.com]
2017-11-08 20:06:06,212 INFO : io.searchbox.client.JestClientFactory - Using multi thread/connection supporting pooling connection manager
2017-11-08 20:06:06,282 INFO : io.searchbox.client.JestClientFactory - Using custom ObjectMapper instance
2017-11-08 20:06:06,282 INFO : io.searchbox.client.JestClientFactory - Node Discovery disabled...
2017-11-08 20:06:06,282 INFO : io.searchbox.client.JestClientFactory - Idle connection reaping disabled...
2017-11-08 20:06:06,602 INFO : org.graylog2.shared.buffers.ProcessBuffer - Initialized ProcessBuffer with ring size <65536> and wait strategy <BlockingWaitStrategy>.
2017-11-08 20:06:08,208 INFO : org.graylog2.bindings.providers.RulesEngineProvider - No static rules file loaded.
2017-11-08 20:06:08,379 INFO : org.graylog2.buffers.OutputBuffer - Initialized OutputBuffer with ring size <65536> and wait strategy <BlockingWaitStrategy>.
2017-11-08 20:06:09,219 INFO : org.graylog2.bootstrap.ServerBootstrap - Graylog server 2.3.2+3df951e starting up
2017-11-08 20:06:09,219 INFO : org.graylog2.bootstrap.ServerBootstrap - JRE: Oracle Corporation 1.8.0_141 on Linux 4.4.0-1022-aws
2017-11-08 20:06:09,219 INFO : org.graylog2.bootstrap.ServerBootstrap - Deployment: docker
2017-11-08 20:06:09,219 INFO : org.graylog2.bootstrap.ServerBootstrap - OS: Debian GNU/Linux 9 (stretch) (debian)
2017-11-08 20:06:09,220 INFO : org.graylog2.bootstrap.ServerBootstrap - Arch: amd64
2017-11-08 20:06:09,222 WARN : org.graylog2.shared.events.DeadEventLoggingListener - Received unhandled event of type <org.graylog2.plugin.lifecycles.Lifecycle> from event bus <AsyncEventBus{graylog-eventbus}>
2017-11-08 20:06:09,255 INFO : org.graylog2.shared.initializers.PeriodicalsService - Starting 26 periodicals ...
2017-11-08 20:06:09,256 INFO : org.graylog2.periodical.Periodicals - Starting [org.graylog2.periodical.ThroughputCalculator] periodical in [0s], polling every [1s].
2017-11-08 20:06:09,259 INFO : org.graylog2.periodical.Periodicals - Starting [org.graylog2.periodical.AlertScannerThread] periodical in [10s], polling every [60s].
2017-11-08 20:06:09,259 INFO : org.graylog2.periodical.Periodicals - Starting [org.graylog2.periodical.BatchedElasticSearchOutputFlushThread] periodical in [0s], polling every [1s].
2017-11-08 20:06:09,270 INFO : org.graylog2.periodical.Periodicals - Starting [org.graylog2.periodical.ClusterHealthCheckThread] periodical in [120s], polling every [20s].
2017-11-08 20:06:09,271 INFO : org.graylog2.periodical.Periodicals - Starting [org.graylog2.periodical.ContentPackLoaderPeriodical] periodical, running forever.
2017-11-08 20:06:09,272 INFO : org.graylog2.periodical.Periodicals - Starting [org.graylog2.periodical.GarbageCollectionWarningThread] periodical, running forever.
2017-11-08 20:06:09,272 INFO : org.graylog2.periodical.Periodicals - Starting [org.graylog2.periodical.IndexerClusterCheckerThread] periodical in [0s], polling every [30s].
2017-11-08 20:06:09,273 INFO : org.graylog2.periodical.Periodicals - Starting [org.graylog2.periodical.IndexRetentionThread] periodical in [0s], polling every [300s].
2017-11-08 20:06:09,273 INFO : org.graylog2.periodical.Periodicals - Starting [org.graylog2.periodical.IndexRotationThread] periodical in [0s], polling every [10s].
2017-11-08 20:06:09,276 INFO : org.mongodb.driver.connection - Opened connection [connectionId{localValue:3, serverValue:183}] to mongo:27017
2017-11-08 20:06:09,281 INFO : org.graylog2.periodical.Periodicals - Starting [org.graylog2.periodical.NodePingThread] periodical in [0s], polling every [1s].
2017-11-08 20:06:09,282 INFO : org.graylog2.periodical.Periodicals - Starting [org.graylog2.periodical.VersionCheckThread] periodical in [300s], polling every [1800s].
2017-11-08 20:06:09,282 INFO : org.graylog2.periodical.Periodicals - Starting [org.graylog2.periodical.ThrottleStateUpdaterThread] periodical in [1s], polling every [1s].
2017-11-08 20:06:09,282 INFO : org.graylog2.periodical.Periodicals - Starting [org.graylog2.events.ClusterEventPeriodical] periodical in [0s], polling every [1s].
2017-11-08 20:06:09,283 INFO : org.graylog2.periodical.Periodicals - Starting [org.graylog2.events.ClusterEventCleanupPeriodical] periodical in [0s], polling every [86400s].
2017-11-08 20:06:09,286 INFO : org.graylog2.periodical.Periodicals - Starting [org.graylog2.periodical.ClusterIdGeneratorPeriodical] periodical, running forever.
2017-11-08 20:06:09,287 INFO : org.graylog2.periodical.Periodicals - Starting [org.graylog2.periodical.IndexRangesMigrationPeriodical] periodical, running forever.
2017-11-08 20:06:09,287 INFO : org.graylog2.periodical.Periodicals - Starting [org.graylog2.periodical.IndexRangesCleanupPeriodical] periodical in [15s], polling every [3600s].
2017-11-08 20:06:09,309 INFO : org.mongodb.driver.connection - Opened connection [connectionId{localValue:11, serverValue:191}] to mongo:27017
2017-11-08 20:06:09,311 INFO : org.mongodb.driver.connection - Opened connection [connectionId{localValue:4, serverValue:184}] to mongo:27017
2017-11-08 20:06:09,311 INFO : org.mongodb.driver.connection - Opened connection [connectionId{localValue:5, serverValue:185}] to mongo:27017
2017-11-08 20:06:09,313 INFO : org.mongodb.driver.connection - Opened connection [connectionId{localValue:8, serverValue:188}] to mongo:27017
2017-11-08 20:06:09,315 INFO : org.mongodb.driver.connection - Opened connection [connectionId{localValue:6, serverValue:186}] to mongo:27017
2017-11-08 20:06:09,318 INFO : org.mongodb.driver.connection - Opened connection [connectionId{localValue:10, serverValue:190}] to mongo:27017
2017-11-08 20:06:09,342 INFO : org.mongodb.driver.connection - Opened connection [connectionId{localValue:9, serverValue:189}] to mongo:27017
2017-11-08 20:06:09,342 INFO : org.mongodb.driver.connection - Opened connection [connectionId{localValue:7, serverValue:187}] to mongo:27017
2017-11-08 20:06:09,350 INFO : org.graylog2.shared.initializers.PeriodicalsService - Not starting [org.graylog2.periodical.UserPermissionMigrationPeriodical] periodical. Not configured to run on this node.
2017-11-08 20:06:09,350 INFO : org.graylog2.periodical.Periodicals - Starting [org.graylog2.periodical.AlarmCallbacksMigrationPeriodical] periodical, running forever.
2017-11-08 20:06:09,352 INFO : org.graylog2.periodical.Periodicals - Starting [org.graylog2.periodical.ConfigurationManagementPeriodical] periodical, running forever.
2017-11-08 20:06:09,363 INFO : org.graylog2.periodical.Periodicals - Starting [org.graylog2.periodical.LdapGroupMappingMigration] periodical, running forever.
2017-11-08 20:06:09,394 INFO : org.graylog2.periodical.Periodicals - Starting [org.graylog2.periodical.IndexFailuresPeriodical] periodical, running forever.
2017-11-08 20:06:09,394 INFO : org.graylog2.periodical.Periodicals - Starting [org.graylog.plugins.usagestatistics.UsageStatsNodePeriodical] periodical in [300s], polling every [21600s].
2017-11-08 20:06:09,400 INFO : org.graylog2.periodical.Periodicals - Starting [org.graylog.plugins.usagestatistics.UsageStatsClusterPeriodical] periodical in [300s], polling every [21600s].
2017-11-08 20:06:09,405 INFO : org.graylog2.periodical.Periodicals - Starting [org.graylog.plugins.pipelineprocessor.periodical.LegacyDefaultStreamMigration] periodical, running forever.
2017-11-08 20:06:09,413 INFO : org.graylog2.periodical.Periodicals - Starting [org.graylog.plugins.collector.periodical.PurgeExpiredCollectorsThread] periodical in [0s], polling every [3600s].
2017-11-08 20:06:09,426 INFO : org.graylog.plugins.pipelineprocessor.periodical.LegacyDefaultStreamMigration - Legacy default stream has no connections, no migration needed.
2017-11-08 20:06:09,725 INFO : org.graylog2.shared.initializers.JerseyService - Enabling CORS for HTTP endpoint
2017-11-08 20:06:19,440 INFO : org.glassfish.grizzly.http.server.NetworkListener - Started listener bound to [0.0.0.0:9000]
2017-11-08 20:06:19,441 INFO : org.glassfish.grizzly.http.server.HttpServer - [HttpServer] Started.
2017-11-08 20:06:19,441 INFO : org.graylog2.shared.initializers.JerseyService - Started REST API at <http://0.0.0.0:9000/api/>
2017-11-08 20:06:19,441 INFO : org.graylog2.shared.initializers.JerseyService - Started Web Interface at <http://0.0.0.0:9000/>
2017-11-08 20:06:19,443 INFO : org.graylog2.bootstrap.ServerBootstrap - Services started, startup times in ms: {BufferSynchronizerService [RUNNING]=8, OutputSetupService [RUNNING]=9, KafkaJournal [RUNNING]=16, InputSetupService [RUNNING]=32, ConfigurationEtagService [RUNNING]=91, JournalReader [RUNNING]=92, LookupTableService [RUNNING]=99, StreamCacheService [RUNNING]=136, PeriodicalsService [RUNNING]=202, JerseyService [RUNNING]=10198}
2017-11-08 20:06:19,445 INFO : org.graylog2.shared.initializers.ServiceManagerListener - Services are healthy
2017-11-08 20:06:19,446 INFO : org.graylog2.bootstrap.ServerBootstrap - Graylog server up and running.
2017-11-08 20:06:19,447 INFO : org.graylog2.shared.initializers.InputSetupService - Triggering launching persisted inputs, node transitioned from Uninitialized [LB:DEAD] to Running [LB:ALIVE]
2017-11-08 20:06:19,466 INFO : org.graylog2.inputs.InputStateListener - Input [Syslog TCP/59f0a5c9b7167c000151ea4b] is now STARTING
2017-11-08 20:06:19,514 WARN : org.graylog2.plugin.inputs.transports.NettyTransport - receiveBufferSize (SO_RCVBUF) for input SyslogTCPInput{title=syslog-input, type=org.graylog2.inputs.syslog.tcp.SyslogTCPInput, nodeId=null} should be 1048576 but is 212992.
2017-11-08 20:06:19,522 INFO : org.graylog2.inputs.InputStateListener - Input [Syslog TCP/59f0a5c9b7167c000151ea4b] is now RUNNING
joschi commented 6 years ago

the default options in the docker container seems to be different than those in the documentation

Correct, see https://github.com/Graylog2/graylog-docker/blob/2.3.2-1/config/graylog.conf for the configuration file used in the graylog/graylog Docker image.

What's in the logs of the Graylog node when the search request fails?

jean-francois-labbe commented 6 years ago

I can't find the log file.

/usr/share/graylog/data/log and /usr/share/graylog/data/log are empty inside the container

Where should I look for the log file inside the docker?

jean-francois-labbe commented 6 years ago

I can see that I have errors with statistics on the /system/logging page

joschi commented 6 years ago

I can't find the log file.

The logs of the Graylog Docker image are printed on standard out, so that the Docker logging mechanism can pick them up: https://github.com/Graylog2/graylog-docker/blob/2.3.2-1/config/log4j2.xml#L4-L6

jean-francois-labbe commented 6 years ago

Ok so as I see it there are no more logs than those in the issue.

But I see my number of document increasing in AWS elasticsearch, I can see them with Kibana but Graylog is unable to perform a query.

I've no idea where to look cause the graylog setup worked with an elasticsearch container linked to the docker container but now that elastic is on AWS it can't perform search.

joschi commented 6 years ago

@jean-francois-labbe Please provide the configuration of your Graylog node (e. g. the environment variables set for the Docker container) and the complete logs after (re-) starting the Graylog node.

jean-francois-labbe commented 6 years ago

When graylog performs: http://graylog/api/system/fields it gets: 

{message: "Couldn't read cluster state for indices graylog_*,test_*", details: []}

Graylog says:

{type: "ApiError", message: "Couldn't read Elasticsearch cluster health"} Name

But I can see the indices throught the Graylog API

Here is the Graylog configuration:

version: '2'
services:
  graylog-app:
    image: graylog/graylog:2.3.2-1
    environment:
      GRAYLOG_ELASTICSEARCH_HOSTS: https://vpc-graylog.eu-west-1.es.amazonaws.com
      GRAYLOG_PASSWORD_SECRET: f9fsd5f6q4f6sqd
      GRAYLOG_ROOT_PASSWORD_SHA2: 23fef88b72d95a5938f4ce655e5b3638a
      GRAYLOG_WEB_ENDPOINT_URI: http://graylog.internal/api
      GRAYLOG_ELASTICSEARCH_DISCOVERY_ENABLED: 'false'
      GRAYLOG_ELASTICSEARCH_COMPRESSION_ENABLED: 'false'
    volumes:
    - graylog-data:/usr/share/graylog/data
    - graylog-plugin:/usr/share/graylog/plugin
    links:
    - mongo:mongo
  graylog:
    image: rancher/lb-service-haproxy:v0.7.9
    ports:
    - 80:80/tcp

Logs after a restart:

2017-11-09 10:22:28,864 INFO : org.graylog2.bootstrap.CmdLineTool - Loaded plugin: Elastic Beats Input 2.3.2 [org.graylog.plugins.beats.BeatsInputPlugin]
2017-11-09 10:22:28,866 INFO : org.graylog2.bootstrap.CmdLineTool - Loaded plugin: Collector 2.3.2 [org.graylog.plugins.collector.CollectorPlugin]
2017-11-09 10:22:28,867 INFO : org.graylog2.bootstrap.CmdLineTool - Loaded plugin: Enterprise Integration Plugin 2.3.2 [org.graylog.plugins.enterprise_integration.EnterpriseIntegrationPlugin]
2017-11-09 10:22:28,867 INFO : org.graylog2.bootstrap.CmdLineTool - Loaded plugin: MapWidgetPlugin 2.3.2 [org.graylog.plugins.map.MapWidgetPlugin]
2017-11-09 10:22:28,875 INFO : org.graylog2.bootstrap.CmdLineTool - Loaded plugin: Pipeline Processor Plugin 2.3.2 [org.graylog.plugins.pipelineprocessor.ProcessorPlugin]
2017-11-09 10:22:28,876 INFO : org.graylog2.bootstrap.CmdLineTool - Loaded plugin: QuickValuesPlusWidget 3.0.1 [org.graylog.plugins.quickvaluesplus.QuickValuesPlusWidgetPlugin]
2017-11-09 10:22:28,877 INFO : org.graylog2.bootstrap.CmdLineTool - Loaded plugin: Anonymous Usage Statistics 2.3.2 [org.graylog.plugins.usagestatistics.UsageStatsPlugin]
2017-11-09 10:22:28,877 INFO : org.graylog2.bootstrap.CmdLineTool - Loaded plugin: Slack 2.4.0 [org.graylog2.plugins.slack.callback.SlackAlarmCallback]
2017-11-09 10:22:29,159 INFO : org.graylog2.bootstrap.CmdLineTool - Running with JVM arguments: -XX:+UnlockExperimentalVMOptions -XX:+UseCGroupMemoryLimitForHeap -XX:NewRatio=1 -XX:MaxMetaspaceSize=256m -XX:+ResizeTLAB -XX:+UseConcMarkSweepGC -XX:+CMSConcurrentMTEnabled -XX:+CMSClassUnloadingEnabled -XX:+UseParNewGC -XX:-OmitStackTraceInFastThrow -Dlog4j.configurationFile=/usr/share/graylog/data/config/log4j2.xml -Djava.library.path=/usr/share/graylog/lib/sigar/ -Dgraylog2.installation_source=docker
2017-11-09 10:22:29,354 INFO : org.hibernate.validator.internal.util.Version - HV000001: Hibernate Validator null
2017-11-09 10:22:32,355 INFO : org.graylog2.shared.buffers.InputBufferImpl - Message journal is enabled.
2017-11-09 10:22:32,390 INFO : org.graylog2.plugin.system.NodeId - Node ID: 2f00df0f-166d-43e2-9abd-97b115d618cb
2017-11-09 10:22:32,680 INFO : kafka.log.LogManager - Loading logs.
2017-11-09 10:22:32,831 INFO : kafka.log.LogManager - Logs loading complete.
2017-11-09 10:22:32,831 INFO : org.graylog2.shared.journal.KafkaJournal - Initialized Kafka based journal at /usr/share/graylog/data/journal
2017-11-09 10:22:32,874 INFO : org.graylog2.shared.buffers.InputBufferImpl - Initialized InputBufferImpl with ring size <65536> and wait strategy <BlockingWaitStrategy>, running 2 parallel message handlers.
2017-11-09 10:22:32,893 INFO : org.mongodb.driver.cluster - Cluster created with settings {hosts=[mongo:27017], mode=SINGLE, requiredClusterType=UNKNOWN, serverSelectionTimeout='30000 ms', maxWaitQueueSize=500}
2017-11-09 10:22:32,939 INFO : org.mongodb.driver.cluster - No server chosen by ReadPreferenceServerSelector{readPreference=primary} from cluster description ClusterDescription{type=UNKNOWN, connectionMode=SINGLE, serverDescriptions=[ServerDescription{address=mongo:27017, type=UNKNOWN, state=CONNECTING}]}. Waiting for 30000 ms before timing out
2017-11-09 10:22:32,950 INFO : org.mongodb.driver.connection - Opened connection [connectionId{localValue:1, serverValue:29}] to mongo:27017
2017-11-09 10:22:32,952 INFO : org.mongodb.driver.cluster - Monitor thread successfully connected to server with description ServerDescription{address=mongo:27017, type=STANDALONE, state=CONNECTED, ok=true, version=ServerVersion{versionList=[3, 2, 17]}, minWireVersion=0, maxWireVersion=4, maxDocumentSize=16777216, roundTripTimeNanos=512837}
2017-11-09 10:22:32,958 INFO : org.mongodb.driver.connection - Opened connection [connectionId{localValue:2, serverValue:30}] to mongo:27017
2017-11-09 10:22:33,334 INFO : io.searchbox.client.AbstractJestClient - Setting server pool to a list of 1 servers: [https://vpc-graylog.eu-west-1.es.amazonaws.com]
2017-11-09 10:22:33,335 INFO : io.searchbox.client.JestClientFactory - Using multi thread/connection supporting pooling connection manager
2017-11-09 10:22:33,405 INFO : io.searchbox.client.JestClientFactory - Using custom ObjectMapper instance
2017-11-09 10:22:33,405 INFO : io.searchbox.client.JestClientFactory - Node Discovery disabled...
2017-11-09 10:22:33,406 INFO : io.searchbox.client.JestClientFactory - Idle connection reaping disabled...
2017-11-09 10:22:33,935 INFO : org.graylog2.shared.buffers.ProcessBuffer - Initialized ProcessBuffer with ring size <65536> and wait strategy <BlockingWaitStrategy>.
2017-11-09 10:22:36,258 INFO : org.graylog2.bindings.providers.RulesEngineProvider - No static rules file loaded.
2017-11-09 10:22:36,454 INFO : org.graylog2.buffers.OutputBuffer - Initialized OutputBuffer with ring size <65536> and wait strategy <BlockingWaitStrategy>.
2017-11-09 10:22:37,756 INFO : org.mongodb.driver.connection - Opened connection [connectionId{localValue:3, serverValue:31}] to mongo:27017
2017-11-09 10:22:37,893 INFO : org.graylog2.bootstrap.ServerBootstrap - Graylog server 2.3.2+3df951e starting up
2017-11-09 10:22:37,893 INFO : org.graylog2.bootstrap.ServerBootstrap - JRE: Oracle Corporation 1.8.0_141 on Linux 4.4.0-1039-aws
2017-11-09 10:22:37,893 INFO : org.graylog2.bootstrap.ServerBootstrap - Deployment: docker
2017-11-09 10:22:37,893 INFO : org.graylog2.bootstrap.ServerBootstrap - OS: Debian GNU/Linux 9 (stretch) (debian)
2017-11-09 10:22:37,893 INFO : org.graylog2.bootstrap.ServerBootstrap - Arch: amd64
2017-11-09 10:22:37,896 WARN : org.graylog2.shared.events.DeadEventLoggingListener - Received unhandled event of type <org.graylog2.plugin.lifecycles.Lifecycle> from event bus <AsyncEventBus{graylog-eventbus}>
2017-11-09 10:22:37,946 INFO : org.graylog2.shared.initializers.PeriodicalsService - Starting 26 periodicals ...
2017-11-09 10:22:37,946 INFO : org.graylog2.periodical.Periodicals - Starting [org.graylog2.periodical.ThroughputCalculator] periodical in [0s], polling every [1s].
2017-11-09 10:22:37,981 INFO : org.graylog2.periodical.Periodicals - Starting [org.graylog2.periodical.AlertScannerThread] periodical in [10s], polling every [60s].
2017-11-09 10:22:37,982 INFO : org.graylog2.periodical.Periodicals - Starting [org.graylog2.periodical.BatchedElasticSearchOutputFlushThread] periodical in [0s], polling every [1s].
2017-11-09 10:22:37,984 INFO : org.graylog2.periodical.Periodicals - Starting [org.graylog2.periodical.ClusterHealthCheckThread] periodical in [120s], polling every [20s].
2017-11-09 10:22:37,986 INFO : org.graylog2.periodical.Periodicals - Starting [org.graylog2.periodical.ContentPackLoaderPeriodical] periodical, running forever.
2017-11-09 10:22:37,986 INFO : org.graylog2.periodical.Periodicals - Starting [org.graylog2.periodical.GarbageCollectionWarningThread] periodical, running forever.
2017-11-09 10:22:37,987 INFO : org.graylog2.periodical.Periodicals - Starting [org.graylog2.periodical.IndexerClusterCheckerThread] periodical in [0s], polling every [30s].
2017-11-09 10:22:37,988 INFO : org.graylog2.periodical.Periodicals - Starting [org.graylog2.periodical.IndexRetentionThread] periodical in [0s], polling every [300s].
2017-11-09 10:22:37,989 INFO : org.graylog2.periodical.Periodicals - Starting [org.graylog2.periodical.IndexRotationThread] periodical in [0s], polling every [10s].
2017-11-09 10:22:37,991 INFO : org.graylog2.periodical.Periodicals - Starting [org.graylog2.periodical.NodePingThread] periodical in [0s], polling every [1s].
2017-11-09 10:22:37,992 INFO : org.graylog2.periodical.Periodicals - Starting [org.graylog2.periodical.VersionCheckThread] periodical in [300s], polling every [1800s].
2017-11-09 10:22:37,992 INFO : org.graylog2.periodical.Periodicals - Starting [org.graylog2.periodical.ThrottleStateUpdaterThread] periodical in [1s], polling every [1s].
2017-11-09 10:22:37,992 INFO : org.graylog2.periodical.Periodicals - Starting [org.graylog2.events.ClusterEventPeriodical] periodical in [0s], polling every [1s].
2017-11-09 10:22:37,995 INFO : org.graylog2.periodical.Periodicals - Starting [org.graylog2.events.ClusterEventCleanupPeriodical] periodical in [0s], polling every [86400s].
2017-11-09 10:22:37,995 INFO : org.graylog2.periodical.Periodicals - Starting [org.graylog2.periodical.ClusterIdGeneratorPeriodical] periodical, running forever.
2017-11-09 10:22:37,995 INFO : org.graylog2.periodical.Periodicals - Starting [org.graylog2.periodical.IndexRangesMigrationPeriodical] periodical, running forever.
2017-11-09 10:22:37,997 INFO : org.graylog2.periodical.Periodicals - Starting [org.graylog2.periodical.IndexRangesCleanupPeriodical] periodical in [15s], polling every [3600s].
2017-11-09 10:22:38,001 INFO : org.mongodb.driver.connection - Opened connection [connectionId{localValue:4, serverValue:32}] to mongo:27017
2017-11-09 10:22:38,002 INFO : org.mongodb.driver.connection - Opened connection [connectionId{localValue:5, serverValue:33}] to mongo:27017
2017-11-09 10:22:38,015 INFO : org.mongodb.driver.connection - Opened connection [connectionId{localValue:6, serverValue:34}] to mongo:27017
2017-11-09 10:22:38,044 INFO : org.mongodb.driver.connection - Opened connection [connectionId{localValue:7, serverValue:35}] to mongo:27017
2017-11-09 10:22:38,070 INFO : org.graylog2.shared.initializers.PeriodicalsService - Not starting [org.graylog2.periodical.UserPermissionMigrationPeriodical] periodical. Not configured to run on this node.
2017-11-09 10:22:38,070 INFO : org.graylog2.periodical.Periodicals - Starting [org.graylog2.periodical.AlarmCallbacksMigrationPeriodical] periodical, running forever.
2017-11-09 10:22:38,074 INFO : org.graylog2.periodical.Periodicals - Starting [org.graylog2.periodical.ConfigurationManagementPeriodical] periodical, running forever.
2017-11-09 10:22:38,081 INFO : org.graylog2.periodical.Periodicals - Starting [org.graylog2.periodical.LdapGroupMappingMigration] periodical, running forever.
2017-11-09 10:22:38,083 INFO : org.graylog2.periodical.Periodicals - Starting [org.graylog2.periodical.IndexFailuresPeriodical] periodical, running forever.
2017-11-09 10:22:38,088 INFO : org.graylog2.periodical.Periodicals - Starting [org.graylog.plugins.usagestatistics.UsageStatsNodePeriodical] periodical in [300s], polling every [21600s].
2017-11-09 10:22:38,089 INFO : org.graylog2.periodical.Periodicals - Starting [org.graylog.plugins.usagestatistics.UsageStatsClusterPeriodical] periodical in [300s], polling every [21600s].
2017-11-09 10:22:38,093 INFO : org.graylog2.periodical.Periodicals - Starting [org.graylog.plugins.pipelineprocessor.periodical.LegacyDefaultStreamMigration] periodical, running forever.
2017-11-09 10:22:38,093 INFO : org.graylog2.periodical.Periodicals - Starting [org.graylog.plugins.collector.periodical.PurgeExpiredCollectorsThread] periodical in [0s], polling every [3600s].
2017-11-09 10:22:38,101 INFO : org.graylog.plugins.pipelineprocessor.periodical.LegacyDefaultStreamMigration - Legacy default stream has no connections, no migration needed.
2017-11-09 10:22:38,492 INFO : org.graylog2.shared.initializers.JerseyService - Enabling CORS for HTTP endpoint
2017-11-09 10:22:51,380 INFO : org.glassfish.grizzly.http.server.NetworkListener - Started listener bound to [0.0.0.0:9000]
2017-11-09 10:22:51,382 INFO : org.glassfish.grizzly.http.server.HttpServer - [HttpServer] Started.
2017-11-09 10:22:51,382 INFO : org.graylog2.shared.initializers.JerseyService - Started REST API at <http://0.0.0.0:9000/api/>
2017-11-09 10:22:51,382 INFO : org.graylog2.shared.initializers.JerseyService - Started Web Interface at <http://0.0.0.0:9000/>
2017-11-09 10:22:51,383 INFO : org.graylog2.shared.initializers.ServiceManagerListener - Services are healthy
2017-11-09 10:22:51,385 INFO : org.graylog2.bootstrap.ServerBootstrap - Services started, startup times in ms: {OutputSetupService [RUNNING]=28, BufferSynchronizerService [RUNNING]=28, KafkaJournal [RUNNING]=53, JournalReader [RUNNING]=98, StreamCacheService [RUNNING]=146, LookupTableService [RUNNING]=147, ConfigurationEtagService [RUNNING]=148, PeriodicalsService [RUNNING]=167, InputSetupService [RUNNING]=172, JerseyService [RUNNING]=13441}
2017-11-09 10:22:51,386 INFO : org.graylog2.shared.initializers.InputSetupService - Triggering launching persisted inputs, node transitioned from Uninitialized [LB:DEAD] to Running [LB:ALIVE]
2017-11-09 10:22:51,394 INFO : org.graylog2.bootstrap.ServerBootstrap - Graylog server up and running.
2017-11-09 10:22:51,422 INFO : org.graylog2.inputs.InputStateListener - Input [Syslog TCP/59f0a5c9b7167c000151ea4b] is now STARTING
2017-11-09 10:22:51,472 WARN : org.graylog2.plugin.inputs.transports.NettyTransport - receiveBufferSize (SO_RCVBUF) for input SyslogTCPInput{title=syslog-input, type=org.graylog2.inputs.syslog.tcp.SyslogTCPInput, nodeId=null} should be 1048576 but is 212992.
2017-11-09 10:22:51,475 INFO : org.graylog2.inputs.InputStateListener - Input [Syslog TCP/59f0a5c9b7167c000151ea4b] is now RUNNING
jean-francois-labbe commented 6 years ago

Logs after an Index range rebuild through API /api/system/indices/ranges/rebuild

2017-11-09 10:31:15,812 INFO : org.graylog2.system.jobs.SystemJobManager - Submitted SystemJob <1d939b30-c539-11e7-93f1-028f7b3617cc> [org.graylog2.indexer.ranges.RebuildIndexRangesJob]
2017-11-09 10:31:15,812 INFO : org.graylog2.indexer.ranges.RebuildIndexRangesJob - Recalculating index ranges.
2017-11-09 10:31:15,829 INFO : org.graylog2.indexer.ranges.RebuildIndexRangesJob - Recalculating index ranges for index set rancher-infra-2 (rancher_stacks_*): 1 indices affected.
2017-11-09 10:31:15,833 INFO : org.graylog2.indexer.ranges.RebuildIndexRangesJob - Recalculating index ranges for index set Default index set (graylog_*): 1 indices affected.
2017-11-09 10:31:15,837 INFO : org.graylog2.indexer.ranges.RebuildIndexRangesJob - Done calculating index ranges for 2 indices. Took 9ms.
2017-11-09 10:31:15,838 INFO : org.graylog2.system.jobs.SystemJobManager - SystemJob <1d939b30-c539-11e7-93f1-028f7b3617cc> [org.graylog2.indexer.ranges.RebuildIndexRangesJob] finished in 25ms.
jean-francois-labbe commented 6 years ago

It seems to be an issue with the GRAYLOG_WEB_ENDPOINT_URI, I just exposed the graylog-app port and could make it work.

What I don't understand is that my only change was the elasticsearch cluster

joschi commented 6 years ago

@jean-francois-labbe The URI provided in web_endpoint_uri must be accessible for the clients of the Graylog web interface.