search

Graylog2 / graylog2-server

Free and open log management
https://www.graylog.org
Other
7.37k stars 1.06k forks source link

Graylog 2.0.0 Https web config in Amazon Instance - Unable to call on node problem #2193

Closed onedaywillcome1 closed 8 years ago

onedaywillcome1 commented 8 years ago

Problem description

I installed graylog 2.0.0 in my Amazon Instance and reached web dashboard via http. Everything was Ok. Then, for security reasons, I enabled https feature in graylog config. I again successfully reached web dashboard via https. However, after clicking signin button, I noticed that in every seconds, warning message is coming to log file. I didnt understand which problem causes to that problem. Here is my graylog config: 

is_master = true
password_secret  <my secret>
root_password_sha2 <my password> 
rest_listen_uri = https://0.0.0.0:12900/
rest_enable_tls = true
rest_tls_cert_file = /etc/graylog/server/x509key.pem
rest_tls_key_file = /etc/graylog/server/pkc8key.pem
web_endpoint_uri = https://<public-ip>:12900
web_listen_uri = https://0.0.0.0:9000/
web_enable_tls = true
web_tls_cert_file = /etc/graylog/server/x509key.pem
web_tls_key_file = /etc/graylog/server/pkc8key.pem
elasticsearch_cluster_name = graylog
elasticsearch_discovery_zen_ping_unicast_hosts = 127.0.0.1:9300

I created my certificates as follow:

sudo openssl genrsa -out key.pem 2048
sudo openssl req -new -key key.pem -out csr.pem
sudo openssl req -x509 -days 3650 -key key.pem -in csr.pem -out x509key.pem
sudo openssl pkcs8 -in key.pem -topk8 -out pkc8key.pem -v1 PBE-SHA1-3DES -nocrypt

Here is my log file:

2016-05-05T13:37:39.160Z INFO  [CmdLineTool] Loaded plugins: [Anonymous Usage Statistics 2.0.0 [org.graylog.plugins.usagestatistics.UsageStatsPlugin], Pipeline Processor Plugin 1.0.0-beta.2 [org.graylog.plugins.pipelineprocessor.ProcessorPlugin], Collector 1.0.0 [org.graylog.plugins.collector.CollectorPlugin], MapWidgetPlugin 1.0.0 [org.graylog.plugins.map.MapWidgetPlugin], Enterprise Integration Plugin 1.0.0 [org.graylog.plugins.enterprise_integration.EnterpriseIntegrationPlugin]]
2016-05-05T13:37:39.273Z INFO  [CmdLineTool] Running with JVM arguments: -Xms1g -Xmx1g -XX:NewRatio=1 -XX:+ResizeTLAB -XX:+UseConcMarkSweepGC -XX:+CMSConcurrentMTEnabled -XX:+CMSClassUnloadingEnabled -XX:+UseParNewGC -XX:-OmitStackTraceInFastThrow -Djava.library.path=/usr/share/graylog-server/lib/sigar -Dlog4j.configurationFile=file:///etc/graylog/server/log4j2.xml -Dgraylog2.installation_source=rpm
2016-05-05T13:37:41.801Z INFO  [InputBufferImpl] Message journal is enabled.
2016-05-05T13:37:41.992Z INFO  [LogManager] Loading logs.
2016-05-05T13:37:42.088Z INFO  [LogManager] Logs loading complete.
2016-05-05T13:37:42.088Z INFO  [KafkaJournal] Initialized Kafka based journal at /var/lib/graylog-server/journal
2016-05-05T13:37:42.110Z INFO  [InputBufferImpl] Initialized InputBufferImpl with ring size <65536> and wait strategy <BlockingWaitStrategy>, running 2 parallel message handlers.
2016-05-05T13:37:42.147Z INFO  [cluster] Cluster created with settings {hosts=[localhost:27017], mode=SINGLE, requiredClusterType=UNKNOWN, serverSelectionTimeout='30000 ms', maxWaitQueueSize=5000}
2016-05-05T13:37:42.191Z INFO  [cluster] No server chosen by ReadPreferenceServerSelector{readPreference=primary} from cluster description ClusterDescription{type=UNKNOWN, connectionMode=SINGLE, all=[ServerDescription{address=localhost:27017, type=UNKNOWN, state=CONNECTING}]}. Waiting for 30000 ms before timing out
2016-05-05T13:37:42.229Z INFO  [connection] Opened connection [connectionId{localValue:1, serverValue:272}] to localhost:27017
2016-05-05T13:37:42.230Z INFO  [cluster] Monitor thread successfully connected to server with description ServerDescription{address=localhost:27017, type=STANDALONE, state=CONNECTED, ok=true, version=ServerVersion{versionList=[3, 2, 6]}, minWireVersion=0, maxWireVersion=4, maxDocumentSize=16777216, roundTripTimeNanos=590309}
2016-05-05T13:37:42.238Z INFO  [connection] Opened connection [connectionId{localValue:2, serverValue:273}] to localhost:27017
2016-05-05T13:37:42.454Z INFO  [NodeId] Node ID: fea7225f-ff92-4cd7-ac17-2adffb9d2c61
2016-05-05T13:37:42.517Z INFO  [node] [graylog-fea7225f-ff92-4cd7-ac17-2adffb9d2c61] version[2.3.1], pid[5923], build[bd98092/2016-04-04T12:25:05Z]
2016-05-05T13:37:42.518Z INFO  [node] [graylog-fea7225f-ff92-4cd7-ac17-2adffb9d2c61] initializing ...
2016-05-05T13:37:42.525Z INFO  [plugins] [graylog-fea7225f-ff92-4cd7-ac17-2adffb9d2c61] modules [], plugins [graylog-monitor], sites []
2016-05-05T13:37:44.425Z INFO  [node] [graylog-fea7225f-ff92-4cd7-ac17-2adffb9d2c61] initialized
2016-05-05T13:37:44.525Z INFO  [Version] HV000001: Hibernate Validator 5.2.4.Final
2016-05-05T13:37:44.651Z INFO  [ProcessBuffer] Initialized ProcessBuffer with ring size <65536> and wait strategy <BlockingWaitStrategy>.
2016-05-05T13:37:46.817Z INFO  [RulesEngineProvider] No static rules file loaded.
2016-05-05T13:37:46.940Z WARN  [GeoIpResolverEngine] GeoIP database file does not exist: /tmp/GeoLite2-City.mmdb
2016-05-05T13:37:46.947Z INFO  [OutputBuffer] Initialized OutputBuffer with ring size <65536> and wait strategy <BlockingWaitStrategy>.
2016-05-05T13:37:47.663Z INFO  [ServerBootstrap] Graylog server 2.0.0 (2dc6c03) starting up
2016-05-05T13:37:47.663Z INFO  [ServerBootstrap] JRE: Oracle Corporation 1.8.0_66 on Linux 4.1.10-17.31.amzn1.x86_64
2016-05-05T13:37:47.663Z INFO  [ServerBootstrap] Deployment: rpm
2016-05-05T13:37:47.664Z INFO  [ServerBootstrap] OS: Amazon Linux AMI 2016.03 (amzn)
2016-05-05T13:37:47.664Z INFO  [ServerBootstrap] Arch: amd64
2016-05-05T13:37:47.669Z WARN  [DeadEventLoggingListener] Received unhandled event of type <org.graylog2.plugin.lifecycles.Lifecycle> from event bus <AsyncEventBus{graylog-eventbus}>
2016-05-05T13:37:47.694Z INFO  [PeriodicalsService] Starting 24 periodicals ...
2016-05-05T13:37:47.694Z INFO  [Periodicals] Starting [org.graylog2.periodical.ThroughputCalculator] periodical in [0s], polling every [1s].
2016-05-05T13:37:47.701Z INFO  [node] [graylog-fea7225f-ff92-4cd7-ac17-2adffb9d2c61] starting ...
2016-05-05T13:37:47.710Z INFO  [Periodicals] Starting [org.graylog2.periodical.AlertScannerThread] periodical in [10s], polling every [60s].
2016-05-05T13:37:47.713Z INFO  [Periodicals] Starting [org.graylog2.periodical.BatchedElasticSearchOutputFlushThread] periodical in [0s], polling every [1s].
2016-05-05T13:37:47.713Z INFO  [Periodicals] Starting [org.graylog2.periodical.ClusterHealthCheckThread] periodical in [0s], polling every [20s].
2016-05-05T13:37:47.716Z INFO  [Periodicals] Starting [org.graylog2.periodical.ContentPackLoaderPeriodical] periodical, running forever.
2016-05-05T13:37:47.716Z INFO  [Periodicals] Starting [org.graylog2.periodical.GarbageCollectionWarningThread] periodical, running forever.
2016-05-05T13:37:47.717Z INFO  [Periodicals] Starting [org.graylog2.periodical.IndexerClusterCheckerThread] periodical in [0s], polling every [30s].
2016-05-05T13:37:47.718Z INFO  [Periodicals] Starting [org.graylog2.periodical.IndexRetentionThread] periodical in [0s], polling every [300s].
2016-05-05T13:37:47.724Z INFO  [IndexerClusterCheckerThread] Indexer not fully initialized yet. Skipping periodic cluster check.
2016-05-05T13:37:47.725Z INFO  [Periodicals] Starting [org.graylog2.periodical.IndexRotationThread] periodical in [0s], polling every [10s].
2016-05-05T13:37:47.725Z INFO  [IndexRetentionThread] Elasticsearch cluster not available, skipping index retention checks.
2016-05-05T13:37:47.725Z INFO  [Periodicals] Starting [org.graylog2.periodical.NodePingThread] periodical in [0s], polling every [1s].
2016-05-05T13:37:47.726Z INFO  [Periodicals] Starting [org.graylog2.periodical.VersionCheckThread] periodical in [300s], polling every [1800s].
2016-05-05T13:37:47.726Z INFO  [Periodicals] Starting [org.graylog2.periodical.ThrottleStateUpdaterThread] periodical in [1s], polling every [1s].
2016-05-05T13:37:47.726Z INFO  [Periodicals] Starting [org.graylog2.events.ClusterEventPeriodical] periodical in [0s], polling every [1s].
2016-05-05T13:37:47.728Z INFO  [Periodicals] Starting [org.graylog2.events.ClusterEventCleanupPeriodical] periodical in [0s], polling every [300s].
2016-05-05T13:37:47.729Z INFO  [Periodicals] Starting [org.graylog2.periodical.ClusterIdGeneratorPeriodical] periodical, running forever.
2016-05-05T13:37:47.729Z INFO  [Periodicals] Starting [org.graylog2.periodical.IndexRangesMigrationPeriodical] periodical, running forever.
2016-05-05T13:37:47.730Z INFO  [Periodicals] Starting [org.graylog2.periodical.IndexRangesCleanupPeriodical] periodical in [15s], polling every [3600s].
2016-05-05T13:37:47.745Z INFO  [connection] Opened connection [connectionId{localValue:7, serverValue:278}] to localhost:27017
2016-05-05T13:37:47.747Z INFO  [connection] Opened connection [connectionId{localValue:3, serverValue:274}] to localhost:27017
2016-05-05T13:37:47.757Z INFO  [connection] Opened connection [connectionId{localValue:4, serverValue:275}] to localhost:27017
2016-05-05T13:37:47.761Z INFO  [connection] Opened connection [connectionId{localValue:9, serverValue:280}] to localhost:27017
2016-05-05T13:37:47.768Z INFO  [connection] Opened connection [connectionId{localValue:6, serverValue:277}] to localhost:27017
2016-05-05T13:37:47.770Z INFO  [connection] Opened connection [connectionId{localValue:8, serverValue:279}] to localhost:27017
2016-05-05T13:37:47.776Z INFO  [connection] Opened connection [connectionId{localValue:5, serverValue:276}] to localhost:27017
2016-05-05T13:37:47.814Z INFO  [PeriodicalsService] Not starting [org.graylog2.periodical.UserPermissionMigrationPeriodical] periodical. Not configured to run on this node.
2016-05-05T13:37:47.814Z INFO  [Periodicals] Starting [org.graylog2.periodical.AlarmCallbacksMigrationPeriodical] periodical, running forever.
2016-05-05T13:37:47.818Z INFO  [Periodicals] Starting [org.graylog2.periodical.ConfigurationManagementPeriodical] periodical, running forever.
2016-05-05T13:37:47.827Z INFO  [Periodicals] Starting [org.graylog2.periodical.LdapGroupMappingMigration] periodical, running forever.
2016-05-05T13:37:47.837Z INFO  [Periodicals] Starting [org.graylog.plugins.usagestatistics.UsageStatsNodePeriodical] periodical in [300s], polling every [21600s].
2016-05-05T13:37:47.840Z INFO  [Periodicals] Starting [org.graylog.plugins.usagestatistics.UsageStatsClusterPeriodical] periodical in [300s], polling every [21600s].
2016-05-05T13:37:47.841Z INFO  [Periodicals] Starting [org.graylog.plugins.collector.periodical.PurgeExpiredCollectorsThread] periodical in [0s], polling every [3600s].
2016-05-05T13:37:48.103Z INFO  [transport] [graylog-fea7225f-ff92-4cd7-ac17-2adffb9d2c61] publish_address {127.0.0.1:9350}, bound_addresses {[::1]:9350}, {127.0.0.1:9350}
2016-05-05T13:37:48.121Z INFO  [discovery] [graylog-fea7225f-ff92-4cd7-ac17-2adffb9d2c61] graylog/O4qTRaIpSrWiu7vDxDPRUA
2016-05-05T13:37:48.347Z INFO  [AbstractJerseyService] Enabling CORS for HTTP endpoint
2016-05-05T13:37:51.133Z WARN  [discovery] [graylog-fea7225f-ff92-4cd7-ac17-2adffb9d2c61] waited for 3s and no initial state was set by the discovery
2016-05-05T13:37:51.134Z INFO  [node] [graylog-fea7225f-ff92-4cd7-ac17-2adffb9d2c61] started
2016-05-05T13:37:51.300Z INFO  [service] [graylog-fea7225f-ff92-4cd7-ac17-2adffb9d2c61] detected_master {Deadhead}{Xt0uILKXQ82RTj_mfB6uKA}{127.0.0.1}{127.0.0.1:9300}, added {{Deadhead}{Xt0uILKXQ82RTj_mfB6uKA}{127.0.0.1}{127.0.0.1:9300},}, reason: zen-disco-receive(from master [{Deadhead}{Xt0uILKXQ82RTj_mfB6uKA}{127.0.0.1}{127.0.0.1:9300}])
2016-05-05T13:37:52.353Z INFO  [NetworkListener] Started listener bound to [0.0.0.0:9000]
2016-05-05T13:37:52.355Z INFO  [HttpServer] [HttpServer] Started.
2016-05-05T13:37:52.356Z INFO  [WebInterfaceService] Started Web Interface at <https://0.0.0.0:9000/>
2016-05-05T13:37:57.045Z INFO  [NetworkListener] Started listener bound to [0.0.0.0:12900]
2016-05-05T13:37:57.046Z INFO  [HttpServer] [HttpServer-1] Started.
2016-05-05T13:37:57.046Z INFO  [RestApiService] Started REST API at <https://0.0.0.0:12900/>
2016-05-05T13:37:57.047Z INFO  [ServiceManagerListener] Services are healthy
2016-05-05T13:37:57.048Z INFO  [ServerBootstrap] Services started, startup times in ms: {OutputSetupService [RUNNING]=60, MetricsReporterService [RUNNING]=61, BufferSynchronizerService [RUNNING]=61, KafkaJournal [RUNNING]=66, InputSetupService [RUNNING]=71, PeriodicalsService [RUNNING]=168, JournalReader [RUNNING]=179, IndexerSetupService [RUNNING]=3650, WebInterfaceService [RUNNING]=4672, RestApiService [RUNNING]=9357}
2016-05-05T13:37:57.049Z INFO  [InputSetupService] Triggering launching persisted inputs, node transitioned from Uninitialized [LB:DEAD] to Running [LB:ALIVE]
2016-05-05T13:37:57.053Z INFO  [ServerBootstrap] Graylog server up and running.
2016-05-05T13:38:02.534Z WARN  [ProxiedResource] Unable to call http://10.0.26.10:12900/system/metrics/multiple on node <fea7225f-ff92-4cd7-ac17-2adffb9d2c61>, caught exception: unexpected end of stream on okhttp3.Address@2b5fd040 (class java.io.IOException)
2016-05-05T13:38:03.627Z WARN  [ProxiedResource] Unable to call http://10.0.26.10:12900/system/metrics/multiple on node <fea7225f-ff92-4cd7-ac17-2adffb9d2c61>, caught exception: unexpected end of stream on okhttp3.Address@2b5fd040 (class java.io.IOException)
2016-05-05T13:38:03.685Z WARN  [ProxiedResource] Unable to call http://10.0.26.10:12900/system/metrics/multiple on node <fea7225f-ff92-4cd7-ac17-2adffb9d2c61>, caught exception: unexpected end of stream on okhttp3.Address@2b5fd040 (class java.io.IOException)
2016-05-05T13:38:04.836Z WARN  [ProxiedResource] Unable to call http://10.0.26.10:12900/system/metrics/multiple on node <fea7225f-ff92-4cd7-ac17-2adffb9d2c61>, caught exception: unexpected end of stream on okhttp3.Address@2b5fd040 (class java.io.IOException)
2016-05-05T13:38:04.940Z WARN  [ProxiedResource] Unable to call http://10.0.26.10:12900/system/metrics/multiple on node <fea7225f-ff92-4cd7-ac17-2adffb9d2c61>, caught exception: unexpected end of stream on okhttp3.Address@2b5fd040 (class java.io.IOException)
2016-05-05T13:38:05.073Z WARN  [ProxiedResource] Unable to call http://10.0.26.10:12900/system/metrics/multiple on node <fea7225f-ff92-4cd7-ac17-2adffb9d2c61>, caught exception: unexpected end of stream on okhttp3.Address@2b5fd040 (class java.io.IOException)
2016-05-05T13:38:05.170Z WARN  [ProxiedResource] Unable to call http://10.0.26.10:12900/system/metrics/multiple on node <fea7225f-ff92-4cd7-ac17-2adffb9d2c61>, caught exception: unexpected end of stream on okhttp3.Address@2b5fd040 (class java.io.IOException)
2016-05-05T13:38:05.888Z WARN  [ProxiedResource] Unable to call http://10.0.26.10:12900/system/metrics/multiple on node <fea7225f-ff92-4cd7-ac17-2adffb9d2c61>, caught exception: unexpected end of stream on okhttp3.Address@2b5fd040 (class java.io.IOException)
2016-05-05T13:38:06.003Z WARN  [ProxiedResource] Unable to call http://10.0.26.10:12900/system/inputstates on node <fea7225f-ff92-4cd7-ac17-2adffb9d2c61>, caught exception: unexpected end of stream on okhttp3.Address@2b5fd040 (class java.io.IOException)
2016-05-05T13:38:06.143Z WARN  [ProxiedResource] Unable to call http://10.0.26.10:12900/system/metrics/multiple on node <fea7225f-ff92-4cd7-ac17-2adffb9d2c61>, caught exception: unexpected end of stream on okhttp3.Address@2b5fd040 (class java.io.IOException)
2016-05-05T13:38:06.780Z WARN  [ProxiedResource] Unable to call http://10.0.26.10:12900/system/metrics/multiple on node <fea7225f-ff92-4cd7-ac17-2adffb9d2c61>, caught exception: unexpected end of stream on okhttp3.Address@2b5fd040 (class java.io.IOException)
^C
[ec2-user@ip-10-0-26-10 server]$

Environment

onedaywillcome1 commented 8 years ago

I saw similar problem in post [https://github.com/Graylog2/graylog2-server/issues/2186], tried to add my openssl x509 pem file into jdk keytool as below, but still getting "Unable to calll" warning messages in graylog log file :

jdk1.8.0_66/bin/keytool -trustcacerts -keystore cacerts -storepass changeit -noprompt -importcert -file /etc/graylog/server/x509key.pem

Logs:

2016-05-05T13:38:02.534Z WARN  [ProxiedResource] Unable to call http://10.0.26.10:12900/system/metrics/multiple on node <fea7225f-ff92-4cd7-ac17-2adffb9d2c61>, caught exception: unexpected end of stream on okhttp3.Address@2b5fd040 (class java.io.IOException)
2016-05-05T13:38:03.627Z WARN  [ProxiedResource] Unable to call http://10.0.26.10:12900/system/metrics/multiple on node <fea7225f-ff92-4cd7-ac17-2adffb9d2c61>, caught exception: unexpected end of stream on okhttp3.Address@2b5fd040 (class java.io.IOException)
2016-05-05T13:38:03.685Z WARN  [ProxiedResource] Unable to call http://10.0.26.10:12900/system/metrics/multiple on node <fea7225f-ff92-4cd7-ac17-2adffb9d2c61>, caught exception: unexpected end of stream on okhttp3.Address@2b5fd040 (class java.io.IOException)
2016-05-05T13:38:04.836Z WARN  [ProxiedResource] Unable to call http://10.0.26.10:12900/system/metrics/multiple on node <fea7225f-ff92-4cd7-ac17-2adffb9d2c61>, caught exception: unexpected end of stream on okhttp3.Address@2b5fd040 (class java.io.IOException)
2016-05-05T13:38:04.940Z WARN  [ProxiedResource] Unable to call http://10.0.26.10:12900/system/metrics/multiple on node <fea7225f-ff92-4cd7-ac17-2adffb9d2c61>, caught exception: unexpected end of stream on okhttp3.Address@2b5fd040 (class java.io.IOException)
2016-05-05T13:38:05.073Z WARN  [ProxiedResource] Unable to call http://10.0.26.10:12900/system/metrics/multiple on node <fea7225f-ff92-4cd7-ac17-2adffb9d2c61>, caught exception: unexpected end of stream on okhttp3.Address@2b5fd040 (class java.io.IOException)
.
.
dennisoelkers commented 8 years ago

What happens when you try to get http://10.0.26.10:12900/system/metrics/multiple using valid credentials from the Graylog server machine you are connecting against with the web interface?

onedaywillcome1 commented 8 years ago

@dennisoelkers http://10.0.26.10:12900/system/metrics/multiple is not reachable. But, when I try to get https://10.0.26.10:12900/system/metrics/multiple

It gives following message: {"type":"ApiError","message":"HTTP 405 Method Not Allowed"}

I noticed that, I am not starting input after creating it via https in addition to getting warning messages. Web UI gives a following error:

Input 'GELF TCP' could not be started Request to start input 'GELF TCP' failed. Check your Graylog logs for more information.

By the way, I tried to reach http://10.0.26.10:12900/system/metrics/multiple when I disable https features in graylog config. But, I am still getting following message: {"type":"ApiError","message":"HTTP 405 Method Not Allowed"}

onedaywillcome1 commented 8 years ago

I enabled rest_transport_uri and disabled web_endpoint_uri in graylog config. I am still getting warning messages. But, this time warning messages changed.

Graylog config: 

rest_listen_uri = https://0.0.0.0:12900/
rest_transport_uri = https://<public-ip>:12900
rest_enable_tls = true
rest_tls_cert_file = /etc/graylog/server/x509key.pem
rest_tls_key_file = /etc/graylog/server/pkc8key.pem

web_listen_uri = https://0.0.0.0:9000/
#web_endpoint_uri = https://<public-ip>:12900   (Commented)
web_enable_tls = true
web_tls_cert_file = /etc/graylog/server/x509key.pem
web_tls_key_file = /etc/graylog/server/pkc8key.pem

Warning messages

2016-05-09T11:09:29.838Z WARN  [ProxiedResource] Unable to call https://10.0.26.10:12900/system/metrics/multiple on node <78638bca-cc35-45b6-96f3-89c5e8cde52b>, caught exception: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target (class javax.net.ssl.SSLHandshakeException)
2016-05-09T11:09:30.008Z WARN  [ProxiedResource] Unable to call https://10.0.26.10:12900/system/metrics/multiple on node <78638bca-cc35-45b6-96f3-89c5e8cde52b>, caught exception: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target (class javax.net.ssl.SSLHandshakeException)
2016-05-09T11:09:31.971Z WARN  [ProxiedResource] Unable to call https://10.0.26.10:12900/system/metrics/multiple on node <78638bca-cc35-45b6-96f3-89c5e8cde52b>, caught exception: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target (class javax.net.ssl.SSLHandshakeException)
2016-05-09T11:09:34.416Z WARN  [ProxiedResource] Unable to call https://10.0.26.10:12900/system/inputstates on node <78638bca-cc35-45b6-96f3-89c5e8cde52b>, caught exception: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target (class javax.net.ssl.SSLHandshakeException)
2016-05-09T11:09:34.979Z WARN  [ProxiedResource] Unable to call https://10.0.26.10:12900/system/metrics/multiple on node <78638bca-cc35-45b6-96f3-89c5e8cde52b>, caught exception: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target (class javax.net.ssl.SSLHandshakeException)
2016-05-09T11:09:36.002Z WARN  [ProxiedResource] Unable to call https://10.0.26.10:12900/system/inputstates on node <78638bca-cc35-45b6-96f3-89c5e8cde52b>, caught exception: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target (class javax.net.ssl.SSLHandshakeException)

When I try to get https://10.0.26.10:12900/system/inputstates in web browser, I get following message:

{"states":[{"id":"572c93c3a6c2e4014d9c0f11","state":"RUNNING","started_at":"2016-05-09T11:09:23.240Z","detailed_message":null,"message_input":{"title":"GELF TCP","global":false,"name":"GELF TCP","content_pack":null,"created_at":"2016-05-06T12:53:23.382Z","type":"org.graylog2.inputs.gelf.tcp.GELFTCPInput","creator_user_id":"admin","attributes":{"recv_buffer_size":1048576,"tcp_keepalive":false,"use_null_delimiter":true,"tls_client_auth_cert_file":"","bind_address":"0.0.0.0","tls_cert_file":"","port":12201,"tls_key_file":"","tls_enable":false,"tls_key_password":"","max_message_size":2097152,"tls_client_auth":"disabled","override_source":null},"static_fields":{},"node":"78638bca-cc35-45b6-96f3-89c5e8cde52b","id":"572c93c3a6c2e4014d9c0f11"}}]}

When I try to get https://10.0.26.10:12900/system/metrics/multiple

{"type":"ApiError","message":"HTTP 405 Method Not Allowed"}

I tried to import my x509key.pem into keytool by following command and restarted graylog-server, but still getting above warning messages:

[ec2-user@ip-10-0-26-10 bin]$ sudo /jdk1.8.0_66/jre/bin/keytool -trustcacerts -keystore ~/jdk1.8.0_66/jre/lib/security/cacerts -storepass changeit -alias tomcat7 -noprompt -importcert -file /etc/graylog/server/x509key.pem

[ec2-user@ip-10-0-26-10 bin]$ /jdk1.8.0_66/jre/bin/keytool -list -keystore ~/jdk1.8.0_66/jre/lib/security/cacerts 
Enter keystore password:  

Keystore type: JKS
Keystore provider: SUN

Your keystore contains 3 entries

tomcat7, May 9, 2016, trustedCertEntry, 
Certificate fingerprint (SHA1): C2:D3:0C:11:AB:50:EE:18:1D:D8:78:88:1D:7B:1D:3D:AC:E7:D0:BB
tomcat5, May 6, 2016, trustedCertEntry, 
Certificate fingerprint (SHA1): C2:D3:0C:11:AB:50:EE:18:1D:D8:78:88:1D:7B:1D:3D:AC:E7:D0:BB
mykey, May 6, 2016, trustedCertEntry, 
Certificate fingerprint (SHA1): C2:D3:0C:11:AB:50:EE:18:1D:D8:78:88:1D:7B:1D:3D:AC:E7:D0:BB
dennisoelkers commented 8 years ago

Configuring rest_transport_uri is correct, but you still need to configure web_endpoint_uri if you set web_listen_uri to 0.0.0.0. The remaining error (sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target (class javax.net.ssl.SSLHandshakeException)) means that you Graylog server does not trust the certificate of the Graylog server it connects to (even if it connects back to itself). Therefore make sure you have added the correct certificate (preferably the CA cert) to the correct keystore used by the Graylog server.

As this is not a bug in the Graylog server, I will close this issue now. If you need additional help, please use the Graylog mailing list (see https://www.graylog.org/community-support).