search

elastic / kibana

Your window into the Elastic Stack
https://www.elastic.co/products/kibana
Other
19.63k stars 8.22k forks source link

elasticsearch.hosts with multiple nodes #102684

Closed CR-ItsikHarel closed 2 years ago

CR-ItsikHarel commented 3 years ago

Kibana version: v 7.10.1

Elasticsearch version: v 7.10.1

Server OS version: Centos 7 , uanme -a:3.10.0-1127.19.1.el7.x86_64

Browser version: Chrome

Browser OS version: macos

Original install method (e.g. download page, yum, from source, etc.):

Describe the bug: In kibana.yml, when i specify a single node or multiple valid nodes everything works, exmaple: elasticsearch.hosts: '["https://cr-dr-performance-us-420-client1.dr.com:9200","https://cr-dr-performance-us-420-client2.dr.com:9200","https://cr-dr-performance-us-420-client3.dr.come:9200"]'

If all nodes are valid and responsive everything works, if one becomes unavailable kibana stop working.

Steps to reproduce:

  1. added an invalid or unavailable to url to kibana.yml elasticsearch.hosts
  2. kibana service will start but the UI will fail to start

Expected behavior: i would expect the application to start as long as a single hostname is valid, regardless of unavailable or invalid hosts

Screenshots (if relevant):

Errors in browser console (if relevant): Version: 7.10.1 Build: 36063 Error: Internal Server Error at Fetch._callee3$ (https://cr-dr-performance-us_kibana-2.testing.luminatesec.com/36063/bundles/core/core.entry.js:6:59535) at l (https://cr-dr-performance-us_kibana-2.testing.luminatesec.com/36063/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:380:1740520) at Generator._invoke (https://cr-dr-performance-us_kibana-2.testing.luminatesec.com/36063/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:380:1740273) at Generator.forEach.e. [as next] (https://cr-dr-performance-us_kibana-2.testing.luminatesec.com/36063/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:380:1740877) at fetch_asyncGeneratorStep (https://cr-dr-performance-us_kibana-2.testing.luminatesec.com/36063/bundles/core/core.entry.js:6:52652) at _next (https://cr-dr-performance-us_kibana-2.testing.luminatesec.com/36063/bundles/core/core.entry.js:6:52968)

Provide logs and/or server output (if relevant):

{"type":"log","@timestamp":"2021-06-20T13:08:53Z","tags":["info","plugins-system"],"pid":1626,"message":"Setting up [96] plugins: [taskManager,licensing,globalSearch,globalSearchProviders,code,usageCollection,xpackLegacy,telemetryCollectionManager,telemetry,telemetryCollectionXpack,kibanaUsageCollection,securityOss,newsfeed,mapsLegacy,kibanaLegacy,translations,share,legacyExport,embeddable,uiActionsEnhanced,expressions,data,home,observability,cloud,console,consoleExtensions,apmOss,searchprofiler,painlessLab,grokdebugger,management,indexPatternManagement,advancedSettings,fileUpload,savedObjects,dashboard,visualizations,visTypeVega,visTypeTimelion,timelion,features,upgradeAssistant,security,snapshotRestore,enterpriseSearch,encryptedSavedObjects,ingestManager,indexManagement,remoteClusters,crossClusterReplication,indexLifecycleManagement,dashboardMode,beatsManagement,transform,ingestPipelines,maps,licenseManagement,graph,dataEnhanced,visTypeTable,visTypeMarkdown,tileMap,regionMap,inputControlVis,visualize,esUiShared,bfetch,canvas,charts,lens,visTypeVislib,visTypeTimeseries,rollup,visTypeTagcloud,visTypeMetric,watcher,discover,discoverEnhanced,savedObjectsManagement,spaces,reporting,lists,eventLog,actions,case,alerts,stackAlerts,triggersActionsUi,ml,securitySolution,infra,monitoring,logstash,apm,uptime]"} {"type":"log","@timestamp":"2021-06-20T13:08:54Z","tags":["warning","plugins","security","config"],"pid":1626,"message":"Generating a random key for xpack.security.encryptionKey. To prevent sessions from being invalidated on restart, please set xpack.security.encryptionKey in kibana.yml"} {"type":"log","@timestamp":"2021-06-20T13:08:54Z","tags":["warning","plugins","security","config"],"pid":1626,"message":"Session cookies will be transmitted over insecure connections. This is not recommended."} {"type":"log","@timestamp":"2021-06-20T13:08:54Z","tags":["warning","plugins","encryptedSavedObjects","config"],"pid":1626,"message":"Generating a random key for xpack.encryptedSavedObjects.encryptionKey. To be able to decrypt encrypted saved objects attributes after restart, please set xpack.encryptedSavedObjects.encryptionKey in kibana.yml"} {"type":"log","@timestamp":"2021-06-20T13:08:54Z","tags":["warning","plugins","ingestManager"],"pid":1626,"message":"Fleet APIs are disabled due to the Encrypted Saved Objects plugin using an ephemeral encryption key. Please set xpack.encryptedSavedObjects.encryptionKey in kibana.yml."} {"type":"log","@timestamp":"2021-06-20T13:08:54Z","tags":["warning","plugins","reporting","config"],"pid":1626,"message":"Generating a random key for xpack.reporting.encryptionKey. To prevent sessions from being invalidated on restart, please set xpack.reporting.encryptionKey in kibana.yml"} {"type":"log","@timestamp":"2021-06-20T13:08:54Z","tags":["warning","plugins","reporting","config"],"pid":1626,"message":"Chromium sandbox provides an additional layer of protection, but is not supported for Linux CentOS 7.6.1810 OS. Automatically setting 'xpack.reporting.capture.browser.chromium.disableSandbox: true'."} {"type":"log","@timestamp":"2021-06-20T13:08:54Z","tags":["warning","plugins","actions","actions"],"pid":1626,"message":"APIs are disabled due to the Encrypted Saved Objects plugin using an ephemeral encryption key. Please set xpack.encryptedSavedObjects.encryptionKey in kibana.yml."} {"type":"log","@timestamp":"2021-06-20T13:08:54Z","tags":["warning","plugins","alerts","plugins","alerting"],"pid":1626,"message":"APIs are disabled due to the Encrypted Saved Objects plugin using an ephemeral encryption key. Please set xpack.encryptedSavedObjects.encryptionKey in kibana.yml."} {"type":"log","@timestamp":"2021-06-20T13:08:54Z","tags":["info","plugins","monitoring","monitoring"],"pid":1626,"message":"config sourced from: production cluster"} {"type":"log","@timestamp":"2021-06-20T13:08:54Z","tags":["info","savedobjects-service"],"pid":1626,"message":"Waiting until all Elasticsearch nodes are compatible with Kibana before starting saved objects migrations..."} {"type":"log","@timestamp":"2021-06-20T13:08:56Z","tags":["info","savedobjects-service"],"pid":1626,"message":"Starting saved objects migrations"} {"type":"log","@timestamp":"2021-06-20T13:09:06Z","tags":["error","elasticsearch","data"],"pid":1626,"message":"[ConnectionError]: getaddrinfo ENOTFOUND cr-dr-performance-us-420-client3.dr.testing.net cr-dr-performance-us-420-client3.dr.testing.net:9200"} {"type":"log","@timestamp":"2021-06-20T13:09:06Z","tags":["warning","savedobjects-service"],"pid":1626,"message":"Unable to connect to Elasticsearch. Error: getaddrinfo ENOTFOUND cr-dr-performance-us-420-client3.dr.testing.net cr-dr-performance-us-420-client3.dr.testing.net:9200"} {"type":"log","@timestamp":"2021-06-20T13:09:11Z","tags":["error","elasticsearch","data"],"pid":1626,"message":"[ConnectionError]: getaddrinfo ENOTFOUND cr-dr-performance-us-420-client3.dr.testing.net cr-dr-performance-us-420-client3.dr.testing.net:9200"} {"type":"log","@timestamp":"2021-06-20T13:09:11Z","tags":["warning","savedobjects-service"],"pid":1626,"message":"Unable to connect to Elasticsearch. Error: getaddrinfo ENOTFOUND cr-dr-performance-us-420-client3.dr.testing.net cr-dr-performance-us-420-client3.dr.testing.net:9200"} {"type":"log","@timestamp":"2021-06-20T13:09:14Z","tags":["info","plugins-system"],"pid":1626,"message":"Starting [96] plugins: [taskManager,licensing,globalSearch,globalSearchProviders,code,usageCollection,xpackLegacy,telemetryCollectionManager,telemetry,telemetryCollectionXpack,kibanaUsageCollection,securityOss,newsfeed,mapsLegacy,kibanaLegacy,translations,share,legacyExport,embeddable,uiActionsEnhanced,expressions,data,home,observability,cloud,console,consoleExtensions,apmOss,searchprofiler,painlessLab,grokdebugger,management,indexPatternManagement,advancedSettings,fileUpload,savedObjects,dashboard,visualizations,visTypeVega,visTypeTimelion,timelion,features,upgradeAssistant,security,snapshotRestore,enterpriseSearch,encryptedSavedObjects,ingestManager,indexManagement,remoteClusters,crossClusterReplication,indexLifecycleManagement,dashboardMode,beatsManagement,transform,ingestPipelines,maps,licenseManagement,graph,dataEnhanced,visTypeTable,visTypeMarkdown,tileMap,regionMap,inputControlVis,visualize,esUiShared,bfetch,canvas,charts,lens,visTypeVislib,visTypeTimeseries,rollup,visTypeTagcloud,visTypeMetric,watcher,discover,discoverEnhanced,savedObjectsManagement,spaces,reporting,lists,eventLog,actions,case,alerts,stackAlerts,triggersActionsUi,ml,securitySolution,infra,monitoring,logstash,apm,uptime]"} {"type":"log","@timestamp":"2021-06-20T13:09:14Z","tags":["info","plugins","taskManager","taskManager"],"pid":1626,"message":"TaskManager is identified by the Kibana UUID: dc8dd0b0-fe0c-4c12-b13a-b455cbcecdf0"} @ "kibana.log" 71L, 32553C {"type":"response","@timestamp":"2021-06-20T13:09:36Z","tags":[],"pid":1626,"method":"head","statusCode":302,"req":{"url":"/","method":"head","headers":{"host":"10.240.96.54:5601","user-agent":"ApplicationHealthService/1.0","connection":"close"},"remoteAddress":"10.239.2.10","userAgent":"ApplicationHealthService/1.0"},"res":{"statusCode":302,"responseTime":1,"contentLength":9},"message":"HEAD / 302 1ms - 9.0B"} {"type":"response","@timestamp":"2021-06-20T13:09:36Z","tags":[],"pid":1626,"method":"head","statusCode":302,"req":{"url":"/","method":"head","headers":{"host":"10.240.96.54:5601","user-agent":"ApplicationHealthService/1.0","connection":"close"},"remoteAddress":"10.239.2.10","userAgent":"ApplicationHealthService/1.0"},"res":{"statusCode":302,"responseTime":1,"contentLength":9},"message":"HEAD / 302 1ms - 9.0B"} {"type":"response","@timestamp":"2021-06-20T13:09:36Z","tags":[],"pid":1626,"method":"head","statusCode":302,"req":{"url":"/","method":"head","headers":{"host":"10.240.96.54:5601","user-agent":"ApplicationHealthService/1.0","connection":"close"},"remoteAddress":"10.239.2.10","userAgent":"ApplicationHealthService/1.0"},"res":{"statusCode":302,"responseTime":1,"contentLength":9},"message":"HEAD / 302 1ms - 9.0B"} {"type":"response","@timestamp":"2021-06-20T13:09:36Z","tags":[],"pid":1626,"method":"head","statusCode":302,"req":{"url":"/","method":"head","headers":{"host":"10.240.96.54:5601","user-agent":"ApplicationHealthService/1.0","connection":"close"},"remoteAddress":"10.239.2.10","userAgent":"ApplicationHealthService/1.0"},"res":{"statusCode":302,"responseTime":1,"contentLength":9},"message":"HEAD / 302 1ms - 9.0B"} {"type":"response","@timestamp":"2021-06-20T13:09:36Z","tags":[],"pid":1626,"method":"head","statusCode":302,"req":{"url":"/","method":"head","headers":{"host":"10.240.96.54:5601","user-agent":"ApplicationHealthService/1.0","connection":"close"},"remoteAddress":"10.239.2.10","userAgent":"ApplicationHealthService/1.0"},"res":{"statusCode":302,"responseTime":1,"contentLength":9},"message":"HEAD / 302 1ms - 9.0B"} {"type":"response","@timestamp":"2021-06-20T13:09:36Z","tags":[],"pid":1626,"method":"head","statusCode":302,"req":{"url":"/","method":"head","headers":{"host":"10.240.96.54:5601","user-agent":"ApplicationHealthService/1.0","connection":"close"},"remoteAddress":"10.239.2.10","userAgent":"ApplicationHealthService/1.0"},"res":{"statusCode":302,"responseTime":1,"contentLength":9},"message":"HEAD / 302 1ms - 9.0B"} {"type":"response","@timestamp":"2021-06-20T13:09:36Z","tags":[],"pid":1626,"method":"head","statusCode":302,"req":{"url":"/","method":"head","headers":{"host":"10.240.96.54:5601","user-agent":"ApplicationHealthService/1.0","connection":"close"},"remoteAddress":"10.239.2.10","userAgent":"ApplicationHealthService/1.0"},"res":{"statusCode":302,"responseTime":1,"contentLength":9},"message":"HEAD / 302 1ms - 9.0B"} {"type":"response","@timestamp":"2021-06-20T13:09:36Z","tags":[],"pid":1626,"method":"head","statusCode":302,"req":{"url":"/","method":"head","headers":{"host":"10.240.96.54:5601","user-agent":"ApplicationHealthService/1.0","connection":"close"},"remoteAddress":"10.239.2.10","userAgent":"ApplicationHealthService/1.0"},"res":{"statusCode":302,"responseTime":1,"contentLength":9},"message":"HEAD / 302 1ms - 9.0B"} {"type":"response","@timestamp":"2021-06-20T13:09:36Z","tags":[],"pid":1626,"method":"head","statusCode":302,"req":{"url":"/","method":"head","headers":{"host":"10.240.96.54:5601","user-agent":"ApplicationHealthService/1.0","connection":"close"},"remoteAddress":"10.239.2.10","userAgent":"ApplicationHealthService/1.0"},"res":{"statusCode":302,"responseTime":0,"contentLength":9},"message":"HEAD / 302 0ms - 9.0B"} {"type":"response","@timestamp":"2021-06-20T13:09:36Z","tags":[],"pid":1626,"method":"head","statusCode":302,"req":{"url":"/","method":"head","headers":{"host":"10.240.96.54:5601","user-agent":"ApplicationHealthService/1.0","connection":"close"},"remoteAddress":"10.239.2.10","userAgent":"ApplicationHealthService/1.0"},"res":{"statusCode":302,"responseTime":1,"contentLength":9},"message":"HEAD / 302 1ms - 9.0B"} {"type":"response","@timestamp":"2021-06-20T13:09:36Z","tags":[],"pid":1626,"method":"head","statusCode":302,"req":{"url":"/","method":"head","headers":{"host":"10.240.96.54:5601","user-agent":"ApplicationHealthService/1.0","connection":"close"},"remoteAddress":"10.239.2.10","userAgent":"ApplicationHealthService/1.0"},"res":{"statusCode":302,"responseTime":1,"contentLength":9},"message":"HEAD / 302 1ms - 9.0B"} {"type":"response","@timestamp":"2021-06-20T13:09:36Z","tags":[],"pid":1626,"method":"head","statusCode":302,"req":{"url":"/","method":"head","headers":{"host":"10.240.96.54:5601","user-agent":"ApplicationHealthService/1.0","connection":"close"},"remoteAddress":"10.239.2.10","userAgent":"ApplicationHealthService/1.0"},"res":{"statusCode":302,"responseTime":1,"contentLength":9},"message":"HEAD / 302 1ms - 9.0B"} {"type":"response","@timestamp":"2021-06-20T13:09:36Z","tags":[],"pid":1626,"method":"head","statusCode":302,"req":{"url":"/","method":"head","headers":{"host":"10.240.96.54:5601","user-agent":"ApplicationHealthService/1.0","connection":"close"},"remoteAddress":"10.239.2.10","userAgent":"ApplicationHealthService/1.0"},"res":{"statusCode":302,"responseTime":1,"contentLength":9},"message":"HEAD / 302 1ms - 9.0B"} {"type":"response","@timestamp":"2021-06-20T13:09:36Z","tags":[],"pid":1626,"method":"head","statusCode":302,"req":{"url":"/","method":"head","headers":{"host":"10.240.96.54:5601","user-agent":"ApplicationHealthService/1.0","connection":"close"},"remoteAddress":"10.239.2.10","userAgent":"ApplicationHealthService/1.0"},"res":{"statusCode":302,"responseTime":1,"contentLength":9},"message":"HEAD / 302 1ms - 9.0B"} {"type":"response","@timestamp":"2021-06-20T13:09:36Z","tags":[],"pid":1626,"method":"head","statusCode":302,"req":{"url":"/","method":"head","headers":{"host":"10.240.96.54:5601","user-agent":"ApplicationHealthService/1.0","connection":"close"},"remoteAddress":"10.239.2.10","userAgent":"ApplicationHealthService/1.0"},"res":{"statusCode":302,"responseTime":2,"contentLength":9},"message":"HEAD / 302 2ms - 9.0B"} {"type":"log","@timestamp":"2021-06-20T13:09:42Z","tags":["error","elasticsearch","security"],"pid":1626,"message":"Request error, retrying\nPOST https://cr-dr-performance-us-420-client3.dr.testing.net:9200/_security/user/_has_privileges => getaddrinfo ENOTFOUND cr-dr-performance-us-420-client3.dr.testing.net cr-dr-performance-us-420-client3.dr.testing.net:9200"} ~ ~ {"type":"response","@timestamp":"2021-06-20T13:09:36Z","tags":[],"pid":1626,"method":"head","statusCode":302,"req":{"url":"/","method":"head","headers":{"host":"10.240.96.54:5601","user-agent":"ApplicationHealthService/1.0","connection":"close"},"remoteAddress":"10.239.2.10","userAgent":"ApplicationHealthService/1.0"},"res":{"statusCode":302,"responseTime":1,"contentLength":9},"message":"HEAD / 302 1ms - 9.0B"} {"type":"response","@timestamp":"2021-06-20T13:09:36Z","tags":[],"pid":1626,"method":"head","statusCode":302,"req":{"url":"/","method":"head","headers":{"host":"10.240.96.54:5601","user-agent":"ApplicationHealthService/1.0","connection":"close"},"remoteAddress":"10.239.2.10","userAgent":"ApplicationHealthService/1.0"},"res":{"statusCode":302,"responseTime":1,"contentLength":9},"message":"HEAD / 302 1ms - 9.0B"} {"type":"response","@timestamp":"2021-06-20T13:09:36Z","tags":[],"pid":1626,"method":"head","statusCode":302,"req":{"url":"/","method":"head","headers":{"host":"10.240.96.54:5601","user-agent":"ApplicationHealthService/1.0","connection":"close"},"remoteAddress":"10.239.2.10","userAgent":"ApplicationHealthService/1.0"},"res":{"statusCode":302,"responseTime":1,"contentLength":9},"message":"HEAD / 302 1ms - 9.0B"} {"type":"response","@timestamp":"2021-06-20T13:09:36Z","tags":[],"pid":1626,"method":"head","statusCode":302,"req":{"url":"/","method":"head","headers":{"host":"10.240.96.54:5601","user-agent":"ApplicationHealthService/1.0","connection":"close"},"remoteAddress":"10.239.2.10","userAgent":"ApplicationHealthService/1.0"},"res":{"statusCode":302,"responseTime":1,"contentLength":9},"message":"HEAD / 302 1ms - 9.0B"} {"type":"response","@timestamp":"2021-06-20T13:09:36Z","tags":[],"pid":1626,"method":"head","statusCode":302,"req":{"url":"/","method":"head","headers":{"host":"10.240.96.54:5601","user-agent":"ApplicationHealthService/1.0","connection":"close"},"remoteAddress":"10.239.2.10","userAgent":"ApplicationHealthService/1.0"},"res":{"statusCode":302,"responseTime":1,"contentLength":9},"message":"HEAD / 302 1ms - 9.0B"} {"type":"response","@timestamp":"2021-06-20T13:09:36Z","tags":[],"pid":1626,"method":"head","statusCode":302,"req":{"url":"/","method":"head","headers":{"host":"10.240.96.54:5601","user-agent":"ApplicationHealthService/1.0","connection":"close"},"remoteAddress":"10.239.2.10","userAgent":"ApplicationHealthService/1.0"},"res":{"statusCode":302,"responseTime":1,"contentLength":9},"message":"HEAD / 302 1ms - 9.0B"} {"type":"response","@timestamp":"2021-06-20T13:09:36Z","tags":[],"pid":1626,"method":"head","statusCode":302,"req":{"url":"/","method":"head","headers":{"host":"10.240.96.54:5601","user-agent":"ApplicationHealthService/1.0","connection":"close"},"remoteAddress":"10.239.2.10","userAgent":"ApplicationHealthService/1.0"},"res":{"statusCode":302,"responseTime":1,"contentLength":9},"message":"HEAD / 302 1ms - 9.0B"} {"type":"response","@timestamp":"2021-06-20T13:09:36Z","tags":[],"pid":1626,"method":"head","statusCode":302,"req":{"url":"/","method":"head","headers":{"host":"10.240.96.54:5601","user-agent":"ApplicationHealthService/1.0","connection":"close"},"remoteAddress":"10.239.2.10","userAgent":"ApplicationHealthService/1.0"},"res":{"statusCode":302,"responseTime":0,"contentLength":9},"message":"HEAD / 302 0ms - 9.0B"} {"type":"response","@timestamp":"2021-06-20T13:09:36Z","tags":[],"pid":1626,"method":"head","statusCode":302,"req":{"url":"/","method":"head","headers":{"host":"10.240.96.54:5601","user-agent":"ApplicationHealthService/1.0","connection":"close"},"remoteAddress":"10.239.2.10","userAgent":"ApplicationHealthService/1.0"},"res":{"statusCode":302,"responseTime":1,"contentLength":9},"message":"HEAD / 302 1ms - 9.0B"} {"type":"response","@timestamp":"2021-06-20T13:09:36Z","tags":[],"pid":1626,"method":"head","statusCode":302,"req":{"url":"/","method":"head","headers":{"host":"10.240.96.54:5601","user-agent":"ApplicationHealthService/1.0","connection":"close"},"remoteAddress":"10.239.2.10","userAgent":"ApplicationHealthService/1.0"},"res":{"statusCode":302,"responseTime":1,"contentLength":9},"message":"HEAD / 302 1ms - 9.0B"} {"type":"response","@timestamp":"2021-06-20T13:09:36Z","tags":[],"pid":1626,"method":"head","statusCode":302,"req":{"url":"/","method":"head","headers":{"host":"10.240.96.54:5601","user-agent":"ApplicationHealthService/1.0","connection":"close"},"remoteAddress":"10.239.2.10","userAgent":"ApplicationHealthService/1.0"},"res":{"statusCode":302,"responseTime":1,"contentLength":9},"message":"HEAD / 302 1ms - 9.0B"} {"type":"response","@timestamp":"2021-06-20T13:09:36Z","tags":[],"pid":1626,"method":"head","statusCode":302,"req":{"url":"/","method":"head","headers":{"host":"10.240.96.54:5601","user-agent":"ApplicationHealthService/1.0","connection":"close"},"remoteAddress":"10.239.2.10","userAgent":"ApplicationHealthService/1.0"},"res":{"statusCode":302,"responseTime":1,"contentLength":9},"message":"HEAD / 302 1ms - 9.0B"} {"type":"response","@timestamp":"2021-06-20T13:09:36Z","tags":[],"pid":1626,"method":"head","statusCode":302,"req":{"url":"/","method":"head","headers":{"host":"10.240.96.54:5601","user-agent":"ApplicationHealthService/1.0","connection":"close"},"remoteAddress":"10.239.2.10","userAgent":"ApplicationHealthService/1.0"},"res":{"statusCode":302,"responseTime":1,"contentLength":9},"message":"HEAD / 302 1ms - 9.0B"} {"type":"response","@timestamp":"2021-06-20T13:09:36Z","tags":[],"pid":1626,"method":"head","statusCode":302,"req":{"url":"/","method":"head","headers":{"host":"10.240.96.54:5601","user-agent":"ApplicationHealthService/1.0","connection":"close"},"remoteAddress":"10.239.2.10","userAgent":"ApplicationHealthService/1.0"},"res":{"statusCode":302,"responseTime":2,"contentLength":9},"message":"HEAD / 302 2ms - 9.0B"} {"type":"log","@timestamp":"2021-06-20T13:09:42Z","tags":["error","elasticsearch","security"],"pid":1626,"message":"Request error, retrying\nPOST https://cr-dr-performance-us-420-client3.dr.testing.net:9200/_security/user/_has_privileges => getaddrinfo ENOTFOUND cr-dr-performance-us-420-client3.dr.testing.net cr-dr-performance-us-420-client3.dr.testing.net:9200"} ~ ~ ~ ~ ~ {"type":"log","@timestamp":"2021-06-20T13:08:53Z","tags":["info","plugins-service"],"pid":1626,"message":"Plugin \"visTypeXy\" is disabled."} {"type":"log","@timestamp":"2021-06-20T13:08:53Z","tags":["info","plugins-service"],"pid":1626,"message":"Plugin \"auditTrail\" is disabled."} {"type":"log","@timestamp":"2021-06-20T13:08:53Z","tags":["warning","config","deprecation"],"pid":1626,"message":"Config key [monitoring.cluster_alerts.email_notifications.email_address] will be required for email notifications to work in 8.0.\""} {"type":"log","@timestamp":"2021-06-20T13:08:53Z","tags":["info","plugins-system"],"pid":1626,"message":"Setting up [96] plugins: [taskManager,licensing,globalSearch,globalSearchProviders,code,usageCollection,xpackLegacy,telemetryCollectionManager,telemetry,telemetryCollectionXpack,kibanaUsageCollection,securityOss,newsfeed,mapsLegacy,kibanaLegacy,translations,share,legacyExport,embeddable,uiActionsEnhanced,expressions,data,home,observability,cloud,console,consoleExtensions,apmOss,searchprofiler,painlessLab,grokdebugger,management,indexPatternManagement,advancedSettings,fileUpload,savedObjects,dashboard,visualizations,visTypeVega,visTypeTimelion,timelion,features,upgradeAssistant,security,snapshotRestore,enterpriseSearch,encryptedSavedObjects,ingestManager,indexManagement,remoteClusters,crossClusterReplication,indexLifecycleManagement,dashboardMode,beatsManagement,transform,ingestPipelines,maps,licenseManagement,graph,dataEnhanced,visTypeTable,visTypeMarkdown,tileMap,regionMap,inputControlVis,visualize,esUiShared,bfetch,canvas,charts,lens,visTypeVislib,visTypeTimeseries,rollup,visTypeTagcloud,visTypeMetric,watcher,discover,discoverEnhanced,savedObjectsManagement,spaces,reporting,lists,eventLog,actions,case,alerts,stackAlerts,triggersActionsUi,ml,securitySolution,infra,monitoring,logstash,apm,uptime]"} {"type":"log","@timestamp":"2021-06-20T13:08:54Z","tags":["warning","plugins","security","config"],"pid":1626,"message":"Generating a random key for xpack.security.encryptionKey. To prevent sessions from being invalidated on restart, please set xpack.security.encryptionKey in kibana.yml"} {"type":"log","@timestamp":"2021-06-20T13:08:54Z","tags":["warning","plugins","security","config"],"pid":1626,"message":"Session cookies will be transmitted over insecure connections. This is not recommended."} {"type":"log","@timestamp":"2021-06-20T13:08:54Z","tags":["warning","plugins","encryptedSavedObjects","config"],"pid":1626,"message":"Generating a random key for xpack.encryptedSavedObjects.encryptionKey. To be able to decrypt encrypted saved objects attributes after restart, please set xpack.encryptedSavedObjects.encryptionKey in kibana.yml"} {"type":"log","@timestamp":"2021-06-20T13:08:54Z","tags":["warning","plugins","ingestManager"],"pid":1626,"message":"Fleet APIs are disabled due to the Encrypted Saved Objects plugin using an ephemeral encryption key. Please set xpack.encryptedSavedObjects.encryptionKey in kibana.yml."} {"type":"log","@timestamp":"2021-06-20T13:08:54Z","tags":["warning","plugins","reporting","config"],"pid":1626,"message":"Generating a random key for xpack.reporting.encryptionKey. To prevent sessions from being invalidated on restart, please set xpack.reporting.encryptionKey in kibana.yml"} {"type":"log","@timestamp":"2021-06-20T13:08:54Z","tags":["warning","plugins","reporting","config"],"pid":1626,"message":"Chromium sandbox provides an additional layer of protection, but is not supported for Linux CentOS 7.6.1810 OS. Automatically setting 'xpack.reporting.capture.browser.chromium.disableSandbox: true'."} {"type":"log","@timestamp":"2021-06-20T13:08:54Z","tags":["warning","plugins","actions","actions"],"pid":1626,"message":"APIs are disabled due to the Encrypted Saved Objects plugin using an ephemeral encryption key. Please set xpack.encryptedSavedObjects.encryptionKey in kibana.yml."} {"type":"log","@timestamp":"2021-06-20T13:08:54Z","tags":["warning","plugins","alerts","plugins","alerting"],"pid":1626,"message":"APIs are disabled due to the Encrypted Saved Objects plugin using an ephemeral encryption key. Please set xpack.encryptedSavedObjects.encryptionKey in kibana.yml."} {"type":"log","@timestamp":"2021-06-20T13:08:54Z","tags":["info","plugins","monitoring","monitoring"],"pid":1626,"message":"config sourced from: production cluster"} {"type":"log","@timestamp":"2021-06-20T13:08:54Z","tags":["info","savedobjects-service"],"pid":1626,"message":"Waiting until all Elasticsearch nodes are compatible with Kibana before starting saved objects migrations..."} {"type":"log","@timestamp":"2021-06-20T13:08:56Z","tags":["info","savedobjects-service"],"pid":1626,"message":"Starting saved objects migrations"} {"type":"log","@timestamp":"2021-06-20T13:09:06Z","tags":["error","elasticsearch","data"],"pid":1626,"message":"[ConnectionError]: getaddrinfo ENOTFOUND cr-dr-performance-us-420-client3.dr.testing.net cr-dr-performance-us-420-client3.dr.testing.net:9200"} {"type":"log","@timestamp":"2021-06-20T13:09:06Z","tags":["warning","savedobjects-service"],"pid":1626,"message":"Unable to connect to Elasticsearch. Error: getaddrinfo ENOTFOUND cr-dr-performance-us-420-client3.dr.testing.net cr-dr-performance-us-420-client3.dr.testing.net:9200"} {"type":"log","@timestamp":"2021-06-20T13:09:11Z","tags":["error","elasticsearch","data"],"pid":1626,"message":"[ConnectionError]: getaddrinfo ENOTFOUND cr-dr-performance-us-420-client3.dr.testing.net cr-dr-performance-us-420-client3.dr.testing.net:9200"} {"type":"log","@timestamp":"2021-06-20T13:09:11Z","tags":["warning","savedobjects-service"],"pid":1626,"message":"Unable to connect to Elasticsearch. Error: getaddrinfo ENOTFOUND cr-dr-performance-us-420-client3.dr.testing.net cr-dr-performance-us-420-client3.dr.testing.net:9200"} {"type":"log","@timestamp":"2021-06-20T13:09:14Z","tags":["info","plugins-system"],"pid":1626,"message":"Starting [96] plugins: [taskManager,licensing,globalSearch,globalSearchProviders,code,usageCollection,xpackLegacy,telemetryCollectionManager,telemetry,telemetryCollectionXpack,kibanaUsageCollection,securityOss,newsfeed,mapsLegacy,kibanaLegacy,translations,share,legacyExport,embeddable,uiActionsEnhanced,expressions,data,home,observability,cloud,console,consoleExtensions,apmOss,searchprofiler,painlessLab,grokdebugger,management,indexPatternManagement,advancedSettings,fileUpload,savedObjects,dashboard,visualizations,visTypeVega,visTypeTimelion,timelion,features,upgradeAssistant,security,snapshotRestore,enterpriseSearch,encryptedSavedObjects,ingestManager,indexManagement,remoteClusters,crossClusterReplication,indexLifecycleManagement,dashboardMode,beatsManagement,transform,ingestPipelines,maps,licenseManagement,graph,dataEnhanced,visTypeTable,visTypeMarkdown,tileMap,regionMap,inputControlVis,visualize,esUiShared,bfetch,canvas,charts,lens,visTypeVislib,visTypeTimeseries,rollup,visTypeTagcloud,visTypeMetric,watcher,discover,discoverEnhanced,savedObjectsManagement,spaces,reporting,lists,eventLog,actions,case,alerts,stackAlerts,triggersActionsUi,ml,securitySolution,infra,monitoring,logstash,apm,uptime]"} {"type":"log","@timestamp":"2021-06-20T13:09:14Z","tags":["info","plugins","taskManager","taskManager"],"pid":1626,"message":"TaskManager is identified by the Kibana UUID: dc8dd0b0-fe0c-4c12-b13a-b455cbcecdf0"} @ {"type":"log","@timestamp":"2021-06-20T13:09:24Z","tags":["error","elasticsearch","data"],"pid":1626,"message":"[version_conflict_engine_exception]: [task:Alerting-alerting_telemetry]: version conflict, document already exists (current version [25])"} {"type":"response","@timestamp":"2021-06-20T13:09:24Z","tags":["api"],"pid":1626,"method":"get","statusCode":200,"req":{"url":"/bootstrap.js","method":"get","headers":{"host":"10.240.96.54:5601","user-agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.77 Safari/537.36","accept":"/","accept-encoding":"gzip, deflate, br","accept-language":"en-US,en;q=0.9,he;q=0.8","if-none-match":"\"67cbb1336c012d765ee0288bb94cb4e35c24c888-gzip\"","referer":"http://10.240.96.54:5601/login?next=%2Fapp%2Fkibana","sec-ch-ua":"\" Not;A Brand\";v=\"99\", \"Google Chrome\";v=\"91\", \"Chromium\";v=\"91\"","sec-ch-ua-mobile":"?0","sec-fetch-dest":"script","sec-fetch-mode":"no-cors","sec-fetch-site":"same-origin","x-forwarded-for":"5.22.128.254","x-forwarded-host":"cr-dr-performance-us_kibana-2.testing.luminatesec.com","x-forwarded-proto":"HTTPS","connection":"close"},"remoteAddress":"10.239.2.10","userAgent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.77 Safari/537.36","referer":"http://10.240.96.54:5601/login?next=%2Fapp%2Fkibana"},"res":{"statusCode":200,"responseTime":20,"contentLength":9},"message":"GET /bootstrap.js 200 20ms - 9.0B"} {"type":"response","@timestamp":"2021-06-20T13:09:25Z","tags":[],"pid":1626,"method":"get","statusCode":200,"req":{"url":"/ui/favicons/favicon.ico","method":"get","headers":{"host":"10.240.96.54:5601","user-agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.77 Safari/537.36","accept":"image/avif,image/webp,image/apng,image/svg+xml,image/,/;q=0.8","accept-encoding":"gzip, deflate, br","accept-language":"en-US,en;q=0.9,he;q=0.8","referer":"http://10.240.96.54:5601/login?next=%2Fapp%2Fkibana","sec-ch-ua":"\" Not;A Brand\";v=\"99\", \"Google Chrome\";v=\"91\", \"Chromium\";v=\"91\"","sec-ch-ua-mobile":"?0","sec-fetch-dest":"image","sec-fetch-mode":"no-cors","sec-fetch-site":"same-origin","x-forwarded-for":"5.22.128.254","x-forwarded-host":"cr-dr-performance-us_kibana-2.testing.luminatesec.com","x-forwarded-proto":"HTTPS","connection":"close"},"remoteAddress":"10.239.2.10","userAgent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.77 Safari/537.36","referer":"http://10.240.96.54:5601/login?next=%2Fapp%2Fkibana"},"res":{"statusCode":200,"responseTime":7,"contentLength":9},"message":"GET /ui/favicons/favicon.ico 200 7ms - 9.0B"} {"type":"response","@timestamp":"2021-06-20T13:09:25Z","tags":[],"pid":1626,"method":"get","statusCode":200,"req":{"url":"/node_modules/@kbn/ui-framework/dist/kui_light.css","method":"get","headers":{"host":"10.240.96.54:5601","user-agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.77 Safari/537.36","accept":"text/css,/;q=0.1","accept-encoding":"gzip, deflate, br","accept-language":"en-US,en;q=0.9,he;q=0.8","referer":"http://10.240.96.54:5601/login?next=%2Fapp%2Fkibana","sec-ch-ua":"\" Not;A Brand\";v=\"99\", \"Google Chrome\";v=\"91\", \"Chromium\";v=\"91\"","sec-ch-ua-mobile":"?0","sec-fetch-dest":"style","sec-fetch-mode":"no-cors","sec-fetch-site":"same-origin","x-forwarded-for":"5.22.128.254","x-forwarded-host":"cr-dr-performance-us_kibana-2.testing.luminatesec.com","x-forwarded-proto":"HTTPS","connection":"close"},"remoteAddress":"10.239.2.10","userAgent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.77 Safari/537.36","referer":"http://10.240.96.54:5601/login?next=%2Fapp%2Fkibana"},"res":{"statusCode":200,"responseTime":5,"contentLength":9},"message":"GET /node_modules/@kbn/ui-framework/dist/kui_light.css 200 5ms - 9.0B"} {"type":"response","@timestamp":"2021-06-20T13:09:26Z","tags":[],"pid":1626,"method":"get","statusCode":200,"req":{"url":"/ui/legacy_light_theme.css","method":"get","headers":{"host":"10.240.96.54:5601","user-agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.77 Safari/537.36","accept":"text/css,/;q=0.1","accept-encoding":"gzip, deflate, br","accept-language":"en-US,en;q=0.9,he;q=0.8","referer":"http://10.240.96.54:5601/login?next=%2Fapp%2Fkibana","sec-ch-ua":"\" Not;A Brand\";v=\"99\", \"Google Chrome\";v=\"91\", \"Chromium\";v=\"91\"","sec-ch-ua-mobile":"?0","sec-fetch-dest":"style","sec-fetch-mode":"no-cors","sec-fetch-site":"same-origin","x-forwarded-for":"5.22.128.254","x-forwarded-host":"cr-dr-performance-us_kibana-2.testing.luminatesec.com","x-forwarded-proto":"HTTPS","connection":"close"},"remoteAddress":"10.239.2.10","userAgent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.77 Safari/537.36","referer":"http://10.240.96.54:5601/login?next=%2Fapp%2Fkibana"},"res":{"statusCode":200,"responseTime":5,"contentLength":9},"message":"GET /ui/legacy_light_theme.css 200 5ms - 9.0B"} {"type":"response","@timestamp":"2021-06-20T13:09:26Z","tags":[],"pid":1626,"method":"get","statusCode":304,"req":{"url":"/translations/en.json","method":"get","headers":{"host":"10.240.96.54:5601","user-agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.77 Safari/537.36","accept":"/","accept-encoding":"gzip, deflate, br","accept-language":"en-US,en;q=0.9,he;q=0.8","if-none-match":"\"37992637719f97813c3068cfbf877b2d3bb43b97\"","referer":"http://10.240.96.54:5601/login?next=%2Fapp%2Fkibana","sec-ch-ua":"\" Not;A Brand\";v=\"99\", \"Google Chrome\";v=\"91\", \"Chromium\";v=\"91\"","sec-ch-ua-mobile":"?0","sec-fetch-dest":"empty","sec-fetch-mode":"cors","sec-fetch-site":"same-origin","x-forwarded-for":"5.22.128.254","x-forwarded-host":"cr-dr-performance-us_kibana-2.testing.luminatesec.com","x-forwarded-proto":"HTTPS","connection":"close"},"remoteAddress":"10.239.2.10","userAgent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.77 Safari/537.36","referer":"http://10.240.96.54:5601/login?next=%2Fapp%2Fkibana"},"res":{"statusCode":304,"responseTime":2,"contentLength":9},"message":"GET /translations/en.json 304 2ms - 9.0B"} {"type":"response","@timestamp":"2021-06-20T13:09:26Z","tags":[],"pid":1626,"method":"get","statusCode":200,"req":{"url":"/ui/fonts/inter_ui/Inter-UI-Regular.woff2","method":"get","headers":{"host":"10.240.96.54:5601","user-agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.77 Safari/537.36","accept":"/*","accept-encoding":"gzip, deflate, br","accept-language":"en-US,en;q=0.9,he;q=0.8","origin":"http://10.240.96.54:5601","referer":"http://10.240.96.54:5601/login?next=%2Fapp%2Fkibana","sec-ch-ua":"\" Not;A Brand\";v=\"99\", \"Google Chrome\";v=\"91\", \"Chromium\";v=\"91\"","sec-ch-ua-mobile":"?0","sec-fetch-dest":"font","sec-fetch-mode":"cors","sec-fetch-site":"same-origin","x-forwarded-for":"5.22.128.254","x-forwarded-host":"cr-dr-performance-us_kibana-2.testing.luminatesec.com","x-forwarded-proto":"HTTPS","connection":"close"},"remoteAddress":"10.239.2.10","userAgent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.77 Safari/537.36","referer":"http://10.240.96.54:5601/login?next=%2Fapp%2Fkibana"},"res":{"statusCode":200,"responseTime":5,"contentLength":9},"message":"GET /ui/fonts/inter_ui/Inter-UI-Regular.woff2 200 5ms - 9.0B"} {"type":"log","@timestamp":"2021-06-20T13:09:27Z","tags":["error","elasticsearch","data"],"pid":1626,"message":"[version_conflict_engine_exception]: [task:endpoint:user-artifact-packager:1.0.0]: version conflict, document already exists (current version [30325])"} {"type":"log","@timestamp":"2021-06-20T13:09:28Z","tags":["error","elasticsearch","data"],"pid":1626,"message":"[version_conflict_engine_exception]: [task:apm-telemetry-task]: version conflict, document already exists (current version [46])"} {"type":"log","@timestamp":"2021-06-20T13:09:28Z","tags":["error","elasticsearch","security"],"pid":1626,"message":"Request error, retrying\nHEAD https://cr-dr-performance-us-420-client3.dr.testing.net:9200/_template/.kibana_security_session_index_template_1 => getaddrinfo ENOTFOUND cr-dr-performance-us-420-client3.dr.testing.net cr-dr-performance-us-420-client3.dr.testing.net:9200"} {"type":"response","@timestamp":"2021-06-20T13:09:36Z","tags":[],"pid":1626,"method":"head","statusCode":302,"req":{"url":"/","method":"head","headers":{"host":"10.240.96.54:5601","user-agent":"ApplicationHealthService/1.0","connection":"close"},"remoteAddress":"10.239.2.10","userAgent":"ApplicationHealthService/1.0"},"res":{"statusCode":302,"responseTime":1,"contentLength":9},"message":"HEAD / 302 1ms - 9.0B"} {"type":"response","@timestamp":"2021-06-20T13:09:36Z","tags":[],"pid":1626,"method":"head","statusCode":302,"req":{"url":"/","method":"head","headers":{"host":"10.240.96.54:5601","user-agent":"ApplicationHealthService/1.0","connection":"close"},"remoteAddress":"10.239.2.10","userAgent":"ApplicationHealthService/1.0"},"res":{"statusCode":302,"responseTime":1,"contentLength":9},"message":"HEAD / 302 1ms - 9.0B"} {"type":"response","@timestamp":"2021-06-20T13:09:36Z","tags":[],"pid":1626,"method":"head","statusCode":302,"req":{"url":"/","method":"head","headers":{"host":"10.240.96.54:5601","user-agent":"ApplicationHealthService/1.0","connection":"close"},"remoteAddress":"10.239.2.10","userAgent":"ApplicationHealthService/1.0"},"res":{"statusCode":302,"responseTime":1,"contentLength":9},"message":"HEAD / 302 1ms - 9.0B"} {"type":"response","@timestamp":"2021-06-20T13:09:36Z","tags":[],"pid":1626,"method":"head","statusCode":302,"req":{"url":"/","method":"head","headers":{"host":"10.240.96.54:5601","user-agent":"ApplicationHealthService/1.0","connection":"close"},"remoteAddress":"10.239.2.10","userAgent":"ApplicationHealthService/1.0"},"res":{"statusCode":302,"responseTime":1,"contentLength":9},"message":"HEAD / 302 1ms - 9.0B"} {"type":"response","@timestamp":"2021-06-20T13:09:36Z","tags":[],"pid":1626,"method":"head","statusCode":302,"req":{"url":"/","method":"head","headers":{"host":"10.240.96.54:5601","user-agent":"ApplicationHealthService/1.0","connection":"close"},"remoteAddress":"10.239.2.10","userAgent":"ApplicationHealthService/1.0"},"res":{"statusCode":302,"responseTime":1,"contentLength":9},"message":"HEAD / 302 1ms - 9.0B"} {"type":"response","@timestamp":"2021-06-20T13:09:36Z","tags":[],"pid":1626,"method":"head","statusCode":302,"req":{"url":"/","method":"head","headers":{"host":"10.240.96.54:5601","user-agent":"ApplicationHealthService/1.0","connection":"close"},"remoteAddress":"10.239.2.10","userAgent":"ApplicationHealthService/1.0"},"res":{"statusCode":302,"responseTime":1,"contentLength":9},"message":"HEAD / 302 1ms - 9.0B"} {"type":"response","@timestamp":"2021-06-20T13:09:36Z","tags":[],"pid":1626,"method":"head","statusCode":302,"req":{"url":"/","method":"head","headers":{"host":"10.240.96.54:5601","user-agent":"ApplicationHealthService/1.0","connection":"close"},"remoteAddress":"10.239.2.10","userAgent":"ApplicationHealthService/1.0"},"res":{"statusCode":302,"responseTime":1,"contentLength":9},"message":"HEAD / 302 1ms - 9.0B"} {"type":"response","@timestamp":"2021-06-20T13:09:36Z","tags":[],"pid":1626,"method":"head","statusCode":302,"req":{"url":"/","method":"head","headers":{"host":"10.240.96.54:5601","user-agent":"ApplicationHealthService/1.0","connection":"close"},"remoteAddress":"10.239.2.10","userAgent":"ApplicationHealthService/1.0"},"res":{"statusCode":302,"responseTime":1,"contentLength":9},"message":"HEAD / 302 1ms - 9.0B"} {"type":"response","@timestamp":"2021-06-20T13:09:36Z","tags":[],"pid":1626,"method":"head","statusCode":302,"req":{"url":"/","method":"head","headers":{"host":"10.240.96.54:5601","user-agent":"ApplicationHealthService/1.0","connection":"close"},"remoteAddress":"10.239.2.10","userAgent":"ApplicationHealthService/1.0"},"res":{"statusCode":302,"responseTime":0,"contentLength":9},"message":"HEAD / 302 0ms - 9.0B"} {"type":"response","@timestamp":"2021-06-20T13:09:36Z","tags":[],"pid":1626,"method":"head","statusCode":302,"req":{"url":"/","method":"head","headers":{"host":"10.240.96.54:5601","user-agent":"ApplicationHealthService/1.0","connection":"close"},"remoteAddress":"10.239.2.10","userAgent":"ApplicationHealthService/1.0"},"res":{"statusCode":302,"responseTime":1,"contentLength":9},"message":"HEAD / 302 1ms - 9.0B"} {"type":"response","@timestamp":"2021-06-20T13:09:36Z","tags":[],"pid":1626,"method":"head","statusCode":302,"req":{"url":"/","method":"head","headers":{"host":"10.240.96.54:5601","user-agent":"ApplicationHealthService/1.0","connection":"close"},"remoteAddress":"10.239.2.10","userAgent":"ApplicationHealthService/1.0"},"res":{"statusCode":302,"responseTime":1,"contentLength":9},"message":"HEAD / 302 1ms - 9.0B"} {"type":"response","@timestamp":"2021-06-20T13:09:36Z","tags":[],"pid":1626,"method":"head","statusCode":302,"req":{"url":"/","method":"head","headers":{"host":"10.240.96.54:5601","user-agent":"ApplicationHealthService/1.0","connection":"close"},"remoteAddress":"10.239.2.10","userAgent":"ApplicationHealthService/1.0"},"res":{"statusCode":302,"responseTime":1,"contentLength":9},"message":"HEAD / 302 1ms - 9.0B"} {"type":"response","@timestamp":"2021-06-20T13:09:36Z","tags":[],"pid":1626,"method":"head","statusCode":302,"req":{"url":"/","method":"head","headers":{"host":"10.240.96.54:5601","user-agent":"ApplicationHealthService/1.0","connection":"close"},"remoteAddress":"10.239.2.10","userAgent":"ApplicationHealthService/1.0"},"res":{"statusCode":302,"responseTime":1,"contentLength":9},"message":"HEAD / 302 1ms - 9.0B"} {"type":"response","@timestamp":"2021-06-20T13:09:36Z","tags":[],"pid":1626,"method":"head","statusCode":302,"req":{"url":"/","method":"head","headers":{"host":"10.240.96.54:5601","user-agent":"ApplicationHealthService/1.0","connection":"close"},"remoteAddress":"10.239.2.10","userAgent":"ApplicationHealthService/1.0"},"res":{"statusCode":302,"responseTime":1,"contentLength":9},"message":"HEAD / 302 1ms - 9.0B"} {"type":"response","@timestamp":"2021-06-20T13:09:36Z","tags":[],"pid":1626,"method":"head","statusCode":302,"req":{"url":"/","method":"head","headers":{"host":"10.240.96.54:5601","user-agent":"ApplicationHealthService/1.0","connection":"close"},"remoteAddress":"10.239.2.10","userAgent":"ApplicationHealthService/1.0"},"res":{"statusCode":302,"responseTime":2,"contentLength":9},"message":"HEAD / 302 2ms - 9.0B"} {"type":"log","@timestamp":"2021-06-20T13:09:42Z","tags":["error","elasticsearch","security"],"pid":1626,"message":"Request error, retrying\nPOST https://cr-dr-performance-us-420-client3.dr.testing.net:9200/_security/user/_has_privileges => getaddrinfo ENOTFOUND cr-dr-performance-us-420-client3.dr.testing.net cr-dr-performance-us-420-client3.dr.testing.net:9200"}

Any additional context:

elasticmachine commented 3 years ago

Pinging @elastic/kibana-core (Team:Core)

mshustov commented 3 years ago

I think it's https://github.com/elastic/elasticsearch-js/issues/1426 Would you mind trying out a recommendation from https://github.com/elastic/elasticsearch-js/issues/1426#issuecomment-809470034 ?

pgayvallet commented 2 years ago

From https://github.com/elastic/elasticsearch-js/issues/1426#issuecomment-809470034, this is the expected behavior from the ES client. The favored solution is to enable sniffing to have the client remove the missing nodes from the cluster.

Closing.