search

elastic / cloud-on-k8s

Elastic Cloud on Kubernetes
Other
55 stars 707 forks source link

Kibana fails to start after upgrading from 7.2.1 to 7.5.0 #2352

Closed sebgl closed 4 years ago

sebgl commented 4 years ago

I got this problem once while running ECK 1.0.0-rc4 and was not able to reproduce it.

  1. Deploy ES and Kibana in version 6.8.5, then 7.1.1, then 7.2.1 (not sure if the early upgrades matter)
  2. Upgrade both to version 7.5.0

Kibana fails to start:

bash-4.2$ curl https://localhost:5601 -k                                                                                                                                                                          
Kibana server is not ready yetbash-4.2$

Logs: 

⟩ k logs -f  kb-apm-sample-kb-68f5d765d4-mf7cs -n kibana-namespace
{"type":"log","@timestamp":"2020-01-06T13:10:25Z","tags":["info","plugins-system"],"pid":6,"message":"Setting up [15] plugins: [licensing,security,code,timelion,features,spaces,translations,uiActions,newsfeed,i
nspector,embeddable,advancedUiActions,expressions,eui_utils,data]"}
{"type":"log","@timestamp":"2020-01-06T13:10:25Z","tags":["info","plugins","licensing"],"pid":6,"message":"Setting up plugin"}
{"type":"log","@timestamp":"2020-01-06T13:10:25Z","tags":["info","plugins","security"],"pid":6,"message":"Setting up plugin"}
{"type":"log","@timestamp":"2020-01-06T13:10:25Z","tags":["warning","plugins","security","config"],"pid":6,"message":"Generating a random key for xpack.security.encryptionKey. To prevent sessions from being inv
alidated on restart, please set xpack.security.encryptionKey in kibana.yml"}
{"type":"log","@timestamp":"2020-01-06T13:10:25Z","tags":["info","plugins","code"],"pid":6,"message":"Setting up plugin"}
{"type":"log","@timestamp":"2020-01-06T13:10:25Z","tags":["info","plugins","timelion"],"pid":6,"message":"Setting up plugin"}
{"type":"log","@timestamp":"2020-01-06T13:10:25Z","tags":["info","plugins","features"],"pid":6,"message":"Setting up plugin"}
{"type":"log","@timestamp":"2020-01-06T13:10:25Z","tags":["info","plugins","spaces"],"pid":6,"message":"Setting up plugin"}
{"type":"log","@timestamp":"2020-01-06T13:10:25Z","tags":["info","plugins","translations"],"pid":6,"message":"Setting up plugin"}
{"type":"log","@timestamp":"2020-01-06T13:10:25Z","tags":["info","plugins","data"],"pid":6,"message":"Setting up plugin"}
{"type":"log","@timestamp":"2020-01-06T13:10:45Z","tags":["warning","legacy-plugins"],"pid":6,"path":"/usr/share/kibana/src/legacy/core_plugins/visualizations","message":"Skipping non-plugin directory at /usr/s
hare/kibana/src/legacy/core_plugins/visualizations"}
{"type":"log","@timestamp":"2020-01-06T13:10:46Z","tags":["info","plugins-system"],"pid":6,"message":"Starting [8] plugins: [licensing,security,code,timelion,features,spaces,translations,data]"}
{"type":"log","@timestamp":"2020-01-06T13:10:46Z","tags":["info","plugins","licensing"],"pid":6,"message":"Imported changed license information from Elasticsearch for the [data] cluster: type: basic | status: a
ctive"}
{"type":"log","@timestamp":"2020-01-06T13:10:46Z","tags":["info","migrations"],"pid":6,"message":"Creating index .kibana_task_manager_2."}
{"type":"log","@timestamp":"2020-01-06T13:10:46Z","tags":["warning","migrations"],"pid":6,"message":"Unable to connect to Elasticsearch. Error: [resource_already_exists_exception] index [.kibana_task_manager_2/
jzQ9e04nRQeAlb3tRIjTDQ] already exists, with { index_uuid=\"jzQ9e04nRQeAlb3tRIjTDQ\" & index=\".kibana_task_manager_2\" }"}
{"type":"log","@timestamp":"2020-01-06T13:10:46Z","tags":["warning","migrations"],"pid":6,"message":"Another Kibana instance appears to be migrating the index. Waiting for that migration to complete. If no othe
r Kibana instance is attempting migrations, you can get past this message by deleting index .kibana_task_manager_2 and restarting Kibana."}

According to the last lines it looks like 2 Kibana instances were running at the same time? It is supposed not to happen since https://github.com/elastic/cloud-on-k8s/pull/2137.

sebgl commented 4 years ago

Looking at events on the cluster I can only see one Pod being created in version 7.5.0 (kb-apm-sample-kb-68f5d765d4-frttt). Also the creation of the 7.5.0 Pod (kb-apm-sample-kb-68f5d765d4-frttt - 2020-01-06T12:48:04Z) seems to happen after the deletion of the 7.2.1 Pod (kb-apm-sample-kb-59f588dd5b-6wd8f - 2020-01-06T12:48:01Z).

Dump of the events: events.txt events-json.txt

Note Pod kb-apm-sample-kb-68f5d765d4-mf7cs in those events was created following a manual deletion of Pod kb-apm-sample-kb-68f5d765d4-frttt.

sebgl commented 4 years ago

I actually managed to reproduce by going through the same 6.8.5 -> 7.1.1 -> 7.2.1 -> 7.5.0 path. I get the same error (Another Kibana instance appears to be migrating the index. Waiting for that migration to complete. If no othe r Kibana instance is attempting migrations, you can get past this message by deleting index .kibana_task_manager_2 and restarting Kibana) but also another one leading to Kibana crashing:

⟩ k logs -f kb-apm-sample-kb-74fd7d7bcf-tb4jp -n kibana-namespace                                                                                                                                                 
{"type":"log","@timestamp":"2020-01-06T13:47:42Z","tags":["info","plugins-system"],"pid":6,"message":"Setting up [15] plugins: [licensing,security,code,timelion,features,spaces,translations,uiActions,newsfeed,e
xpressions,inspector,embeddable,advancedUiActions,eui_utils,data]"}                                                                                                                                               
{"type":"log","@timestamp":"2020-01-06T13:47:42Z","tags":["info","plugins","licensing"],"pid":6,"message":"Setting up plugin"}                                                                                    
{"type":"log","@timestamp":"2020-01-06T13:47:42Z","tags":["info","plugins","security"],"pid":6,"message":"Setting up plugin"}                                                                                     
{"type":"log","@timestamp":"2020-01-06T13:47:42Z","tags":["warning","plugins","security","config"],"pid":6,"message":"Generating a random key for xpack.security.encryptionKey. To prevent sessions from being inv
alidated on restart, please set xpack.security.encryptionKey in kibana.yml"}                                                                                                                                      
{"type":"log","@timestamp":"2020-01-06T13:47:42Z","tags":["info","plugins","code"],"pid":6,"message":"Setting up plugin"}                                                                                         
{"type":"log","@timestamp":"2020-01-06T13:47:42Z","tags":["info","plugins","timelion"],"pid":6,"message":"Setting up plugin"}                                                                                     
{"type":"log","@timestamp":"2020-01-06T13:47:42Z","tags":["info","plugins","features"],"pid":6,"message":"Setting up plugin"}                                                                                     
{"type":"log","@timestamp":"2020-01-06T13:47:42Z","tags":["info","plugins","spaces"],"pid":6,"message":"Setting up plugin"}                                                                                       
{"type":"log","@timestamp":"2020-01-06T13:47:42Z","tags":["info","plugins","translations"],"pid":6,"message":"Setting up plugin"}                                                                                 
{"type":"log","@timestamp":"2020-01-06T13:47:42Z","tags":["info","plugins","data"],"pid":6,"message":"Setting up plugin"}                                                                                         
{"type":"log","@timestamp":"2020-01-06T13:48:03Z","tags":["warning","legacy-plugins"],"pid":6,"path":"/usr/share/kibana/src/legacy/core_plugins/visualizations","message":"Skipping non-plugin directory at /usr/s
hare/kibana/src/legacy/core_plugins/visualizations"}                                                                                                                                                              
{"type":"log","@timestamp":"2020-01-06T13:48:04Z","tags":["info","plugins-system"],"pid":6,"message":"Starting [8] plugins: [licensing,security,code,timelion,features,spaces,translations,data]"}                
{"type":"log","@timestamp":"2020-01-06T13:48:04Z","tags":["info","plugins","licensing"],"pid":6,"message":"Imported changed license information from Elasticsearch for the [data] cluster: type: basic | status: a
ctive"}                                                                                                                                                                                                           
{"type":"log","@timestamp":"2020-01-06T13:48:05Z","tags":["info","migrations"],"pid":6,"message":"Creating index .kibana_task_manager_2."}                                                                        
{"type":"log","@timestamp":"2020-01-06T13:48:05Z","tags":["warning","migrations"],"pid":6,"message":"Unable to connect to Elasticsearch. Error: [resource_already_exists_exception] index [.kibana_task_manager_2/
Q4VBuu4SQ_-Nv48qwyqSJQ] already exists, with { index_uuid=\"Q4VBuu4SQ_-Nv48qwyqSJQ\" & index=\".kibana_task_manager_2\" }"}                                                                                       
{"type":"log","@timestamp":"2020-01-06T13:48:05Z","tags":["warning","migrations"],"pid":6,"message":"Another Kibana instance appears to be migrating the index. Waiting for that migration to complete. If no othe
r Kibana instance is attempting migrations, you can get past this message by deleting index .kibana_task_manager_2 and restarting Kibana."}                                                                       
{"type":"log","@timestamp":"2020-01-06T13:48:05Z","tags":["info","migrations"],"pid":6,"message":"Detected mapping change in \"properties.space\""}                                                               
{"type":"log","@timestamp":"2020-01-06T13:48:05Z","tags":["info","migrations"],"pid":6,"message":"Creating index .kibana_4."}                                                                                     
{"type":"log","@timestamp":"2020-01-06T13:48:05Z","tags":["warning","migrations"],"pid":6,"message":"Unable to connect to Elasticsearch. Error: [mapper_parsing_exception] No handler for type [flattened] declare
d on field [state]"}                                                                                                                                                                                              
{"type":"log","@timestamp":"2020-01-06T13:48:05Z","tags":["fatal","root"],"pid":6,"message":"{ Error: [mapper_parsing_exception] No handler for type [flattened] declared on field [state]\n    at respond (/usr/s
hare/kibana/node_modules/elasticsearch/src/lib/transport.js:349:15)\n    at checkRespForFailure (/usr/share/kibana/node_modules/elasticsearch/src/lib/transport.js:306:7)\n    at HttpConnector.<anonymous> (/usr/
share/kibana/node_modules/elasticsearch/src/lib/connectors/http.js:173:7)\n    at IncomingMessage.wrapper (/usr/share/kibana/node_modules/elasticsearch/node_modules/lodash/lodash.js:4929:19)\n    at IncomingMes
sage.emit (events.js:194:15)\n    at endReadableNT (_stream_readable.js:1103:12)\n    at process._tickCallback (internal/process/next_tick.js:63:19)\n  status: 400,\n  displayName: 'BadRequest',\n  message:\n  
 '[mapper_parsing_exception] No handler for type [flattened] declared on field [state]',\n  path: '/.kibana_4',\n  query: {},\n  body:\n   { error:\n      { root_cause: [Array],\n        type: 'mapper_parsing_e
xception',\n        reason:\n         'Failed to parse mapping [_doc]: No handler for type [flattened] declared on field [state]',\n        caused_by: [Object] },\n     status: 400 },\n  statusCode: 400,\n  res
ponse:\n   '{\"error\":{\"root_cause\":[{\"type\":\"mapper_parsing_exception\",\"reason\":\"No handler for type [flattened] declared on field [state]\"}],\"type\":\"mapper_parsing_exception\",\"reason\":\"Faile
d to parse mapping [_doc]: No handler for type [flattened] declared on field [state]\",\"caused_by\":{\"type\":\"mapper_parsing_exception\",\"reason\":\"No handler for type [flattened] declared on field [state]
\"}},\"status\":400}',\n  toString: [Function],\n  toJSON: [Function] }"}                                                                                                                                         
{"type":"log","@timestamp":"2020-01-06T13:48:05Z","tags":["info","plugins-system"],"pid":6,"message":"Stopping all plugins."}  
{"type":"log","@timestamp":"2020-01-06T13:48:05Z","tags":["info","plugins","data"],"pid":6,"message":"Stopping plugin"}        
{"type":"log","@timestamp":"2020-01-06T13:48:05Z","tags":["info","plugins","translations"],"pid":6,"message":"Stopping plugin"}
{"type":"log","@timestamp":"2020-01-06T13:48:05Z","tags":["info","plugins","spaces"],"pid":6,"message":"Stopping plugin"}  
{"type":"log","@timestamp":"2020-01-06T13:48:05Z","tags":["info","plugins","features"],"pid":6,"message":"Stopping plugin"}
{"type":"log","@timestamp":"2020-01-06T13:48:05Z","tags":["info","plugins","timelion"],"pid":6,"message":"Stopping plugin"}

FATAL  [mapper_parsing_exception] No handler for type [flattened] declared on field [state] :: {"path":"/.kibana_4","query":{},"body":"{\"mappings\":{\"dynamic\":\"strict\",\"properties\":{\"apm-telemetry\":{\
"properties\":{\"has_any_services\":{\"type\":\"boolean\"},\"services_per_agent\":{\"properties\":{\"python\":{\"type\":\"long\",\"null_value\":0},\"java\":{\"type\":\"long\",\"null_value\":0},\"nodejs\":{\"typ
e\":\"long\",\"null_value\":0},\"js-base\":{\"type\":\"long\",\"null_value\":0},\"rum-js\":{\"type\":\"long\",\"null_value\":0},\"dotnet\":{\"type\":\"long\",\"null_value\":0},\"ruby\":{\"type\":\"long\",\"null
_value\":0},\"go\":{\"type\":\"long\",\"null_value\":0}}}}},\"canvas-element\":{\"dynamic\":false,\"properties\":{\"name\":{\"type\":\"text\",\"fields\":{\"keyword\":{\"type\":\"keyword\"}}},\"help\":{\"type\":
\"text\"},\"content\":{\"type\":\"text\"},\"image\":{\"type\":\"text\"},\"@timestamp\":{\"type\":\"date\"},\"@created\":{\"type\":\"date\"}}},\"canvas-workpad\":{\"dynamic\":false,\"properties\":{\"name\":{\"ty
pe\":\"text\",\"fields\":{\"keyword\":{\"type\":\"keyword\"}}},\"@timestamp\":{\"type\":\"date\"},\"@created\":{\"type\":\"date\"}}},\"config\":{\"dynamic\":\"true\",\"properties\":{\"buildNum\":{\"type\":\"key
word\"}}},\"dashboard\":{\"properties\":{\"description\":{\"type\":\"text\"},\"hits\":{\"type\":\"integer\"},\"kibanaSavedObjectMeta\":{\"properties\":{\"searchSourceJSON\":{\"type\":\"text\"}}},\"optionsJSON\"
:{\"type\":\"text\"},\"panelsJSON\":{\"type\":\"text\"},\"refreshInterval\":{\"properties\":{\"display\":{\"type\":\"keyword\"},\"pause\":{\"type\":\"boolean\"},\"section\":{\"type\":\"integer\"},\"value\":{\"t
ype\":\"integer\"}}},\"timeFrom\":{\"type\":\"keyword\"},\"timeRestore\":{\"type\":\"boolean\"},\"timeTo\":{\"type\":\"keyword\"},\"title\":{\"type\":\"text\"},\"version\":{\"type\":\"integer\"}}},\"graph-works
pace\":{\"properties\":{\"description\":{\"type\":\"text\"},\"kibanaSavedObjectMeta\":{\"properties\":{\"searchSourceJSON\":{\"type\":\"text\"}}},\"numLinks\":{\"type\":\"integer\"},\"numVertices\":{\"type\":\"
integer\"},\"title\":{\"type\":\"text\"},\"version\":{\"type\":\"integer\"},\"wsState\":{\"type\":\"text\"}}},\"index-pattern\":{\"properties\":{\"fieldFormatMap\":{\"type\":\"text\"},\"fields\":{\"type\":\"tex
t\"},\"intervalName\":{\"type\":\"keyword\"},\"notExpandable\":{\"type\":\"boolean\"},\"sourceFilters\":{\"type\":\"text\"},\"timeFieldName\":{\"type\":\"keyword\"},\"title\":{\"type\":\"text\"},\"type\":{\"typ
e\":\"keyword\"},\"typeMeta\":{\"type\":\"keyword\"}}},\"infrastructure-ui-source\":{\"properties\":{\"name\":{\"type\":\"text\"},\"description\":{\"type\":\"text\"},\"metricAlias\":{\"type\":\"keyword\"},\"log
Alias\":{\"type\":\"keyword\"},\"fields\":{\"properties\":{\"container\":{\"type\":\"keyword\"},\"host\":{\"type\":\"keyword\"},\"pod\":{\"type\":\"keyword\"},\"tiebreaker\":{\"type\":\"keyword\"},\"timestamp\"
:{\"type\":\"keyword\"}}},\"logColumns\":{\"type\":\"nested\",\"properties\":{\"timestampColumn\":{\"properties\":{\"id\":{\"type\":\"keyword\"}}},\"messageColumn\":{\"properties\":{\"id\":{\"type\":\"keyword\"
}}},\"fieldColumn\":{\"properties\":{\"id\":{\"type\":\"keyword\"},\"field\":{\"type\":\"keyword\"}}}}}}},\"kql-telemetry\":{\"properties\":{\"optInCount\":{\"type\":\"long\"},\"optOutCount\":{\"type\":\"long\"}}},\"map\":{\"properties\":{\"description\":{\"type\":\"text\"},\"title\":{\"type\":\"text\"},\"version\":{\"type\":\"integer\"},\"bounds\":{\"type\":\"geo_shape\"},\"mapStateJSON\":{\"type\":\"text\"},\"layer
ListJSON\":{\"type\":\"text\"},\"uiStateJSON\":{\"type\":\"text\"}}},\"maps-telemetry\":{\"properties\":{\"mapsTotalCount\":{\"type\":\"long\"},\"timeCaptured\":{\"type\":\"date\"},\"attributesPerMap\":{\"prope
rties\":{\"dataSourcesCount\":{\"properties\":{\"min\":{\"type\":\"long\"},\"max\":{\"type\":\"long\"},\"avg\":{\"type\":\"long\"}}},\"layersCount\":{\"properties\":{\"min\":{\"type\":\"long\"},\"max\":{\"type\
":\"long\"},\"avg\":{\"type\":\"long\"}}},\"layerTypesCount\":{\"dynamic\":\"true\",\"properties\":{}},\"emsVectorLayersCount\":{\"dynamic\":\"true\",\"properties\":{}}}}}},\"migrationVersion\":{\"dynamic\":\"t
rue\",\"type\":\"object\"},\"ml-telemetry\":{\"properties\":{\"file_data_visualizer\":{\"properties\":{\"index_creation_count\":{\"type\":\"long\"}}}}},\"namespace\":{\"type\":\"keyword\"},\"references\":{\"typ
e\":\"nested\",\"properties\":{\"name\":{\"type\":\"keyword\"},\"type\":{\"type\":\"keyword\"},\"id\":{\"type\":\"keyword\"}}},\"sample-data-telemetry\":{\"properties\":{\"installCount\":{\"type\":\"long\"},\"u
nInstallCount\":{\"type\":\"long\"}}},\"search\":{\"properties\":{\"columns\":{\"type\":\"keyword\"},\"description\":{\"type\":\"text\"},\"hits\":{\"type\":\"integer\"},\"kibanaSavedObjectMeta\":{\"properties\"
:{\"searchSourceJSON\":{\"type\":\"text\"}}},\"sort\":{\"type\":\"keyword\"},\"title\":{\"type\":\"text\"},\"version\":{\"type\":\"integer\"}}},\"server\":{\"properties\":{\"uuid\":{\"type\":\"keyword\"}}},\"si
em-ui-timeline\":{\"properties\":{\"columns\":{\"properties\":{\"aggregatable\":{\"type\":\"boolean\"},\"category\":{\"type\":\"keyword\"},\"columnHeaderType\":{\"type\":\"keyword\"},\"description\":{\"type\":\
"text\"},\"example\":{\"type\":\"text\"},\"indexes\":{\"type\":\"keyword\"},\"id\":{\"type\":\"keyword\"},\"name\":{\"type\":\"text\"},\"placeholder\":{\"type\":\"text\"},\"searchable\":{\"type\":\"boolean\"},\
"type\":{\"type\":\"keyword\"}}},\"dataProviders\":{\"properties\":{\"id\":{\"type\":\"keyword\"},\"name\":{\"type\":\"text\"},\"enabled\":{\"type\":\"boolean\"},\"excluded\":{\"type\":\"boolean\"},\"kqlQuery\"
:{\"type\":\"text\"},\"queryMatch\":{\"properties\":{\"field\":{\"type\":\"text\"},\"displayField\":{\"type\":\"text\"},\"value\":{\"type\":\"text\"},\"displayValue\":{\"type\":\"text\"},\"operator\":{\"type\":
\"text\"}}},\"and\":{\"properties\":{\"id\":{\"type\":\"keyword\"},\"name\":{\"type\":\"text\"},\"enabled\":{\"type\":\"boolean\"},\"excluded\":{\"type\":\"boolean\"},\"kqlQuery\":{\"type\":\"text\"},\"queryMat
ch\":{\"properties\":{\"field\":{\"type\":\"text\"},\"displayField\":{\"type\":\"text\"},\"value\":{\"type\":\"text\"},\"displayValue\":{\"type\":\"text\"},\"operator\":{\"type\":\"text\"}}}}}}},\"description\"
:{\"type\":\"text\"},\"favorite\":{\"properties\":{\"keySearch\":{\"type\":\"text\"},\"fullName\":{\"type\":\"text\"},\"userName\":{\"type\":\"text\"},\"favoriteDate\":{\"type\":\"date\"}}},\"kqlMode\":{\"type\
":\"keyword\"},\"kqlQuery\":{\"properties\":{\"filterQuery\":{\"properties\":{\"kuery\":{\"properties\":{\"kind\":{\"type\":\"keyword\"},\"expression\":{\"type\":\"text\"}}},\"serializedQuery\":{\"type\":\"text
\"}}}}},\"title\":{\"type\":\"text\"},\"dateRange\":{\"properties\":{\"start\":{\"type\":\"date\"},\"end\":{\"type\":\"date\"}}},\"sort\":{\"properties\":{\"columnId\":{\"type\":\"keyword\"},\"sortDirection\":{
\"type\":\"keyword\"}}},\"created\":{\"type\":\"date\"},\"createdBy\":{\"type\":\"text\"},\"updated\":{\"type\":\"date\"},\"updatedBy\":{\"type\":\"text\"}}},\"siem-ui-timeline-note\":{\"properties\":{\"timelin
eId\":{\"type\":\"keyword\"},\"eventId\":{\"type\":\"keyword\"},\"note\":{\"type\":\"text\"},\"created\":{\"type\":\"date\"},\"createdBy\":{\"type\":\"text\"},\"updated\":{\"type\":\"date\"},\"updatedBy\":{\"ty
pe\":\"text\"}}},\"siem-ui-timeline-pinned-event\":{\"properties\":{\"timelineId\":{\"type\":\"keyword\"},\"eventId\":{\"type\":\"keyword\"},\"created\":{\"type\":\"date\"},\"createdBy\":{\"type\":\"text\"},\"u
pdated\":{\"type\":\"date\"},\"updatedBy\":{\"type\":\"text\"}}},\"space\":{\"properties\":{\"name\":{\"type\":\"text\",\"fields\":{\"keyword\":{\"type\":\"keyword\",\"ignore_above\":2048}}},\"description\":{\"
type\":\"text\"},\"initials\":{\"type\":\"keyword\"},\"color\":{\"type\":\"keyword\"},\"disabledFeatures\":{\"type\":\"keyword\"},\"imageUrl\":{\"type\":\"text\",\"index\":false},\"_reserved\":{\"type\":\"boole
an\"}}},\"telemetry\":{\"properties\":{\"enabled\":{\"type\":\"boolean\"},\"sendUsageFrom\":{\"ignore_above\":256,\"type\":\"keyword\"},\"lastReported\":{\"type\":\"date\"},\"lastVersionChecked\":{\"ignore_abov
e\":256,\"type\":\"keyword\"},\"userHasSeenNotice\":{\"type\":\"boolean\"}}},\"timelion-sheet\":{\"properties\":{\"description\":{\"type\":\"text\"},\"hits\":{\"type\":\"integer\"},\"kibanaSavedObjectMeta\":{\"
properties\":{\"searchSourceJSON\":{\"type\":\"text\"}}},\"timelion_chart_height\":{\"type\":\"integer\"},\"timelion_columns\":{\"type\":\"integer\"},\"timelion_interval\":{\"type\":\"keyword\"},\"timelion_othe
r_interval\":{\"type\":\"keyword\"},\"timelion_rows\":{\"type\":\"integer\"},\"timelion_sheet\":{\"type\":\"text\"},\"title\":{\"type\":\"text\"},\"version\":{\"type\":\"integer\"}}},\"type\":{\"type\":\"keywor
d\"},\"ui-metric\":{\"properties\":{\"count\":{\"type\":\"integer\"}}},\"updated_at\":{\"type\":\"date\"},\"upgrade-assistant-reindex-operation\":{\"dynamic\":true,\"properties\":{\"indexName\":{\"type\":\"keyw
ord\"},\"status\":{\"type\":\"integer\"}}},\"upgrade-assistant-telemetry\":{\"properties\":{\"ui_open\":{\"properties\":{\"overview\":{\"type\":\"long\",\"null_value\":0},\"cluster\":{\"type\":\"long\",\"null_v
alue\":0},\"indices\":{\"type\":\"long\",\"null_value\":0}}},\"ui_reindex\":{\"properties\":{\"close\":{\"type\":\"long\",\"null_value\":0},\"open\":{\"type\":\"long\",\"null_value\":0},\"start\":{\"type\":\"lo
ng\",\"null_value\":0},\"stop\":{\"type\":\"long\",\"null_value\":0}}},\"features\":{\"properties\":{\"deprecation_logging\":{\"properties\":{\"enabled\":{\"type\":\"boolean\",\"null_value\":true}}}}}}},\"url\"
:{\"properties\":{\"accessCount\":{\"type\":\"long\"},\"accessDate\":{\"type\":\"date\"},\"createDate\":{\"type\":\"date\"},\"url\":{\"type\":\"text\",\"fields\":{\"keyword\":{\"type\":\"keyword\",\"ignore_abov
e\":2048}}}}},\"visualization\":{\"properties\":{\"description\":{\"type\":\"text\"},\"kibanaSavedObjectMeta\":{\"properties\":{\"searchSourceJSON\":{\"type\":\"text\"}}},\"savedSearchRefName\":{\"type\":\"keyw
ord\"},\"title\":{\"type\":\"text\"},\"uiStateJSON\":{\"type\":\"text\"},\"version\":{\"type\":\"integer\"},\"visState\":{\"type\":\"text\"}}},\"metrics-explorer-view\":{\"properties\":{\"name\":{\"type\":\"key
word\"},\"options\":{\"properties\":{\"metrics\":{\"type\":\"nested\",\"properties\":{\"aggregation\":{\"type\":\"keyword\"},\"field\":{\"type\":\"keyword\"},\"color\":{\"type\":\"keyword\"},\"label\":{\"type\"
:\"keyword\"}}},\"limit\":{\"type\":\"integer\"},\"groupBy\":{\"type\":\"keyword\"},\"filterQuery\":{\"type\":\"keyword\"},\"aggregation\":{\"type\":\"keyword\"}}},\"chartOptions\":{\"properties\":{\"type\":{\"
type\":\"keyword\"},\"yAxisMode\":{\"type\":\"keyword\"},\"stack\":{\"type\":\"boolean\"}}},\"currentTimerange\":{\"properties\":{\"from\":{\"type\":\"keyword\"},\"to\":{\"type\":\"keyword\"},\"interval\":{\"ty
pe\":\"keyword\"}}}}},\"inventory-view\":{\"properties\":{\"name\":{\"type\":\"keyword\"},\"metric\":{\"properties\":{\"type\":{\"type\":\"keyword\"}}},\"groupBy\":{\"type\":\"nested\",\"properties\":{\"label\"
:{\"type\":\"keyword\"},\"field\":{\"type\":\"keyword\"}}},\"nodeType\":{\"type\":\"keyword\"},\"view\":{\"type\":\"keyword\"},\"customOptions\":{\"type\":\"nested\",\"properties\":{\"text\":{\"type\":\"keyword
\"},\"field\":{\"type\":\"keyword\"}}},\"boundsOverride\":{\"properties\":{\"max\":{\"type\":\"integer\"},\"min\":{\"type\":\"integer\"}}},\"autoBounds\":{\"type\":\"boolean\"},\"time\":{\"type\":\"integer\"},\
"autoReload\":{\"type\":\"boolean\"},\"filterQuery\":{\"properties\":{\"kind\":{\"type\":\"keyword\"},\"expression\":{\"type\":\"keyword\"}}}}},\"file-upload-telemetry\":{\"properties\":{\"filesUploadedTotalCou
nt\":{\"type\":\"long\"}}},\"lens\":{\"properties\":{\"title\":{\"type\":\"text\"},\"visualizationType\":{\"type\":\"keyword\"},\"state\":{\"type\":\"flattened\"},\"expression\":{\"index\":false,\"type\":\"keyw
ord\"}}},\"lens-ui-telemetry\":{\"properties\":{\"name\":{\"type\":\"keyword\"},\"type\":{\"type\":\"keyword\"},\"date\":{\"type\":\"date\"},\"count\":{\"type\":\"integer\"}}},\"query\":{\"properties\":{\"title
\":{\"type\":\"text\"},\"description\":{\"type\":\"text\"},\"query\":{\"properties\":{\"language\":{\"type\":\"keyword\"},\"query\":{\"type\":\"keyword\",\"index\":false}}},\"filters\":{\"type\":\"object\",\"en
abled\":false},\"timefilter\":{\"type\":\"object\",\"enabled\":false}}}},\"_meta\":{\"migrationMappingPropertyHashes\":{\"config\":\"87aca8fdb053154f11383fce3dbf3edf\",\"migrationVersion\":\"4a1746014a75ade3a71
4e1db5763276f\",\"type\":\"2f4316de49999235636386fe51dc06c1\",\"namespace\":\"2f4316de49999235636386fe51dc06c1\",\"updated_at\":\"00da57df13e94e9d98437d13ace4bfe0\",\"references\":\"7997cf5a56cc02bdc9c93361bde7
32b0\",\"graph-workspace\":\"cd7ba1330e6682e9cc00b78850874be1\",\"space\":\"c5ca8acafa0beaa4d08d014a97b6bc6b\",\"ml-telemetry\":\"257fd1d4b4fdbb9cb4b8a3b27da201e9\",\"apm-telemetry\":\"07ee1939fa4302c62ddc052ec
03fed90\",\"map\":\"23d7aa4a720d4938ccde3983f87bd58d\",\"maps-telemetry\":\"a4229f8b16a6820c6d724b7e0c1f729d\",\"canvas-workpad\":\"b0a1706d356228dbdcb4a17e6b9eb231\",\"canvas-element\":\"7390014e1091044523666d
97247392fc\",\"infrastructure-ui-source\":\"ddc0ecb18383f6b26101a2fadb2dab0c\",\"metrics-explorer-view\":\"53c5365793677328df0ccb6138bf3cdd\",\"inventory-view\":\"84b320fd67209906333ffce261128462\",\"siem-ui-ti
meline\":\"1f6f0860ad7bc0dba3e42467ca40470d\",\"siem-ui-timeline-note\":\"8874706eedc49059d4cf0f5094559084\",\"siem-ui-timeline-pinned-event\":\"20638091112f0e14f0e443d512301c29\",\"upgrade-assistant-reindex-op
eration\":\"a53a20fe086b72c9a86da3cc12dad8a6\",\"upgrade-assistant-telemetry\":\"56702cec857e0a9dacfb696655b4ff7b\",\"file-upload-telemetry\":\"0ed4d3e1983d1217a30982630897092e\",\"lens\":\"21c3ea0763beb1ecb016
2529706b88c5\",\"lens-ui-telemetry\":\"509bfa5978586998e05f9e303c07a327\",\"query\":\"11aaeb7f5f7fa5bb43f25e18ce26e7d9\",\"index-pattern\":\"66eccb05066c5a89924f48a9e9736499\",\"visualization\":\"52d7a13ad68a15
0c4525b292d23e12cc\",\"search\":\"181661168bbadd1eff5902361e2a0d5c\",\"dashboard\":\"d00f614b29a80360e1190193fd333bab\",\"url\":\"c7f66a0df8b1b52f17c28c4adb111105\",\"server\":\"ec97f1c5da1a19609a60874e5af1100c
\",\"kql-telemetry\":\"d12a98a6f19a2d273696597547e064ee\",\"sample-data-telemetry\":\"7d3cfeb915303c9641c59681967ffeb4\",\"telemetry\":\"358ffaa88ba34a97d55af0933a117de4\",\"timelion-sheet\":\"9a2a2748877c7a7b5
82fef201ab1d4cf\",\"ui-metric\":\"0d409297dc5ebe1e3a1da691c6ee32e3\"}}},\"settings\":{\"number_of_shards\":1,\"auto_expand_replicas\":\"0-1\"}}","statusCode":400,"response":"{\"error\":{\"root_cause\":[{\"type\
":\"mapper_parsing_exception\",\"reason\":\"No handler for type [flattened] declared on field [state]\"}],\"type\":\"mapper_parsing_exception\",\"reason\":\"Failed to parse mapping [_doc]: No handler for type [
flattened] declared on field [state]\",\"caused_by\":{\"type\":\"mapper_parsing_exception\",\"reason\":\"No handler for type [flattened] declared on field [state]\"}},\"status\":400}"}
barkbay commented 4 years ago

I got the same while doing some upgrading tests to 7.6.0 : 

{"type":"log","@timestamp":"2020-01-06T13:49:07Z","tags":["info","migrations"],"pid":8,"message":"Creating index .kibana_task_manager_1."}
{"type":"log","@timestamp":"2020-01-06T13:49:07Z","tags":["warning","migrations"],"pid":8,"message":"Unable to connect to Elasticsearch. Error: [resource_already_exists_exception] index [.kibana_task_manager_1/2Yld1pq_S06WdQZck6XBWQ] already exists, with { index_uuid=\"2Yld1pq_S06WdQZck6XBWQ\" & index=\".kibana_task_manager_1\" }"}
{"type":"log","@timestamp":"2020-01-06T13:49:07Z","tags":["warning","migrations"],"pid":8,"message":"Another Kibana instance appears to be migrating the index. Waiting for that migration to complete. If no other Kibana instance is attempting migrations, you can get past this message by deleting index .kibana_task_manager_1 and restarting Kibana."}
{"type":"log","@timestamp":"2020-01-06T13:49:07Z","tags":["info","migrations"],"pid":8,"message":"Detected mapping change in \"properties.space\""}
{"type":"log","@timestamp":"2020-01-06T13:49:07Z","tags":["info","migrations"],"pid":8,"message":"Creating index .kibana_2."}
{"type":"log","@timestamp":"2020-01-06T13:49:07Z","tags":["info","migrations"],"pid":8,"message":"Migrating .kibana_1 saved objects to .kibana_2"}
{"type":"log","@timestamp":"2020-01-06T13:49:08Z","tags":["info","migrations"],"pid":8,"message":"Pointing alias .kibana to .kibana_2."}
{"type":"log","@timestamp":"2020-01-06T13:49:08Z","tags":["info","migrations"],"pid":8,"message":"Finished in 703ms."}

I'm pretty sure that there was only 1 Kibana instance running at a given point in time because I made a typo in the image name of the previous deployment:

NAME                                    READY   STATUS             RESTARTS   AGE     IP           NODE                                             NOMINATED NODE
pod/kb-apm-sample-kb-5f9cbc6d69-f8rbc   0/1     Running            3          9m55s   10.28.2.14   gke-michael-dev-ssd-default-pool-642cd34d-1lgw   <none>
pod/kb-apm-sample-kb-8c9699f79-dbbwz    0/1     ImagePullBackOff   0          11m     10.28.0.14   gke-michael-dev-ssd-default-pool-6a186a41-n8f9   <none>
sebgl commented 4 years ago

Browsing through Kibana issues, it looks like this is the the problem: https://github.com/elastic/kibana/issues/52569

If Elasticsearch allocation is disabled when Kibana is upgraded, the upgrade will get stuck, requiring intervention, even after allocation has been re-enabled.

In our case, we disable shards allocation for Elasticsearch rolling upgrade.

Until it gets fixed, ensuring Kibana version upgrade happens after Elasticsearch version upgrade may help. But the problem also happens if Kibana version upgrade happens while any other Elasticsearch rolling upgrade is happening (version upgrade or not).

anyasabo commented 4 years ago

I tried spinning up a Kibana 7.2.0 instance connected to an ES 7.5.0 instance. I disabled replica allocation on ES then updated Kibana to 7.5.0 and it went through, so I think we may be okay during rolling upgrades?

DaveCTurner commented 4 years ago

The exception No handler for type [flattened] declared on field [state] suggests that the flattened mapper plugin is not installed. Was this a change from the default distribution to an OSS one, for instance? Could someone share with me the full logs from an upgrade that's failing in this way from all the nodes involved (both Elasticsearch and Kibana) for the whole upgrade process please?

DaveCTurner commented 4 years ago

Anya quite rightly points out that the flattened mapper plugin was only released in 7.3 so I think it's to be expected that a 7.5.0 Kibana would encounter this when started against Elasticsearch 7.2.1.

sebgl commented 4 years ago

I've spent some time trying to reproduce the issue. The only way I could reproduce it was by upgrading Elasticsearch and Kibana versions concurrently. Which, as stated above, is unsupported and may lead to a fatal error. Elasticsearch must be upgraded before Kibana.

I did not have any problem with regular Elasticsearch rolling upgrade (non version related).

I don't know if there's anything that could be improved so Kibana eventually recovers. Some details about my test scenarios:

  1. Successful upgrade case

Upgrading both Elasticsearch and Kibana from 6.8 -> 7.1.0 -> 7.2.0 -> 7.3.0 -> 7.4.0 -> 7.5.0: no problem. When Kibana starts in a new version, and until Elasticsearch is fully upgraded to that version, Kibana logs messages like the following for all plugins:


{"type":"log","@timestamp":"2020-01-24T10:28:43Z","tags":["status","plugin:elasticsearch@undefined
","info"],"pid":1,"state":"green","message":"Status changed from red to green - Ready","prevState"
:"red","prevMsg":"This version of Kibana requires Elasticsearch v7.1.0 on all nodes. I found the f
ollowing incompatible nodes in your cluster: v6.8.0 @ 10.28.1.19:9200 (10.28.1.19)"}

Once Elasticsearch is fully upgraded, Kibana works as expected. It does not crash.

  1. Unsuccessful upgrade case

Upgrading both Elasticsearch and Kibana from 7.1.0 to 7.5.0 leads to the current issue. When Kibana 7.5.0 first starts (and Elasticsearch 7.1 is still in the process of being upgraded), it logs the following:

{"type":"log","@timestamp":"2020-01-24T11:40:08Z","tags":["info","plugins-system"],"pid":6,"message":"Setting up [15] plugins: [security,licensing,code,timelion,features,spaces,translations,uiActions,eui_utils,newsfeed,inspector,embeddable,advancedUiActions,expressio
ns,data]"}                                                                                                                                                                                                                                                                 
{"type":"log","@timestamp":"2020-01-24T11:40:08Z","tags":["info","plugins","security"],"pid":6,"message":"Setting up plugin"}                                                                                                                                              
{"type":"log","@timestamp":"2020-01-24T11:40:08Z","tags":["info","plugins","licensing"],"pid":6,"message":"Setting up plugin"}                                                                                                                                             
{"type":"log","@timestamp":"2020-01-24T11:40:08Z","tags":["info","plugins","code"],"pid":6,"message":"Setting up plugin"}                                                                                                                                                  
{"type":"log","@timestamp":"2020-01-24T11:40:08Z","tags":["info","plugins","timelion"],"pid":6,"message":"Setting up plugin"}                                                                                                                                              
{"type":"log","@timestamp":"2020-01-24T11:40:08Z","tags":["info","plugins","features"],"pid":6,"message":"Setting up plugin"}                                                                                                                                              
{"type":"log","@timestamp":"2020-01-24T11:40:08Z","tags":["info","plugins","spaces"],"pid":6,"message":"Setting up plugin"}                                                                                                                                                
{"type":"log","@timestamp":"2020-01-24T11:40:08Z","tags":["info","plugins","translations"],"pid":6,"message":"Setting up plugin"}                                                                                                                                          
{"type":"log","@timestamp":"2020-01-24T11:40:08Z","tags":["info","plugins","data"],"pid":6,"message":"Setting up plugin"}                                                                                                                                                  
{"type":"log","@timestamp":"2020-01-24T11:40:37Z","tags":["warning","legacy-plugins"],"pid":6,"path":"/usr/share/kibana/src/legacy/core_plugins/visualizations","message":"Skipping non-plugin directory at /usr/share/kibana/src/legacy/core_plugins/visualizations"}     
{"type":"log","@timestamp":"2020-01-24T11:40:38Z","tags":["info","plugins-system"],"pid":6,"message":"Starting [8] plugins: [security,licensing,code,timelion,features,spaces,translations,data]"}                                                                         
{"type":"log","@timestamp":"2020-01-24T11:40:38Z","tags":["info","plugins","licensing"],"pid":6,"message":"Imported changed license information from Elasticsearch for the [data] cluster: type: basic | status: active"}                                                  
{"type":"log","@timestamp":"2020-01-24T11:40:38Z","tags":["info","migrations"],"pid":6,"message":"Creating index .kibana_task_manager_1."}                                                                                                                                 
{"type":"log","@timestamp":"2020-01-24T11:40:38Z","tags":["info","migrations"],"pid":6,"message":"Creating index .kibana_2."}                                                                                                                                              
{"type":"log","@timestamp":"2020-01-24T11:40:38Z","tags":["warning","migrations"],"pid":6,"message":"Unable to connect to Elasticsearch. Error: [mapper_parsing_exception] No handler for type [flattened] declared on field [state]"}                                     
{"type":"log","@timestamp":"2020-01-24T11:40:38Z","tags":["fatal","root"],"pid":6,"message":"{ Error: [mapper_parsing_exception] No handler for type [flattened] declared on field [state]\n    at respond (/usr/share/kibana/node_modules/elasticsearch/src/lib/transport.
js:349:15)\n    at checkRespForFailure (/usr/share/kibana/node_modules/elasticsearch/src/lib/transport.js:306:7)\n    at HttpConnector.<anonymous> (/usr/share/kibana/node_modules/elasticsearch/src/lib/connectors/http.js:173:7)\n    at IncomingMessage.wrapper (/usr/sh
are/kibana/node_modules/elasticsearch/node_modules/lodash/lodash.js:4929:19)\n    at IncomingMessage.emit (events.js:194:15)\n    at endReadableNT (_stream_readable.js:1103:12)\n    at process._tickCallback (internal/process/next_tick.js:63:19)\n  status: 400,\n  dis
playName: 'BadRequest',\n  message:\n   '[mapper_parsing_exception] No handler for type [flattened] declared on field [state]',\n  path: '/.kibana_2',\n  query: {},\n  body:\n   { error:\n      { root_cause: [Array],\n        type: 'mapper_parsing_exception',\n      
  reason:\n         'Failed to parse mapping [_doc]: No handler for type [flattened] declared on field [state]',\n        caused_by: [Object] },\n     status: 400 },\n  statusCode: 400,\n  response:\n   '{\"error\":{\"root_cause\":[{\"type\":\"mapper_parsing_exceptio
n\",\"reason\":\"No handler for type [flattened] declared on field [state]\"}],\"type\":\"mapper_parsing_exception\",\"reason\":\"Failed to parse mapping [_doc]: No handler for type [flattened] declared on field [state]\",\"caused_by\":{\"type\":\"mapper_parsing_exce
ption\",\"reason\":\"No handler for type [flattened] declared on field [state]\"}},\"status\":400}',\n  toString: [Function],\n  toJSON: [Function] }"}                                                                                                                    
{"type":"log","@timestamp":"2020-01-24T11:40:38Z","tags":["info","plugins-system"],"pid":6,"message":"Stopping all plugins."}                                                                                                                                              
{"type":"log","@timestamp":"2020-01-24T11:40:38Z","tags":["info","plugins","data"],"pid":6,"message":"Stopping plugin"}                                                                                                                                                    
{"type":"log","@timestamp":"2020-01-24T11:40:38Z","tags":["info","plugins","translations"],"pid":6,"message":"Stopping plugin"}                                                                                                                                            
{"type":"log","@timestamp":"2020-01-24T11:40:38Z","tags":["info","plugins","spaces"],"pid":6,"message":"Stopping plugin"}                                                                                                                                                  
{"type":"log","@timestamp":"2020-01-24T11:40:38Z","tags":["info","plugins","features"],"pid":6,"message":"Stopping plugin"}                                                                                                                                                
{"type":"log","@timestamp":"2020-01-24T11:40:38Z","tags":["info","plugins","timelion"],"pid":6,"message":"Stopping plugin"}                                                                                                                                                
{"type":"log","@timestamp":"2020-01-24T11:40:38Z","tags":["info","plugins","code"],"pid":6,"message":"Stopping plugin"}                                                                                                                                                    
{"type":"log","@timestamp":"2020-01-24T11:40:38Z","tags":["info","plugins","licensing"],"pid":6,"message":"Stopping plugin"}                                                                                                                                               
{"type":"log","@timestamp":"2020-01-24T11:40:38Z","tags":["info","plugins","security"],"pid":6,"message":"Stopping plugin"}  

FATAL  [mapper_parsing_exception] No handler for type [flattened] declared on field [state] :: {"path":"/.kibana_2","query":{},"body":"{\"mappings\":{\"dynamic\":\"strict\",\"properties\":{\"apm-telemetry\":{\"properties\":{\"has_any_services\":{\"type\":\"boolean\"
},\"services_per_agent\":{\"properties\":{\"python\":{\"type\":\"long\",\"null_value\":0},\"java\":{\"type\":\"long\",\"null_value\":0},\"nodejs\":{\"type\":\"long\",\"null_value\":0},\"js-base\":{\"type\":\"long\",\"null_value\":0},\"rum-js\":{\"type\":\"long\",\"nu
ll_value\":0},\"dotnet\":{\"type\":\"long\",\"null_value\":0},\"ruby\":{\"type\":\"long\",\"null_value\":0},\"go\":{\"type\":\"long\",\"null_value\":0}}}}},\"canvas-workpad\":{\"dynamic\":false,\"properties\":{\"name\":{\"type\":\"text\",\"fields\":{\"keyword\":{\"ty
pe\":\"keyword\"}}},\"@timestamp\":{\"type\":\"date\"},\"@created\":{\"type\":\"date\"}}},\"config\":{\"dynamic\":\"true\",\"properties\":{\"buildNum\":{\"type\":\"keyword\"}}},\"dashboard\":{\"properties\":{\"description\":{\"type\":\"text\"},\"hits\":{\"type\":\"in
teger\"},\"kibanaSavedObjectMeta\":{\"properties\":{\"searchSourceJSON\":{\"type\":\"text\"}}},\"optionsJSON\":{\"type\":\"text\"},\"panelsJSON\":{\"type\":\"text\"},\"refreshInterval\":{\"properties\":{\"display\":{\"type\":\"keyword\"},\"pause\":{\"type\":\"boolean
\"},\"section\":{\"type\":\"integer\"},\"value\":{\"type\":\"integer\"}}},\"timeFrom\":{\"type\":\"keyword\"},\"timeRestore\":{\"type\":\"boolean\"},\"timeTo\":{\"type\":\"keyword\"},\"title\":{\"type\":\"text\"},\"version\":{\"type\":\"integer\"}}},\"graph-workspace
\":{\"properties\":{\"description\":{\"type\":\"text\"},\"kibanaSavedObjectMeta\":{\"properties\":{\"searchSourceJSON\":{\"type\":\"text\"}}},\"numLinks\":{\"type\":\"integer\"},\"numVertices\":{\"type\":\"integer\"},\"title\":{\"type\":\"text\"},\"version\":{\"type\
":\"integer\"},\"wsState\":{\"type\":\"text\"}}},\"index-pattern\":{\"properties\":{\"fieldFormatMap\":{\"type\":\"text\"},\"fields\":{\"type\":\"text\"},\"intervalName\":{\"type\":\"keyword\"},\"notExpandable\":{\"type\":\"boolean\"},\"sourceFilters\":{\"type\":\"te
xt\"},\"timeFieldName\":{\"type\":\"keyword\"},\"title\":{\"type\":\"text\"},\"type\":{\"type\":\"keyword\"},\"typeMeta\":{\"type\":\"keyword\"}}},\"infrastructure-ui-source\":{\"properties\":{\"name\":{\"type\":\"text\"},\"description\":{\"type\":\"text\"},\"metricA
lias\":{\"type\":\"keyword\"},\"logAlias\":{\"type\":\"keyword\"},\"fields\":{\"properties\":{\"container\":{\"type\":\"keyword\"},\"host\":{\"type\":\"keyword\"},\"pod\":{\"type\":\"keyword\"},\"tiebreaker\":{\"type\":\"keyword\"},\"timestamp\":{\"type\":\"keyword\"
}}},\"logColumns\":{\"type\":\"nested\",\"properties\":{\"timestampColumn\":{\"properties\":{\"id\":{\"type\":\"keyword\"}}},\"messageColumn\":{\"properties\":{\"id\":{\"type\":\"keyword\"}}},\"fieldColumn\":{\"properties\":{\"id\":{\"type\":\"keyword\"},\"field\":{\
"type\":\"keyword\"}}}}}}},\"kql-telemetry\":{\"properties\":{\"optInCount\":{\"type\":\"long\"},\"optOutCount\":{\"type\":\"long\"}}},\"map\":{\"properties\":{\"description\":{\"type\":\"text\"},\"title\":{\"type\":\"text\"},\"version\":{\"type\":\"integer\"},\"boun
ds\":{\"type\":\"geo_shape\"},\"mapStateJSON\":{\"type\":\"text\"},\"layerListJSON\":{\"type\":\"text\"},\"uiStateJSON\":{\"type\":\"text\"}}},\"maps-telemetry\":{\"properties\":{\"mapsTotalCount\":{\"type\":\"long\"},\"timeCaptured\":{\"type\":\"date\"},\"attributes
PerMap\":{\"properties\":{\"dataSourcesCount\":{\"properties\":{\"min\":{\"type\":\"long\"},\"max\":{\"type\":\"long\"},\"avg\":{\"type\":\"long\"}}},\"layersCount\":{\"properties\":{\"min\":{\"type\":\"long\"},\"max\":{\"type\":\"long\"},\"avg\":{\"type\":\"long\"}}
},\"layerTypesCount\":{\"dynamic\":\"true\",\"properties\":{}},\"emsVectorLayersCount\":{\"dynamic\":\"true\",\"properties\":{}}}}}},\"migrationVersion\":{\"dynamic\":\"true\",\"type\":\"object\"},\"ml-telemetry\":{\"properties\":{\"file_data_visualizer\":{\"properti
es\":{\"index_creation_count\":{\"type\":\"long\"}}}}},\"namespace\":{\"type\":\"keyword\"},\"references\":{\"type\":\"nested\",\"properties\":{\"name\":{\"type\":\"keyword\"},\"type\":{\"type\":\"keyword\"},\"id\":{\"type\":\"keyword\"}}},\"search\":{\"properties\":
{\"columns\":{\"type\":\"keyword\"},\"description\":{\"type\":\"text\"},\"hits\":{\"type\":\"integer\"},\"kibanaSavedObjectMeta\":{\"properties\":{\"searchSourceJSON\":{\"type\":\"text\"}}},\"sort\":{\"type\":\"keyword\"},\"title\":{\"type\":\"text\"},\"version\":{\"
type\":\"integer\"}}},\"server\":{\"properties\":{\"uuid\":{\"type\":\"keyword\"}}},\"space\":{\"properties\":{\"name\":{\"type\":\"text\",\"fields\":{\"keyword\":{\"type\":\"keyword\",\"ignore_above\":2048}}},\"description\":{\"type\":\"text\"},\"initials\":{\"type\
":\"keyword\"},\"color\":{\"type\":\"keyword\"},\"disabledFeatures\":{\"type\":\"keyword\"},\"imageUrl\":{\"type\":\"text\",\"index\":false},\"_reserved\":{\"type\":\"boolean\"}}},\"telemetry\":{\"properties\":{\"enabled\":{\"type\":\"boolean\"},\"sendUsageFrom\":{\"
ignore_above\":256,\"type\":\"keyword\"},\"lastReported\":{\"type\":\"date\"},\"lastVersionChecked\":{\"ignore_above\":256,\"type\":\"keyword\"},\"userHasSeenNotice\":{\"type\":\"boolean\"}}},\"timelion-sheet\":{\"properties\":{\"description\":{\"type\":\"text\"},\"h
its\":{\"type\":\"integer\"},\"kibanaSavedObjectMeta\":{\"properties\":{\"searchSourceJSON\":{\"type\":\"text\"}}},\"timelion_chart_height\":{\"type\":\"integer\"},\"timelion_columns\":{\"type\":\"integer\"},\"timelion_interval\":{\"type\":\"keyword\"},\"timelion_oth
er_interval\":{\"type\":\"keyword\"},\"timelion_rows\":{\"type\":\"integer\"},\"timelion_sheet\":{\"type\":\"text\"},\"title\":{\"type\":\"text\"},\"version\":{\"type\":\"integer\"}}},\"type\":{\"type\":\"keyword\"},\"updated_at\":{\"type\":\"date\"},\"upgrade-assist
ant-reindex-operation\":{\"dynamic\":true,\"properties\":{\"indexName\":{\"type\":\"keyword\"},\"status\":{\"type\":\"integer\"}}},\"upgrade-assistant-telemetry\":{\"properties\":{\"ui_open\":{\"properties\":{\"overview\":{\"type\":\"long\",\"null_value\":0},\"cluste
r\":{\"type\":\"long\",\"null_value\":0},\"indices\":{\"type\":\"long\",\"null_value\":0}}},\"ui_reindex\":{\"properties\":{\"close\":{\"type\":\"long\",\"null_value\":0},\"open\":{\"type\":\"long\",\"null_value\":0},\"start\":{\"type\":\"long\",\"null_value\":0},\"s
top\":{\"type\":\"long\",\"null_value\":0}}},\"features\":{\"properties\":{\"deprecation_logging\":{\"properties\":{\"enabled\":{\"type\":\"boolean\",\"null_value\":true}}}}}}},\"url\":{\"properties\":{\"accessCount\":{\"type\":\"long\"},\"accessDate\":{\"type\":\"da
te\"},\"createDate\":{\"type\":\"date\"},\"url\":{\"type\":\"text\",\"fields\":{\"keyword\":{\"type\":\"keyword\",\"ignore_above\":2048}}}}},\"visualization\":{\"properties\":{\"description\":{\"type\":\"text\"},\"kibanaSavedObjectMeta\":{\"properties\":{\"searchSour
ceJSON\":{\"type\":\"text\"}}},\"savedSearchRefName\":{\"type\":\"keyword\"},\"title\":{\"type\":\"text\"},\"uiStateJSON\":{\"type\":\"text\"},\"version\":{\"type\":\"integer\"},\"visState\":{\"type\":\"text\"}}},\"canvas-element\":{\"dynamic\":false,\"properties\":{
\"name\":{\"type\":\"text\",\"fields\":{\"keyword\":{\"type\":\"keyword\"}}},\"help\":{\"type\":\"text\"},\"content\":{\"type\":\"text\"},\"image\":{\"type\":\"text\"},\"@timestamp\":{\"type\":\"date\"},\"@created\":{\"type\":\"date\"}}},\"metrics-explorer-view\":{\"
properties\":{\"name\":{\"type\":\"keyword\"},\"options\":{\"properties\":{\"metrics\":{\"type\":\"nested\",\"properties\":{\"aggregation\":{\"type\":\"keyword\"},\"field\":{\"type\":\"keyword\"},\"color\":{\"type\":\"keyword\"},\"label\":{\"type\":\"keyword\"}}},\"l
imit\":{\"type\":\"integer\"},\"groupBy\":{\"type\":\"keyword\"},\"filterQuery\":{\"type\":\"keyword\"},\"aggregation\":{\"type\":\"keyword\"}}},\"chartOptions\":{\"properties\":{\"type\":{\"type\":\"keyword\"},\"yAxisMode\":{\"type\":\"keyword\"},\"stack\":{\"type\"
:\"boolean\"}}},\"currentTimerange\":{\"properties\":{\"from\":{\"type\":\"keyword\"},\"to\":{\"type\":\"keyword\"},\"interval\":{\"type\":\"keyword\"}}}}},\"inventory-view\":{\"properties\":{\"name\":{\"type\":\"keyword\"},\"metric\":{\"properties\":{\"type\":{\"typ
e\":\"keyword\"}}},\"groupBy\":{\"type\":\"nested\",\"properties\":{\"label\":{\"type\":\"keyword\"},\"field\":{\"type\":\"keyword\"}}},\"nodeType\":{\"type\":\"keyword\"},\"view\":{\"type\":\"keyword\"},\"customOptions\":{\"type\":\"nested\",\"properties\":{\"text\"
:{\"type\":\"keyword\"},\"field\":{\"type\":\"keyword\"}}},\"boundsOverride\":{\"properties\":{\"max\":{\"type\":\"integer\"},\"min\":{\"type\":\"integer\"}}},\"autoBounds\":{\"type\":\"boolean\"},\"time\":{\"type\":\"integer\"},\"autoReload\":{\"type\":\"boolean\"},
\"filterQuery\":{\"properties\":{\"kind\":{\"type\":\"keyword\"},\"expression\":{\"type\":\"keyword\"}}}}},\"siem-ui-timeline\":{\"properties\":{\"columns\":{\"properties\":{\"aggregatable\":{\"type\":\"boolean\"},\"category\":{\"type\":\"keyword\"},\"columnHeaderTyp
e\":{\"type\":\"keyword\"},\"description\":{\"type\":\"text\"},\"example\":{\"type\":\"text\"},\"indexes\":{\"type\":\"keyword\"},\"id\":{\"type\":\"keyword\"},\"name\":{\"type\":\"text\"},\"placeholder\":{\"type\":\"text\"},\"searchable\":{\"type\":\"boolean\"},\"ty
pe\":{\"type\":\"keyword\"}}},\"dataProviders\":{\"properties\":{\"id\":{\"type\":\"keyword\"},\"name\":{\"type\":\"text\"},\"enabled\":{\"type\":\"boolean\"},\"excluded\":{\"type\":\"boolean\"},\"kqlQuery\":{\"type\":\"text\"},\"queryMatch\":{\"properties\":{\"field
\":{\"type\":\"text\"},\"displayField\":{\"type\":\"text\"},\"value\":{\"type\":\"text\"},\"displayValue\":{\"type\":\"text\"},\"operator\":{\"type\":\"text\"}}},\"and\":{\"properties\":{\"id\":{\"type\":\"keyword\"},\"name\":{\"type\":\"text\"},\"enabled\":{\"type\"
:\"boolean\"},\"excluded\":{\"type\":\"boolean\"},\"kqlQuery\":{\"type\":\"text\"},\"queryMatch\":{\"properties\":{\"field\":{\"type\":\"text\"},\"displayField\":{\"type\":\"text\"},\"value\":{\"type\":\"text\"},\"displayValue\":{\"type\":\"text\"},\"operator\":{\"ty
pe\":\"text\"}}}}}}},\"description\":{\"type\":\"text\"},\"favorite\":{\"properties\":{\"keySearch\":{\"type\":\"text\"},\"fullName\":{\"type\":\"text\"},\"userName\":{\"type\":\"text\"},\"favoriteDate\":{\"type\":\"date\"}}},\"kqlMode\":{\"type\":\"keyword\"},\"kqlQ
uery\":{\"properties\":{\"filterQuery\":{\"properties\":{\"kuery\":{\"properties\":{\"kind\":{\"type\":\"keyword\"},\"expression\":{\"type\":\"text\"}}},\"serializedQuery\":{\"type\":\"text\"}}}}},\"title\":{\"type\":\"text\"},\"dateRange\":{\"properties\":{\"start\"
:{\"type\":\"date\"},\"end\":{\"type\":\"date\"}}},\"sort\":{\"properties\":{\"columnId\":{\"type\":\"keyword\"},\"sortDirection\":{\"type\":\"keyword\"}}},\"created\":{\"type\":\"date\"},\"createdBy\":{\"type\":\"text\"},\"updated\":{\"type\":\"date\"},\"updatedBy\"
:{\"type\":\"text\"}}},\"siem-ui-timeline-note\":{\"properties\":{\"timelineId\":{\"type\":\"keyword\"},\"eventId\":{\"type\":\"keyword\"},\"note\":{\"type\":\"text\"},\"created\":{\"type\":\"date\"},\"createdBy\":{\"type\":\"text\"},\"updated\":{\"type\":\"date\"},\
"updatedBy\":{\"type\":\"text\"}}},\"siem-ui-timeline-pinned-event\":{\"properties\":{\"timelineId\":{\"type\":\"keyword\"},\"eventId\":{\"type\":\"keyword\"},\"created\":{\"type\":\"date\"},\"createdBy\":{\"type\":\"text\"},\"updated\":{\"type\":\"date\"},\"updatedB
y\":{\"type\":\"text\"}}},\"file-upload-telemetry\":{\"properties\":{\"filesUploadedTotalCount\":{\"type\":\"long\"}}},\"lens\":{\"properties\":{\"title\":{\"type\":\"text\"},\"visualizationType\":{\"type\":\"keyword\"},\"state\":{\"type\":\"flattened\"},\"expression
\":{\"index\":false,\"type\":\"keyword\"}}},\"lens-ui-telemetry\":{\"properties\":{\"name\":{\"type\":\"keyword\"},\"type\":{\"type\":\"keyword\"},\"date\":{\"type\":\"date\"},\"count\":{\"type\":\"integer\"}}},\"query\":{\"properties\":{\"title\":{\"type\":\"text\"}
,\"description\":{\"type\":\"text\"},\"query\":{\"properties\":{\"language\":{\"type\":\"keyword\"},\"query\":{\"type\":\"keyword\",\"index\":false}}},\"filters\":{\"type\":\"object\",\"enabled\":false},\"timefilter\":{\"type\":\"object\",\"enabled\":false}}},\"sampl
e-data-telemetry\":{\"properties\":{\"installCount\":{\"type\":\"long\"},\"unInstallCount\":{\"type\":\"long\"}}},\"ui-metric\":{\"properties\":{\"count\":{\"type\":\"integer\"}}}},\"_meta\":{\"migrationMappingPropertyHashes\":{\"config\":\"87aca8fdb053154f11383fce3d
bf3edf\",\"migrationVersion\":\"4a1746014a75ade3a714e1db5763276f\",\"type\":\"2f4316de49999235636386fe51dc06c1\",\"namespace\":\"2f4316de49999235636386fe51dc06c1\",\"updated_at\":\"00da57df13e94e9d98437d13ace4bfe0\",\"references\":\"7997cf5a56cc02bdc9c93361bde732b0\"
,\"graph-workspace\":\"cd7ba1330e6682e9cc00b78850874be1\",\"space\":\"c5ca8acafa0beaa4d08d014a97b6bc6b\",\"ml-telemetry\":\"257fd1d4b4fdbb9cb4b8a3b27da201e9\",\"apm-telemetry\":\"07ee1939fa4302c62ddc052ec03fed90\",\"map\":\"23d7aa4a720d4938ccde3983f87bd58d\",\"maps-t
elemetry\":\"a4229f8b16a6820c6d724b7e0c1f729d\",\"canvas-workpad\":\"b0a1706d356228dbdcb4a17e6b9eb231\",\"canvas-element\":\"7390014e1091044523666d97247392fc\",\"infrastructure-ui-source\":\"ddc0ecb18383f6b26101a2fadb2dab0c\",\"metrics-explorer-view\":\"53c5365793677
328df0ccb6138bf3cdd\",\"inventory-view\":\"84b320fd67209906333ffce261128462\",\"siem-ui-timeline\":\"1f6f0860ad7bc0dba3e42467ca40470d\",\"siem-ui-timeline-note\":\"8874706eedc49059d4cf0f5094559084\",\"siem-ui-timeline-pinned-event\":\"20638091112f0e14f0e443d512301c29
\",\"upgrade-assistant-reindex-operation\":\"a53a20fe086b72c9a86da3cc12dad8a6\",\"upgrade-assistant-telemetry\":\"56702cec857e0a9dacfb696655b4ff7b\",\"file-upload-telemetry\":\"0ed4d3e1983d1217a30982630897092e\",\"lens\":\"21c3ea0763beb1ecb0162529706b88c5\",\"lens-ui
-telemetry\":\"509bfa5978586998e05f9e303c07a327\",\"query\":\"11aaeb7f5f7fa5bb43f25e18ce26e7d9\",\"index-pattern\":\"66eccb05066c5a89924f48a9e9736499\",\"visualization\":\"52d7a13ad68a150c4525b292d23e12cc\",\"search\":\"181661168bbadd1eff5902361e2a0d5c\",\"dashboard\
":\"d00f614b29a80360e1190193fd333bab\",\"url\":\"c7f66a0df8b1b52f17c28c4adb111105\",\"server\":\"ec97f1c5da1a19609a60874e5af1100c\",\"kql-telemetry\":\"d12a98a6f19a2d273696597547e064ee\",\"sample-data-telemetry\":\"7d3cfeb915303c9641c59681967ffeb4\",\"telemetry\":\"3
58ffaa88ba34a97d55af0933a117de4\",\"timelion-sheet\":\"9a2a2748877c7a7b582fef201ab1d4cf\",\"ui-metric\":\"0d409297dc5ebe1e3a1da691c6ee32e3\"}}},\"settings\":{\"number_of_shards\":1,\"auto_expand_replicas\":\"0-1\"}}","statusCode":400,"response":"{\"error\":{\"root_ca
use\":[{\"type\":\"mapper_parsing_exception\",\"reason\":\"No handler for type [flattened] declared on field [state]\"}],\"type\":\"mapper_parsing_exception\",\"reason\":\"Failed to parse mapping [_doc]: No handler for type [flattened] declared on field [state]\",\"c
aused_by\":{\"type\":\"mapper_parsing_exception\",\"reason\":\"No handler for type [flattened] declared on field [state]\"}},\"status\":400}"}

Then, it crashes, gets restarted automatically, and logs the following:

{"type":"log","@timestamp":"2020-01-24T11:40:45Z","tags":["info","plugins-system"],"pid":6,"message":"Setting up [15] plugins: [security,licensing,code,timelion,features,spaces,translations,uiActions,newsfeed,inspector,embeddable,advancedUiActions,expressions,eui_uti
ls,data]"}                                                        
{"type":"log","@timestamp":"2020-01-24T11:40:45Z","tags":["info","plugins","security"],"pid":6,"message":"Setting up plugin"}                                                                                                                                              
{"type":"log","@timestamp":"2020-01-24T11:40:45Z","tags":["info","plugins","licensing"],"pid":6,"message":"Setting up plugin"}                                                                                                                                             
{"type":"log","@timestamp":"2020-01-24T11:40:45Z","tags":["info","plugins","code"],"pid":6,"message":"Setting up plugin"}                                                                                                                                                  
{"type":"log","@timestamp":"2020-01-24T11:40:45Z","tags":["info","plugins","timelion"],"pid":6,"message":"Setting up plugin"}                                                                                                                                              
{"type":"log","@timestamp":"2020-01-24T11:40:45Z","tags":["info","plugins","features"],"pid":6,"message":"Setting up plugin"}                                                                                                                                              
{"type":"log","@timestamp":"2020-01-24T11:40:45Z","tags":["info","plugins","spaces"],"pid":6,"message":"Setting up plugin"}                                                                                                                                                
{"type":"log","@timestamp":"2020-01-24T11:40:45Z","tags":["info","plugins","translations"],"pid":6,"message":"Setting up plugin"}                                                                                                                                          
{"type":"log","@timestamp":"2020-01-24T11:40:45Z","tags":["info","plugins","data"],"pid":6,"message":"Setting up plugin"}                                                                                                                                                  
{"type":"log","@timestamp":"2020-01-24T11:41:15Z","tags":["warning","plugins","licensing"],"pid":6,"message":"License information could not be obtained from Elasticsearch for the [data] cluster. Error: Request Timeout after 30000ms"}
{"type":"log","@timestamp":"2020-01-24T11:41:16Z","tags":["warning","legacy-plugins"],"pid":6,"path":"/usr/share/kibana/src/legacy/core_plugins/visualizations","message":"Skipping non-plugin directory at /usr/share/kibana/src/legacy/core_plugins/visualizations"}
{"type":"log","@timestamp":"2020-01-24T11:41:17Z","tags":["info","plugins-system"],"pid":6,"message":"Starting [8] plugins: [security,licensing,code,timelion,features,spaces,translations,data]"}                                                                         
{"type":"log","@timestamp":"2020-01-24T11:41:17Z","tags":["info","plugins","licensing"],"pid":6,"message":"Imported changed license information from Elasticsearch for the [data] cluster: type: basic | status: active"}
{"type":"log","@timestamp":"2020-01-24T11:41:17Z","tags":["info","migrations"],"pid":6,"message":"Creating index .kibana_task_manager_1."}                                                                                                                                 
{"type":"log","@timestamp":"2020-01-24T11:41:17Z","tags":["warning","migrations"],"pid":6,"message":"Unable to connect to Elasticsearch. Error: [resource_already_exists_exception] index [.kibana_task_manager_1/NOIsDqumSiSU6C2iYG__-g] already exists, with { index_uuid
=\"NOIsDqumSiSU6C2iYG__-g\" & index=\".kibana_task_manager_1\" }"}
{"type":"log","@timestamp":"2020-01-24T11:41:17Z","tags":["warning","migrations"],"pid":6,"message":"Another Kibana instance appears to be migrating the index. Waiting for that migration to complete. If no other Kibana instance is attempting migrations, you can get p
ast this message by deleting index .kibana_task_manager_1 and restarting Kibana."}                                                   
{"type":"log","@timestamp":"2020-01-24T11:41:17Z","tags":["info","migrations"],"pid":6,"message":"Creating index .kibana_2."}                                                                                                                                              
{"type":"log","@timestamp":"2020-01-24T11:41:17Z","tags":["warning","migrations"],"pid":6,"message":"Unable to connect to Elasticsearch. Error: [mapper_parsing_exception] No handler for type [flattened] declared on field [state]"}
{"type":"log","@timestamp":"2020-01-24T11:41:17Z","tags":["fatal","root"],"pid":6,"message":"{ Error: [mapper_parsing_exception] No handler for type [flattened] declared on field [state]\n    at respond (/usr/share/kibana/node_modules/elasticsearch/src/lib/transport.
js:349:15)\n    at checkRespForFailure (/usr/share/kibana/node_modules/elasticsearch/src/lib/transport.js:306:7)\n    at HttpConnector.<anonymous> (/usr/share/kibana/node_modules/elasticsearch/src/lib/connectors/http.js:173:7)\n    at IncomingMessage.wrapper (/usr/sh
are/kibana/node_modules/elasticsearch/node_modules/lodash/lodash.js:4929:19)\n    at IncomingMessage.emit (events.js:194:15)\n    at endReadableNT (_stream_readable.js:1103:12)\n    at process._tickCallback (internal/process/next_tick.js:63:19)\n  status: 400,\n  dis
playName: 'BadRequest',\n  message:\n   '[mapper_parsing_exception] No handler for type [flattened] declared on field [state]',\n  path: '/.kibana_2',\n  query: {},\n  body:\n   { error:\n      { root_cause: [Array],\n        type: 'mapper_parsing_exception',\n      
  reason:\n         'Failed to parse mapping [_doc]: No handler for type [flattened] declared on field [state]',\n        caused_by: [Object] },\n     status: 400 },\n  statusCode: 400,\n  response:\n   '{\"error\":{\"root_cause\":[{\"type\":\"mapper_parsing_exceptio
n\",\"reason\":\"No handler for type [flattened] declared on field [state]\"}],\"type\":\"mapper_parsing_exception\",\"reason\":\"Failed to parse mapping [_doc]: No handler for type [flattened] declared on field [state]\",\"caused_by\":{\"type\":\"mapper_parsing_exce
ption\",\"reason\":\"No handler for type [flattened] declared on field [state]\"}},\"status\":400}',\n  toString: [Function],\n  toJSON: [Function] }"}                                                                                                                    
{"type":"log","@timestamp":"2020-01-24T11:41:17Z","tags":["info","plugins-system"],"pid":6,"message":"Stopping all plugins."}                                                                                                                                              
{"type":"log","@timestamp":"2020-01-24T11:41:17Z","tags":["info","plugins","data"],"pid":6,"message":"Stopping plugin"}                                                                                                                                                    
{"type":"log","@timestamp":"2020-01-24T11:41:17Z","tags":["info","plugins","translations"],"pid":6,"message":"Stopping plugin"}                                                                                                                                            
{"type":"log","@timestamp":"2020-01-24T11:41:17Z","tags":["info","plugins","spaces"],"pid":6,"message":"Stopping plugin"}                                                                                                                                                  
{"type":"log","@timestamp":"2020-01-24T11:41:17Z","tags":["info","plugins","features"],"pid":6,"message":"Stopping plugin"}                                                                                                                                                
{"type":"log","@timestamp":"2020-01-24T11:41:17Z","tags":["info","plugins","timelion"],"pid":6,"message":"Stopping plugin"}                                                                                                                                                
{"type":"log","@timestamp":"2020-01-24T11:41:17Z","tags":["info","plugins","code"],"pid":6,"message":"Stopping plugin"}                                                                                                                                                    
{"type":"log","@timestamp":"2020-01-24T11:41:17Z","tags":["info","plugins","licensing"],"pid":6,"message":"Stopping plugin"}                                                                                                                                               
{"type":"log","@timestamp":"2020-01-24T11:41:17Z","tags":["info","plugins","security"],"pid":6,"message":"Stopping plugin"}

FATAL  [mapper_parsing_exception] No handler for type [flattened] declared on field [state] :: {"path":"/.kibana_2","query":{},"body":"{\"mappings\":{\"dynamic\":\"strict\",\"properties\":{\"apm-telemetry\":{\"properties\":{\"has_any_services\":{\"type\":\"boolean\"
},\"services_per_agent\":{\"properties\":{\"python\":{\"type\":\"long\",\"null_value\":0},\"java\":{\"type\":\"long\",\"null_value\":0},\"nodejs\":{\"type\":\"long\",\"null_value\":0},\"js-base\":{\"type\":\"long\",\"null_value\":0},\"rum-js\":{\"type\":\"long\",\"nu
ll_value\":0},\"dotnet\":{\"type\":\"long\",\"null_value\":0},\"ruby\":{\"type\":\"long\",\"null_value\":0},\"go\":{\"type\":\"long\",\"null_value\":0}}}}},\"canvas-workpad\":{\"dynamic\":false,\"properties\":{\"name\":{\"type\":\"text\",\"fields\":{\"keyword\":{\"ty
pe\":\"keyword\"}}},\"@timestamp\":{\"type\":\"date\"},\"@created\":{\"type\":\"date\"}}},\"config\":{\"dynamic\":\"true\",\"properties\":{\"buildNum\":{\"type\":\"keyword\"}}},\"dashboard\":{\"properties\":{\"description\":{\"type\":\"text\"},\"hits\":{\"type\":\"in
teger\"},\"kibanaSavedObjectMeta\":{\"properties\":{\"searchSourceJSON\":{\"type\":\"text\"}}},\"optionsJSON\":{\"type\":\"text\"},\"panelsJSON\":{\"type\":\"text\"},\"refreshInterval\":{\"properties\":{\"display\":{\"type\":\"keyword\"},\"pause\":{\"type\":\"boolean
\"},\"section\":{\"type\":\"integer\"},\"value\":{\"type\":\"integer\"}}},\"timeFrom\":{\"type\":\"keyword\"},\"timeRestore\":{\"type\":\"boolean\"},\"timeTo\":{\"type\":\"keyword\"},\"title\":{\"type\":\"text\"},\"version\":{\"type\":\"integer\"}}},\"graph-workspace
\":{\"properties\":{\"description\":{\"type\":\"text\"},\"kibanaSavedObjectMeta\":{\"properties\":{\"searchSourceJSON\":{\"type\":\"text\"}}},\"numLinks\":{\"type\":\"integer\"},\"numVertices\":{\"type\":\"integer\"},\"title\":{\"type\":\"text\"},\"version\":{\"type\
":\"integer\"},\"wsState\":{\"type\":\"text\"}}},\"index-pattern\":{\"properties\":{\"fieldFormatMap\":{\"type\":\"text\"},\"fields\":{\"type\":\"text\"},\"intervalName\":{\"type\":\"keyword\"},\"notExpandable\":{\"type\":\"boolean\"},\"sourceFilters\":{\"type\":\"te
xt\"},\"timeFieldName\":{\"type\":\"keyword\"},\"title\":{\"type\":\"text\"},\"type\":{\"type\":\"keyword\"},\"typeMeta\":{\"type\":\"keyword\"}}},\"infrastructure-ui-source\":{\"properties\":{\"name\":{\"type\":\"text\"},\"description\":{\"type\":\"text\"},\"metricA
lias\":{\"type\":\"keyword\"},\"logAlias\":{\"type\":\"keyword\"},\"fields\":{\"properties\":{\"container\":{\"type\":\"keyword\"},\"host\":{\"type\":\"keyword\"},\"pod\":{\"type\":\"keyword\"},\"tiebreaker\":{\"type\":\"keyword\"},\"timestamp\":{\"type\":\"keyword\"
}}},\"logColumns\":{\"type\":\"nested\",\"properties\":{\"timestampColumn\":{\"properties\":{\"id\":{\"type\":\"keyword\"}}},\"messageColumn\":{\"properties\":{\"id\":{\"type\":\"keyword\"}}},\"fieldColumn\":{\"properties\":{\"id\":{\"type\":\"keyword\"},\"field\":{\
"type\":\"keyword\"}}}}}}},\"kql-telemetry\":{\"properties\":{\"optInCount\":{\"type\":\"long\"},\"optOutCount\":{\"type\":\"long\"}}},\"map\":{\"properties\":{\"description\":{\"type\":\"text\"},\"title\":{\"type\":\"text\"},\"version\":{\"type\":\"integer\"},\"boun
ds\":{\"type\":\"geo_shape\"},\"mapStateJSON\":{\"type\":\"text\"},\"layerListJSON\":{\"type\":\"text\"},\"uiStateJSON\":{\"type\":\"text\"}}},\"maps-telemetry\":{\"properties\":{\"mapsTotalCount\":{\"type\":\"long\"},\"timeCaptured\":{\"type\":\"date\"},\"attributes
PerMap\":{\"properties\":{\"dataSourcesCount\":{\"properties\":{\"min\":{\"type\":\"long\"},\"max\":{\"type\":\"long\"},\"avg\":{\"type\":\"long\"}}},\"layersCount\":{\"properties\":{\"min\":{\"type\":\"long\"},\"max\":{\"type\":\"long\"},\"avg\":{\"type\":\"long\"}}
},\"layerTypesCount\":{\"dynamic\":\"true\",\"properties\":{}},\"emsVectorLayersCount\":{\"dynamic\":\"true\",\"properties\":{}}}}}},\"migrationVersion\":{\"dynamic\":\"true\",\"type\":\"object\"},\"ml-telemetry\":{\"properties\":{\"file_data_visualizer\":{\"properti
es\":{\"index_creation_count\":{\"type\":\"long\"}}}}},\"namespace\":{\"type\":\"keyword\"},\"references\":{\"type\":\"nested\",\"properties\":{\"name\":{\"type\":\"keyword\"},\"type\":{\"type\":\"keyword\"},\"id\":{\"type\":\"keyword\"}}},\"search\":{\"properties\":
{\"columns\":{\"type\":\"keyword\"},\"description\":{\"type\":\"text\"},\"hits\":{\"type\":\"integer\"},\"kibanaSavedObjectMeta\":{\"properties\":{\"searchSourceJSON\":{\"type\":\"text\"}}},\"sort\":{\"type\":\"keyword\"},\"title\":{\"type\":\"text\"},\"version\":{\"
type\":\"integer\"}}},\"server\":{\"properties\":{\"uuid\":{\"type\":\"keyword\"}}},\"space\":{\"properties\":{\"name\":{\"type\":\"text\",\"fields\":{\"keyword\":{\"type\":\"keyword\",\"ignore_above\":2048}}},\"description\":{\"type\":\"text\"},\"initials\":{\"type\
":\"keyword\"},\"color\":{\"type\":\"keyword\"},\"disabledFeatures\":{\"type\":\"keyword\"},\"imageUrl\":{\"type\":\"text\",\"index\":false},\"_reserved\":{\"type\":\"boolean\"}}},\"telemetry\":{\"properties\":{\"enabled\":{\"type\":\"boolean\"},\"sendUsageFrom\":{\"
ignore_above\":256,\"type\":\"keyword\"},\"lastReported\":{\"type\":\"date\"},\"lastVersionChecked\":{\"ignore_above\":256,\"type\":\"keyword\"},\"userHasSeenNotice\":{\"type\":\"boolean\"}}},\"timelion-sheet\":{\"properties\":{\"description\":{\"type\":\"text\"},\"h
its\":{\"type\":\"integer\"},\"kibanaSavedObjectMeta\":{\"properties\":{\"searchSourceJSON\":{\"type\":\"text\"}}},\"timelion_chart_height\":{\"type\":\"integer\"},\"timelion_columns\":{\"type\":\"integer\"},\"timelion_interval\":{\"type\":\"keyword\"},\"timelion_oth
er_interval\":{\"type\":\"keyword\"},\"timelion_rows\":{\"type\":\"integer\"},\"timelion_sheet\":{\"type\":\"text\"},\"title\":{\"type\":\"text\"},\"version\":{\"type\":\"integer\"}}},\"type\":{\"type\":\"keyword\"},\"updated_at\":{\"type\":\"date\"},\"upgrade-assist
ant-reindex-operation\":{\"dynamic\":true,\"properties\":{\"indexName\":{\"type\":\"keyword\"},\"status\":{\"type\":\"integer\"}}},\"upgrade-assistant-telemetry\":{\"properties\":{\"ui_open\":{\"properties\":{\"overview\":{\"type\":\"long\",\"null_value\":0},\"cluste
r\":{\"type\":\"long\",\"null_value\":0},\"indices\":{\"type\":\"long\",\"null_value\":0}}},\"ui_reindex\":{\"properties\":{\"close\":{\"type\":\"long\",\"null_value\":0},\"open\":{\"type\":\"long\",\"null_value\":0},\"start\":{\"type\":\"long\",\"null_value\":0},\"s
top\":{\"type\":\"long\",\"null_value\":0}}},\"features\":{\"properties\":{\"deprecation_logging\":{\"properties\":{\"enabled\":{\"type\":\"boolean\",\"null_value\":true}}}}}}},\"url\":{\"properties\":{\"accessCount\":{\"type\":\"long\"},\"accessDate\":{\"type\":\"da
te\"},\"createDate\":{\"type\":\"date\"},\"url\":{\"type\":\"text\",\"fields\":{\"keyword\":{\"type\":\"keyword\",\"ignore_above\":2048}}}}},\"visualization\":{\"properties\":{\"description\":{\"type\":\"text\"},\"kibanaSavedObjectMeta\":{\"properties\":{\"searchSour
ceJSON\":{\"type\":\"text\"}}},\"savedSearchRefName\":{\"type\":\"keyword\"},\"title\":{\"type\":\"text\"},\"uiStateJSON\":{\"type\":\"text\"},\"version\":{\"type\":\"integer\"},\"visState\":{\"type\":\"text\"}}},\"canvas-element\":{\"dynamic\":false,\"properties\":{
\"name\":{\"type\":\"text\",\"fields\":{\"keyword\":{\"type\":\"keyword\"}}},\"help\":{\"type\":\"text\"},\"content\":{\"type\":\"text\"},\"image\":{\"type\":\"text\"},\"@timestamp\":{\"type\":\"date\"},\"@created\":{\"type\":\"date\"}}},\"metrics-explorer-view\":{\"
properties\":{\"name\":{\"type\":\"keyword\"},\"options\":{\"properties\":{\"metrics\":{\"type\":\"nested\",\"properties\":{\"aggregation\":{\"type\":\"keyword\"},\"field\":{\"type\":\"keyword\"},\"color\":{\"type\":\"keyword\"},\"label\":{\"type\":\"keyword\"}}},\"l
imit\":{\"type\":\"integer\"},\"groupBy\":{\"type\":\"keyword\"},\"filterQuery\":{\"type\":\"keyword\"},\"aggregation\":{\"type\":\"keyword\"}}},\"chartOptions\":{\"properties\":{\"type\":{\"type\":\"keyword\"},\"yAxisMode\":{\"type\":\"keyword\"},\"stack\":{\"type\"
:\"boolean\"}}},\"currentTimerange\":{\"properties\":{\"from\":{\"type\":\"keyword\"},\"to\":{\"type\":\"keyword\"},\"interval\":{\"type\":\"keyword\"}}}}},\"inventory-view\":{\"properties\":{\"name\":{\"type\":\"keyword\"},\"metric\":{\"properties\":{\"type\":{\"typ
e\":\"keyword\"}}},\"groupBy\":{\"type\":\"nested\",\"properties\":{\"label\":{\"type\":\"keyword\"},\"field\":{\"type\":\"keyword\"}}},\"nodeType\":{\"type\":\"keyword\"},\"view\":{\"type\":\"keyword\"},\"customOptions\":{\"type\":\"nested\",\"properties\":{\"text\"
:{\"type\":\"keyword\"},\"field\":{\"type\":\"keyword\"}}},\"boundsOverride\":{\"properties\":{\"max\":{\"type\":\"integer\"},\"min\":{\"type\":\"integer\"}}},\"autoBounds\":{\"type\":\"boolean\"},\"time\":{\"type\":\"integer\"},\"autoReload\":{\"type\":\"boolean\"},
\"filterQuery\":{\"properties\":{\"kind\":{\"type\":\"keyword\"},\"expression\":{\"type\":\"keyword\"}}}}},\"siem-ui-timeline\":{\"properties\":{\"columns\":{\"properties\":{\"aggregatable\":{\"type\":\"boolean\"},\"category\":{\"type\":\"keyword\"},\"columnHeaderTyp
e\":{\"type\":\"keyword\"},\"description\":{\"type\":\"text\"},\"example\":{\"type\":\"text\"},\"indexes\":{\"type\":\"keyword\"},\"id\":{\"type\":\"keyword\"},\"name\":{\"type\":\"text\"},\"placeholder\":{\"type\":\"text\"},\"searchable\":{\"type\":\"boolean\"},\"ty
pe\":{\"type\":\"keyword\"}}},\"dataProviders\":{\"properties\":{\"id\":{\"type\":\"keyword\"},\"name\":{\"type\":\"text\"},\"enabled\":{\"type\":\"boolean\"},\"excluded\":{\"type\":\"boolean\"},\"kqlQuery\":{\"type\":\"text\"},\"queryMatch\":{\"properties\":{\"field
\":{\"type\":\"text\"},\"displayField\":{\"type\":\"text\"},\"value\":{\"type\":\"text\"},\"displayValue\":{\"type\":\"text\"},\"operator\":{\"type\":\"text\"}}},\"and\":{\"properties\":{\"id\":{\"type\":\"keyword\"},\"name\":{\"type\":\"text\"},\"enabled\":{\"type\"
:\"boolean\"},\"excluded\":{\"type\":\"boolean\"},\"kqlQuery\":{\"type\":\"text\"},\"queryMatch\":{\"properties\":{\"field\":{\"type\":\"text\"},\"displayField\":{\"type\":\"text\"},\"value\":{\"type\":\"text\"},\"displayValue\":{\"type\":\"text\"},\"operator\":{\"ty
pe\":\"text\"}}}}}}},\"description\":{\"type\":\"text\"},\"favorite\":{\"properties\":{\"keySearch\":{\"type\":\"text\"},\"fullName\":{\"type\":\"text\"},\"userName\":{\"type\":\"text\"},\"favoriteDate\":{\"type\":\"date\"}}},\"kqlMode\":{\"type\":\"keyword\"},\"kqlQ
uery\":{\"properties\":{\"filterQuery\":{\"properties\":{\"kuery\":{\"properties\":{\"kind\":{\"type\":\"keyword\"},\"expression\":{\"type\":\"text\"}}},\"serializedQuery\":{\"type\":\"text\"}}}}},\"title\":{\"type\":\"text\"},\"dateRange\":{\"properties\":{\"start\"
:{\"type\":\"date\"},\"end\":{\"type\":\"date\"}}},\"sort\":{\"properties\":{\"columnId\":{\"type\":\"keyword\"},\"sortDirection\":{\"type\":\"keyword\"}}},\"created\":{\"type\":\"date\"},\"createdBy\":{\"type\":\"text\"},\"updated\":{\"type\":\"date\"},\"updatedBy\"
:{\"type\":\"text\"}}},\"siem-ui-timeline-note\":{\"properties\":{\"timelineId\":{\"type\":\"keyword\"},\"eventId\":{\"type\":\"keyword\"},\"note\":{\"type\":\"text\"},\"created\":{\"type\":\"date\"},\"createdBy\":{\"type\":\"text\"},\"updated\":{\"type\":\"date\"},\
"updatedBy\":{\"type\":\"text\"}}},\"siem-ui-timeline-pinned-event\":{\"properties\":{\"timelineId\":{\"type\":\"keyword\"},\"eventId\":{\"type\":\"keyword\"},\"created\":{\"type\":\"date\"},\"createdBy\":{\"type\":\"text\"},\"updated\":{\"type\":\"date\"},\"updatedB
y\":{\"type\":\"text\"}}},\"file-upload-telemetry\":{\"properties\":{\"filesUploadedTotalCount\":{\"type\":\"long\"}}},\"lens\":{\"properties\":{\"title\":{\"type\":\"text\"},\"visualizationType\":{\"type\":\"keyword\"},\"state\":{\"type\":\"flattened\"},\"expression
\":{\"index\":false,\"type\":\"keyword\"}}},\"lens-ui-telemetry\":{\"properties\":{\"name\":{\"type\":\"keyword\"},\"type\":{\"type\":\"keyword\"},\"date\":{\"type\":\"date\"},\"count\":{\"type\":\"integer\"}}},\"query\":{\"properties\":{\"title\":{\"type\":\"text\"}
,\"description\":{\"type\":\"text\"},\"query\":{\"properties\":{\"language\":{\"type\":\"keyword\"},\"query\":{\"type\":\"keyword\",\"index\":false}}},\"filters\":{\"type\":\"object\",\"enabled\":false},\"timefilter\":{\"type\":\"object\",\"enabled\":false}}},\"sampl
e-data-telemetry\":{\"properties\":{\"installCount\":{\"type\":\"long\"},\"unInstallCount\":{\"type\":\"long\"}}},\"ui-metric\":{\"properties\":{\"count\":{\"type\":\"integer\"}}}},\"_meta\":{\"migrationMappingPropertyHashes\":{\"config\":\"87aca8fdb053154f11383fce3d
bf3edf\",\"migrationVersion\":\"4a1746014a75ade3a714e1db5763276f\",\"type\":\"2f4316de49999235636386fe51dc06c1\",\"namespace\":\"2f4316de49999235636386fe51dc06c1\",\"updated_at\":\"00da57df13e94e9d98437d13ace4bfe0\",\"references\":\"7997cf5a56cc02bdc9c93361bde732b0\"
,\"graph-workspace\":\"cd7ba1330e6682e9cc00b78850874be1\",\"space\":\"c5ca8acafa0beaa4d08d014a97b6bc6b\",\"ml-telemetry\":\"257fd1d4b4fdbb9cb4b8a3b27da201e9\",\"apm-telemetry\":\"07ee1939fa4302c62ddc052ec03fed90\",\"map\":\"23d7aa4a720d4938ccde3983f87bd58d\",\"maps-t
elemetry\":\"a4229f8b16a6820c6d724b7e0c1f729d\",\"canvas-workpad\":\"b0a1706d356228dbdcb4a17e6b9eb231\",\"canvas-element\":\"7390014e1091044523666d97247392fc\",\"infrastructure-ui-source\":\"ddc0ecb18383f6b26101a2fadb2dab0c\",\"metrics-explorer-view\":\"53c5365793677
328df0ccb6138bf3cdd\",\"inventory-view\":\"84b320fd67209906333ffce261128462\",\"siem-ui-timeline\":\"1f6f0860ad7bc0dba3e42467ca40470d\",\"siem-ui-timeline-note\":\"8874706eedc49059d4cf0f5094559084\",\"siem-ui-timeline-pinned-event\":\"20638091112f0e14f0e443d512301c29
\",\"upgrade-assistant-reindex-operation\":\"a53a20fe086b72c9a86da3cc12dad8a6\",\"upgrade-assistant-telemetry\":\"56702cec857e0a9dacfb696655b4ff7b\",\"file-upload-telemetry\":\"0ed4d3e1983d1217a30982630897092e\",\"lens\":\"21c3ea0763beb1ecb0162529706b88c5\",\"lens-ui
-telemetry\":\"509bfa5978586998e05f9e303c07a327\",\"query\":\"11aaeb7f5f7fa5bb43f25e18ce26e7d9\",\"index-pattern\":\"66eccb05066c5a89924f48a9e9736499\",\"visualization\":\"52d7a13ad68a150c4525b292d23e12cc\",\"search\":\"181661168bbadd1eff5902361e2a0d5c\",\"dashboard\
":\"d00f614b29a80360e1190193fd333bab\",\"url\":\"c7f66a0df8b1b52f17c28c4adb111105\",\"server\":\"ec97f1c5da1a19609a60874e5af1100c\",\"kql-telemetry\":\"d12a98a6f19a2d273696597547e064ee\",\"sample-data-telemetry\":\"7d3cfeb915303c9641c59681967ffeb4\",\"telemetry\":\"3
58ffaa88ba34a97d55af0933a117de4\",\"timelion-sheet\":\"9a2a2748877c7a7b582fef201ab1d4cf\",\"ui-metric\":\"0d409297dc5ebe1e3a1da691c6ee32e3\"}}},\"settings\":{\"number_of_shards\":1,\"auto_expand_replicas\":\"0-1\"}}","statusCode":400,"response":"{\"error\":{\"root_ca
use\":[{\"type\":\"mapper_parsing_exception\",\"reason\":\"No handler for type [flattened] declared on field [state]\"}],\"type\":\"mapper_parsing_exception\",\"reason\":\"Failed to parse mapping [_doc]: No handler for type [flattened] declared on field [state]\",\"c
aused_by\":{\"type\":\"mapper_parsing_exception\",\"reason\":\"No handler for type [flattened] declared on field [state]\"}},\"status\":400}"}

Then, it crashes and gets restarted automatically, over and over again, until Elasticsearch version upgrade is completely done. After which Kibana stays stuck forever after having logged the following:

{"type":"log","@timestamp":"2020-01-24T11:44:26Z","tags":["info","plugins-system"],"pid":6,"message":"Setting up [15] plugins: [licensing,security,code,timelion,features,spaces,translations,inspector,newsfeed,uiActions,embeddable,advancedUiActions,expressions,eui_utils,data]"}
{"type":"log","@timestamp":"2020-01-24T11:44:26Z","tags":["info","plugins","licensing"],"pid":6,"message":"Setting up plugin"}
{"type":"log","@timestamp":"2020-01-24T11:44:26Z","tags":["info","plugins","security"],"pid":6,"message":"Setting up plugin"}
{"type":"log","@timestamp":"2020-01-24T11:44:26Z","tags":["info","plugins","code"],"pid":6,"message":"Setting up plugin"}
{"type":"log","@timestamp":"2020-01-24T11:44:26Z","tags":["info","plugins","timelion"],"pid":6,"message":"Setting up plugin"}
{"type":"log","@timestamp":"2020-01-24T11:44:26Z","tags":["info","plugins","features"],"pid":6,"message":"Setting up plugin"}
{"type":"log","@timestamp":"2020-01-24T11:44:26Z","tags":["info","plugins","spaces"],"pid":6,"message":"Setting up plugin"}
{"type":"log","@timestamp":"2020-01-24T11:44:26Z","tags":["info","plugins","translations"],"pid":6,"message":"Setting up plugin"}
{"type":"log","@timestamp":"2020-01-24T11:44:26Z","tags":["info","plugins","data"],"pid":6,"message":"Setting up plugin"}
{"type":"log","@timestamp":"2020-01-24T11:44:56Z","tags":["error","elasticsearch","data"],"pid":6,"message":"Request error, retrying\nGET https://elasticsearch-sample-es-http.default.svc:9200/_xpack => Client network socket disconnected before secure TLS connection was established"}
{"type":"log","@timestamp":"2020-01-24T11:44:56Z","tags":["warning","legacy-plugins"],"pid":6,"path":"/usr/share/kibana/src/legacy/core_plugins/visualizations","message":"Skipping non-plugin directory at /usr/share/kibana/src/legacy/core_plugins/visualizations"}
{"type":"log","@timestamp":"2020-01-24T11:44:57Z","tags":["warning","plugins","licensing"],"pid":6,"message":"License information could not be obtained from Elasticsearch for the [data] cluster. Error: Request Timeout after 30000ms"}
{"type":"log","@timestamp":"2020-01-24T11:44:57Z","tags":["info","plugins-system"],"pid":6,"message":"Starting [8] plugins: [licensing,security,code,timelion,features,spaces,translations,data]"}
{"type":"log","@timestamp":"2020-01-24T11:44:57Z","tags":["info","migrations"],"pid":6,"message":"Creating index .kibana_task_manager_1."}
{"type":"log","@timestamp":"2020-01-24T11:44:57Z","tags":["warning","migrations"],"pid":6,"message":"Unable to connect to Elasticsearch. Error: [resource_already_exists_exception] index [.kibana_task_manager_1/NOIsDqumSiSU6C2iYG__-g] already exists, with { index_uuid=\"NOIsDqumSiSU6C2iYG__-g\" & index=\".kibana_task_manager_1\" }"}
{"type":"log","@timestamp":"2020-01-24T11:44:57Z","tags":["warning","migrations"],"pid":6,"message":"Another Kibana instance appears to be migrating the index. Waiting for that migration to complete. If no other Kibana instance is attempting migrations, you can get past this message by deleting index .kibana_task_manager_1 and restarting Kibana."}
{"type":"log","@timestamp":"2020-01-24T11:44:58Z","tags":["info","plugins","licensing"],"pid":6,"message":"Imported changed license information from Elasticsearch for the [data] cluster: type: basic | status: active"}
{"type":"log","@timestamp":"2020-01-24T11:44:58Z","tags":["info","migrations"],"pid":6,"message":"Creating index .kibana_2."}
{"type":"log","@timestamp":"2020-01-24T11:44:58Z","tags":["info","migrations"],"pid":6,"message":"Migrating .kibana_1 saved objects to .kibana_2"}
{"type":"log","@timestamp":"2020-01-24T11:44:58Z","tags":["info","migrations"],"pid":6,"message":"Pointing alias .kibana to .kibana_2."}
{"type":"log","@timestamp":"2020-01-24T11:44:59Z","tags":["info","migrations"],"pid":6,"message":"Finished in 766ms."}

I don't see anything relevant in Elasticsearch logs, which mostly reports "updating [cluster.routing.allocation.enable] from [all] to [primaries] (and the other way around), matching the rolling upgrade procedure.

In short, what seems to happen is:

At this point I think we can close this issue. The only way to end up in that situation seems to incorrectly update Kibana concurrently/before Elasticsearch, which is unsupported. In an "ideal" world, I wish Kibana would have detected early on that it's targeting a different Elasticsearch cluster and should just wait instead of crashing. Similar to what seems to happen on a more regular version upgrade (test scenario 1). https://github.com/elastic/cloud-on-k8s/issues/2353 covers the version upgrade ordering problem in ECK.

CODE-REaD commented 4 years ago

@sebgl comments,

The only way to end up in that situation seems to incorrectly update Kibana concurrently/before Elasticsearch, which is unsupported.

I think I found a fairly common way:

yum update

This turned my Wazuh site from working to failing with the browser message Kibana is not ready and repeating errors as described above in /var/log/messages.

To resolve the issue, I:

  1. Stopped kibana (systemctl stop kibana)
  2. Deleted indexes: 
    curl -XDELETE http://localhost:9200/.kibana_1
curl -XDELETE http://localhost:9200/.kibana_2
curl -XDELETE http://localhost:9200/.kibana_3
curl -XDELETE http://localhost:9200/.kibana_4
  3. Restarted kibana
  4. Used the web interface to add indexes (Management->Kibana->Index Patterns->Create Index Pattern

Environment: CentOS Linux 7 Kibana 7.7.0 wazuh-manager 3.12.3 elasticsearch 7.7.0

jeff419 commented 4 years ago

@sebgl comments,

The only way to end up in that situation seems to incorrectly update Kibana concurrently/before Elasticsearch, which is unsupported.

I think I found a fairly common way:

yum update

This turned my Wazuh site from working to failing with the browser message Kibana is not ready and repeating errors as described above in /var/log/messages.

To resolve the issue, I:

  1. Stopped kibana (systemctl stop kibana)
  2. Deleted indexes:
curl -XDELETE http://localhost:9200/.kibana_1
curl -XDELETE http://localhost:9200/.kibana_2
curl -XDELETE http://localhost:9200/.kibana_3
curl -XDELETE http://localhost:9200/.kibana_4
  1. Restarted kibana
  2. Used the web interface to add indexes (Management->Kibana->Index Patterns->Create Index Pattern

Environment: CentOS Linux 7 Kibana 7.7.0 wazuh-manager 3.12.3 elasticsearch 7.7.0

hi, my solution is as follows:

I tried your steps first, but I failed.

First, I turned off kibana, elasticsearch, metricbeat, logstash services. Then I would run "rm -vf /var/lib/eleasticsearch/nodes" command on all clusters. You should note that this command will delete all data. I was able to delete it very easily because I did not need it.

Finally, I opened all services to kibana, elasticsearch, metricbeat, logstash. ELK worked successfully but in exchange for losing all my data.

all version 7.8.0

ebuildy commented 4 years ago

@jeff419 this is not an upgrade since you have removed all data! but a fresh install.

ionescu77 commented 3 years ago

@CODE-REaD Hey the suggestion did help me. Thanks!

I found kibana was stuck migrating (Centos 7).

The sugestion was actually in the logs (I run kibana with log to file, /etc/kibana/kibana.yml):

# Enables you specify a file where Kibana stores log output.
logging.dest: /var/log/kibana/kibana.log
eating index .kibana_9."}
{"type":"log","@timestamp":"2020-12-23T13:34:18Z","tags":["warning","savedobjects-service"],"pid":7569,"message":"Unable to connect to Elasticsearch. Error: [resource_already_exists_exception] index [.kibana_task_manager_5/q_CGKXHvSnO5RfyMm34eCQ] already exists, with { index_uuid=\"q_CGKXHvSnO5RfyMm34eCQ\" & index=\".kibana_task_manager_5\" }"}
{"type":"log","@timestamp":"2020-12-23T13:34:18Z","tags":["warning","savedobjects-service"],"pid":7569,"message":"Another Kibana instance appears to be migrating the index. Waiting for that migration to complete. If no other Kibana instance is attempting migrations, you can get past this message by deleting index .kibana_task_manager_5 and restarting Kibana."}
{"type":"log","@timestamp":"2020-12-23T13:34:18Z","tags":["warning","savedobjects-service"],"pid":7569,"message":"Unable to connect to Elasticsearch. Error: [resource_already_exists_exception] index [.kibana_9/bD5ChYtnT_qV37N6ovIAAQ] already exists, with { index_uuid=\"bD5ChYtnT_qV37N6ovIAAQ\" & index=\".kibana_9\" }"}
{"type":"log","@timestamp":"2020-12-23T13:34:18Z","tags":["warning","savedobjects-service"],"pid":7569,"message":"Another Kibana instance appears to be migrating the index. Waiting for that migration to complete. If no other Kibana instance is attempting migrations, you can get past this message by deleting index .kibana_9 and restarting Kibana."}

I performed the steps sugested,a.

systemctl stop kibana
curl -XDELETE http://localhost:9200/.kibana_9
{"acknowledged":true}

curl -XDELETE http://localhost:9200/.kibana_task_manager_5
{"acknowledged":true}

systemctl start kibana

Kibana is again available at localhost:5601

PS I might have tried to many kibana restarts, but in my case I was also running out of disk space, and had to fix elastic first. But after elastic was there kibana was stuck. The above fixed my issue...

izzy commented 3 years ago

I've run into this as well and honestly reading this issue is just another needle sting I got from the whole elastic stack.

Are you seriously suggesting that using the linux system upgrade mechanisms like apt or yum on packages elastic provides is not supported?!

This is clearly a bug! If the packages need to be upgraded one after the other they MUST implement a locking or version check either in the package metadata itself or in the post-upgrade-scripts provided with the packages or in the software itself.

Saying "this thing that makes half of your cluster unusable without manual intervention(that includes potential data loss) isn't really a problem" is literally like Apple saying you're holding your phone wrong. No, this is just downplaying the problems the packages elastic provides created.

After needing to fix this bs(and also some other problems with the last updates) from your packages that just get updated with the rest of the system because they lack checks and repairs for breaking changes I have to admit I'm starting to think the value I get from elastic isn't as big as the pain from every. single. upgrade.