search

elastic / kibana

Your window into the Elastic Stack
https://www.elastic.co/products/kibana
Other
19.53k stars 8.07k forks source link

[Migrations] [illegal_argument_exception]: mapper [property] cannot be changed from type [x] to [y] #163289

Closed gsoldevila closed 1 week ago

gsoldevila commented 12 months ago

We have spotted a couple of occurrences of that error during saved object migrations:

The first one, on a flaky test: https://github.com/elastic/kibana/issues/127545

[00:02:13] proc [kibana] [2022-03-11T20:45:43.189+00:00][ERROR][savedobjects-service] [.kibana_task_manager] Unexpected Elasticsearch ResponseError: statusCode: 400, method: PUT, url: /.kibana_task_manager_8.2.0_001/_mapping?timeout=60s error: [illegal_argument_exception]: mapper [task.status] cannot be changed from type [text] to [keyword],
[00:02:13] proc [kibana] [2022-03-11T20:45:43.190+00:00][ERROR][http] Error: Unable to complete saved object migrations for the [.kibana_task_manager] index. Please check the health of your Elasticsearch cluster and try again. Unexpected Elasticsearch ResponseError: statusCode: 400, method: PUT, url: /.kibana_task_manager_8.2.0_001/_mapping?timeout=60s error: [illegal_argument_exception]: mapper [task.status] cannot be changed from type [text] to [keyword],

The second one, on an SDH: https://github.com/elastic/sdh-kibana/issues/3976

Unable to complete saved object migrations for the [.kibana] index. Error: [{"type":"illegal_argument_exception","reason":"mapper [canvas-workpad.pages.elements.position.left] cannot be changed from type [long] to [float]"}]

For the SDH occurrence, the issue seemed to occur during the REINDEX_SOURCE_TO_TEMP_INDEX_BULK step, when the migrators are indexing documents into the temporary indices.

elasticmachine commented 12 months ago

Pinging @elastic/kibana-core (Team:Core)

pgayvallet commented 11 months ago

For the SDH occurrence, the issue seemed to occur during the REINDEX_SOURCE_TO_TEMP_INDEX_BULK step, when the migrators are indexing documents into the temporary indices.

Even if the temp index mapping is dynamic: false, it seems that internally, ES still needs to identify the type of the field.

From my understanding, this error between long and float is caused by the fact that the first document using this field has a "shorter" value, creating the field as a float, but then another document has a value that can't be contained in a float, so using a long value in java, causing the mapper exception, as the field already exists as a float type.

We should probably try to confirm this hypothesis, but if confirmed, I'm not 100% sure what we could do about it, aside from actually using the mappings from the types (in addition to "dynamic: false") for the temp index?

Note that this wouldn't explain the FTR error between text and keyword, so maybe my guess is just wrong...

pgayvallet commented 3 months ago

@gsoldevila did we do anything for this issue / shall we keep it open or close it?

rudolf commented 1 week ago

I think this was another case of https://github.com/elastic/kibana/pull/158182 where one node is bulk indexing and another finishes and deletes the temp index. The node that continues to bulk index then causes ES to auto-create an index with the incorrect mappings.