Bulk requests ignore "coerce" option of dynamically created fields

[andrershov]

Suppose I’ve created an index my_index with no properties.

PUT my_index

Now I perform the following bulk operation:

/_bulk
{ "index" : { "_index" : "my_index", "_type" : "_doc", "_id" : "1" } }
{"my_number": 10}
{ "index" : { "_index" : "my_index", "_type" : "_doc", "_id" : "2" } }
{"my_number": 2.5}

This bulk operation will partially fail with the following error:

"reason": "mapper [my_number] cannot be changed from type [float] to [long]"

This happens because before bulk there was no “my_number” field defined and both index operations try to create this field - one with the type long, another with the type float. One of them wins, the second one fails. If I execute this bulk request the second time, it will index both documents, because at this point there is already “my_number” field with the type “long” or “float” and because of coercion both of the inserts are successful. The more real-life example of this is the reindex API, that uses bulk API. Consider you have index “my_index” with dynamic mapping. Now you successfully insert values like 2 and 2.5, because of coercion. But now if you want to re-index “my_index” to “my_index2”, you will get an error. If you execute re-index for the second time it will work without a problem. We need to understand how this issue is related to https://github.com/elastic/elasticsearch/issues/25861.

[elasticmachine]

Pinging @elastic/es-search

[elasticmachine]

Pinging @elastic/es-distributed

[andrershov]

Reproduction works fine Elasticsearch 6.5.1 and could not be reproduced on Elasticsearch 7.0.0. The reason for this is that the default number of shards has been changed from 5 to 1 starting with 7.0.0. In order to reproduce the issue with 7.0.0, you need to create an index with more than 1 shard and make sure documents are routed to different shards.

[andrershov]

We discussed the issue on the team meeting. The issue is happening because TransportBulkRequest spawns TransportShardBulkRequests in parallel. And these TransportShardBulkRequests do not synchronize with each other. One of them, when indexing the document, tries to update the mapping with the type long and another one tries to update the mapping with the type float. Please note that inside one TransportShardBulkRequest all indexing operations are executed sequentially and if mapping update is needed, the primary node waits for this mapping update to happen. The possible solution here could be to include the current mapping version in the mapping update exception. This way the node executing TransportShardBulkRequest can compare its mapping version and received mapping version - if local mapping version is less than in the exception - we can retry the update.