`MapperService.assertSerialization` assertion tripping

[DaveCTurner]

Seen in a recent 8.15.0-SNAPSHOT:

[2024-04-23T10:50:49,865][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [runTask-0] fatal error in thread [elasticsearch[runTask-0][masterService#updateTask][T#6]], exiting java.lang.AssertionError: Mapping serialization result is different from source.
--> Source [{"_doc":{"dynamic":"false","_meta":{"namespace":"default","kibana":{"version":"8.15.0"},"managed":true},"_data_stream_timestamp":{"enabled":true},"properties":{"@timestamp":{"type":"date","ignore_malformed":false},"api_config":{"properties":{"action_type_id":{"type":"keyword"},"connector_id":{"type":"keyword"},"default_system_prompt_id":{"type":"keyword"},"model":{"type":"keyword"},"provider":{"type":"keyword"}}},"created_at":{"type":"date"},"id":{"type":"keyword"},"is_default":{"type":"boolean"},"messages":{"type":"nested","properties":{"@timestamp":{"type":"date"},"content":{"type":"text"},"is_error":{"type":"boolean"},"reader":{"type":"object"},"role":{"type":"keyword"},"trace_data":{"properties":{"trace_id":{"type":"keyword"},"transaction_id":{"type":"keyword"}}}}},"replacements":{"properties":{"uuid":{"type":"keyword"},"value":{"type":"keyword"}}},"summary":{"properties":{"@timestamp":{"type":"date"},"confidence":{"type":"keyword"},"content":{"type":"text"},"public":{"type":"boolean"}}},"title":{"type":"keyword"},"updated_at":{"type":"date"},"users":{"type":"nested","properties":{"id":{"type":"keyword"},"name":{"type":"keyword"}}}}}}]
--> Result [{"_doc":{"dynamic":"false","_meta":{"namespace":"default","kibana":{"version":"8.15.0"},"managed":true},"_data_stream_timestamp":{"enabled":true},"properties":{"@timestamp":{"type":"date","ignore_malformed":false},"api_config":{"properties":{"action_type_id":{"type":"keyword"},"connector_id":{"type":"keyword"},"default_system_prompt_id":{"type":"keyword"},"model":{"type":"keyword"},"provider":{"type":"keyword"}}},"created_at":{"type":"date"},"id":{"type":"keyword"},"is_default":{"type":"boolean"},"messages":{"type":"nested","properties":{"@timestamp":{"type":"date"},"content":{"type":"text"},"is_error":{"type":"boolean"},"reader":{"type":"object"},"role":{"type":"keyword"},"trace_data":{"properties":{"trace_id":{"type":"keyword"},"transaction_id":{"type":"keyword"}}}}},"replacements":{"properties":{"uuid":{"type":"keyword"},"value":{"type":"keyword"}}},"summary":{"properties":{"@timestamp":{"type":"date","ignore_malformed":false},"confidence":{"type":"keyword"},"content":{"type":"text"},"public":{"type":"boolean"}}},"title":{"type":"keyword"},"updated_at":{"type":"date"},"users":{"type":"nested","properties":{"id":{"type":"keyword"},"name":{"type":"keyword"}}}}}}]
        at org.elasticsearch.server@8.15.0-SNAPSHOT/org.elasticsearch.index.mapper.MapperService.assertSerialization(MapperService.java:632)
        at org.elasticsearch.server@8.15.0-SNAPSHOT/org.elasticsearch.index.mapper.MapperService.doMerge(MapperService.java:545)
        at org.elasticsearch.server@8.15.0-SNAPSHOT/org.elasticsearch.index.mapper.MapperService.merge(MapperService.java:532)
        at org.elasticsearch.server@8.15.0-SNAPSHOT/org.elasticsearch.cluster.metadata.MetadataMappingService$PutMappingExecutor.applyRequest(MetadataMappingService.java:168)
        at org.elasticsearch.server@8.15.0-SNAPSHOT/org.elasticsearch.cluster.metadata.MetadataMappingService$PutMappingExecutor.execute(MetadataMappingService.java:115)
        at org.elasticsearch.server@8.15.0-SNAPSHOT/org.elasticsearch.cluster.service.MasterService.innerExecuteTasks(MasterService.java:1039)
        at org.elasticsearch.server@8.15.0-SNAPSHOT/org.elasticsearch.cluster.service.MasterService.executeTasks(MasterService.java:1004)
        at org.elasticsearch.server@8.15.0-SNAPSHOT/org.elasticsearch.cluster.service.MasterService.executeAndPublishBatch(MasterService.java:232)
        at org.elasticsearch.server@8.15.0-SNAPSHOT/org.elasticsearch.cluster.service.MasterService$BatchingTaskQueue$Processor.lambda$run$2(MasterService.java:1645)
        at org.elasticsearch.server@8.15.0-SNAPSHOT/org.elasticsearch.action.ActionListener.run(ActionListener.java:356)
        at org.elasticsearch.server@8.15.0-SNAPSHOT/org.elasticsearch.cluster.service.MasterService$BatchingTaskQueue$Processor.run(MasterService.java:1642)
        at org.elasticsearch.server@8.15.0-SNAPSHOT/org.elasticsearch.cluster.service.MasterService$5.lambda$doRun$0(MasterService.java:1237)
        at org.elasticsearch.server@8.15.0-SNAPSHOT/org.elasticsearch.action.ActionListener.run(ActionListener.java:356)
        at org.elasticsearch.server@8.15.0-SNAPSHOT/org.elasticsearch.cluster.service.MasterService$5.doRun(MasterService.java:1216)
        at org.elasticsearch.server@8.15.0-SNAPSHOT/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:984)
        at org.elasticsearch.server@8.15.0-SNAPSHOT/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
        at java.base/java.lang.Thread.run(Thread.java:1583)

Possibly a consequence of #99346? The difference is the ,"ignore_malformed":false added to the summary.@timestamp field.

[elasticsearchmachine]

Pinging @elastic/es-search (Team:Search)

[hop-dev]

Commenting out the field definitions for messages.@timestamp and summary.@timestamp in https://github.com/elastic/kibana/blob/master/x-pack/plugins/elastic_assistant/server/ai_assistant_data_clients/conversations/field_maps_configuration.ts seems to have stopped the error from happening, I have an elasticsearch running for 24h now whereas before it would never get past 1hr 30 mins.

[benwtrent]

@hop-dev how long has this been occurring? The issue that @DaveCTurner linked is from 8.11. Its weird that we are just noticing this now. Something else has changed I think.

[javanna]

The naming summary.@timestamp caught my eye, and I think I found at least one problem with #99346 . I opened #109948 .

[aleksmaus]

Stumbled upon a similar issue today with freshly installed security serverless. The ES indexing pod crashing on this error. Looks like the issue is still not fixed. Any updates?

[elasticsearchmachine]

Pinging @elastic/es-search-foundations (Team:Search Foundations)