Ignore_malformed fires mapper_parsing_exception for keyword type on 7.10.1 (it worked on 7.1.0)

[RicardoGralhoz]

Elasticsearch version: 7.10.1 (also seen on 7.8.1) This used to work on 7.1.0. It may be a breaking change in 7.4+ that apparently was not documented.

Plugins installed: none

JVM version : OpenJDK 64-Bit Server VM (build 11.0.3+7-Ubuntu-1ubuntu218.04.1)

OS version : Ubuntu 18.04.2 LTS

Description of the problem including expected versus actual behavior: Please see this forum topic

Using ignore_malformed on a dynamic_template mapping of keyword type fires a mapper_parsing_exception when trying to index a document with a new field that matches the template. Indexing a document with an existing field that was matched before upgrading from 7.1.0 works fine, even if it's a keyword field.

There are differences in the docs when comparing versions 7.3 and 7.4. From 7.4 on it says:

The ignore_malformed setting is currently supported by the following mapping types: Numeric, Date, Date nanoseconds, Geo-point, Geo-shape, IP

– which excludes other types such as keyword.

Steps to reproduce:

1. index creation & mapping

PUT /my_index
{
  "mappings": {
    "dynamic_templates": [
      {
        "template_stdField": {
          "mapping": {
            "ignore_malformed": true,
            "type": "keyword"
          },
          "path_match": "*"
        }
      }
    ]
  }
}

2. creating a document

PUT /my_index/_doc/1
{
  "mykeyword": "hello world"
}

Provide logs:

{
  "error": {
    "root_cause": [{
      "type": "mapper_parsing_exception",
      "reason": "unknown parameter [ignore_malformed] on mapper [mykeyword] of type [keyword]",
      "stack_trace": "MapperParsingException[unknown parameter [ignore_malformed] on mapper [mykeyword] of type [keyword]]\n\tat org.elasticsearch.index.mapper.ParametrizedFieldMapper$Builder.parse(ParametrizedFieldMapper.java:614)\n\tat org.elasticsearch.index.mapper.ParametrizedFieldMapper$TypeParser.parse(ParametrizedFieldMapper.java:662)\n\tat org.elasticsearch.index.mapper.ParametrizedFieldMapper$TypeParser.parse(ParametrizedFieldMapper.java:647)\n\tat org.elasticsearch.index.mapper.RootObjectMapper.findTemplateBuilder(RootObjectMapper.java:278)\n\tat org.elasticsearch.index.mapper.RootObjectMapper.findTemplateBuilder(RootObjectMapper.java:251)\n\tat org.elasticsearch.index.mapper.DocumentParser.createBuilderFromDynamicValue(DocumentParser.java:704)\n\tat org.elasticsearch.index.mapper.DocumentParser.parseDynamicValue(DocumentParser.java:765)\n\tat org.elasticsearch.index.mapper.DocumentParser.parseValue(DocumentParser.java:620)\n\tat org.elasticsearch.index.mapper.DocumentParser.innerParseObject(DocumentParser.java:424)\n\tat org.elasticsearch.index.mapper.DocumentParser.parseObjectOrNested(DocumentParser.java:395)\n\tat org.elasticsearch.index.mapper.DocumentParser.internalParseDocument(DocumentParser.java:112)\n\tat org.elasticsearch.index.mapper.DocumentParser.parseDocument(DocumentParser.java:71)\n\tat org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:227)\n\tat org.elasticsearch.index.shard.IndexShard.prepareIndex(IndexShard.java:803)\n\tat org.elasticsearch.index.shard.IndexShard.applyIndexOperation(IndexShard.java:780)\n\tat org.elasticsearch.index.shard.IndexShard.applyIndexOperationOnPrimary(IndexShard.java:752)\n\tat org.elasticsearch.action.bulk.TransportShardBulkAction.executeBulkItemRequest(TransportShardBulkAction.java:285)\n\tat org.elasticsearch.action.bulk.TransportShardBulkAction$2.doRun(TransportShardBulkAction.java:175)\n\tat org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)\n\tat org.elasticsearch.action.bulk.TransportShardBulkAction.performOnPrimary(TransportShardBulkAction.java:220)\n\tat org.elasticsearch.action.bulk.TransportShardBulkAction.dispatchedShardOperationOnPrimary(TransportShardBulkAction.java:126)\n\tat org.elasticsearch.action.bulk.TransportShardBulkAction.dispatchedShardOperationOnPrimary(TransportShardBulkAction.java:85)\n\tat org.elasticsearch.action.support.replication.TransportWriteAction$1.doRun(TransportWriteAction.java:179)\n\tat org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:737)\n\tat org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630)\n\tat java.base/java.lang.Thread.run(Thread.java:832)\n"
    }],
    "type": "mapper_parsing_exception",
    "reason": "unknown parameter [ignore_malformed] on mapper [mykeyword] of type [keyword]",
    "stack_trace": "MapperParsingException[unknown parameter [ignore_malformed] on mapper [mykeyword] of type [keyword]]\n\tat org.elasticsearch.index.mapper.ParametrizedFieldMapper$Builder.parse(ParametrizedFieldMapper.java:614)\n\tat org.elasticsearch.index.mapper.ParametrizedFieldMapper$TypeParser.parse(ParametrizedFieldMapper.java:662)\n\tat org.elasticsearch.index.mapper.ParametrizedFieldMapper$TypeParser.parse(ParametrizedFieldMapper.java:647)\n\tat org.elasticsearch.index.mapper.RootObjectMapper.findTemplateBuilder(RootObjectMapper.java:278)\n\tat org.elasticsearch.index.mapper.RootObjectMapper.findTemplateBuilder(RootObjectMapper.java:251)\n\tat org.elasticsearch.index.mapper.DocumentParser.createBuilderFromDynamicValue(DocumentParser.java:704)\n\tat org.elasticsearch.index.mapper.DocumentParser.parseDynamicValue(DocumentParser.java:765)\n\tat org.elasticsearch.index.mapper.DocumentParser.parseValue(DocumentParser.java:620)\n\tat org.elasticsearch.index.mapper.DocumentParser.innerParseObject(DocumentParser.java:424)\n\tat org.elasticsearch.index.mapper.DocumentParser.parseObjectOrNested(DocumentParser.java:395)\n\tat org.elasticsearch.index.mapper.DocumentParser.internalParseDocument(DocumentParser.java:112)\n\tat org.elasticsearch.index.mapper.DocumentParser.parseDocument(DocumentParser.java:71)\n\tat org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:227)\n\tat org.elasticsearch.index.shard.IndexShard.prepareIndex(IndexShard.java:803)\n\tat org.elasticsearch.index.shard.IndexShard.applyIndexOperation(IndexShard.java:780)\n\tat org.elasticsearch.index.shard.IndexShard.applyIndexOperationOnPrimary(IndexShard.java:752)\n\tat org.elasticsearch.action.bulk.TransportShardBulkAction.executeBulkItemRequest(TransportShardBulkAction.java:285)\n\tat org.elasticsearch.action.bulk.TransportShardBulkAction$2.doRun(TransportShardBulkAction.java:175)\n\tat org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)\n\tat org.elasticsearch.action.bulk.TransportShardBulkAction.performOnPrimary(TransportShardBulkAction.java:220)\n\tat org.elasticsearch.action.bulk.TransportShardBulkAction.dispatchedShardOperationOnPrimary(TransportShardBulkAction.java:126)\n\tat org.elasticsearch.action.bulk.TransportShardBulkAction.dispatchedShardOperationOnPrimary(TransportShardBulkAction.java:85)\n\tat org.elasticsearch.action.support.replication.TransportWriteAction$1.doRun(TransportWriteAction.java:179)\n\tat org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:737)\n\tat org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630)\n\tat java.base/java.lang.Thread.run(Thread.java:832)\n"
  },
  "status": 400
}

[RicardoGralhoz]

it looks related to #64999 and #64982

[elasticmachine]

Pinging @elastic/es-search (Team:Search)

[cbuescher]

I can confirm that 7.1.0 allows using "ignore_malformed" in templates for keywordtype. However it already rejects the same parameter when directly used in a keyword type mapping. With 7.10 we issue a deprecation warning when putting the template, but we should avoid throwing this error for existing malformed templates when adding documents with new fields.

[cbuescher]

I believe the problem is more general than just the "ignore_malformed" parameter on keyword fields. With 7.10 we can put any unknown parameter in a "dynamic_templates" section for a type. We currently emit a deprecation warning, e.g.

PUT /my_index
{
  "mappings": {
    "dynamic_templates": [
      {
        "template_stdField": {
          "mapping": {
            "bla": true,
            "type": "text"
          },
          "path_match": "*"
        }
      }
    ]
  }
}

Will warn with

#! Deprecation: dynamic template [template_stdField] has invalid content [{"path_match":"*","mapping":{"bla":true,"type":"text"}}], caused by [unknown parameter [bla] on mapper [__dynamic__template_stdField] of type [text]]

However this opens the door for any indexed doc that matches the definition to throw an error since we try to parse the mapping just then and its illegal except for the few parameter names specifically defined in `ParametrizedFieldMapper#DEPRECATED_PARAMS" (like "boost", "meta" etc...).