opensearch-project / OpenSearch

🔎 Open source distributed and RESTful search engine.
https://opensearch.org/docs/latest/opensearch/index/
Apache License 2.0
9.29k stars 1.71k forks source link

[FEATURE] Allow `dynamic_templates` but disallow other dynamic mappings #11276

Closed esamattis closed 3 weeks ago

esamattis commented 9 months ago

Is your feature request related to a problem? Please describe.

Currently it is possible to set dynamic: "strict" which prevents adding fields that do not have a static mapping in the "properties" but it also disables usage of dynamic_templates.

For example

{
    "dynamic": "strict",
    "properties": {
        "url": { "type": "keyword" },
    },
    "dynamic_templates": [
        {
            "custom_dates": {
                "match": "date__*",
                "mapping": {
                    "type": "date",
                },
            },
        },
    ],
}

This will disallow documents like

{ "bad": "something" }

while allowing

{ "url": "https://example" }

but it does not allow

{ "date__created": "2023-11-20T16:11:20.598Z" }

which is unfortunate 😞

Describe the solution you'd like

Option like "dynamic": "templates" which allows the date__created and url but disallows the bad field would be really useful to avoid accidental data in the index.

Describe alternatives you've considered

There's a hacky workaround. One can map any remaining fields to a "broken mapping"

{
    "properties": {
        "url": { "type": "keyword" },
    },
    "dynamic_templates": [
        {
            "dates": {
                "match": "date__*",
                "mapping": {
                    "type": "date",
                },
            },
        },
        {
            "reject_rest": {
                "match": "*",
                "mapping": {
                    // Something invalid according to opensearch
                    "__NOT_ALLOWED__": "__NOT_ALLOWED__"
                },
            },
        },
    ],
}

And this works like described but emits bit weird error messages.

unknown parameter [__NOT_ALLOWED__] on mapper [bad] of type [null]
peternied commented 6 months ago

[Triage - attendees 1 2 3 4] @esamattis Thanks for filing, we'd welcome a pull request for this feature.

gaobinlong commented 3 weeks ago

Closed by https://github.com/opensearch-project/OpenSearch/pull/14555.