After upgrading from 8.11 to 8.12 some transforms have failed with the following message:
task encountered more than 10 failures; latest failure: Fielddata is disabled on [ece.runner] in [.ds-service-proxy-requests-filebeat-7.12.0-azure-eastus-2022.11.18-000332]. Text fields are not optimised for operations that require per-document field data like aggregations and sorting, so these operations are disabled by default. Please use a keyword field instead. Alternatively, set fielddata=true on [ece.runner] in order to load field data by uninverting the inverted index. Note that this can use significant memory.
Such an error message means that the transform encountered a source index with mapping type for the ece.runner field being text rather than keyword.
Interestingly, the index with this out-of-date mapping for ece.runner field existed for a long time (year 2022) but the transform was able to deal with it.
This issue is to find out and potentially fix regression in 8.12 which made this transform start failing.
More details:
The problem occurred despite transform having filter:
{
"took": 4907,
"timed_out": false,
"_shards": {
"total": 624,
"successful": 576,
"skipped": 486,
"failed": 48,
"failures": [
{
"shard": 0,
"index": ".ds-service-proxy-requests-filebeat-7.12.0-aws-us-gov-east-1-2022.07.19-2022.11.19-000335",
"node": "Sj3o1omwQleQBnIEtGWBHQ",
"reason": {
"type": "illegal_argument_exception",
"reason": "Fielddata is disabled on [ece.runner] in [.ds-service-proxy-requests-filebeat-7.12.0-aws-us-gov-east-1-2022.07.19-2022.11.19-000335]. Text fields are not optimised for operations that require per-document field data like aggregations and sorting, so these operations are disabled by default. Please use a keyword field instead. Alternatively, set fielddata=true on [ece.runner] in order to load field data by uninverting the inverted index. Note that this can use significant memory."
}
},
so in a sense this behavior in transform could also be expected. But it worked just fine before the upgrade to 8.12.
Discussed mitigations:
a) delete the indices which have out-of-date mapping and restart the transform
This requires identifying and deleting the indices and then restarting the affected transforms.
b) filter out the offending indices using source.query.
This requires update of each affected transform and does not guarantee that the problem will not happen in the future.
c) make the ece.runner field always render as keyword using source.runtime_mappings.
This requires update of each affected transform but has a better chance of fixing the problem permanently.
Don't know how using runtime field affects transform performance.
Here is part of the relevant _update request:
I wasn't able to reproduce the issue locally.
For version:
8.7 - the transform failed due to "fielddata" problem without retrying
8.11 and 8.12 - the transform failed due to "fielddata" problem after 10 retries
I did not see a situation where a transform would just work properly with the out-of-date text mapping for the ece.runner field.
But if it was a long-running transform then some additional validations could kick in after the upgrade. This needs investigation.
Elasticsearch Version
8.12
Installed Plugins
No response
Java Version
bundled
OS Version
n/a
Problem Description
After upgrading from
8.11
to8.12
some transforms have failed with the following message:Such an error message means that the transform encountered a source index with mapping type for the
ece.runner
field beingtext
rather thankeyword
. Interestingly, the index with this out-of-date mapping forece.runner
field existed for a long time (year 2022) but the transform was able to deal with it. This issue is to find out and potentially fix regression in8.12
which made this transform start failing.More details:
The problem occurred despite transform having filter:
That's because the offending index has never been moved to "frozen" but was "hot" when this problem occurred.
When simply running the aggregation (no transform involved):
the result is:
so in a sense this behavior in transform could also be expected. But it worked just fine before the upgrade to
8.12
.a) delete the indices which have out-of-date mapping and restart the transform This requires identifying and deleting the indices and then restarting the affected transforms.
b) filter out the offending indices using
source.query
. This requires update of each affected transform and does not guarantee that the problem will not happen in the future.c) make the
ece.runner
field always render askeyword
usingsource.runtime_mappings
. This requires update of each affected transform but has a better chance of fixing the problem permanently. Don't know how using runtime field affects transform performance. Here is part of the relevant_update
request:Steps to Reproduce
I wasn't able to reproduce the issue locally. For version:
8.7
- the transform failed due to "fielddata" problem without retrying8.11
and8.12
- the transform failed due to "fielddata" problem after 10 retriesI did not see a situation where a transform would just work properly with the out-of-date
text
mapping for theece.runner
field. But if it was a long-running transform then some additional validations could kick in after the upgrade. This needs investigation.Logs (if relevant)
No response