search

elastic / elasticsearch

Free and Open Source, Distributed, RESTful Search Engine
https://www.elastic.co/products/elasticsearch
Other
69.6k stars 24.63k forks source link

Support indices created in 7.x with camelcase date format in rollups #110961

Open pgomulka opened 2 months ago

pgomulka commented 2 months ago

Elasticsearch Version

8.x

Installed Plugins

No response

Java Version

bundled

OS Version

n/a

Problem Description

when an index was created in 7.x with camel case date format (dateOptionalTime for instance) we should still support this in every usecase of Elasticsearch in 8.x as per issue https://github.com/elastic/elasticsearch/issues/84199 we have done this for most of the usecases with this PR https://github.com/elastic/elasticsearch/pull/88914 however for rollups it will still fail as we have seen with this stacktrace

[2024-07-15T01:15:00,905][WARN ][org.elasticsearch.xpack.rollup.job.RollupJobTask] [instance-0000000071] Rollup job [osgsumarizado3] failed with an exception: 
org.elasticsearch.action.search.SearchPhaseExecutionException: 
    at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseFailure(AbstractSearchAsyncAction.java:712) ~[elasticsearch-8.13.2.jar:?]
    at org.elasticsearch.action.search.FetchSearchPhase$1.onFailure(FetchSearchPhase.java:92) ~[elasticsearch-8.13.2.jar:?]
    at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:28) ~[elasticsearch-8.13.2.jar:?]
    at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33) ~[elasticsearch-8.13.2.jar:?]
    at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:984) ~[elasticsearch-8.13.2.jar:?]
    at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) ~[elasticsearch-8.13.2.jar:?]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]
    at java.lang.Thread.run(Thread.java:1583) ~[?:?]
Caused by: java.lang.IllegalArgumentException: Invalid format: [dateOptionalTime]: Unknown pattern letter: t
    at org.elasticsearch.common.time.DateFormatters.forPattern(DateFormatters.java:2130) ~[elasticsearch-8.13.2.jar:?]
    at org.elasticsearch.common.time.DateFormatter.forPattern(DateFormatter.java:126) ~[elasticsearch-8.13.2.jar:?]
    at org.elasticsearch.common.time.DateFormatter.forPattern(DateFormatter.java:104) ~[elasticsearch-8.13.2.jar:?]
    at org.elasticsearch.search.DocValueFormat$DateTime.<init>(DocValueFormat.java:240) ~[elasticsearch-8.13.2.jar:?]
    at org.elasticsearch.common.io.stream.NamedWriteableAwareStreamInput.readNamedWriteable(NamedWriteableAwareStreamInput.java:56) ~[elasticsearch-8.13.2.jar:?]
    at org.elasticsearch.common.io.stream.NamedWriteableAwareStreamInput.readNamedWriteable(NamedWriteableAwareStreamInput.java:32) ~[elasticsearch-8.13.2.jar:?]
    at org.elasticsearch.search.aggregations.metrics.InternalNumericMetricsAggregation.<init>(InternalNumericMetricsAggregation.java:146) ~[elasticsearch-8.13.2.jar:?]
    at org.elasticsearch.search.aggregations.metrics.InternalNumericMetricsAggregation.<init>(InternalNumericMetricsAggregation.java:136) ~[elasticsearch-8.13.2.jar:?]
    at org.elasticsearch.search.aggregations.metrics.InternalNumericMetricsAggregation$SingleValue.<init>(InternalNumericMetricsAggregation.java:37) ~[elasticsearch-8.13.2.jar:?]
    at org.elasticsearch.search.aggregations.metrics.Max.<init>(Max.java:47) ~[elasticsearch-8.13.2.jar:?]
    at org.elasticsearch.common.io.stream.NamedWriteableAwareStreamInput.readNamedWriteable(NamedWriteableAwareStreamInput.java:56) ~[elasticsearch-8.13.2.jar:?]
    at org.elasticsearch.common.io.stream.NamedWriteableAwareStreamInput.readNamedWriteable(NamedWriteableAwareStreamInput.java:32) ~[elasticsearch-8.13.2.jar:?]
    at org.elasticsearch.search.aggregations.InternalAggregations.lambda$readFrom$0(InternalAggregations.java:168) ~[elasticsearch-8.13.2.jar:?]
    at org.elasticsearch.common.io.stream.StreamInput.readCollection(StreamInput.java:1216) ~[elasticsearch-8.13.2.jar:?]
    at org.elasticsearch.common.io.stream.StreamInput.readCollectionAsList(StreamInput.java:1088) ~[elasticsearch-8.13.2.jar:?]
    at org.elasticsearch.search.aggregations.InternalAggregations.readFrom(InternalAggregations.java:168) ~[elasticsearch-8.13.2.jar:?]
    at org.elasticsearch.search.aggregations.bucket.composite.InternalComposite$InternalBucket.<init>(InternalComposite.java:386) ~[elasticsearch-8.13.2.jar:?]
    at org.elasticsearch.search.aggregations.bucket.composite.InternalComposite.lambda$new$1(InternalComposite.java:103) ~[elasticsearch-8.13.2.jar:?]
    at org.elasticsearch.common.io.stream.StreamInput.readCollection(StreamInput.java:1216) ~[elasticsearch-8.13.2.jar:?]
    at org.elasticsearch.common.io.stream.StreamInput.readCollectionAsList(StreamInput.java:1088) ~[elasticsearch-8.13.2.jar:?]
    at org.elasticsearch.search.aggregations.bucket.composite.InternalComposite.<init>(InternalComposite.java:103) ~[elasticsearch-8.13.2.jar:?]
    at org.elasticsearch.common.io.stream.NamedWriteableAwareStreamInput.readNamedWriteable(NamedWriteableAwareStreamInput.java:56) ~[elasticsearch-8.13.2.jar:?]
    at org.elasticsearch.common.io.stream.NamedWriteableAwareStreamInput.readNamedWriteable(NamedWriteableAwareStreamInput.java:32) ~[elasticsearch-8.13.2.jar:?]
    at org.elasticsearch.search.aggregations.InternalAggregations.lambda$readFrom$0(InternalAggregations.java:168) ~[elasticsearch-8.13.2.jar:?]
    at org.elasticsearch.common.io.stream.StreamInput.readCollection(StreamInput.java:1216) ~[elasticsearch-8.13.2.jar:?]
    at org.elasticsearch.common.io.stream.StreamInput.readCollectionAsList(StreamInput.java:1088) ~[elasticsearch-8.13.2.jar:?]
    at org.elasticsearch.search.aggregations.InternalAggregations.readFrom(InternalAggregations.java:168) ~[elasticsearch-8.13.2.jar:?]
    at org.elasticsearch.common.io.stream.DelayableWriteable.deserialize(DelayableWriteable.java:237) ~[elasticsearch-8.13.2.jar:?]
    at org.elasticsearch.common.io.stream.DelayableWriteable$Serialized.expand(DelayableWriteable.java:184) ~[elasticsearch-8.13.2.jar:?]
    at org.elasticsearch.search.aggregations.InternalAggregations$1.get(InternalAggregations.java:214) ~[elasticsearch-8.13.2.jar:?]
    at org.elasticsearch.search.aggregations.InternalAggregations$1.get(InternalAggregations.java:211) ~[elasticsearch-8.13.2.jar:?]
    at java.util.AbstractList$Itr.next(AbstractList.java:373) ~[?:?]
    at org.elasticsearch.search.aggregations.InternalAggregations.reduce(InternalAggregations.java:282) ~[elasticsearch-8.13.2.jar:?]
    at org.elasticsearch.search.aggregations.InternalAggregations.topLevelReduce(InternalAggregations.java:234) ~[elasticsearch-8.13.2.jar:?]
    at org.elasticsearch.search.aggregations.InternalAggregations.topLevelReduceDelayable(InternalAggregations.java:222) ~[elasticsearch-8.13.2.jar:?]
    at org.elasticsearch.action.search.SearchPhaseController.reduceAggs(SearchPhaseController.java:669) ~[elasticsearch-8.13.2.jar:?]
    at org.elasticsearch.action.search.SearchPhaseController.reducedQueryPhase(SearchPhaseController.java:632) ~[elasticsearch-8.13.2.jar:?]
    at org.elasticsearch.action.search.QueryPhaseResultConsumer.reduce(QueryPhaseResultConsumer.java:139) ~[elasticsearch-8.13.2.jar:?]
    at org.elasticsearch.action.search.FetchSearchPhase.innerRun(FetchSearchPhase.java:99) ~[elasticsearch-8.13.2.jar:?]
    at org.elasticsearch.action.search.FetchSearchPhase$1.doRun(FetchSearchPhase.java:87) ~[elasticsearch-8.13.2.jar:?]
    at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) ~[elasticsearch-8.13.2.jar:?]
    ... 6 more
Caused by: java.lang.IllegalArgumentException: Unknown pattern letter: t
    at java.time.format.DateTimeFormatterBuilder.parsePattern(DateTimeFormatterBuilder.java:2009) ~[?:?]
    at java.time.format.DateTimeFormatterBuilder.appendPattern(DateTimeFormatterBuilder.java:1899) ~[?:?]
    at org.elasticsearch.common.time.DateFormatters.forPattern(DateFormatters.java:2126) ~[elasticsearch-8.13.2.jar:?]
    at org.elasticsearch.common.time.DateFormatter.forPattern(DateFormatter.java:126) ~[elasticsearch-8.13.2.jar:?]
    at org.elasticsearch.common.time.DateFormatter.forPattern(DateFormatter.java:104) ~[elasticsearch-8.13.2.jar:?]
    at org.elasticsearch.search.DocValueFormat$DateTime.<init>(DocValueFormat.java:240) ~[elasticsearch-8.13.2.jar:?]
    at org.elasticsearch.common.io.stream.NamedWriteableAwareStreamInput.readNamedWriteable(NamedWriteableAwareStreamInput.java:56) ~[elasticsearch-8.13.2.jar:?]
    at org.elasticsearch.common.io.stream.NamedWriteableAwareStreamInput.readNamedWriteable(NamedWriteableAwareStreamInput.java:32) ~[elasticsearch-8.13.2.jar:?]
    at org.elasticsearch.search.aggregations.metrics.InternalNumericMetricsAggregation.<init>(InternalNumericMetricsAggregation.java:146) ~[elasticsearch-8.13.2.jar:?]
    at org.elasticsearch.search.aggregations.metrics.InternalNumericMetricsAggregation.<init>(InternalNumericMetricsAggregation.java:136) ~[elasticsearch-8.13.2.jar:?]
    at org.elasticsearch.search.aggregations.metrics.InternalNumericMetricsAggregation$SingleValue.<init>(InternalNumericMetricsAggregation.java:37) ~[elasticsearch-8.13.2.jar:?]
    at org.elasticsearch.search.aggregations.metrics.Max.<init>(Max.java:47) ~[elasticsearch-8.13.2.jar:?]
    at org.elasticsearch.common.io.stream.NamedWriteableAwareStreamInput.readNamedWriteable(NamedWriteableAwareStreamInput.java:56) ~[elasticsearch-8.13.2.jar:?]
    at org.elasticsearch.common.io.stream.NamedWriteableAwareStreamInput.readNamedWriteable(NamedWriteableAwareStreamInput.java:32) ~[elasticsearch-8.13.2.jar:?]
    at org.elasticsearch.search.aggregations.InternalAggregations.lambda$readFrom$0(InternalAggregations.java:168) ~[elasticsearch-8.13.2.jar:?]
    at org.elasticsearch.common.io.stream.StreamInput.readCollection(StreamInput.java:1216) ~[elasticsearch-8.13.2.jar:?]
    at org.elasticsearch.common.io.stream.StreamInput.readCollectionAsList(StreamInput.java:1088) ~[elasticsearch-8.13.2.jar:?]
    at org.elasticsearch.search.aggregations.InternalAggregations.readFrom(InternalAggregations.java:168) ~[elasticsearch-8.13.2.jar:?]
    at org.elasticsearch.search.aggregations.bucket.composite.InternalComposite$InternalBucket.<init>(InternalComposite.java:386) ~[elasticsearch-8.13.2.jar:?]
    at org.elasticsearch.search.aggregations.bucket.composite.InternalComposite.lambda$new$1(InternalComposite.java:103) ~[elasticsearch-8.13.2.jar:?]
    at org.elasticsearch.common.io.stream.StreamInput.readCollection(StreamInput.java:1216) ~[elasticsearch-8.13.2.jar:?]
    at org.elasticsearch.common.io.stream.StreamInput.readCollectionAsList(StreamInput.java:1088) ~[elasticsearch-8.13.2.jar:?]
    at org.elasticsearch.search.aggregations.bucket.composite.InternalComposite.<init>(InternalComposite.java:103) ~[elasticsearch-8.13.2.jar:?]
    at org.elasticsearch.common.io.stream.NamedWriteableAwareStreamInput.readNamedWriteable(NamedWriteableAwareStreamInput.java:56) ~[elasticsearch-8.13.2.jar:?]
    at org.elasticsearch.common.io.stream.NamedWriteableAwareStreamInput.readNamedWriteable(NamedWriteableAwareStreamInput.java:32) ~[elasticsearch-8.13.2.jar:?]
    at org.elasticsearch.search.aggregations.InternalAggregations.lambda$readFrom$0(InternalAggregations.java:168) ~[elasticsearch-8.13.2.jar:?]
    at org.elasticsearch.common.io.stream.StreamInput.readCollection(StreamInput.java:1216) ~[elasticsearch-8.13.2.jar:?]
    at org.elasticsearch.common.io.stream.StreamInput.readCollectionAsList(StreamInput.java:1088) ~[elasticsearch-8.13.2.jar:?]
    at org.elasticsearch.search.aggregations.InternalAggregations.readFrom(InternalAggregations.java:168) ~[elasticsearch-8.13.2.jar:?]
    at org.elasticsearch.common.io.stream.DelayableWriteable.deserialize(DelayableWriteable.java:237) ~[elasticsearch-8.13.2.jar:?]
    at org.elasticsearch.common.io.stream.DelayableWriteable$Serialized.expand(DelayableWriteable.java:184) ~[elasticsearch-8.13.2.jar:?]
    at org.elasticsearch.search.aggregations.InternalAggregations$1.get(InternalAggregations.java:214) ~[elasticsearch-8.13.2.jar:?]
    at org.elasticsearch.search.aggregations.InternalAggregations$1.get(InternalAggregations.java:211) ~[elasticsearch-8.13.2.jar:?]
    at java.util.AbstractList$Itr.next(AbstractList.java:373) ~[?:?]
    at org.elasticsearch.search.aggregations.InternalAggregations.reduce(InternalAggregations.java:282) ~[elasticsearch-8.13.2.jar:?]
    at org.elasticsearch.search.aggregations.InternalAggregations.topLevelReduce(InternalAggregations.java:234) ~[elasticsearch-8.13.2.jar:?]
    at org.elasticsearch.search.aggregations.InternalAggregations.topLevelReduceDelayable(InternalAggregations.java:222) ~[elasticsearch-8.13.2.jar:?]
    at org.elasticsearch.action.search.SearchPhaseController.reduceAggs(SearchPhaseController.java:669) ~[elasticsearch-8.13.2.jar:?]
    at org.elasticsearch.action.search.SearchPhaseController.reducedQueryPhase(SearchPhaseController.java:632) ~[elasticsearch-8.13.2.jar:?]
    at org.elasticsearch.action.search.QueryPhaseResultConsumer.reduce(QueryPhaseResultConsumer.java:139) ~[elasticsearch-8.13.2.jar:?]
    at org.elasticsearch.action.search.FetchSearchPhase.innerRun(FetchSearchPhase.java:99) ~[elasticsearch-8.13.2.jar:?]
    at org.elasticsearch.action.search.FetchSearchPhase$1.doRun(FetchSearchPhase.java:87) ~[elasticsearch-8.13.2.jar:?]
    at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) ~[elasticsearch-8.13.2.jar:?]
    ... 6 more

we should use a Version parameter when creating a date time formatter in:

at org.elasticsearch.common.time.DateFormatter.forPattern(DateFormatter.java:104) ~[elasticsearch-8.13.2.jar:?]
    at org.elasticsearch.search.DocValueFormat$DateTime.<init>(DocValueFormat.java:240) ~[elasticsearch-8.13.2.jar:?]

Steps to Reproduce

n/a

Logs (if relevant)

n/a

elasticsearchmachine commented 2 months ago

Pinging @elastic/es-core-infra (Team:Core/Infra)