search

elastic / elasticsearch

Free and Open Source, Distributed, RESTful Search Engine
https://www.elastic.co/products/elasticsearch
Other
1.14k stars 24.84k forks source link

[CI] EsqlClientYamlIT test {p0=esql/60_usage/Basic ESQL usage output (telemetry)} failing #113502

Closed elasticsearchmachine closed 1 month ago

elasticsearchmachine commented 1 month ago

Build Scans:

Reproduction Line:

./gradlew ":x-pack:plugin:esql:qa:server:mixed-cluster:v8.16.0#yamlRestTest" -Dtests.class="org.elasticsearch.xpack.esql.qa.mixed.EsqlClientYamlIT" -Dtests.method="test {p0=esql/60_usage/Basic ESQL usage output (telemetry)}" -Dtests.seed=721C143E54D2988C -Dtests.bwc=true -Dtests.locale=no-NO -Dtests.timezone=Asia/Hovd -Druntime.java=22

Applicable branches: main

Reproduces locally?: N/A

Failure History: See dashboard&_a=(controlGroupInput:(chainingSystem:HIERARCHICAL,controlStyle:twoLine,ignoreParentSettings:(ignoreFilters:!f,ignoreQuery:!f,ignoreTimerange:!f,ignoreValidations:!t),panels:('0c0c9cb8-ccd2-45c6-9b13-96bac4abc542':(explicitInput:(dataViewId:fbbdc689-be23-4b3d-8057-aa402e9ed0c5,enhancements:(),fieldName:task.keyword,grow:!t,id:'0c0c9cb8-ccd2-45c6-9b13-96bac4abc542',searchTechnique:wildcard,selectedOptions:!(),singleSelect:!t,title:'Gradle%20Task',width:medium),grow:!t,order:0,type:optionsListControl,width:small),'144933da-5c1b-4257-a969-7f43455a7901':(explicitInput:(dataViewId:fbbdc689-be23-4b3d-8057-aa402e9ed0c5,enhancements:(),fieldName:name.keyword,grow:!t,id:'144933da-5c1b-4257-a969-7f43455a7901',searchTechnique:wildcard,selectedOptions:!('test%20%7Bp0%3Desql%2F60_usage%2FBasic%20ESQL%20usage%20output%20(telemetry)%7D'),title:Test,width:medium),grow:!t,order:2,type:optionsListControl,width:medium),'4e6ad9d6-6fdc-4fcc-bf1a-aa6ca79e0850':(explicitInput:(dataViewId:fbbdc689-be23-4b3d-8057-aa402e9ed0c5,enhancements:(),fieldName:className.keyword,grow:!t,id:'4e6ad9d6-6fdc-4fcc-bf1a-aa6ca79e0850',searchTechnique:wildcard,selectedOptions:!('org.elasticsearch.xpack.esql.qa.mixed.EsqlClientYamlIT'),title:Suite,width:medium),grow:!t,order:1,type:optionsListControl,width:medium)))))

Failure Message:

java.lang.AssertionError: Failure at [esql/60_usage:23]: expected [2xx] status code but api [xpack.usage] returned [500 Internal Server Error] [{"error":{"root_cause":[{"type":"transport_serialization_exception","reason":"Failed to deserialize response from handler [ContextRestoreResponseHandler[UnregisterChildTransportResponseHandler[unregisterChildNode=releaseOnce[org.elasticsearch.tasks.TaskManager$$Lambda/0x00007fe0c438edc0@369963e8], handler=[org.elasticsearch.action.ActionListenerImplementations$RunBeforeActionListener/org.elasticsearch.action.ActionListenerImplementations$RunBeforeActionListener/org.elasticsearch.tasks.TaskManager$1{org.elasticsearch.rest.action.RestCancellableNodeClient$1@59c9c58c}{CancellableTask{Task{id=5043, type='transport', action='cluster:monitor/xpack/usage', description='', parentTask=unset, startTime=1727238352068, headers={}, startTimeNanos=1194615182022}, reason='null', isCancelled=false}}/org.elasticsearch.action.support.TransportAction$$Lambda/0x00007fe0c4300b08@23e57dc3/org.elasticsearch.action.support.master.TransportMasterNodeAction$$Lambda/0x00007fe0c4302558@44595e7d], action=cluster:monitor/xpack/usage, childRequest=TermOverridingMasterNodeRequest[newMasterTerm=1 in org.elasticsearch.protocol.xpack.XPackUsageRequest/lyiBRlKgQWeftTd6cHSjmQ:5043], childConnection=NodeChannels[{test-cluster-2}{ARcOX2BwTHKDJkoYo8SGFQ}{zdsv6rkJQuijPsRgp554oA}{test-cluster-2}{127.0.0.1}{127.0.0.1:41691}{cdfhilmrstw}{8.16.0}{7000099-8515000}], taskManager=org.elasticsearch.tasks.TaskManager@1045c79e]]]","stack_trace":"org.elasticsearch.transport.TransportSerializationException: Failed to deserialize response from handler [ContextRestoreResponseHandler[UnregisterChildTransportResponseHandler[unregisterChildNode=releaseOnce[org.elasticsearch.tasks.TaskManager$$Lambda/0x00007fe0c438edc0@369963e8], handler=[org.elasticsearch.action.ActionListenerImplementations$RunBeforeActionListener/org.elasticsearch.action.ActionListenerImplementations$RunBeforeActionListener/org.elasticsearch.tasks.TaskManager$1{org.elasticsearch.rest.action.RestCancellableNodeClient$1@59c9c58c}{CancellableTask{Task{id=5043, type='transport', action='cluster:monitor/xpack/usage', description='', parentTask=unset, startTime=1727238352068, headers={}, startTimeNanos=1194615182022}, reason='null', isCancelled=false}}/org.elasticsearch.action.support.TransportAction$$Lambda/0x00007fe0c4300b08@23e57dc3/org.elasticsearch.action.support.master.TransportMasterNodeAction$$Lambda/0x00007fe0c4302558@44595e7d], action=cluster:monitor/xpack/usage, childRequest=TermOverridingMasterNodeRequest[newMasterTerm=1 in org.elasticsearch.protocol.xpack.XPackUsageRequest/lyiBRlKgQWeftTd6cHSjmQ:5043], childConnection=NodeChannels[{test-cluster-2}{ARcOX2BwTHKDJkoYo8SGFQ}{zdsv6rkJQuijPsRgp554oA}{test-cluster-2}{127.0.0.1}{127.0.0.1:41691}{cdfhilmrstw}{8.16.0}{7000099-8515000}], taskManager=org.elasticsearch.tasks.TaskManager@1045c79e]]]\n\tat org.elasticsearch.server@9.0.0-SNAPSHOT/org.elasticsearch.transport.InboundHandler.doHandleResponse(InboundHandler.java:422)\n\tat org.elasticsearch.server@9.0.0-SNAPSHOT/org.elasticsearch.transport.InboundHandler$2.doRun(InboundHandler.java:390)\n\tat org.elasticsearch.server@9.0.0-SNAPSHOT/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:992)\n\tat org.elasticsearch.server@9.0.0-SNAPSHOT/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:27)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)\n\tat java.base/java.lang.Thread.run(Thread.java:1570)\nCaused by: java.io.IOException: Can't read unknown type [111]\n\tat org.elasticsearch.server@9.0.0-SNAPSHOT/org.elasticsearch.common.io.stream.StreamInput.readGenericValue(StreamInput.java:887)\n\tat org.elasticsearch.server@9.0.0-SNAPSHOT/org.elasticsearch.common.io.stream.StreamInput.readMap(StreamInput.java:741)\n\tat org.elasticsearch.server@9.0.0-SNAPSHOT/org.elasticsearch.common.io.stream.StreamInput.readMap(StreamInput.java:726)\n\tat org.elasticsearch.server@9.0.0-SNAPSHOT/org.elasticsearch.common.io.stream.StreamInput.readGenericValue(StreamInput.java:862)\n\tat org.elasticsearch.server@9.0.0-SNAPSHOT/org.elasticsearch.common.io.stream.StreamInput.readGenericMap(StreamInput.java:789)\n\tat org.elasticsearch.xcore@9.0.0-SNAPSHOT/org.elasticsearch.xpack.core.ml.MachineLearningFeatureSetUsage.<init>(MachineLearningFeatureSetUsage.java:70)\n\tat org.elasticsearch.server@9.0.0-SNAPSHOT/org.elasticsearch.common.io.stream.NamedWriteableAwareStreamInput.readNamedWriteableCollectionAsList(NamedWriteableAwareStreamInput.java:46)\n\tat org.elasticsearch.xcore@9.0.0-SNAPSHOT/org.elasticsearch.xpack.core.action.XPackUsageResponse.<init>(XPackUsageResponse.java:27)\n\tat org.elasticsearch.server@9.0.0-SNAPSHOT/org.elasticsearch.action.ActionListenerResponseHandler.read(ActionListenerResponseHandler.java:64)\n\tat org.elasticsearch.server@9.0.0-SNAPSHOT/org.elasticsearch.action.ActionListenerResponseHandler.read(ActionListenerResponseHandler.java:26)\n\tat org.elasticsearch.server@9.0.0-SNAPSHOT/org.elasticsearch.transport.TransportService$UnregisterChildTransportResponseHandler.read(TransportService.java:1785)\n\tat org.elasticsearch.server@9.0.0-SNAPSHOT/org.elasticsearch.transport.TransportService$UnregisterChildTransportResponseHandler.read(TransportService.java:1754)\n\tat org.elasticsearch.server@9.0.0-SNAPSHOT/org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.read(TransportService.java:1489)\n\tat org.elasticsearch.server@9.0.0-SNAPSHOT/org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.read(TransportService.java:1476)\n\tat org.elasticsearch.server@9.0.0-SNAPSHOT/org.elasticsearch.transport.InboundHandler.doHandleResponse(InboundHandler.java:419)\n\t... 6 more\n"}],"type":"transport_serialization_exception","reason":"Failed to deserialize response from handler [ContextRestoreResponseHandler[UnregisterChildTransportResponseHandler[unregisterChildNode=releaseOnce[org.elasticsearch.tasks.TaskManager$$Lambda/0x00007fe0c438edc0@369963e8], handler=[org.elasticsearch.action.ActionListenerImplementations$RunBeforeActionListener/org.elasticsearch.action.ActionListenerImplementations$RunBeforeActionListener/org.elasticsearch.tasks.TaskManager$1{org.elasticsearch.rest.action.RestCancellableNodeClient$1@59c9c58c}{CancellableTask{Task{id=5043, type='transport', action='cluster:monitor/xpack/usage', description='', parentTask=unset, startTime=1727238352068, headers={}, startTimeNanos=1194615182022}, reason='null', isCancelled=false}}/org.elasticsearch.action.support.TransportAction$$Lambda/0x00007fe0c4300b08@23e57dc3/org.elasticsearch.action.support.master.TransportMasterNodeAction$$Lambda/0x00007fe0c4302558@44595e7d], action=cluster:monitor/xpack/usage, childRequest=TermOverridingMasterNodeRequest[newMasterTerm=1 in org.elasticsearch.protocol.xpack.XPackUsageRequest/lyiBRlKgQWeftTd6cHSjmQ:5043], childConnection=NodeChannels[{test-cluster-2}{ARcOX2BwTHKDJkoYo8SGFQ}{zdsv6rkJQuijPsRgp554oA}{test-cluster-2}{127.0.0.1}{127.0.0.1:41691}{cdfhilmrstw}{8.16.0}{7000099-8515000}], taskManager=org.elasticsearch.tasks.TaskManager@1045c79e]]]","caused_by":{"type":"i_o_exception","reason":"Can't read unknown type [111]","stack_trace":"java.io.IOException: Can't read unknown type [111]\n\tat org.elasticsearch.server@9.0.0-SNAPSHOT/org.elasticsearch.common.io.stream.StreamInput.readGenericValue(StreamInput.java:887)\n\tat org.elasticsearch.server@9.0.0-SNAPSHOT/org.elasticsearch.common.io.stream.StreamInput.readMap(StreamInput.java:741)\n\tat org.elasticsearch.server@9.0.0-SNAPSHOT/org.elasticsearch.common.io.stream.StreamInput.readMap(StreamInput.java:726)\n\tat org.elasticsearch.server@9.0.0-SNAPSHOT/org.elasticsearch.common.io.stream.StreamInput.readGenericValue(StreamInput.java:862)\n\tat org.elasticsearch.server@9.0.0-SNAPSHOT/org.elasticsearch.common.io.stream.StreamInput.readGenericMap(StreamInput.java:789)\n\tat org.elasticsearch.xcore@9.0.0-SNAPSHOT/org.elasticsearch.xpack.core.ml.MachineLearningFeatureSetUsage.<init>(MachineLearningFeatureSetUsage.java:70)\n\tat org.elasticsearch.server@9.0.0-SNAPSHOT/org.elasticsearch.common.io.stream.NamedWriteableAwareStreamInput.readNamedWriteableCollectionAsList(NamedWriteableAwareStreamInput.java:46)\n\tat org.elasticsearch.xcore@9.0.0-SNAPSHOT/org.elasticsearch.xpack.core.action.XPackUsageResponse.<init>(XPackUsageResponse.java:27)\n\tat org.elasticsearch.server@9.0.0-SNAPSHOT/org.elasticsearch.action.ActionListenerResponseHandler.read(ActionListenerResponseHandler.java:64)\n\tat org.elasticsearch.server@9.0.0-SNAPSHOT/org.elasticsearch.action.ActionListenerResponseHandler.read(ActionListenerResponseHandler.java:26)\n\tat org.elasticsearch.server@9.0.0-SNAPSHOT/org.elasticsearch.transport.TransportService$UnregisterChildTransportResponseHandler.read(TransportService.java:1785)\n\tat org.elasticsearch.server@9.0.0-SNAPSHOT/org.elasticsearch.transport.TransportService$UnregisterChildTransportResponseHandler.read(TransportService.java:1754)\n\tat org.elasticsearch.server@9.0.0-SNAPSHOT/org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.read(TransportService.java:1489)\n\tat org.elasticsearch.server@9.0.0-SNAPSHOT/org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.read(TransportService.java:1476)\n\tat org.elasticsearch.server@9.0.0-SNAPSHOT/org.elasticsearch.transport.InboundHandler.doHandleResponse(InboundHandler.java:419)\n\tat org.elasticsearch.server@9.0.0-SNAPSHOT/org.elasticsearch.transport.InboundHandler$2.doRun(InboundHandler.java:390)\n\tat org.elasticsearch.server@9.0.0-SNAPSHOT/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:992)\n\tat org.elasticsearch.server@9.0.0-SNAPSHOT/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:27)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)\n\tat java.base/java.lang.Thread.run(Thread.java:1570)\n"},"stack_trace":"org.elasticsearch.transport.TransportSerializationException: Failed to deserialize response from handler [ContextRestoreResponseHandler[UnregisterChildTransportResponseHandler[unregisterChildNode=releaseOnce[org.elasticsearch.tasks.TaskManager$$Lambda/0x00007fe0c438edc0@369963e8], handler=[org.elasticsearch.action.ActionListenerImplementations$RunBeforeActionListener/org.elasticsearch.action.ActionListenerImplementations$RunBeforeActionListener/org.elasticsearch.tasks.TaskManager$1{org.elasticsearch.rest.action.RestCancellableNodeClient$1@59c9c58c}{CancellableTask{Task{id=5043, type='transport', action='cluster:monitor/xpack/usage', description='', parentTask=unset, startTime=1727238352068, headers={}, startTimeNanos=1194615182022}, reason='null', isCancelled=false}}/org.elasticsearch.action.support.TransportAction$$Lambda/0x00007fe0c4300b08@23e57dc3/org.elasticsearch.action.support.master.TransportMasterNodeAction$$Lambda/0x00007fe0c4302558@44595e7d], action=cluster:monitor/xpack/usage, childRequest=TermOverridingMasterNodeRequest[newMasterTerm=1 in org.elasticsearch.protocol.xpack.XPackUsageRequest/lyiBRlKgQWeftTd6cHSjmQ:5043], childConnection=NodeChannels[{test-cluster-2}{ARcOX2BwTHKDJkoYo8SGFQ}{zdsv6rkJQuijPsRgp554oA}{test-cluster-2}{127.0.0.1}{127.0.0.1:41691}{cdfhilmrstw}{8.16.0}{7000099-8515000}], taskManager=org.elasticsearch.tasks.TaskManager@1045c79e]]]\n\tat org.elasticsearch.server@9.0.0-SNAPSHOT/org.elasticsearch.transport.InboundHandler.doHandleResponse(InboundHandler.java:422)\n\tat org.elasticsearch.server@9.0.0-SNAPSHOT/org.elasticsearch.transport.InboundHandler$2.doRun(InboundHandler.java:390)\n\tat org.elasticsearch.server@9.0.0-SNAPSHOT/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:992)\n\tat org.elasticsearch.server@9.0.0-SNAPSHOT/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:27)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)\n\tat java.base/java.lang.Thread.run(Thread.java:1570)\nCaused by: java.io.IOException: Can't read unknown type [111]\n\tat org.elasticsearch.server@9.0.0-SNAPSHOT/org.elasticsearch.common.io.stream.StreamInput.readGenericValue(StreamInput.java:887)\n\tat org.elasticsearch.server@9.0.0-SNAPSHOT/org.elasticsearch.common.io.stream.StreamInput.readMap(StreamInput.java:741)\n\tat org.elasticsearch.server@9.0.0-SNAPSHOT/org.elasticsearch.common.io.stream.StreamInput.readMap(StreamInput.java:726)\n\tat org.elasticsearch.server@9.0.0-SNAPSHOT/org.elasticsearch.common.io.stream.StreamInput.readGenericValue(StreamInput.java:862)\n\tat org.elasticsearch.server@9.0.0-SNAPSHOT/org.elasticsearch.common.io.stream.StreamInput.readGenericMap(StreamInput.java:789)\n\tat org.elasticsearch.xcore@9.0.0-SNAPSHOT/org.elasticsearch.xpack.core.ml.MachineLearningFeatureSetUsage.<init>(MachineLearningFeatureSetUsage.java:70)\n\tat org.elasticsearch.server@9.0.0-SNAPSHOT/org.elasticsearch.common.io.stream.NamedWriteableAwareStreamInput.readNamedWriteableCollectionAsList(NamedWriteableAwareStreamInput.java:46)\n\tat org.elasticsearch.xcore@9.0.0-SNAPSHOT/org.elasticsearch.xpack.core.action.XPackUsageResponse.<init>(XPackUsageResponse.java:27)\n\tat org.elasticsearch.server@9.0.0-SNAPSHOT/org.elasticsearch.action.ActionListenerResponseHandler.read(ActionListenerResponseHandler.java:64)\n\tat org.elasticsearch.server@9.0.0-SNAPSHOT/org.elasticsearch.action.ActionListenerResponseHandler.read(ActionListenerResponseHandler.java:26)\n\tat org.elasticsearch.server@9.0.0-SNAPSHOT/org.elasticsearch.transport.TransportService$UnregisterChildTransportResponseHandler.read(TransportService.java:1785)\n\tat org.elasticsearch.server@9.0.0-SNAPSHOT/org.elasticsearch.transport.TransportService$UnregisterChildTransportResponseHandler.read(TransportService.java:1754)\n\tat org.elasticsearch.server@9.0.0-SNAPSHOT/org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.read(TransportService.java:1489)\n\tat org.elasticsearch.server@9.0.0-SNAPSHOT/org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.read(TransportService.java:1476)\n\tat org.elasticsearch.server@9.0.0-SNAPSHOT/org.elasticsearch.transport.InboundHandler.doHandleResponse(InboundHandler.java:419)\n\t... 6 more\n"},"status":500}]

Issue Reasons:

Note: This issue was created using new test triage automation. Please report issues or feedback to es-delivery.

elasticsearchmachine commented 1 month ago

This has been muted on branch main

Mute Reasons:

Build Scans:

elasticsearchmachine commented 1 month ago

Pinging @elastic/es-analytical-engine (Team:Analytics)

iverase commented 1 month ago

I had a quick look and the error is:

stack_trace":"java.io.IOException: Can't read unknown type [111]\n\tat org.elasticsearch.server@9.0.0-SNAPSHOT
org.elasticsearch.common.io.stream.StreamInput.readGenericValue(StreamInput.java:887)\n\tat org.elasticsearch.server@9.0.0-SNAPSHOT
org.elasticsearch.common.io.stream.StreamInput.readMap(StreamInput.java:741)\n\tat org.elasticsearch.server@9.0.0-SNAPSHOT
org.elasticsearch.common.io.stream.StreamInput.readMap(StreamInput.java:726)\n\tat org.elasticsearch.server@9.0.0-SNAPSHOT
org.elasticsearch.common.io.stream.StreamInput.readGenericValue(StreamInput.java:862)\n\tat org.elasticsearch.server@9.0.0-SNAPSHOT
org.elasticsearch.common.io.stream.StreamInput.readGenericMap(StreamInput.java:789)\n\tat org.elasticsearch.xcore@9.0.0-SNAPSHOT
org.elasticsearch.xpack.core.ml.MachineLearningFeatureSetUsage.<init>(MachineLearningFeatureSetUsage.java:70)\n\tat org.elasticsearch.server@9.0.0-SNAPSHOT
org.elasticsearch.common.io.stream.NamedWriteableAwareStreamInput.readNamedWriteableCollectionAsList(NamedWriteableAwareStreamInput.java:46)\n\tat org.elasticsearch.xcore@9.0.0-SNAPSHOT

It seems some BWC issues coming from ML feature set?

iverase commented 1 month ago

For what I see the TransportVersions are lying. For what I see we are serializing MachineLearningFeatureSetUsage with 8.16 and deserializing with 9.0 but the transport version is providing the latest one.

thecoop commented 1 month ago

Currently 8.16 and 9.0 have the same transport version, as we need to keep 8.x and main identical until 8.17

iverase commented 1 month ago

I added low risk as it will be fixed shortly.

thecoop commented 1 month ago

For reference this was fixed by https://github.com/elastic/elasticsearch/pull/113516