search

elastic / elasticsearch

Free and Open, Distributed, RESTful Search Engine
https://www.elastic.co/products/elasticsearch
Other
69.51k stars 24.6k forks source link

[CI] IngestCommonClientYamlTestSuiteIT test {yaml=ingest/310_reroute_processor/Test first matching router terminates pipeline} failing #102144

Open thecoop opened 10 months ago

thecoop commented 10 months ago

Build scan: https://gradle-enterprise.elastic.co/s/taoppoclp6vds/tests/:modules:ingest-common:yamlRestTest/org.elasticsearch.ingest.common.IngestCommonClientYamlTestSuiteIT/test%20%7Byaml=ingest%2F310_reroute_processor%2FTest%20first%20matching%20router%20terminates%20pipeline%7D Reproduction line:

./gradlew ':modules:ingest-common:yamlRestTest' --tests "org.elasticsearch.ingest.common.IngestCommonClientYamlTestSuiteIT.test {yaml=ingest/310_reroute_processor/Test first matching router terminates pipeline}" -Dtests.seed=69FAFA380C29E77F -Dtests.locale=es-CU -Dtests.timezone=Atlantic/Faeroe -Druntime.java=21

Applicable branches: main

Reproduces locally?: No

Failure history: https://gradle-enterprise.elastic.co/scans/tests?tests.container=org.elasticsearch.ingest.common.IngestCommonClientYamlTestSuiteIT&tests.test=test%20%7Byaml%3Dingest/310_reroute_processor/Test%20first%20matching%20router%20terminates%20pipeline%7D

Failure excerpt:

java.lang.AssertionError: Failure at [ingest/310_reroute_processor:46]: expected [2xx] status code but api [index] returned [400 Bad Request] [{"error":{"root_cause":[{"type":"document_parsing_exception","reason":"[1:83] failed to parse: data stream timestamp field [@timestamp] is missing","stack_trace":"org.elasticsearch.index.mapper.DocumentParsingException: [1:83] failed to parse: data stream timestamp field [@timestamp] is missing\n\tat org.elasticsearch.index.mapper.DocumentParser.wrapInDocumentParsingException(DocumentParser.java:246)\n\tat org.elasticsearch.index.mapper.DocumentParser.internalParseDocument(DocumentParser.java:153)\n\tat org.elasticsearch.index.mapper.DocumentParser.parseDocument(DocumentParser.java:96)\n\tat org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:92)\n\tat org.elasticsearch.index.shard.IndexShard.prepareIndex(IndexShard.java:1012)\n\tat org.elasticsearch.index.shard.IndexShard.applyIndexOperation(IndexShard.java:960)\n\tat org.elasticsearch.index.shard.IndexShard.applyIndexOperationOnPrimary(IndexShard.java:904)\n\tat org.elasticsearch.action.bulk.TransportShardBulkAction.executeBulkItemRequest(TransportShardBulkAction.java:360)\n\tat org.elasticsearch.action.bulk.TransportShardBulkAction$2.doRun(TransportShardBulkAction.java:224)\n\tat org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)\n\tat org.elasticsearch.action.bulk.TransportShardBulkAction.performOnPrimary(TransportShardBulkAction.java:291)\n\tat org.elasticsearch.action.bulk.TransportShardBulkAction.dispatchedShardOperationOnPrimary(TransportShardBulkAction.java:142)\n\tat org.elasticsearch.action.bulk.TransportShardBulkAction.dispatchedShardOperationOnPrimary(TransportShardBulkAction.java:74)\n\tat org.elasticsearch.action.support.replication.TransportWriteAction$1.doRun(TransportWriteAction.java:216)\n\tat org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)\n\tat org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33)\n\tat org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:983)\n\tat org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)\n\tat java.lang.Thread.run(Thread.java:1583)\nCaused by: java.lang.IllegalArgumentException: data stream timestamp field [@timestamp] is missing\n\tat org.elasticsearch.index.mapper.DataStreamTimestampFieldMapper.postParse(DataStreamTimestampFieldMapper.java:218)\n\tat org.elasticsearch.index.mapper.DocumentParser.internalParseDocument(DocumentParser.java:150)\n\t... 19 more\n"}],"type":"document_parsing_exception","reason":"[1:83] failed to parse: data stream timestamp field [@timestamp] is missing","caused_by":{"type":"illegal_argument_exception","reason":"data stream timestamp field [@timestamp] is missing","stack_trace":"java.lang.IllegalArgumentException: data stream timestamp field [@timestamp] is missing\n\tat org.elasticsearch.index.mapper.DataStreamTimestampFieldMapper.postParse(DataStreamTimestampFieldMapper.java:218)\n\tat org.elasticsearch.index.mapper.DocumentParser.internalParseDocument(DocumentParser.java:150)\n\tat org.elasticsearch.index.mapper.DocumentParser.parseDocument(DocumentParser.java:96)\n\tat org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:92)\n\tat org.elasticsearch.index.shard.IndexShard.prepareIndex(IndexShard.java:1012)\n\tat org.elasticsearch.index.shard.IndexShard.applyIndexOperation(IndexShard.java:960)\n\tat org.elasticsearch.index.shard.IndexShard.applyIndexOperationOnPrimary(IndexShard.java:904)\n\tat org.elasticsearch.action.bulk.TransportShardBulkAction.executeBulkItemRequest(TransportShardBulkAction.java:360)\n\tat org.elasticsearch.action.bulk.TransportShardBulkAction$2.doRun(TransportShardBulkAction.java:224)\n\tat org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)\n\tat org.elasticsearch.action.bulk.TransportShardBulkAction.performOnPrimary(TransportShardBulkAction.java:291)\n\tat org.elasticsearch.action.bulk.TransportShardBulkAction.dispatchedShardOperationOnPrimary(TransportShardBulkAction.java:142)\n\tat org.elasticsearch.action.bulk.TransportShardBulkAction.dispatchedShardOperationOnPrimary(TransportShardBulkAction.java:74)\n\tat org.elasticsearch.action.support.replication.TransportWriteAction$1.doRun(TransportWriteAction.java:216)\n\tat org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)\n\tat org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33)\n\tat org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:983)\n\tat org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)\n\tat java.lang.Thread.run(Thread.java:1583)\n"},"stack_trace":"org.elasticsearch.index.mapper.DocumentParsingException: [1:83] failed to parse: data stream timestamp field [@timestamp] is missing\n\tat org.elasticsearch.index.mapper.DocumentParser.wrapInDocumentParsingException(DocumentParser.java:246)\n\tat org.elasticsearch.index.mapper.DocumentParser.internalParseDocument(DocumentParser.java:153)\n\tat org.elasticsearch.index.mapper.DocumentParser.parseDocument(DocumentParser.java:96)\n\tat org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:92)\n\tat org.elasticsearch.index.shard.IndexShard.prepareIndex(IndexShard.java:1012)\n\tat org.elasticsearch.index.shard.IndexShard.applyIndexOperation(IndexShard.java:960)\n\tat org.elasticsearch.index.shard.IndexShard.applyIndexOperationOnPrimary(IndexShard.java:904)\n\tat org.elasticsearch.action.bulk.TransportShardBulkAction.executeBulkItemRequest(TransportShardBulkAction.java:360)\n\tat org.elasticsearch.action.bulk.TransportShardBulkAction$2.doRun(TransportShardBulkAction.java:224)\n\tat org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)\n\tat org.elasticsearch.action.bulk.TransportShardBulkAction.performOnPrimary(TransportShardBulkAction.java:291)\n\tat org.elasticsearch.action.bulk.TransportShardBulkAction.dispatchedShardOperationOnPrimary(TransportShardBulkAction.java:142)\n\tat org.elasticsearch.action.bulk.TransportShardBulkAction.dispatchedShardOperationOnPrimary(TransportShardBulkAction.java:74)\n\tat org.elasticsearch.action.support.replication.TransportWriteAction$1.doRun(TransportWriteAction.java:216)\n\tat org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)\n\tat org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33)\n\tat org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:983)\n\tat org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)\n\tat java.lang.Thread.run(Thread.java:1583)\nCaused by: java.lang.IllegalArgumentException: data stream timestamp field [@timestamp] is missing\n\tat org.elasticsearch.index.mapper.DataStreamTimestampFieldMapper.postParse(DataStreamTimestampFieldMapper.java:218)\n\tat org.elasticsearch.index.mapper.DocumentParser.internalParseDocument(DocumentParser.java:150)\n\t... 19 more\n"},"status":400}]

  at __randomizedtesting.SeedInfo.seed([69FAFA380C29E77F:E1AEC5E2A2D58A87]:0)
  at org.elasticsearch.test.rest.yaml.ESClientYamlSuiteTestCase.executeSection(ESClientYamlSuiteTestCase.java:572)
  at org.elasticsearch.test.rest.yaml.ESClientYamlSuiteTestCase.test(ESClientYamlSuiteTestCase.java:524)
  at jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
  at java.lang.reflect.Method.invoke(Method.java:580)
  at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1758)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:946)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:982)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:996)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:48)
  at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  at org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45)
  at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
  at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:843)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:490)
  at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:955)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:840)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:891)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:902)
  at org.elasticsearch.test.cluster.local.DefaultLocalElasticsearchCluster$1.evaluate(DefaultLocalElasticsearchCluster.java:47)
  at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
  at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
  at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
  at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
  at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
  at org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:850)
  at java.lang.Thread.run(Thread.java:1583)

  Caused by: java.lang.AssertionError: expected [2xx] status code but api [index] returned [400 Bad Request] [{"error":{"root_cause":[{"type":"document_parsing_exception","reason":"[1:83] failed to parse: data stream timestamp field [@timestamp] is missing","stack_trace":"org.elasticsearch.index.mapper.DocumentParsingException: [1:83] failed to parse: data stream timestamp field [@timestamp] is missing\n\tat org.elasticsearch.index.mapper.DocumentParser.wrapInDocumentParsingException(DocumentParser.java:246)\n\tat org.elasticsearch.index.mapper.DocumentParser.internalParseDocument(DocumentParser.java:153)\n\tat org.elasticsearch.index.mapper.DocumentParser.parseDocument(DocumentParser.java:96)\n\tat org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:92)\n\tat org.elasticsearch.index.shard.IndexShard.prepareIndex(IndexShard.java:1012)\n\tat org.elasticsearch.index.shard.IndexShard.applyIndexOperation(IndexShard.java:960)\n\tat org.elasticsearch.index.shard.IndexShard.applyIndexOperationOnPrimary(IndexShard.java:904)\n\tat org.elasticsearch.action.bulk.TransportShardBulkAction.executeBulkItemRequest(TransportShardBulkAction.java:360)\n\tat org.elasticsearch.action.bulk.TransportShardBulkAction$2.doRun(TransportShardBulkAction.java:224)\n\tat org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)\n\tat org.elasticsearch.action.bulk.TransportShardBulkAction.performOnPrimary(TransportShardBulkAction.java:291)\n\tat org.elasticsearch.action.bulk.TransportShardBulkAction.dispatchedShardOperationOnPrimary(TransportShardBulkAction.java:142)\n\tat org.elasticsearch.action.bulk.TransportShardBulkAction.dispatchedShardOperationOnPrimary(TransportShardBulkAction.java:74)\n\tat org.elasticsearch.action.support.replication.TransportWriteAction$1.doRun(TransportWriteAction.java:216)\n\tat org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)\n\tat org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33)\n\tat org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:983)\n\tat org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)\n\tat java.lang.Thread.run(Thread.java:1583)\nCaused by: java.lang.IllegalArgumentException: data stream timestamp field [@timestamp] is missing\n\tat org.elasticsearch.index.mapper.DataStreamTimestampFieldMapper.postParse(DataStreamTimestampFieldMapper.java:218)\n\tat org.elasticsearch.index.mapper.DocumentParser.internalParseDocument(DocumentParser.java:150)\n\t... 19 more\n"}],"type":"document_parsing_exception","reason":"[1:83] failed to parse: data stream timestamp field [@timestamp] is missing","caused_by":{"type":"illegal_argument_exception","reason":"data stream timestamp field [@timestamp] is missing","stack_trace":"java.lang.IllegalArgumentException: data stream timestamp field [@timestamp] is missing\n\tat org.elasticsearch.index.mapper.DataStreamTimestampFieldMapper.postParse(DataStreamTimestampFieldMapper.java:218)\n\tat org.elasticsearch.index.mapper.DocumentParser.internalParseDocument(DocumentParser.java:150)\n\tat org.elasticsearch.index.mapper.DocumentParser.parseDocument(DocumentParser.java:96)\n\tat org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:92)\n\tat org.elasticsearch.index.shard.IndexShard.prepareIndex(IndexShard.java:1012)\n\tat org.elasticsearch.index.shard.IndexShard.applyIndexOperation(IndexShard.java:960)\n\tat org.elasticsearch.index.shard.IndexShard.applyIndexOperationOnPrimary(IndexShard.java:904)\n\tat org.elasticsearch.action.bulk.TransportShardBulkAction.executeBulkItemRequest(TransportShardBulkAction.java:360)\n\tat org.elasticsearch.action.bulk.TransportShardBulkAction$2.doRun(TransportShardBulkAction.java:224)\n\tat org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)\n\tat org.elasticsearch.action.bulk.TransportShardBulkAction.performOnPrimary(TransportShardBulkAction.java:291)\n\tat org.elasticsearch.action.bulk.TransportShardBulkAction.dispatchedShardOperationOnPrimary(TransportShardBulkAction.java:142)\n\tat org.elasticsearch.action.bulk.TransportShardBulkAction.dispatchedShardOperationOnPrimary(TransportShardBulkAction.java:74)\n\tat org.elasticsearch.action.support.replication.TransportWriteAction$1.doRun(TransportWriteAction.java:216)\n\tat org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)\n\tat org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33)\n\tat org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:983)\n\tat org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)\n\tat java.lang.Thread.run(Thread.java:1583)\n"},"stack_trace":"org.elasticsearch.index.mapper.DocumentParsingException: [1:83] failed to parse: data stream timestamp field [@timestamp] is missing\n\tat org.elasticsearch.index.mapper.DocumentParser.wrapInDocumentParsingException(DocumentParser.java:246)\n\tat org.elasticsearch.index.mapper.DocumentParser.internalParseDocument(DocumentParser.java:153)\n\tat org.elasticsearch.index.mapper.DocumentParser.parseDocument(DocumentParser.java:96)\n\tat org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:92)\n\tat org.elasticsearch.index.shard.IndexShard.prepareIndex(IndexShard.java:1012)\n\tat org.elasticsearch.index.shard.IndexShard.applyIndexOperation(IndexShard.java:960)\n\tat org.elasticsearch.index.shard.IndexShard.applyIndexOperationOnPrimary(IndexShard.java:904)\n\tat org.elasticsearch.action.bulk.TransportShardBulkAction.executeBulkItemRequest(TransportShardBulkAction.java:360)\n\tat org.elasticsearch.action.bulk.TransportShardBulkAction$2.doRun(TransportShardBulkAction.java:224)\n\tat org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)\n\tat org.elasticsearch.action.bulk.TransportShardBulkAction.performOnPrimary(TransportShardBulkAction.java:291)\n\tat org.elasticsearch.action.bulk.TransportShardBulkAction.dispatchedShardOperationOnPrimary(TransportShardBulkAction.java:142)\n\tat org.elasticsearch.action.bulk.TransportShardBulkAction.dispatchedShardOperationOnPrimary(TransportShardBulkAction.java:74)\n\tat org.elasticsearch.action.support.replication.TransportWriteAction$1.doRun(TransportWriteAction.java:216)\n\tat org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)\n\tat org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33)\n\tat org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:983)\n\tat org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)\n\tat java.lang.Thread.run(Thread.java:1583)\nCaused by: java.lang.IllegalArgumentException: data stream timestamp field [@timestamp] is missing\n\tat org.elasticsearch.index.mapper.DataStreamTimestampFieldMapper.postParse(DataStreamTimestampFieldMapper.java:218)\n\tat org.elasticsearch.index.mapper.DocumentParser.internalParseDocument(DocumentParser.java:150)\n\t... 19 more\n"},"status":400}]

    at org.junit.Assert.fail(Assert.java:88)
    at org.elasticsearch.test.rest.yaml.section.DoSection.execute(DoSection.java:374)
    at org.elasticsearch.test.rest.yaml.ESClientYamlSuiteTestCase.executeSection(ESClientYamlSuiteTestCase.java:552)
    at org.elasticsearch.test.rest.yaml.ESClientYamlSuiteTestCase.test(ESClientYamlSuiteTestCase.java:524)
    at jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
    at java.lang.reflect.Method.invoke(Method.java:580)
    at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1758)
    at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:946)
    at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:982)
    at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:996)
    at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
    at org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:48)
    at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
    at org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45)
    at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
    at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
    at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
    at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
    at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:843)
    at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:490)
    at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:955)
    at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:840)
    at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:891)
    at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:902)
    at org.elasticsearch.test.cluster.local.DefaultLocalElasticsearchCluster$1.evaluate(DefaultLocalElasticsearchCluster.java:47)
    at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
    at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
    at org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
    at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
    at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
    at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
    at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
    at org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
    at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
    at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
    at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
    at org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
    at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
    at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
    at com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:850)
    at java.lang.Thread.run(Thread.java:1583)
elasticsearchmachine commented 10 months ago

Pinging @elastic/es-data-management (Team:Data Management)

masseyke commented 10 months ago

Here's the formatted version of that returned stack trace:

i [index] returned [400 Bad Request] [{"error":{"root_cause":[{"type":"document_parsing_exception","reason":"[1:83] failed to parse: data stream timestamp field [@timestamp] is missing","stack_trace":"org.elasticsearch.index.mapper.DocumentParsingException: [1:83] failed to parse: data stream timestamp field [@timestamp] is missing
        at org.elasticsearch.index.mapper.DocumentParser.wrapInDocumentParsingException(DocumentParser.java:246)
        at org.elasticsearch.index.mapper.DocumentParser.internalParseDocument(DocumentParser.java:153)
        at org.elasticsearch.index.mapper.DocumentParser.parseDocument(DocumentParser.java:96)
        at org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:92)
        at org.elasticsearch.index.shard.IndexShard.prepareIndex(IndexShard.java:1012)
        at org.elasticsearch.index.shard.IndexShard.applyIndexOperation(IndexShard.java:960)
        at org.elasticsearch.index.shard.IndexShard.applyIndexOperationOnPrimary(IndexShard.java:904)
        at org.elasticsearch.action.bulk.TransportShardBulkAction.executeBulkItemRequest(TransportShardBulkAction.java:360)
        at org.elasticsearch.action.bulk.TransportShardBulkAction$2.doRun(TransportShardBulkAction.java:224)
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
        at org.elasticsearch.action.bulk.TransportShardBulkAction.performOnPrimary(TransportShardBulkAction.java:291)
        at org.elasticsearch.action.bulk.TransportShardBulkAction.dispatchedShardOperationOnPrimary(TransportShardBulkAction.java:142)
        at org.elasticsearch.action.bulk.TransportShardBulkAction.dispatchedShardOperationOnPrimary(TransportShardBulkAction.java:74)
        at org.elasticsearch.action.support.replication.TransportWriteAction$1.doRun(TransportWriteAction.java:216)
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
        at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33)
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:983)
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
        at java.lang.Thread.run(Thread.java:1583)
Caused by: java.lang.IllegalArgumentException: data stream timestamp field [@timestamp] is missing
        at org.elasticsearch.index.mapper.DataStreamTimestampFieldMapper.postParse(DataStreamTimestampFieldMapper.java:218)
        at org.elasticsearch.index.mapper.DocumentParser.internalParseDocument(DocumentParser.java:150)
        ... 19 more
"}],"type":"document_parsing_exception","reason":"[1:83] failed to parse: data stream timestamp field [@timestamp] is missing","caused_by":{"type":"illegal_argument_exception","reason":"data stream timestamp field [@timestamp] is missing","stack_trace":"java.lang.IllegalArgumentException: data stream timestamp field [@timestamp] is missing
        at org.elasticsearch.index.mapper.DataStreamTimestampFieldMapper.postParse(DataStreamTimestampFieldMapper.java:218)
        at org.elasticsearch.index.mapper.DocumentParser.internalParseDocument(DocumentParser.java:150)
        at org.elasticsearch.index.mapper.DocumentParser.parseDocument(DocumentParser.java:96)
        at org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:92)
        at org.elasticsearch.index.shard.IndexShard.prepareIndex(IndexShard.java:1012)
        at org.elasticsearch.index.shard.IndexShard.applyIndexOperation(IndexShard.java:960)
        at org.elasticsearch.index.shard.IndexShard.applyIndexOperationOnPrimary(IndexShard.java:904)
        at org.elasticsearch.action.bulk.TransportShardBulkAction.executeBulkItemRequest(TransportShardBulkAction.java:360)
        at org.elasticsearch.action.bulk.TransportShardBulkAction$2.doRun(TransportShardBulkAction.java:224)
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
        at org.elasticsearch.action.bulk.TransportShardBulkAction.performOnPrimary(TransportShardBulkAction.java:291)
        at org.elasticsearch.action.bulk.TransportShardBulkAction.dispatchedShardOperationOnPrimary(TransportShardBulkAction.java:142)
        at org.elasticsearch.action.bulk.TransportShardBulkAction.dispatchedShardOperationOnPrimary(TransportShardBulkAction.java:74)
        at org.elasticsearch.action.support.replication.TransportWriteAction$1.doRun(TransportWriteAction.java:216)
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
        at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33)
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:983)
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
        at java.lang.Thread.run(Thread.java:1583)
"},"stack_trace":"org.elasticsearch.index.mapper.DocumentParsingException: [1:83] failed to parse: data stream timestamp field [@timestamp] is missing
        at org.elasticsearch.index.mapper.DocumentParser.wrapInDocumentParsingException(DocumentParser.java:246)
        at org.elasticsearch.index.mapper.DocumentParser.internalParseDocument(DocumentParser.java:153)
        at org.elasticsearch.index.mapper.DocumentParser.parseDocument(DocumentParser.java:96)
        at org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:92)
        at org.elasticsearch.index.shard.IndexShard.prepareIndex(IndexShard.java:1012)
        at org.elasticsearch.index.shard.IndexShard.applyIndexOperation(IndexShard.java:960)
        at org.elasticsearch.index.shard.IndexShard.applyIndexOperationOnPrimary(IndexShard.java:904)
        at org.elasticsearch.action.bulk.TransportShardBulkAction.executeBulkItemRequest(TransportShardBulkAction.java:360)
        at org.elasticsearch.action.bulk.TransportShardBulkAction$2.doRun(TransportShardBulkAction.java:224)
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
        at org.elasticsearch.action.bulk.TransportShardBulkAction.performOnPrimary(TransportShardBulkAction.java:291)
        at org.elasticsearch.action.bulk.TransportShardBulkAction.dispatchedShardOperationOnPrimary(TransportShardBulkAction.java:142)
        at org.elasticsearch.action.bulk.TransportShardBulkAction.dispatchedShardOperationOnPrimary(TransportShardBulkAction.java:74)
        at org.elasticsearch.action.support.replication.TransportWriteAction$1.doRun(TransportWriteAction.java:216)
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
        at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33)
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:983)
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
        at java.lang.Thread.run(Thread.java:1583)
Caused by: java.lang.IllegalArgumentException: data stream timestamp field [@timestamp] is missing
        at org.elasticsearch.index.mapper.DataStreamTimestampFieldMapper.postParse(DataStreamTimestampFieldMapper.java:218)
        at org.elasticsearch.index.mapper.DocumentParser.internalParseDocument(DocumentParser.java:150)
        ... 19 more
"},"status":400}]
masseyke commented 10 months ago

Here is what this test is supposed to be doing, assuming that the template in logs@template.json has been loaded in time (sometimes it hasn't been, and so the flow is a little different. But in this case since we're seeing datastream validation we can be sure it has been): It ingests a simple document into the index logs-generic-default using the pipeline pipeline-with-two-data-stream-processors defined in the test. That pipeline has a reroute processor that reroutes it to an index called logs-first-default. That index matches the logs template from logs@template.json. So it has a default pipeline defined in logs@default-pipeline.json, referenced from logs@settings.json, loaded as part of the logs template. That default pipeline has a set processor that sets @timestamp:

      "set": {
        "description": "If '@timestamp' is missing, set it with the ingest timestamp",
        "field": "@timestamp",
        "override": false,
        "copy_from": "_ingest.timestamp"
      }

In this case, something is going wrong. I can see from the logs that logs-first-default is matches the logs template. And I can see that logs@template.json, logs@settings.json, and logs@default-pipeline.json have all been loaded. But somehow either that set processor is not executing, or _ingest.timestamp is null. Maybe a race condition where the logs@default-pipeline.json pipeline is not yet in the cluster state on the ingest node?

jakelandis commented 10 months ago

another instance : https://gradle-enterprise.elastic.co/s/q4grqsnwwmci4/tests/task/:modules:ingest-common:yamlRestTest/details/org.elasticsearch.ingest.common.IngestCommonClientYamlTestSuiteIT/test%20%7Byaml%3Dingest%2F310_reroute_processor%2FTest%20first%20matching%20router%20terminates%20pipeline%7D?top-execution=1

masseyke commented 8 months ago

From my comment above "But somehow either that set processor is not executing, or _ingest.timestamp is null". I don't think it is possible for _ingest.timestamp to be null, because it is set to ZonedDateTime.now(ZoneOffset.UTC) in IngestDocument's constructor, and is never altered or removed. So it does seem most likely that the set processor is just not being executed, meaning that logs@default-pipeline is somehow not there.

masseyke commented 8 months ago

Actually my last comment was wrong. I can reproduce this if I remove just the set processor from logs@default-pipeline.json. But if I don't load that pipeline at all, the test passes because DataStreamTimestampFieldMapper.enabled is false, so it doesn't check for the existence of @timestamp. It also doesn't fail in this way if _ingest.timestamp does not exist when the set processor runs -- it fails in a different way. The only way I've been able to get it to fail is to not run the set processor. That doesn't seem possible though. But I also don't know how _ingest.timestamp could be null though. So I'm stumped on this one.

masseyke commented 8 months ago

One last note for myself (because I'm putting this ticket aside again for a little while). I had to put a Thread.sleep() call into ESClientYamlSuiteTestCase.test() so that all templates are loaded before the test runs, so that the data stream gets created (which is what happened in the failure). The fact that the server is available before all templates have been loaded is a known issues tracked elsewhere, and is not directly related to the bug reported here.