elasticsearchmachine
commented
1 year ago Pinging @elastic/es-data-management (Team:Data Management)
Closed kingherc closed 1 year ago
elasticsearchmachine
commented
1 year ago Pinging @elastic/es-data-management (Team:Data Management)
gmarouli
commented
1 year ago This is a weird failure, I cannot reproduce it locally and I cannot figure out why would the template not be loaded byStackTemplateRegistry:
{
"index_patterns": ["logs-*-*"],
"priority": 100,
"data_stream": {},
"composed_of": [
"logs-mappings",
"logs-settings",
"logs@custom",
"ecs@dynamic_templates"
],
"ignore_missing_component_templates": ["logs@custom"],
"allow_auto_create": true,
"_meta": {
"description": "default logs template installed by x-pack",
"managed": true
},
"version": ${xpack.stack.template.version}
}I haven't found anything on the logs yet, but I will keep looking
DaveCTurner
commented
1 year ago Two more like this today:
The problem is I think that the index templates get installed asynchronously after the node has started up, and we start running tests before that's all finished:
[2023-09-13T12:11:05,586][INFO ][o.e.c.m.MetadataIndexTemplateService] [test-cluster-0] adding index template [metrics] for index patterns [metrics-*-*]
[2023-09-13T12:11:05,607][INFO ][o.e.c.m.MetadataIndexTemplateService] [test-cluster-0] adding index template [synthetics] for index patterns [synthetics-*-*]
[2023-09-13T12:11:05,621][INFO ][o.e.c.m.MetadataIndexTemplateService] [test-cluster-0] adding index template [.deprecation-indexing-template] for index patterns [.logs-deprecation.*]
[2023-09-13T12:11:05,677][INFO ][o.e.x.i.a.TransportPutLifecycleAction] [test-cluster-0] adding index lifecycle policy [.monitoring-8-ilm-policy]
↓↓↓ TEST FAILS HERE ↓↓↓
[2023-09-13T12:11:05,687][INFO ][o.e.d.DataStreamsClientYamlTestSuiteIT] [test] Dump test yaml [/var/lib/jenkins/workspace/elastic+elasticsearch+main+multijob+platform-support-unix/modules/data-streams/build/resources/yamlRestTest/rest-api-spec/test/data_stream/200_logs_datastream_defaults.yml] on failure:
[2023-09-13T12:11:05,695][INFO ][o.e.d.DataStreamsClientYamlTestSuiteIT] [test] Stash dump on test failure [{
↑↑↑ TEST FAILS HERE ↑↑↑
[2023-09-13T12:11:05,767][INFO ][o.e.x.i.a.TransportPutLifecycleAction] [test-cluster-0] adding index lifecycle policy [behavioral_analytics-events-default_policy]
[2023-09-13T12:11:05,848][INFO ][o.e.x.i.a.TransportPutLifecycleAction] [test-cluster-0] adding index lifecycle policy [ml-size-based-ilm-policy]
[2023-09-13T12:11:05,909][INFO ][o.e.x.i.a.TransportPutLifecycleAction] [test-cluster-0] adding index lifecycle policy [metrics]
[2023-09-13T12:11:05,929][INFO ][o.e.x.m.MachineLearning ] [test-cluster-0] Starting machine learning feature reset
[2023-09-13T12:11:05,950][INFO ][o.e.x.i.a.TransportPutLifecycleAction] [test-cluster-0] adding index lifecycle policy [7-days-default]
[2023-09-13T12:11:06,022][INFO ][o.e.x.i.a.TransportPutLifecycleAction] [test-cluster-0] adding index lifecycle policy [synthetics]
[2023-09-13T12:11:06,123][INFO ][o.e.x.c.t.IndexTemplateRegistry] [test-cluster-0] adding ingest pipeline behavioral_analytics-events-final_pipeline
[2023-09-13T12:11:06,123][INFO ][o.e.x.c.t.IndexTemplateRegistry] [test-cluster-0] adding ingest pipeline ent-search-generic-ingestion
[2023-09-13T12:11:06,124][INFO ][o.e.x.c.t.IndexTemplateRegistry] [test-cluster-0] adding ingest pipeline logs@json-message
[2023-09-13T12:11:06,130][INFO ][o.e.c.m.MetadataIndexTemplateService] [test-cluster-0] adding component template [behavioral_analytics-events-settings]
[2023-09-13T12:11:06,197][INFO ][o.e.c.m.MetadataIndexTemplateService] [test-cluster-0] adding index template [behavioral_analytics-events-default] for index patterns [behavioral_analytics-events-*]
[2023-09-13T12:11:06,239][INFO ][o.e.x.i.a.TransportPutLifecycleAction] [test-cluster-0] adding index lifecycle policy [30-days-default]
[2023-09-13T12:11:06,284][INFO ][o.e.x.i.a.TransportPutLifecycleAction] [test-cluster-0] adding index lifecycle policy [logs]
[2023-09-13T12:11:06,327][INFO ][o.e.x.i.a.TransportPutLifecycleAction] [test-cluster-0] adding index lifecycle policy [365-days-default]
[2023-09-13T12:11:06,364][INFO ][o.e.x.i.a.TransportPutLifecycleAction] [test-cluster-0] adding index lifecycle policy [180-days-default]
[2023-09-13T12:11:06,405][INFO ][o.e.x.i.a.TransportPutLifecycleAction] [test-cluster-0] adding index lifecycle policy [90-days-default]
[2023-09-13T12:11:06,447][INFO ][o.e.x.i.a.TransportPutLifecycleAction] [test-cluster-0] adding index lifecycle policy [slm-history-ilm-policy]
[2023-09-13T12:11:06,484][INFO ][o.e.x.i.a.TransportPutLifecycleAction] [test-cluster-0] adding index lifecycle policy [watch-history-ilm-policy-16]
[2023-09-13T12:11:06,525][INFO ][o.e.x.i.a.TransportPutLifecycleAction] [test-cluster-0] adding index lifecycle policy [.fleet-file-tohost-meta-ilm-policy]
[2023-09-13T12:11:06,561][INFO ][o.e.x.i.a.TransportPutLifecycleAction] [test-cluster-0] adding index lifecycle policy [.fleet-actions-results-ilm-policy]
[2023-09-13T12:11:06,593][INFO ][o.e.x.i.a.TransportPutLifecycleAction] [test-cluster-0] adding index lifecycle policy [.deprecation-indexing-ilm-policy]
[2023-09-13T12:11:06,632][INFO ][o.e.x.i.a.TransportPutLifecycleAction] [test-cluster-0] adding index lifecycle policy [.fleet-file-fromhost-data-ilm-policy]
[2023-09-13T12:11:06,667][INFO ][o.e.x.i.a.TransportPutLifecycleAction] [test-cluster-0] adding index lifecycle policy [.fleet-file-fromhost-meta-ilm-policy]
[2023-09-13T12:11:06,703][INFO ][o.e.x.i.a.TransportPutLifecycleAction] [test-cluster-0] adding index lifecycle policy [.fleet-file-tohost-data-ilm-policy]
[2023-09-13T12:11:06,828][INFO ][o.e.h.n.s.HealthNodeTaskExecutor] [test-cluster-0] Node [{test-cluster-0}{f-utzOFqQduw1rzY1XndlA}] is selected as the current health node.
[2023-09-13T12:11:06,951][INFO ][o.e.l.ClusterStateLicenseService] [test-cluster-0] license [85ae5aec-0054-4359-bf44-e2f047b533cb] mode [basic] - valid
[2023-09-13T12:11:06,953][INFO ][o.e.x.s.a.Realms ] [test-cluster-0] license mode is [basic], currently licensed security realms are [reserved/reserved,file/default_file,native/default_native]
[2023-09-13T12:11:07,097][INFO ][o.e.x.c.t.IndexTemplateRegistry] [test-cluster-0] adding ingest pipeline logs-default-pipeline
[2023-09-13T12:11:07,102][INFO ][o.e.c.m.MetadataIndexTemplateService] [test-cluster-0] adding component template [logs-settings]
↓↓↓ INDEX TEMPLATE ADDED HERE ↓↓↓
[2023-09-13T12:11:07,157][INFO ][o.e.c.m.MetadataIndexTemplateService] [test-cluster-0] adding index template [logs] for index patterns [logs-*-*]
↑↑↑ INDEX TEMPLATE ADDED HERE ↑↑↑
[2023-09-13T12:11:07,304][INFO ][o.e.x.m.MachineLearning ] [test-cluster-0] Finished machine learning feature reset
pgomulka
commented
1 year ago another one with the same exception mesage https://gradle-enterprise.elastic.co/s/2iawhjks6motq/tests/task/:modules:data-streams:yamlRestTest/details/org.elasticsearch.datastreams.DataStreamsClientYamlTestSuiteIT/test%20%7Bp0=data_stream%2F220_logs_default_pipeline%2FTest%20default%20logs-*-*%20pipeline%7D?top-execution=1
luigidellaquila
commented
1 year ago 
masseyke
commented
1 year ago Is this a duplicate of #98163? That there's nothing that stops the server from coming up before all templates are ready?
Build scan: https://gradle-enterprise.elastic.co/s/56ydgs7cr5vyc/tests/:modules:data-streams:yamlRestTest/org.elasticsearch.datastreams.DataStreamsClientYamlTestSuiteIT/test%20%7Bp0=data_stream%2F240_logs_ecs_mappings%2FTest%20Elastic%20Agent%20log%20ECS%20mappings%7D
Reproduction line:
Applicable branches: 8.9
Reproduces locally?: Didn't try
Failure history: https://gradle-enterprise.elastic.co/scans/tests?tests.container=org.elasticsearch.datastreams.DataStreamsClientYamlTestSuiteIT&tests.test=test%20%7Bp0%3Ddata_stream/240_logs_ecs_mappings/Test%20Elastic%20Agent%20log%20ECS%20mappings%7D
Failure excerpt: