search

elastic / elasticsearch

Free and Open Source, Distributed, RESTful Search Engine
https://www.elastic.co/products/elasticsearch
Other
1.13k stars 24.84k forks source link

[CI] MLModelDeploymentFullClusterRestartIT testDeploymentSurvivesRestart {cluster=UPGRADED} failing #112980

Closed elasticsearchmachine closed 3 weeks ago

elasticsearchmachine commented 1 month ago

Build Scans:

Reproduction Line:

./gradlew ":x-pack:qa:full-cluster-restart:v8.0.1#bwcTest" -Dtests.class="org.elasticsearch.xpack.restart.MLModelDeploymentFullClusterRestartIT" -Dtests.method="testDeploymentSurvivesRestart {cluster=UPGRADED}" -Dtests.seed=A7BE2CA36E251E1E -Dtests.bwc=true -Dtests.locale=af-ZA -Dtests.timezone=Antarctica/South_Pole -Druntime.java=22

Applicable branches: main

Reproduces locally?: N/A

Failure History: See dashboard&_a=(controlGroupInput:(chainingSystem:HIERARCHICAL,controlStyle:twoLine,ignoreParentSettings:(ignoreFilters:!f,ignoreQuery:!f,ignoreTimerange:!f,ignoreValidations:!t),panels:('0c0c9cb8-ccd2-45c6-9b13-96bac4abc542':(explicitInput:(dataViewId:fbbdc689-be23-4b3d-8057-aa402e9ed0c5,enhancements:(),fieldName:task.keyword,grow:!t,id:'0c0c9cb8-ccd2-45c6-9b13-96bac4abc542',searchTechnique:wildcard,selectedOptions:!(),singleSelect:!t,title:'Gradle%20Task',width:medium),grow:!t,order:0,type:optionsListControl,width:small),'144933da-5c1b-4257-a969-7f43455a7901':(explicitInput:(dataViewId:fbbdc689-be23-4b3d-8057-aa402e9ed0c5,enhancements:(),fieldName:name.keyword,grow:!t,id:'144933da-5c1b-4257-a969-7f43455a7901',searchTechnique:wildcard,selectedOptions:!('testDeploymentSurvivesRestart%20%7Bcluster%3DUPGRADED%7D'),title:Test,width:medium),grow:!t,order:2,type:optionsListControl,width:medium),'4e6ad9d6-6fdc-4fcc-bf1a-aa6ca79e0850':(explicitInput:(dataViewId:fbbdc689-be23-4b3d-8057-aa402e9ed0c5,enhancements:(),fieldName:className.keyword,grow:!t,id:'4e6ad9d6-6fdc-4fcc-bf1a-aa6ca79e0850',searchTechnique:wildcard,selectedOptions:!('org.elasticsearch.xpack.restart.MLModelDeploymentFullClusterRestartIT'),title:Suite,width:medium),grow:!t,order:1,type:optionsListControl,width:medium)))))

Failure Message:

java.lang.AssertionError: {pipeline_count=0, inference_stats={inference_count=0, missing_all_fields_count=0, cache_miss_count=0, failure_count=0, timestamp=1727583532879}, model_size_stats={model_size_bytes=1630, required_native_memory_bytes=251661500}, model_id=trained-model-full-cluster-restart, deployment_stats={start_time=1727583330770, nodes=[{routing_state={routing_state=started}, node={CGZLJvqlRLSEr-HrhzHFwA={transport_address=127.0.0.1:44015, roles=[data, data_cold, data_content, data_frozen, data_hot, data_warm, ingest, master, ml, remote_cluster_client, transform], name=test-cluster-1, min_index_version=8000099, external_id=test-cluster-1, attributes={ml.allocated_processors_double=32.0, ml.machine_memory=126604148736, xpack.installed=true, transform.config_version=10.0.0, testattr=test, ml.config_version=12.0.0, ml.max_jvm_size=536870912, ml.allocated_processors=32}, max_index_version=8515000, ephemeral_id=ml2aiQ73QZigNycsBzN2nw, version=9.0.0}}, start_time=1727583442708, inference_cache_hit_count=0, inference_count=0, throughput_last_minute=0, threads_per_allocation=1, number_of_allocations=1, number_of_pending_requests=0, inference_cache_hit_count_last_minute=0, peak_throughput_per_minute=0}], threads_per_allocation=1, allocation_status={target_allocation_count=1, state=starting, allocation_count=0}, deployment_id=trained-model-full-cluster-restart, model_id=trained-model-full-cluster-restart, state=started, number_of_allocations=1, priority=normal, queue_capacity=1024, peak_throughput_per_minute=0}}
Expected: "fully_allocated"
     but: was "starting"

Issue Reasons:

Note: This issue was created using new test triage automation. Please report issues or feedback to es-delivery.

elasticsearchmachine commented 1 month ago

Pinging @elastic/es-delivery (Team:Delivery)

elasticsearchmachine commented 1 month ago

Pinging @elastic/ml-core (Team:ML)

elasticsearchmachine commented 1 month ago

This has been muted on branch main

Mute Reasons:

Build Scans: