search

elastic / elasticsearch

Free and Open Source, Distributed, RESTful Search Engine
https://www.elastic.co/products/elasticsearch
Other
1.51k stars 24.89k forks source link

[CI] GCSSnapshotRepoTestKitIT testRepositoryAnalysis failing #116546

Open elasticsearchmachine opened 3 weeks ago

elasticsearchmachine commented 3 weeks ago

Build Scans:

Reproduction Line:

./gradlew ':x-pack:plugin:snapshot-repo-test-kit:qa:gcs:javaRestTest' --tests "org.elasticsearch.repositories.blobstore.testkit.GCSSnapshotRepoTestKitIT.testRepositoryAnalysis" -Dtests.seed=958EC26E665DFFC9 -Dtests.locale=ps-AF -Dtests.timezone=America/Argentina/Jujuy -Druntime.java=22

Applicable branches: 8.15

Reproduces locally?: N/A

Failure History: See dashboard&_a=(controlGroupState:(initialChildControlState:('0c0c9cb8-ccd2-45c6-9b13-96bac4abc542':(dataViewId:fbbdc689-be23-4b3d-8057-aa402e9ed0c5,fieldName:task.keyword,order:0,selectedOptions:!(),title:'GradleTask',type:optionsListControl),'4e6ad9d6-6fdc-4fcc-bf1a-aa6ca79e0850':(dataViewId:fbbdc689-be23-4b3d-8057-aa402e9ed0c5,fieldName:className.keyword,order:1,selectedOptions:!(org.elasticsearch.repositories.blobstore.testkit.GCSSnapshotRepoTestKitIT),title:'Suite',type:optionsListControl),'144933da-5c1b-4257-a969-7f43455a7901':(dataViewId:fbbdc689-be23-4b3d-8057-aa402e9ed0c5,fieldName:name.keyword,order:2,selectedOptions:!(testRepositoryAnalysis),title:'Test',type:optionsListControl)))))

Failure Message:

org.elasticsearch.client.ResponseException: method [POST], host [http://[::1]:45521], URI [/_snapshot/repository/_analyze?blob_count=10&seed=-605332933961545440&max_blob_size=1mb&timeout=120s&concurrency=4], status line [HTTP/1.1 500 Internal Server Error]
{"error":{"root_cause":[{"type":"repository_verification_exception","reason":"[repository] register [test-register-contended-xiI4LAB9TV2RWCIUoShEZA] should have value [10] but instead had value [OptionalBytesReference[MISSING]]"}],"type":"repository_verification_exception","reason":"[repository] analysis failed, you may need to manually remove [temp-analysis-GYCHMvHuREKM3vd994Bsuw]","caused_by":{"type":"repository_verification_exception","reason":"[repository] register [test-register-contended-xiI4LAB9TV2RWCIUoShEZA] should have value [10] but instead had value [OptionalBytesReference[MISSING]]"}},"status":500}

Issue Reasons:

Note: This issue was created using new test triage automation. Please report issues or feedback to es-delivery.

elasticsearchmachine commented 3 weeks ago

Pinging @elastic/es-delivery (Team:Delivery)

elasticsearchmachine commented 2 weeks ago

Pinging @elastic/es-distributed-coordination (Team:Distributed Coordination)

ywangd commented 1 week ago

This failure is for the 8.15 branch. The test class is renamed in #111963 to GCSRepositoryAnalysisRestIT since 8.16.0. The same failure also happened to main and 8.16 (dashboard&_a=(controlGroupState:(initialChildControlState:('0c0c9cb8-ccd2-45c6-9b13-96bac4abc542':(dataViewId:fbbdc689-be23-4b3d-8057-aa402e9ed0c5,exclude:!n,existsSelected:!n,fieldName:task.keyword,grow:!n,hideActionBar:!n,hideExclude:!n,hideExists:!n,hideSort:!n,order:0,placeholder:!n,runPastTimeout:!n,searchTechnique:prefix,selectedOptions:!(),singleSelect:!n,sort:(by:_count,direction:desc),title:GradleTask,type:optionsListControl,width:!n),'144933da-5c1b-4257-a969-7f43455a7901':(dataViewId:fbbdc689-be23-4b3d-8057-aa402e9ed0c5,exclude:!n,existsSelected:!n,fieldName:name.keyword,grow:!n,hideActionBar:!n,hideExclude:!n,hideExists:!n,hideSort:!n,order:2,placeholder:!n,runPastTimeout:!n,searchTechnique:prefix,selectedOptions:!(testRepositoryAnalysis),singleSelect:!n,sort:(by:_count,direction:desc),title:Test,type:optionsListControl,width:!n),'4e6ad9d6-6fdc-4fcc-bf1a-aa6ca79e0850':(dataViewId:fbbdc689-be23-4b3d-8057-aa402e9ed0c5,exclude:!n,existsSelected:!n,fieldName:className.keyword,grow:!n,hideActionBar:!n,hideExclude:!n,hideExists:!n,hideSort:!n,order:1,placeholder:!n,runPastTimeout:!n,searchTechnique:prefix,selectedOptions:!(org.elasticsearch.repositories.blobstore.testkit.analyze.GCSRepositoryAnalysisRestIT),singleSelect:!n,sort:(by:_count,direction:desc),title:Suite,type:optionsListControl,width:!n))),panels:!((gridData:(h:4,i:'705187f8-02ab-4858-971b-6e24e9a23df7',w:12,x:0,y:0),panelConfig:(attributes:(description:'',references:!((id:fbbdc689-be23-4b3d-8057-aa402e9ed0c5,name:indexpattern-datasource-layer-b0324444-043a-4e31-8c4c-de06b757a969,type:index-pattern),(id:fbbdc689-be23-4b3d-8057-aa402e9ed0c5,name:'595b302c-fe27-455a-8270-3f32426b668b',type:index-pattern)),state:(adHocDataViews:(),datasourceStates:(formBased:(layers:(b0324444-043a-4e31-8c4c-de06b757a969:(columnOrder:!('498e0348-1dc7-4487-b57c-7a17513c3d90'),columns:('498e0348-1dc7-4487-b57c-7a17513c3d90':(customLabel:!t,dataType:number,isBucketed:!f,label:'Test%20Executions',operationType:count,params:(emptyAsNull:!t),scale:ratio,sourceField:records)),ignoreGlobalFilters:!f,incompleteColumns:(),sampling:1))),indexpattern:(layers:()),textBased:(layers:())),filters:!(('$state':(store:appState),meta:(alias:!n,disabled:!f,field:suite,index:'595b302c-fe27-455a-8270-3f32426b668b',key:suite,negate:!f,params:(query:!f),type:phrase),query:(match_phrase:(suite:!f)))),internalReferences:!(),query:(language:kuery,query:''),visualization:(layerId:b0324444-043a-4e31-8c4c-de06b757a969,layerType:data,metricAccessor:'498e0348-1dc7-4487-b57c-7a17513c3d90')),title:'',type:lens,visualizationType:lnsMetric),enhancements:(),hidePanelTitles:!t),panelIndex:'705187f8-02ab-4858-971b-6e24e9a23df7',type:lens),(gridData:(h:4,i:'34980d70-46e3-4b1f-a75a-9b5224f496b1',w:12,x:12,y:0),panelConfig:(attributes:(description:'',references:!((id:fbbdc689-be23-4b3d-8057-aa402e9ed0c5,name:indexpattern-datasource-layer-6d8f3e80-f6b4-4cfc-8afe-a77ddfa58dc6,type:index-pattern),(id:fbbdc689-be23-4b3d-8057-aa402e9ed0c5,name:'910c9261-f4e3-4509-affb-74639836b5a2',type:index-pattern)),state:(adHocDataViews:(),datasourceStates:(formBased:(layers:('6d8f3e80-f6b4-4cfc-8afe-a77ddfa58dc6':(columnOrder:!(e456e0ba-6886-4c85-9628-e28eb32b02c7),columns:(e456e0ba-6886-4c85-9628-e28eb32b02c7:(customLabel:!t,dataType:number,filter:(language:kuery,query:'failed%20:%20true'),isBucketed:!f,label:'Failed%20Tests',operationType:count,params:(emptyAsNull:!f),scale:ratio,sourceField:records)),ignoreGlobalFilters:!f,incompleteColumns:(),sampling:1))),indexpattern:(layers:()),textBased:(layers:())),filters:!(('$state':(store:appState),meta:(alias:!n,disabled:!f,field:suite,index:'910c9261-f4e3-4509-affb-74639836b5a2',key:suite,negate:!f,params:(query:!f),type:phrase),query:(match_phrase:(suite:!f)))),internalReferences:!(),query:(language:kuery,query:''),visualization:(color:%23e48673,layerId:'6d8f3e80-f6b4-4cfc-8afe-a77ddfa58dc6',layerType:data,metricAccessor:e456e0ba-6886-4c85-9628-e28eb32b02c7)),title:'',type:lens,visualizationType:lnsMetric),enhancements:()),panelIndex:'34980d70-46e3-4b1f-a75a-9b5224f496b1',type:lens),(gridData:(h:4,i:'1163896b-34b5-47b5-9c6f-b6a532848e66',w:12,x:24,y:0),panelConfig:(attributes:(description:'',references:!((id:fbbdc689-be23-4b3d-8057-aa402e9ed0c5,name:indexpattern-datasource-layer-d20ec2fa-0df6-443f-bf5d-5cc3cc23f06b,type:index-pattern)),state:(adHocDataViews:(),datasourceStates:(formBased:(layers:(d20ec2fa-0df6-443f-bf5d-5cc3cc23f06b:(columnOrder:!('3e642804-8b6c-4b14-82df-83005e80073e'),columns:('3e642804-8b6c-4b14-82df-83005e80073e':(customLabel:!t,dataType:number,isBucketed:!f,label:'Build%20Executions',operationType:unique_count,params:(emptyAsNull:!t),scale:ratio,sourceField:build.scanId.keyword)),ignoreGlobalFilters:!f,incompleteColumns:(),sampling:1))),indexpattern:(layers:()),textBased:(layers:())),filters:!(),internalReferences:!(),query:(language:kuery,query:''),visualization:(layerId:d20ec2fa-0df6-443f-bf5d-5cc3cc23f06b,layerType:data,metricAccessor:'3e642804-8b6c-4b14-82df-83005e80073e')),title:'',type:lens,visualizationType:lnsMetric),enhancements:()),panelIndex:'1163896b-34b5-47b5-9c6f-b6a532848e66',type:lens),(gridData:(h:16,i:d35dbbf9-8bc9-498d-b3a3-b560e8ead693,w:12,x:0,y:4),panelConfig:(attributes:(description:'',references:!((id:fbbdc689-be23-4b3d-8057-aa402e9ed0c5,name:indexpattern-datasource-layer-6f78bf93-e05f-411e-b4a1-9a71d3d1b1bd,type:index-pattern)),state:(adHocDataViews:(),datasourceStates:(formBased:(currentIndexPatternId:fbbdc689-be23-4b3d-8057-aa402e9ed0c5,layers:('6f78bf93-e05f-411e-b4a1-9a71d3d1b1bd':(columnOrder:!('91a2bb54-0196-4223-8ed6-9244aafc40d2','79273e54-80cf-4438-8144-a15cafd15e72','79273e54-80cf-4438-8144-a15cafd15e72X0','79273e54-80cf-4438-8144-a15cafd15e72X3','79273e54-80cf-4438-8144-a15cafd15e72X2','79273e54-80cf-4438-8144-a15cafd15e72X1'),columns:('79273e54-80cf-4438-8144-a15cafd15e72':(customLabel:!f,dataType:number,isBucketed:!f,label:'count()%20%2F%20overall_max(count())',operationType:formula,params:(format:(id:percent,params:(decimals:0)),formula:'count()%20%2F%20overall_max(count())',isFormulaBroken:!f),references:!('79273e54-80cf-4438-8144-a15cafd15e72X3'),scale:ratio),'79273e54-80cf-4438-8144-a15cafd15e72X0':(customLabel:!t,dataType:number,isBucketed:!f,label:'Part%20of%20count()%20%2F%20overall_max(count())',operationType:count,params:(emptyAsNull:!f),scale:ratio,sourceField:records),'79273e54-80cf-4438-8144-a15cafd15e72X1':(customLabel:!t,dataType:number,isBucketed:!f,label:'Part%20of%20count()%20%2F%20overall_max(count())',operationType:count,params:(emptyAsNull:!f),scale:ratio,sourceField:records),'79273e54-80cf-4438-8144-a15cafd15e72X2':(customLabel:!t,dataType:number,isBucketed:!f,label:'Part%20of%20count()%20%2F%20overall_max(count())',operationType:overall_max,references:!('79273e54-80cf-4438-8144-a15cafd15e72X1'),scale:ratio),'79273e54-80cf-4438-8144-a15cafd15e72X3':(customLabel:!t,dataType:number,isBucketed:!f,label:'Part%20of%20count()%20%2F%20overall_max(count())',operationType:math,params:(tinymathAst:(args:!('79273e54-80cf-4438-8144-a15cafd15e72X0','79273e54-80cf-4438-8144-a15cafd15e72X2'),location:(max:30,min:0),name:divide,text:'count()%20%2F%20overall_max(count())',type:function)),references:!('79273e54-80cf-4438-8144-a15cafd15e72X0','79273e54-80cf-4438-8144-a15cafd15e72X2'),scale:ratio),'91a2bb54-0196-4223-8ed6-9244aafc40d2':(dataType:string,isBucketed:!t,label:'Top%2010%20values%20of%20build.tags.keyword',operationType:terms,params:(exclude:!('(CI%7Cmain%7C%5Cd%2B.%5Cd%2B)'),excludeIsRegex:!t,include:!(),includeIsRegex:!f,missingBucket:!f,orderAgg:(dataType:number,isBucketed:!f,label:'Count%20of%20records',operationType:count,params:(emptyAsNull:!t),scale:ratio,sourceField:records),orderBy:(type:custom),orderDirection:desc,otherBucket:!f,parentFormat:(id:terms),size:10),scale:ordinal,sourceField:build.tags.keyword)),ignoreGlobalFilters:!f,incompleteColumns:(),indexPatternId:fbbdc689-be23-4b3d-8057-aa402e9ed0c5,sampling:1))),indexpattern:(layers:()),textBased:(layers:())),filters:!(('$state':(store:appState),meta:(alias:!n,disabled:!f,field:failed,index:c3b65f7a-b534-4b2f-a8f4-a79c70c2b59c,key:failed,negate:!f,params:(query:!t),type:phrase),query:(match_phrase:(failed:!t))),('$state':(store:appState),meta:(alias:!n,disabled:!f,field:suite,index:'7e1054a6-3acb-4a8c-8ac9-0ddab627d9ce',key:suite,negate:!f,params:(query:!f),type:phrase),query:(match_phrase:(suite:!f)))),internalReferences:!(),query:(language:kuery,query:''),visualization:(layers:!((categoryDisplay:default,colorMapping:(assignments:!(),colorMode:(sort:asc,steps:!((colorIndex:0,paletteId:eui_amsterdam_color_blind,touched:!f,type:categorical)),type:gradient),paletteId:eui_amsterdam_color_blind,specialAssignments:!((color:(type:loop),rule:(type:other),touched:!f))),layerId:'6f78bf93-e05f-411e-b4a1-9a71d3d1b1bd',layerType:data,legendDisplay:default,metrics:!('79273e54-80cf-4438-8144-a15cafd15e72'),nestedLegend:!f,numberDisplay:value,percentDecimals:2,primaryGroups:!('91a2bb54-0196-4223-8ed6-9244aafc40d2'))),shape:treemap)),title:'',type:lens,visualizationType:lnsPie),enhancements:(),hidePanelTitles:!f),panelIndex:d35dbbf9-8bc9-498d-b3a3-b560e8ead693,title:'Top%20Failure%20Tags',type:lens),(gridData:(h:4,i:c8202da2-f726-40b8-90ac-cb92e467022e,w:12,x:36,y:0),panelConfig:(attributes:(description:'',references:!((id:fbbdc689-be23-4b3d-8057-aa402e9ed0c5,name:indexpattern-datasource-layer-6d8f3e80-f6b4-4cfc-8afe-a77ddfa58dc6,type:index-pattern)),state:(adHocDataViews:(),datasourceStates:(formBased:(currentIndexPatternId:fbbdc689-be23-4b3d-8057-aa402e9ed0c5,layers:('6d8f3e80-f6b4-4cfc-8afe-a77ddfa58dc6':(columnOrder:!(e456e0ba-6886-4c85-9628-e28eb32b02c7),columns:(e456e0ba-6886-4c85-9628-e28eb32b02c7:(customLabel:!t,dataType:number,filter:(language:kuery,query:'failed%20:%20true'),isBucketed:!f,label:'Failed%20Builds',operationType:unique_count,params:(emptyAsNull:!f),scale:ratio,sourceField:build.scanId.keyword)),ignoreGlobalFilters:!f,incompleteColumns:(),indexPatternId:fbbdc689-be23-4b3d-8057-aa402e9ed0c5,sampling:1))),indexpattern:(layers:()),textBased:(layers:())),filters:!(),internalReferences:!(),query:(language:kuery,query:''),visualization:(color:%23e48673,layerId:'6d8f3e80-f6b4-4cfc-8afe-a77ddfa58dc6',layerType:data,metricAccessor:e456e0ba-6886-4c85-9628-e28eb32b02c7,showBar:!f)),title:'',type:lens,visualizationType:lnsMetric),enhancements:()),panelIndex:c8202da2-f726-40b8-90ac-cb92e467022e,type:lens),(gridData:(h:16,i:d1b26fe0-1a87-469e-a445-1d0baa028707,w:18,x:30,y:4),panelConfig:(attributes:(description:'',references:!((id:fbbdc689-be23-4b3d-8057-aa402e9ed0c5,name:indexpattern-datasource-layer-27fdb8d9-1c3b-4028-8c0b-7cda5064801e,type:index-pattern)),state:(adHocDataViews:(),datasourceStates:(formBased:(currentIndexPatternId:fbbdc689-be23-4b3d-8057-aa402e9ed0c5,layers:('27fdb8d9-1c3b-4028-8c0b-7cda5064801e':(columnOrder:!('0c6336a5-0069-4677-95bf-106bfadb94ae',b996f933-7b55-47d0-a1c6-68aa8f42bbbe,'8908dfa8-6b37-4dff-a218-d253a2a1e4d0'),columns:('0c6336a5-0069-4677-95bf-106bfadb94ae':(dataType:string,isBucketed:!t,label:'Top%204%20values%20of%20build.values.Git%20Branch.keyword',operationType:terms,params:(exclude:!(),excludeIsRegex:!f,include:!('(main%7C%5Cd%2B.(%5Cd%2B%7Cx))'),includeIsRegex:!t,missingBucket:!f,orderBy:(columnId:'8908dfa8-6b37-4dff-a218-d253a2a1e4d0',type:column),orderDirection:desc,otherBucket:!f,parentFormat:(id:terms),size:4),scale:ordinal,sourceField:'build.values.Git%20Branch.keyword'),'8908dfa8-6b37-4dff-a218-d253a2a1e4d0':(customLabel:!t,dataType:number,isBucketed:!f,label:Failures,operationType:count,params:(emptyAsNull:!t),scale:ratio,sourceField:records),b996f933-7b55-47d0-a1c6-68aa8f42bbbe:(dataType:date,isBucketed:!t,label:'@timestamp',operationType:date_histogram,params:(dropPartials:!f,includeEmptyRows:!t,interval:d),scale:interval,sourceField:'@timestamp')),incompleteColumns:(),indexPatternId:fbbdc689-be23-4b3d-8057-aa402e9ed0c5,sampling:1))),indexpattern:(layers:()),textBased:(layers:())),filters:!(('$state':(store:appState),meta:(alias:!n,disabled:!f,field:failed,index:'7734cf97-d0b2-4f99-a820-c7a0224e8cd3',key:failed,negate:!f,params:(query:!t),type:phrase),query:(match_phrase:(failed:!t))),('$state':(store:appState),meta:(alias:!n,disabled:!f,field:suite,index:a40b41a1-bb68-44b7-ba67-b11f4354fdfd,key:suite,negate:!f,params:(query:!f),type:phrase),query:(match_phrase:(suite:!f)))),internalReferences:!(),query:(language:kuery,query:'failed%20:%20true'),visualization:(axisTitlesVisibilitySettings:(x:!f,yLeft:!t,yRight:!t),fittingFunction:None,gridlinesVisibilitySettings:(x:!t,yLeft:!t,yRight:!t),hideEndzones:!t,labelsOrientation:(x:0,yLeft:0,yRight:0),layers:!((accessors:!('8908dfa8-6b37-4dff-a218-d253a2a1e4d0'),layerId:'27fdb8d9-1c3b-4028-8c0b-7cda5064801e',layerType:data,palette:(name:default,type:palette),position:top,seriesType:bar_stacked,showGridlines:!f,splitAccessor:'0c6336a5-0069-4677-95bf-106bfadb94ae',xAccessor:b996f933-7b55-47d0-a1c6-68aa8f42bbbe)),legend:(isInside:!f,isVisible:!t,legendStats:!(),position:top),preferredSeriesType:bar_percentage_stacked,tickLabelsVisibilitySettings:(x:!t,yLeft:!t,yRight:!t),valueLabels:hide)),title:'',type:lens,visualizationType:lnsXY),enhancements:(dynamicActions:(events:!())),hidePanelTitles:!f),panelIndex:d1b26fe0-1a87-469e-a445-1d0baa028707,title:'Failures%20By%20Branch',type:lens),(gridData:(h:16,i:'221ee73c-ddc3-4b33-b744-532f1d8f9ff8',w:18,x:12,y:4),panelConfig:(attributes:(description:'',references:!((id:fbbdc689-be23-4b3d-8057-aa402e9ed0c5,name:indexpattern-datasource-layer-fc28e04a-9d1e-41d4-aefd-c7c86e2ba8c7,type:index-pattern)),state:(adHocDataViews:(),datasourceStates:(formBased:(currentIndexPatternId:fbbdc689-be23-4b3d-8057-aa402e9ed0c5,layers:(fc28e04a-9d1e-41d4-aefd-c7c86e2ba8c7:(columnOrder:!(c284db41-8b0a-4cc2-a184-d8cb3e41fce4,'274fe047-621b-4fbc-bccf-13361bbe8448',fc915caa-c5ea-48f4-a067-e607632f5266),columns:('274fe047-621b-4fbc-bccf-13361bbe8448':(dataType:string,isBucketed:!t,label:Filters,operationType:filters,params:(filters:!((input:(language:kuery,query:'failed:%20false%20and%20skipped:%20false'),label:Success),(input:(language:kuery,query:'skipped:%20true'),label:Skipped),(input:(language:kuery,query:'failed:%20true%20'),label:Failed))),scale:ordinal),c284db41-8b0a-4cc2-a184-d8cb3e41fce4:(dataType:date,isBucketed:!t,label:'@timestamp',operationType:date_histogram,params:(dropPartials:!f,includeEmptyRows:!t,interval:d),scale:interval,sourceField:'@timestamp'),fc915caa-c5ea-48f4-a067-e607632f5266:(customLabel:!t,dataType:number,isBucketed:!f,label:Tests,operationType:count,params:(emptyAsNull:!t),scale:ratio,sourceField:records)),ignoreGlobalFilters:!f,incompleteColumns:(),indexPatternId:fbbdc689-be23-4b3d-8057-aa402e9ed0c5,sampling:1))),indexpattern:(layers:()),textBased:(layers:())),filters:!(('$state':(store:appState),meta:(alias:!n,disabled:!f,field:suite,index:'8e220a72-8586-4840-b7c9-18182cce040d',key:suite,negate:!f,params:(query:!f),type:phrase),query:(match_phrase:(suite:!f)))),internalReferences:!(),query:(language:kuery,query:''),visualization:(axisTitlesVisibilitySettings:(x:!f,yLeft:!t,yRight:!t),fittingFunction:None,gridlinesVisibilitySettings:(x:!t,yLeft:!t,yRight:!t),hideEndzones:!t,labelsOrientation:(x:0,yLeft:0,yRight:0),layers:!((accessors:!(fc915caa-c5ea-48f4-a067-e607632f5266),colorMapping:(assignmentMode:manual,assignments:!((color:(colorCode:%2300cb9c,type:colorCode),rule:(type:matchExactly,values:!(Success)),touched:!t),(color:(colorIndex:1,paletteId:neutral,type:categorical),rule:(type:matchExactly,values:!(Skipped)),touched:!t),(color:(colorIndex:9,paletteId:eui_amsterdam_color_blind,type:categorical),rule:(type:matchExactly,values:!(Failed)),touched:!t)),colorMode:(type:categorical),paletteId:eui_amsterdam_color_blind,specialAssignments:!((color:(colorIndex:1,paletteId:neutral,type:categorical),rule:(type:other),touched:!f))),layerId:fc28e04a-9d1e-41d4-aefd-c7c86e2ba8c7,layerType:data,position:top,seriesType:bar_stacked,showGridlines:!f,splitAccessor:'274fe047-621b-4fbc-bccf-13361bbe8448',xAccessor:c284db41-8b0a-4cc2-a184-d8cb3e41fce4)),legend:(horizontalAlignment:left,isInside:!f,isVisible:!t,position:top,verticalAlignment:bottom),preferredSeriesType:bar_stacked,tickLabelsVisibilitySettings:(x:!t,yLeft:!t,yRight:!t),valueLabels:hide)),title:'',type:lens,visualizationType:lnsXY),enhancements:(),hidePanelTitles:!f),panelIndex:'221ee73c-ddc3-4b33-b744-532f1d8f9ff8',title:'Test%20Results',type:lens),(gridData:(h:16,i:'2208b623-0c03-434a-982f-be95f6ec967c',w:24,x:0,y:20),panelConfig:(attributes:(description:'',references:!((id:fbbdc689-be23-4b3d-8057-aa402e9ed0c5,name:indexpattern-datasource-layer-7d02ffda-ba9c-49c9-92d1-f732438fa2b1,type:index-pattern),(id:fbbdc689-be23-4b3d-8057-aa402e9ed0c5,name:'10ebf466-c9fc-42e1-a77f-d1e64dda7530',type:index-pattern),(id:fbbdc689-be23-4b3d-8057-aa402e9ed0c5,name:'9074483e-b602-4235-9405-7bbdd1976d13',type:index-pattern)),state:(adHocDataViews:(),datasourceStates:(formBased:(layers:('7d02ffda-ba9c-49c9-92d1-f732438fa2b1':(columnOrder:!('3148ddf0-8fdf-43df-8ab5-7f5859a8bf05',b890eb79-8854-4935-925b-5c442caf50dc),columns:('3148ddf0-8fdf-43df-8ab5-7f5859a8bf05':(customLabel:!t,dataType:string,isBucketed:!t,label:Test,operationType:terms,params:(accuracyMode:!f,exclude:!(),excludeIsRegex:!f,include:!(),includeIsRegex:!f,missingBucket:!f,orderBy:(columnId:b890eb79-8854-4935-925b-5c442caf50dc,type:column),orderDirection:desc,otherBucket:!f,parentFormat:(id:multi_terms),secondaryFields:!(className.keyword,name.keyword),size:10),scale:ordinal,sourceField:task.keyword),b890eb79-8854-4935-925b-5c442caf50dc:(customLabel:!t,dataType:number,isBucketed:!f,label:Failures,operationType:count,params:(emptyAsNull:!t),scale:ratio,sourceField:records)),incompleteColumns:(),sampling:1))),indexpattern:(layers:()),textBased:(layers:())),filters:!(('$state':(store:appState),meta:(alias:!n,disabled:!f,field:failed,index:'10ebf466-c9fc-42e1-a77f-d1e64dda7530',key:failed,negate:!f,params:(query:!t),type:phrase),query:(match_phrase:(failed:!t))),('$state':(store:appState),meta:(alias:!n,disabled:!f,field:suite,index:'9074483e-b602-4235-9405-7bbdd1976d13',key:suite,negate:!f,params:(query:!f),type:phrase),query:(match_phrase:(suite:!f)))),internalReferences:!(),query:(language:kuery,query:''),visualization:(columns:!((alignment:left,columnId:'3148ddf0-8fdf-43df-8ab5-7f5859a8bf05',isTransposed:!f,width:832),(columnId:b890eb79-8854-4935-925b-5c442caf50dc,hidden:!f,isTransposed:!f)),layerId:'7d02ffda-ba9c-49c9-92d1-f732438fa2b1',layerType:data)),title:'',type:lens,visualizationType:lnsDatatable),enhancements:(dynamicActions:(events:!())),hidePanelTitles:!f),panelIndex:'2208b623-0c03-434a-982f-be95f6ec967c',title:'Top%20Failures',type:lens),(gridData:(h:16,i:'449c1f21-f509-4fff-b487-c9c3c24a5c60',w:24,x:24,y:20),panelConfig:(attributes:(description:'',references:!((id:fbbdc689-be23-4b3d-8057-aa402e9ed0c5,name:indexpattern-datasource-layer-b8536ffe-162e-4f73-86b1-8be7ed896749,type:index-pattern)),state:(adHocDataViews:(),datasourceStates:(formBased:(currentIndexPatternId:fbbdc689-be23-4b3d-8057-aa402e9ed0c5,layers:(b8536ffe-162e-4f73-86b1-8be7ed896749:(columnOrder:!(b839284a-259b-40a3-aef1-0866ec8eaee6,'28b704e9-f885-4f08-bfdc-8aaf74de1fc5'),columns:('28b704e9-f885-4f08-bfdc-8aaf74de1fc5':(customLabel:!t,dataType:number,isBucketed:!f,label:Failures,operationType:count,params:(emptyAsNull:!t),scale:ratio,sourceField:records),b839284a-259b-40a3-aef1-0866ec8eaee6:(customLabel:!t,dataType:string,isBucketed:!t,label:'Failure%20message',operationType:terms,params:(exclude:!(),excludeIsRegex:!f,include:!(),includeIsRegex:!f,missingBucket:!f,orderBy:(columnId:'28b704e9-f885-4f08-bfdc-8aaf74de1fc5',type:column),orderDirection:desc,otherBucket:!t,parentFormat:(id:terms),size:10),scale:ordinal,sourceField:failureMessage.keyword)),incompleteColumns:(),indexPatternId:fbbdc689-be23-4b3d-8057-aa402e9ed0c5,sampling:1))),indexpattern:(layers:()),textBased:(layers:())),filters:!(),internalReferences:!(),query:(language:kuery,query:''),visualization:(columns:!((columnId:b839284a-259b-40a3-aef1-0866ec8eaee6,isTransposed:!f,width:1002.5),(columnId:'28b704e9-f885-4f08-bfdc-8aaf74de1fc5',isTransposed:!f)),layerId:b8536ffe-162e-4f73-86b1-8be7ed896749,layerType:data,rowHeight:auto)),title:'',type:lens,visualizationType:lnsDatatable),enhancements:(),hidePanelTitles:!f),panelIndex:'449c1f21-f509-4fff-b487-c9c3c24a5c60',title:'Top%20Failure%20Exceptions',type:lens)))))

ywangd commented 1 week ago

I re-adjusted the risk to medium since it also happened on main. IIUC, the tests targeted the real GCS storage service. The failure may indicate a discrepency between how atomic operation works on GCS and our understanding. The failure seems to started&_a=(controlGroupState:(initialChildControlState:('0c0c9cb8-ccd2-45c6-9b13-96bac4abc542':(dataViewId:fbbdc689-be23-4b3d-8057-aa402e9ed0c5,exclude:!n,existsSelected:!n,fieldName:task.keyword,grow:!n,hideActionBar:!n,hideExclude:!n,hideExists:!n,hideSort:!n,order:0,placeholder:!n,runPastTimeout:!n,searchTechnique:prefix,selectedOptions:!(),singleSelect:!n,sort:(by:_count,direction:desc),title:GradleTask,type:optionsListControl,width:!n),'144933da-5c1b-4257-a969-7f43455a7901':(dataViewId:fbbdc689-be23-4b3d-8057-aa402e9ed0c5,exclude:!n,existsSelected:!n,fieldName:name.keyword,grow:!n,hideActionBar:!n,hideExclude:!n,hideExists:!n,hideSort:!n,order:2,placeholder:!n,runPastTimeout:!n,searchTechnique:prefix,selectedOptions:!(testRepositoryAnalysis),singleSelect:!n,sort:(by:_count,direction:desc),title:Test,type:optionsListControl,width:!n),'4e6ad9d6-6fdc-4fcc-bf1a-aa6ca79e0850':(dataViewId:fbbdc689-be23-4b3d-8057-aa402e9ed0c5,exclude:!n,existsSelected:!n,fieldName:className.keyword,grow:!n,hideActionBar:!n,hideExclude:!n,hideExists:!n,hideSort:!n,order:1,placeholder:!n,runPastTimeout:!n,searchTechnique:prefix,selectedOptions:!(org.elasticsearch.repositories.blobstore.testkit.GCSSnapshotRepoTestKitIT),singleSelect:!n,sort:(by:_count,direction:desc),title:Suite,type:optionsListControl,width:!n))),panels:!((gridData:(h:4,i:'705187f8-02ab-4858-971b-6e24e9a23df7',w:12,x:0,y:0),panelConfig:(attributes:(description:'',references:!((id:fbbdc689-be23-4b3d-8057-aa402e9ed0c5,name:indexpattern-datasource-layer-b0324444-043a-4e31-8c4c-de06b757a969,type:index-pattern),(id:fbbdc689-be23-4b3d-8057-aa402e9ed0c5,name:'595b302c-fe27-455a-8270-3f32426b668b',type:index-pattern)),state:(adHocDataViews:(),datasourceStates:(formBased:(layers:(b0324444-043a-4e31-8c4c-de06b757a969:(columnOrder:!('498e0348-1dc7-4487-b57c-7a17513c3d90'),columns:('498e0348-1dc7-4487-b57c-7a17513c3d90':(customLabel:!t,dataType:number,isBucketed:!f,label:'Test%20Executions',operationType:count,params:(emptyAsNull:!t),scale:ratio,sourceField:records)),ignoreGlobalFilters:!f,incompleteColumns:(),sampling:1))),indexpattern:(layers:()),textBased:(layers:())),filters:!(('$state':(store:appState),meta:(alias:!n,disabled:!f,field:suite,index:'595b302c-fe27-455a-8270-3f32426b668b',key:suite,negate:!f,params:(query:!f),type:phrase),query:(match_phrase:(suite:!f)))),internalReferences:!(),query:(language:kuery,query:''),visualization:(layerId:b0324444-043a-4e31-8c4c-de06b757a969,layerType:data,metricAccessor:'498e0348-1dc7-4487-b57c-7a17513c3d90')),title:'',type:lens,visualizationType:lnsMetric),enhancements:(),hidePanelTitles:!t),panelIndex:'705187f8-02ab-4858-971b-6e24e9a23df7',type:lens),(gridData:(h:4,i:'34980d70-46e3-4b1f-a75a-9b5224f496b1',w:12,x:12,y:0),panelConfig:(attributes:(description:'',references:!((id:fbbdc689-be23-4b3d-8057-aa402e9ed0c5,name:indexpattern-datasource-layer-6d8f3e80-f6b4-4cfc-8afe-a77ddfa58dc6,type:index-pattern),(id:fbbdc689-be23-4b3d-8057-aa402e9ed0c5,name:'910c9261-f4e3-4509-affb-74639836b5a2',type:index-pattern)),state:(adHocDataViews:(),datasourceStates:(formBased:(layers:('6d8f3e80-f6b4-4cfc-8afe-a77ddfa58dc6':(columnOrder:!(e456e0ba-6886-4c85-9628-e28eb32b02c7),columns:(e456e0ba-6886-4c85-9628-e28eb32b02c7:(customLabel:!t,dataType:number,filter:(language:kuery,query:'failed%20:%20true'),isBucketed:!f,label:'Failed%20Tests',operationType:count,params:(emptyAsNull:!f),scale:ratio,sourceField:records)),ignoreGlobalFilters:!f,incompleteColumns:(),sampling:1))),indexpattern:(layers:()),textBased:(layers:())),filters:!(('$state':(store:appState),meta:(alias:!n,disabled:!f,field:suite,index:'910c9261-f4e3-4509-affb-74639836b5a2',key:suite,negate:!f,params:(query:!f),type:phrase),query:(match_phrase:(suite:!f)))),internalReferences:!(),query:(language:kuery,query:''),visualization:(color:%23e48673,layerId:'6d8f3e80-f6b4-4cfc-8afe-a77ddfa58dc6',layerType:data,metricAccessor:e456e0ba-6886-4c85-9628-e28eb32b02c7)),title:'',type:lens,visualizationType:lnsMetric),enhancements:()),panelIndex:'34980d70-46e3-4b1f-a75a-9b5224f496b1',type:lens),(gridData:(h:4,i:'1163896b-34b5-47b5-9c6f-b6a532848e66',w:12,x:24,y:0),panelConfig:(attributes:(description:'',references:!((id:fbbdc689-be23-4b3d-8057-aa402e9ed0c5,name:indexpattern-datasource-layer-d20ec2fa-0df6-443f-bf5d-5cc3cc23f06b,type:index-pattern)),state:(adHocDataViews:(),datasourceStates:(formBased:(layers:(d20ec2fa-0df6-443f-bf5d-5cc3cc23f06b:(columnOrder:!('3e642804-8b6c-4b14-82df-83005e80073e'),columns:('3e642804-8b6c-4b14-82df-83005e80073e':(customLabel:!t,dataType:number,isBucketed:!f,label:'Build%20Executions',operationType:unique_count,params:(emptyAsNull:!t),scale:ratio,sourceField:build.scanId.keyword)),ignoreGlobalFilters:!f,incompleteColumns:(),sampling:1))),indexpattern:(layers:()),textBased:(layers:())),filters:!(),internalReferences:!(),query:(language:kuery,query:''),visualization:(layerId:d20ec2fa-0df6-443f-bf5d-5cc3cc23f06b,layerType:data,metricAccessor:'3e642804-8b6c-4b14-82df-83005e80073e')),title:'',type:lens,visualizationType:lnsMetric),enhancements:()),panelIndex:'1163896b-34b5-47b5-9c6f-b6a532848e66',type:lens),(gridData:(h:16,i:d35dbbf9-8bc9-498d-b3a3-b560e8ead693,w:12,x:0,y:4),panelConfig:(attributes:(description:'',references:!((id:fbbdc689-be23-4b3d-8057-aa402e9ed0c5,name:indexpattern-datasource-layer-6f78bf93-e05f-411e-b4a1-9a71d3d1b1bd,type:index-pattern)),state:(adHocDataViews:(),datasourceStates:(formBased:(currentIndexPatternId:fbbdc689-be23-4b3d-8057-aa402e9ed0c5,layers:('6f78bf93-e05f-411e-b4a1-9a71d3d1b1bd':(columnOrder:!('91a2bb54-0196-4223-8ed6-9244aafc40d2','79273e54-80cf-4438-8144-a15cafd15e72','79273e54-80cf-4438-8144-a15cafd15e72X0','79273e54-80cf-4438-8144-a15cafd15e72X3','79273e54-80cf-4438-8144-a15cafd15e72X2','79273e54-80cf-4438-8144-a15cafd15e72X1'),columns:('79273e54-80cf-4438-8144-a15cafd15e72':(customLabel:!f,dataType:number,isBucketed:!f,label:'count()%20%2F%20overall_max(count())',operationType:formula,params:(format:(id:percent,params:(decimals:0)),formula:'count()%20%2F%20overall_max(count())',isFormulaBroken:!f),references:!('79273e54-80cf-4438-8144-a15cafd15e72X3'),scale:ratio),'79273e54-80cf-4438-8144-a15cafd15e72X0':(customLabel:!t,dataType:number,isBucketed:!f,label:'Part%20of%20count()%20%2F%20overall_max(count())',operationType:count,params:(emptyAsNull:!f),scale:ratio,sourceField:records),'79273e54-80cf-4438-8144-a15cafd15e72X1':(customLabel:!t,dataType:number,isBucketed:!f,label:'Part%20of%20count()%20%2F%20overall_max(count())',operationType:count,params:(emptyAsNull:!f),scale:ratio,sourceField:records),'79273e54-80cf-4438-8144-a15cafd15e72X2':(customLabel:!t,dataType:number,isBucketed:!f,label:'Part%20of%20count()%20%2F%20overall_max(count())',operationType:overall_max,references:!('79273e54-80cf-4438-8144-a15cafd15e72X1'),scale:ratio),'79273e54-80cf-4438-8144-a15cafd15e72X3':(customLabel:!t,dataType:number,isBucketed:!f,label:'Part%20of%20count()%20%2F%20overall_max(count())',operationType:math,params:(tinymathAst:(args:!('79273e54-80cf-4438-8144-a15cafd15e72X0','79273e54-80cf-4438-8144-a15cafd15e72X2'),location:(max:30,min:0),name:divide,text:'count()%20%2F%20overall_max(count())',type:function)),references:!('79273e54-80cf-4438-8144-a15cafd15e72X0','79273e54-80cf-4438-8144-a15cafd15e72X2'),scale:ratio),'91a2bb54-0196-4223-8ed6-9244aafc40d2':(dataType:string,isBucketed:!t,label:'Top%2010%20values%20of%20build.tags.keyword',operationType:terms,params:(exclude:!('(CI%7Cmain%7C%5Cd%2B.%5Cd%2B)'),excludeIsRegex:!t,include:!(),includeIsRegex:!f,missingBucket:!f,orderAgg:(dataType:number,isBucketed:!f,label:'Count%20of%20records',operationType:count,params:(emptyAsNull:!t),scale:ratio,sourceField:records),orderBy:(type:custom),orderDirection:desc,otherBucket:!f,parentFormat:(id:terms),size:10),scale:ordinal,sourceField:build.tags.keyword)),ignoreGlobalFilters:!f,incompleteColumns:(),indexPatternId:fbbdc689-be23-4b3d-8057-aa402e9ed0c5,sampling:1))),indexpattern:(layers:()),textBased:(layers:())),filters:!(('$state':(store:appState),meta:(alias:!n,disabled:!f,field:failed,index:c3b65f7a-b534-4b2f-a8f4-a79c70c2b59c,key:failed,negate:!f,params:(query:!t),type:phrase),query:(match_phrase:(failed:!t))),('$state':(store:appState),meta:(alias:!n,disabled:!f,field:suite,index:'7e1054a6-3acb-4a8c-8ac9-0ddab627d9ce',key:suite,negate:!f,params:(query:!f),type:phrase),query:(match_phrase:(suite:!f)))),internalReferences:!(),query:(language:kuery,query:''),visualization:(layers:!((categoryDisplay:default,colorMapping:(assignments:!(),colorMode:(sort:asc,steps:!((colorIndex:0,paletteId:eui_amsterdam_color_blind,touched:!f,type:categorical)),type:gradient),paletteId:eui_amsterdam_color_blind,specialAssignments:!((color:(type:loop),rule:(type:other),touched:!f))),layerId:'6f78bf93-e05f-411e-b4a1-9a71d3d1b1bd',layerType:data,legendDisplay:default,metrics:!('79273e54-80cf-4438-8144-a15cafd15e72'),nestedLegend:!f,numberDisplay:value,percentDecimals:2,primaryGroups:!('91a2bb54-0196-4223-8ed6-9244aafc40d2'))),shape:treemap)),title:'',type:lens,visualizationType:lnsPie),enhancements:(),hidePanelTitles:!f),panelIndex:d35dbbf9-8bc9-498d-b3a3-b560e8ead693,title:'Top%20Failure%20Tags',type:lens),(gridData:(h:4,i:c8202da2-f726-40b8-90ac-cb92e467022e,w:12,x:36,y:0),panelConfig:(attributes:(description:'',references:!((id:fbbdc689-be23-4b3d-8057-aa402e9ed0c5,name:indexpattern-datasource-layer-6d8f3e80-f6b4-4cfc-8afe-a77ddfa58dc6,type:index-pattern)),state:(adHocDataViews:(),datasourceStates:(formBased:(currentIndexPatternId:fbbdc689-be23-4b3d-8057-aa402e9ed0c5,layers:('6d8f3e80-f6b4-4cfc-8afe-a77ddfa58dc6':(columnOrder:!(e456e0ba-6886-4c85-9628-e28eb32b02c7),columns:(e456e0ba-6886-4c85-9628-e28eb32b02c7:(customLabel:!t,dataType:number,filter:(language:kuery,query:'failed%20:%20true'),isBucketed:!f,label:'Failed%20Builds',operationType:unique_count,params:(emptyAsNull:!f),scale:ratio,sourceField:build.scanId.keyword)),ignoreGlobalFilters:!f,incompleteColumns:(),indexPatternId:fbbdc689-be23-4b3d-8057-aa402e9ed0c5,sampling:1))),indexpattern:(layers:()),textBased:(layers:())),filters:!(),internalReferences:!(),query:(language:kuery,query:''),visualization:(color:%23e48673,layerId:'6d8f3e80-f6b4-4cfc-8afe-a77ddfa58dc6',layerType:data,metricAccessor:e456e0ba-6886-4c85-9628-e28eb32b02c7,showBar:!f)),title:'',type:lens,visualizationType:lnsMetric),enhancements:()),panelIndex:c8202da2-f726-40b8-90ac-cb92e467022e,type:lens),(gridData:(h:16,i:d1b26fe0-1a87-469e-a445-1d0baa028707,w:18,x:30,y:4),panelConfig:(attributes:(description:'',references:!((id:fbbdc689-be23-4b3d-8057-aa402e9ed0c5,name:indexpattern-datasource-layer-27fdb8d9-1c3b-4028-8c0b-7cda5064801e,type:index-pattern)),state:(adHocDataViews:(),datasourceStates:(formBased:(currentIndexPatternId:fbbdc689-be23-4b3d-8057-aa402e9ed0c5,layers:('27fdb8d9-1c3b-4028-8c0b-7cda5064801e':(columnOrder:!('0c6336a5-0069-4677-95bf-106bfadb94ae',b996f933-7b55-47d0-a1c6-68aa8f42bbbe,'8908dfa8-6b37-4dff-a218-d253a2a1e4d0'),columns:('0c6336a5-0069-4677-95bf-106bfadb94ae':(dataType:string,isBucketed:!t,label:'Top%204%20values%20of%20build.values.Git%20Branch.keyword',operationType:terms,params:(exclude:!(),excludeIsRegex:!f,include:!('(main%7C%5Cd%2B.(%5Cd%2B%7Cx))'),includeIsRegex:!t,missingBucket:!f,orderBy:(columnId:'8908dfa8-6b37-4dff-a218-d253a2a1e4d0',type:column),orderDirection:desc,otherBucket:!f,parentFormat:(id:terms),size:4),scale:ordinal,sourceField:'build.values.Git%20Branch.keyword'),'8908dfa8-6b37-4dff-a218-d253a2a1e4d0':(customLabel:!t,dataType:number,isBucketed:!f,label:Failures,operationType:count,params:(emptyAsNull:!t),scale:ratio,sourceField:records),b996f933-7b55-47d0-a1c6-68aa8f42bbbe:(dataType:date,isBucketed:!t,label:'@timestamp',operationType:date_histogram,params:(dropPartials:!f,includeEmptyRows:!t,interval:d),scale:interval,sourceField:'@timestamp')),incompleteColumns:(),indexPatternId:fbbdc689-be23-4b3d-8057-aa402e9ed0c5,sampling:1))),indexpattern:(layers:()),textBased:(layers:())),filters:!(('$state':(store:appState),meta:(alias:!n,disabled:!f,field:failed,index:'7734cf97-d0b2-4f99-a820-c7a0224e8cd3',key:failed,negate:!f,params:(query:!t),type:phrase),query:(match_phrase:(failed:!t))),('$state':(store:appState),meta:(alias:!n,disabled:!f,field:suite,index:a40b41a1-bb68-44b7-ba67-b11f4354fdfd,key:suite,negate:!f,params:(query:!f),type:phrase),query:(match_phrase:(suite:!f)))),internalReferences:!(),query:(language:kuery,query:'failed%20:%20true'),visualization:(axisTitlesVisibilitySettings:(x:!f,yLeft:!t,yRight:!t),fittingFunction:None,gridlinesVisibilitySettings:(x:!t,yLeft:!t,yRight:!t),hideEndzones:!t,labelsOrientation:(x:0,yLeft:0,yRight:0),layers:!((accessors:!('8908dfa8-6b37-4dff-a218-d253a2a1e4d0'),layerId:'27fdb8d9-1c3b-4028-8c0b-7cda5064801e',layerType:data,palette:(name:default,type:palette),position:top,seriesType:bar_stacked,showGridlines:!f,splitAccessor:'0c6336a5-0069-4677-95bf-106bfadb94ae',xAccessor:b996f933-7b55-47d0-a1c6-68aa8f42bbbe)),legend:(isInside:!f,isVisible:!t,legendStats:!(),position:top),preferredSeriesType:bar_percentage_stacked,tickLabelsVisibilitySettings:(x:!t,yLeft:!t,yRight:!t),valueLabels:hide)),title:'',type:lens,visualizationType:lnsXY),enhancements:(dynamicActions:(events:!())),hidePanelTitles:!f),panelIndex:d1b26fe0-1a87-469e-a445-1d0baa028707,title:'Failures%20By%20Branch',type:lens),(gridData:(h:16,i:'221ee73c-ddc3-4b33-b744-532f1d8f9ff8',w:18,x:12,y:4),panelConfig:(attributes:(description:'',references:!((id:fbbdc689-be23-4b3d-8057-aa402e9ed0c5,name:indexpattern-datasource-layer-fc28e04a-9d1e-41d4-aefd-c7c86e2ba8c7,type:index-pattern)),state:(adHocDataViews:(),datasourceStates:(formBased:(currentIndexPatternId:fbbdc689-be23-4b3d-8057-aa402e9ed0c5,layers:(fc28e04a-9d1e-41d4-aefd-c7c86e2ba8c7:(columnOrder:!(c284db41-8b0a-4cc2-a184-d8cb3e41fce4,'274fe047-621b-4fbc-bccf-13361bbe8448',fc915caa-c5ea-48f4-a067-e607632f5266),columns:('274fe047-621b-4fbc-bccf-13361bbe8448':(dataType:string,isBucketed:!t,label:Filters,operationType:filters,params:(filters:!((input:(language:kuery,query:'failed:%20false%20and%20skipped:%20false'),label:Success),(input:(language:kuery,query:'skipped:%20true'),label:Skipped),(input:(language:kuery,query:'failed:%20true%20'),label:Failed))),scale:ordinal),c284db41-8b0a-4cc2-a184-d8cb3e41fce4:(dataType:date,isBucketed:!t,label:'@timestamp',operationType:date_histogram,params:(dropPartials:!f,includeEmptyRows:!t,interval:d),scale:interval,sourceField:'@timestamp'),fc915caa-c5ea-48f4-a067-e607632f5266:(customLabel:!t,dataType:number,isBucketed:!f,label:Tests,operationType:count,params:(emptyAsNull:!t),scale:ratio,sourceField:records)),ignoreGlobalFilters:!f,incompleteColumns:(),indexPatternId:fbbdc689-be23-4b3d-8057-aa402e9ed0c5,sampling:1))),indexpattern:(layers:()),textBased:(layers:())),filters:!(('$state':(store:appState),meta:(alias:!n,disabled:!f,field:suite,index:'8e220a72-8586-4840-b7c9-18182cce040d',key:suite,negate:!f,params:(query:!f),type:phrase),query:(match_phrase:(suite:!f)))),internalReferences:!(),query:(language:kuery,query:''),visualization:(axisTitlesVisibilitySettings:(x:!f,yLeft:!t,yRight:!t),fittingFunction:None,gridlinesVisibilitySettings:(x:!t,yLeft:!t,yRight:!t),hideEndzones:!t,labelsOrientation:(x:0,yLeft:0,yRight:0),layers:!((accessors:!(fc915caa-c5ea-48f4-a067-e607632f5266),colorMapping:(assignmentMode:manual,assignments:!((color:(colorCode:%2300cb9c,type:colorCode),rule:(type:matchExactly,values:!(Success)),touched:!t),(color:(colorIndex:1,paletteId:neutral,type:categorical),rule:(type:matchExactly,values:!(Skipped)),touched:!t),(color:(colorIndex:9,paletteId:eui_amsterdam_color_blind,type:categorical),rule:(type:matchExactly,values:!(Failed)),touched:!t)),colorMode:(type:categorical),paletteId:eui_amsterdam_color_blind,specialAssignments:!((color:(colorIndex:1,paletteId:neutral,type:categorical),rule:(type:other),touched:!f))),layerId:fc28e04a-9d1e-41d4-aefd-c7c86e2ba8c7,layerType:data,position:top,seriesType:bar_stacked,showGridlines:!f,splitAccessor:'274fe047-621b-4fbc-bccf-13361bbe8448',xAccessor:c284db41-8b0a-4cc2-a184-d8cb3e41fce4)),legend:(horizontalAlignment:left,isInside:!f,isVisible:!t,position:top,verticalAlignment:bottom),preferredSeriesType:bar_stacked,tickLabelsVisibilitySettings:(x:!t,yLeft:!t,yRight:!t),valueLabels:hide)),title:'',type:lens,visualizationType:lnsXY),enhancements:(),hidePanelTitles:!f),panelIndex:'221ee73c-ddc3-4b33-b744-532f1d8f9ff8',title:'Test%20Results',type:lens),(gridData:(h:16,i:'2208b623-0c03-434a-982f-be95f6ec967c',w:24,x:0,y:20),panelConfig:(attributes:(description:'',references:!((id:fbbdc689-be23-4b3d-8057-aa402e9ed0c5,name:indexpattern-datasource-layer-7d02ffda-ba9c-49c9-92d1-f732438fa2b1,type:index-pattern),(id:fbbdc689-be23-4b3d-8057-aa402e9ed0c5,name:'10ebf466-c9fc-42e1-a77f-d1e64dda7530',type:index-pattern),(id:fbbdc689-be23-4b3d-8057-aa402e9ed0c5,name:'9074483e-b602-4235-9405-7bbdd1976d13',type:index-pattern)),state:(adHocDataViews:(),datasourceStates:(formBased:(layers:('7d02ffda-ba9c-49c9-92d1-f732438fa2b1':(columnOrder:!('3148ddf0-8fdf-43df-8ab5-7f5859a8bf05',b890eb79-8854-4935-925b-5c442caf50dc),columns:('3148ddf0-8fdf-43df-8ab5-7f5859a8bf05':(customLabel:!t,dataType:string,isBucketed:!t,label:Test,operationType:terms,params:(accuracyMode:!f,exclude:!(),excludeIsRegex:!f,include:!(),includeIsRegex:!f,missingBucket:!f,orderBy:(columnId:b890eb79-8854-4935-925b-5c442caf50dc,type:column),orderDirection:desc,otherBucket:!f,parentFormat:(id:multi_terms),secondaryFields:!(className.keyword,name.keyword),size:10),scale:ordinal,sourceField:task.keyword),b890eb79-8854-4935-925b-5c442caf50dc:(customLabel:!t,dataType:number,isBucketed:!f,label:Failures,operationType:count,params:(emptyAsNull:!t),scale:ratio,sourceField:records)),incompleteColumns:(),sampling:1))),indexpattern:(layers:()),textBased:(layers:())),filters:!(('$state':(store:appState),meta:(alias:!n,disabled:!f,field:failed,index:'10ebf466-c9fc-42e1-a77f-d1e64dda7530',key:failed,negate:!f,params:(query:!t),type:phrase),query:(match_phrase:(failed:!t))),('$state':(store:appState),meta:(alias:!n,disabled:!f,field:suite,index:'9074483e-b602-4235-9405-7bbdd1976d13',key:suite,negate:!f,params:(query:!f),type:phrase),query:(match_phrase:(suite:!f)))),internalReferences:!(),query:(language:kuery,query:''),visualization:(columns:!((alignment:left,columnId:'3148ddf0-8fdf-43df-8ab5-7f5859a8bf05',isTransposed:!f,width:832),(columnId:b890eb79-8854-4935-925b-5c442caf50dc,hidden:!f,isTransposed:!f)),layerId:'7d02ffda-ba9c-49c9-92d1-f732438fa2b1',layerType:data)),title:'',type:lens,visualizationType:lnsDatatable),enhancements:(dynamicActions:(events:!())),hidePanelTitles:!f),panelIndex:'2208b623-0c03-434a-982f-be95f6ec967c',title:'Top%20Failures',type:lens),(gridData:(h:16,i:'449c1f21-f509-4fff-b487-c9c3c24a5c60',w:24,x:24,y:20),panelConfig:(attributes:(description:'',references:!((id:fbbdc689-be23-4b3d-8057-aa402e9ed0c5,name:indexpattern-datasource-layer-b8536ffe-162e-4f73-86b1-8be7ed896749,type:index-pattern)),state:(adHocDataViews:(),datasourceStates:(formBased:(currentIndexPatternId:fbbdc689-be23-4b3d-8057-aa402e9ed0c5,layers:(b8536ffe-162e-4f73-86b1-8be7ed896749:(columnOrder:!(b839284a-259b-40a3-aef1-0866ec8eaee6,'28b704e9-f885-4f08-bfdc-8aaf74de1fc5'),columns:('28b704e9-f885-4f08-bfdc-8aaf74de1fc5':(customLabel:!t,dataType:number,isBucketed:!f,label:Failures,operationType:count,params:(emptyAsNull:!t),scale:ratio,sourceField:records),b839284a-259b-40a3-aef1-0866ec8eaee6:(customLabel:!t,dataType:string,isBucketed:!t,label:'Failure%20message',operationType:terms,params:(exclude:!(),excludeIsRegex:!f,include:!(),includeIsRegex:!f,missingBucket:!f,orderBy:(columnId:'28b704e9-f885-4f08-bfdc-8aaf74de1fc5',type:column),orderDirection:desc,otherBucket:!t,parentFormat:(id:terms),size:10),scale:ordinal,sourceField:failureMessage.keyword)),incompleteColumns:(),indexPatternId:fbbdc689-be23-4b3d-8057-aa402e9ed0c5,sampling:1))),indexpattern:(layers:()),textBased:(layers:())),filters:!(),internalReferences:!(),query:(language:kuery,query:''),visualization:(columns:!((columnId:b839284a-259b-40a3-aef1-0866ec8eaee6,isTransposed:!f,width:1002.5),(columnId:'28b704e9-f885-4f08-bfdc-8aaf74de1fc5',isTransposed:!f)),layerId:b8536ffe-162e-4f73-86b1-8be7ed896749,layerType:data,rowHeight:auto)),title:'',type:lens,visualizationType:lnsDatatable),enhancements:(),hidePanelTitles:!f),panelIndex:'449c1f21-f509-4fff-b487-c9c3c24a5c60',title:'Top%20Failure%20Exceptions',type:lens)))) on 2024-03-04 with the 8.12 branch. The build-stats goes back to only 2023-10 so that it could have started early. Since it seems to fail regularly every 1-2 month, I intend to believe it did start with 8.12. It could also be a problem with the repo analysis code which would make it a low-risk. I am taking a more cautious approach for the risk estimation.

ywangd commented 1 week ago

@DaveCTurner The test failures are all about compare-and-register with both uncontended and contended actions (it supposed to observe a value but got MISSING instead). I looked through the changes in 8.12 but could not find anything suspicious. Looking at the code, the following line in compareAndExchangeRegister seems odd to me.

https://github.com/elastic/elasticsearch/blob/3b0d7e0eee83bf201ea7d0a0d147efdb858eefde/modules/repository-gcs/src/main/java/org/elasticsearch/repositories/gcs/GoogleCloudStorageBlobStore.java#L711-L712

The method returns MISSING when the server responds 429. I don't think we do the same thing for either s3 and Azure? Intuitvely, we should instead retry or throw exception?

DaveCTurner commented 1 week ago

Yeah this is a little tricky. For the contended register analysis a MISSING here triggers a retry, whereas ignoring 429s would mean we rethrow the exception which would cause the analysis to fail. But since then we added the uncontended register checks which we assert should not fail. The only thing we can do on a 429 to satisfy that would be to retry.

ywangd commented 1 week ago

It also failed for contended register

[repository] register [test-register-contended-VdLtbAL9Rl6TQTaO6ecSeQ] should have value [10] but instead had value [OptionalBytesReference[MISSING]]

Even though we retry on MISSING for contended register, the final verification is performed with compareAndExchangeRegister underlying which can still fail due to 429.

https://github.com/elastic/elasticsearch/blob/9854fdc21572d47cb8feeda87894391ebaa2bb7d/x-pack/plugin/snapshot-repo-test-kit/src/main/java/org/elasticsearch/repositories/blobstore/testkit/analyze/RepositoryAnalyzeAction.java#L669-L686

DaveCTurner commented 1 week ago

Even though we retry on MISSING for contended register, the final verification

Right yeah the final verification is also uncontended so we expect it to succeed. Again, I think we should handle 429s internally rather than bubbling them up to the caller as MISSING to trigger a higher-level retry.