elasticmachine
commented
2 years ago Pinging @elastic/es-distributed (Team:Distributed)
Open fcofdez opened 2 years ago
elasticmachine
commented
2 years ago Pinging @elastic/es-distributed (Team:Distributed)
pxsalehi
commented
2 years ago I've looked into this. Unfortunately no clue why it fails. That being said, this is nothing specific to 7.17. The test has failed only 3 times since 1 Jan (~2800 runs!), and that's across all branches. I haven't been able to reproduce it.
It seems both tests (indices segment and cat segment) in that suite will result in the same request type (IndicesSegmentsRequest), but only one of those tests has failed!
Since both tests are basically doing the same thing, my best guess is either there is some interference between the two (we could avoid sharing the cluster and use a ESIntegTestCase.Scope.TEST, or we simplify the assertions that the test makes. By the latter I mean, e.g. here if we have already asserted that we have received the cancellation exception, do we still need to reach into the internals of the node and verify the task status? (CC: @DaveCTurner and @fcofdez)
DaveCTurner
commented
2 years ago FWIW I ran tens of thousands of iterations of the whole IndicesSegmentsRestCancellationIT suite on a CPU-stressed machine (usually good for triggering rare concurrency issues) without a single problem. I wonder if we're receiving a segments request from an external influence (e.g. another test suite running concurrently).
pxsalehi
commented
2 years ago Yes. I also tried 5k-6k runs of the suite on two branches, with no failure.
pxsalehi
commented
2 years ago I double checked this with @original-brownbear and he agrees that there is not much to do for this failure other than increasing the timeout, which was done a few weeks ago. We can just leave this open until next week and then close it if there are no new failures.
pxsalehi
commented
2 years ago 
kingherc
commented
1 year ago Re-appeared in 8.9 at https://gradle-enterprise.elastic.co/s/szrl2yl34rtz2
pxsalehi
commented
1 year ago This issue is from one year ago! I think a new issue would be more fitting. The code state one year apart might be very different, and re-opening this instead of a new one might just be misleading when investigating the issue.
kingherc
commented
1 year ago I believe the error is so similar, that the history might be relevant. I see commits referenced, that might have been close to the root cause, but not exactly. So, I'd leave this to the judgement of whoever handles this ticket. They can close this and open a new one.
albertzaharovits
commented
1 year ago New failure today on main: https://gradle-enterprise.elastic.co/s/zlqhwimhv7cju/tests/:qa:smoke-test-http:javaRestTest/org.elasticsearch.http.IndicesSegmentsRestCancellationIT/testCatSegmentsRestCancellation?top-execution=1
junit.framework.AssertionFailedError: task 56/indices:monitor/segments[n] not cancelled
at __randomizedtesting.SeedInfo.seed([52231FF4B54560F8:554258C7DC097E3]:0)
at junit.framework.Assert.fail(Assert.java:57)
at junit.framework.Assert.assertTrue(Assert.java:22)
at junit.framework.TestCase.assertTrue(TestCase.java:192)
at org.elasticsearch.test.TaskAssertions.lambda$assertAllCancellableTasksAreCancelled$2(TaskAssertions.java:74)
at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:1173)
at org.elasticsearch.test.TaskAssertions.assertAllCancellableTasksAreCancelled(TaskAssertions.java:65)
at org.elasticsearch.http.BlockedSearcherRestCancellationTestCase.runTest(BlockedSearcherRestCancellationTestCase.java:111)
at org.elasticsearch.http.IndicesSegmentsRestCancellationIT.testCatSegmentsRestCancellation(IndicesSegmentsRestCancellationIT.java:21) And 4 more in the past 7 days https://gradle-enterprise.elastic.co/scans/tests?search.startTimeMax=1689086885271&search.startTimeMin=1688418000000&search.timeZoneId=Europe/Bucharest&tests.container=org.elasticsearch.http.IndicesSegmentsRestCancellationIT&tests.test=testCatSegmentsRestCancellation on various branches.
volodk85
commented
8 months ago According to CI stats this test hasn't failed for 5+ months. Closing for now. Please re-open if new failures get encountered
alex-spies
commented
6 months ago 
cbuescher
commented
4 months ago 
nielsbauman
commented
4 months ago 
mhl-b
commented
2 months ago Spent more time figuring out whats happening. Tried few hypothesis, none worked:
I noticed in one gradle scan that we failed on final assertion to finish tasks, which means task somehow appear after we successfully cancelled and asserted that everything is cancelled. I think @DaveCTurner comment about external interference is most reasonable. I opened PR #110680 that should notify of possibility of external influence.
DaveCTurner
commented
5 days ago Thanks to #110680 the failure at https://gradle-enterprise.elastic.co/s/6bz3jjrcspj22 confirms that we're failing to cancel the task that we should be cancelling, which suggests that maybe we're cancelling it too early because, perhaps, of some other request hitting the block in acquireSearcher.
I've opened #113844 to add more logging.
Build scan: https://gradle-enterprise.elastic.co/s/2o4ugc7mhz6si/tests/:qa:smoke-test-http:javaRestTest/org.elasticsearch.http.IndicesSegmentsRestCancellationIT/testCatSegmentsRestCancellation
Reproduction line:
./gradlew ':qa:smoke-test-http:javaRestTest' --tests "org.elasticsearch.http.IndicesSegmentsRestCancellationIT.testCatSegmentsRestCancellation" -Dtests.seed=8D552D969E2896B -Dtests.locale=ar-IQ -Dtests.timezone=MIT -Druntime.java=8Applicable branches: 7.17
Reproduces locally?: Didn't try
Failure history: https://gradle-enterprise.elastic.co/scans/tests?tests.container=org.elasticsearch.http.IndicesSegmentsRestCancellationIT&tests.test=testCatSegmentsRestCancellation
90-day delivery stats dashboard&_a=(controlGroupState:(initialChildControlState:('0c0c9cb8-ccd2-45c6-9b13-96bac4abc542':(dataViewId:fbbdc689-be23-4b3d-8057-aa402e9ed0c5,exclude:!n,existsSelected:!n,fieldName:task.keyword,grow:!t,hideActionBar:!n,hideExclude:!n,hideExists:!n,hideSort:!n,order:0,placeholder:!n,runPastTimeout:!n,searchTechnique:wildcard,selectedOptions:!(),singleSelect:!t,sort:(by:_count,direction:desc),title:'Gradle%20Task',type:optionsListControl,width:medium),'144933da-5c1b-4257-a969-7f43455a7901':(dataViewId:fbbdc689-be23-4b3d-8057-aa402e9ed0c5,exclude:!n,existsSelected:!n,fieldName:name.keyword,grow:!t,hideActionBar:!n,hideExclude:!n,hideExists:!n,hideSort:!n,order:2,placeholder:!n,runPastTimeout:!n,searchTechnique:wildcard,selectedOptions:!(),singleSelect:!n,sort:(by:_count,direction:desc),title:Test,type:optionsListControl,width:medium),'4e6ad9d6-6fdc-4fcc-bf1a-aa6ca79e0850':(dataViewId:fbbdc689-be23-4b3d-8057-aa402e9ed0c5,exclude:!n,existsSelected:!n,fieldName:className.keyword,grow:!t,hideActionBar:!n,hideExclude:!n,hideExists:!n,hideSort:!n,order:1,placeholder:!n,runPastTimeout:!n,searchTechnique:wildcard,selectedOptions:!(org.elasticsearch.http.IndicesSegmentsRestCancellationIT),singleSelect:!n,sort:(by:_count,direction:desc),title:Suite,type:optionsListControl,width:medium))),panels:!((embeddableConfig:(attributes:(description:'',references:!((id:fbbdc689-be23-4b3d-8057-aa402e9ed0c5,name:indexpattern-datasource-layer-b0324444-043a-4e31-8c4c-de06b757a969,type:index-pattern),(id:fbbdc689-be23-4b3d-8057-aa402e9ed0c5,name:'595b302c-fe27-455a-8270-3f32426b668b',type:index-pattern)),state:(adHocDataViews:(),datasourceStates:(formBased:(layers:(b0324444-043a-4e31-8c4c-de06b757a969:(columnOrder:!('498e0348-1dc7-4487-b57c-7a17513c3d90'),columns:('498e0348-1dc7-4487-b57c-7a17513c3d90':(customLabel:!t,dataType:number,isBucketed:!f,label:'Test%20Executions',operationType:count,params:(emptyAsNull:!t),scale:ratio,sourceField:records)),ignoreGlobalFilters:!f,incompleteColumns:(),sampling:1))),indexpattern:(layers:()),textBased:(layers:())),filters:!(('$state':(store:appState),meta:(alias:!n,disabled:!f,field:suite,index:'595b302c-fe27-455a-8270-3f32426b668b',key:suite,negate:!f,params:(query:!f),type:phrase),query:(match_phrase:(suite:!f)))),internalReferences:!(),query:(language:kuery,query:''),visualization:(layerId:b0324444-043a-4e31-8c4c-de06b757a969,layerType:data,metricAccessor:'498e0348-1dc7-4487-b57c-7a17513c3d90')),title:'',type:lens,visualizationType:lnsMetric),enhancements:(),hidePanelTitles:!t),gridData:(h:4,i:'705187f8-02ab-4858-971b-6e24e9a23df7',w:12,x:0,y:0),panelIndex:'705187f8-02ab-4858-971b-6e24e9a23df7',type:lens),(embeddableConfig:(attributes:(description:'',references:!((id:fbbdc689-be23-4b3d-8057-aa402e9ed0c5,name:indexpattern-datasource-layer-6d8f3e80-f6b4-4cfc-8afe-a77ddfa58dc6,type:index-pattern),(id:fbbdc689-be23-4b3d-8057-aa402e9ed0c5,name:'910c9261-f4e3-4509-affb-74639836b5a2',type:index-pattern)),state:(adHocDataViews:(),datasourceStates:(formBased:(layers:('6d8f3e80-f6b4-4cfc-8afe-a77ddfa58dc6':(columnOrder:!(e456e0ba-6886-4c85-9628-e28eb32b02c7),columns:(e456e0ba-6886-4c85-9628-e28eb32b02c7:(customLabel:!t,dataType:number,filter:(language:kuery,query:'failed%20:%20true'),isBucketed:!f,label:'Failed%20Tests',operationType:count,params:(emptyAsNull:!f),scale:ratio,sourceField:records)),ignoreGlobalFilters:!f,incompleteColumns:(),sampling:1))),indexpattern:(layers:()),textBased:(layers:())),filters:!(('$state':(store:appState),meta:(alias:!n,disabled:!f,field:suite,index:'910c9261-f4e3-4509-affb-74639836b5a2',key:suite,negate:!f,params:(query:!f),type:phrase),query:(match_phrase:(suite:!f)))),internalReferences:!(),query:(language:kuery,query:''),visualization:(color:%23e48673,layerId:'6d8f3e80-f6b4-4cfc-8afe-a77ddfa58dc6',layerType:data,metricAccessor:e456e0ba-6886-4c85-9628-e28eb32b02c7)),title:'',type:lens,visualizationType:lnsMetric),enhancements:()),gridData:(h:4,i:'34980d70-46e3-4b1f-a75a-9b5224f496b1',w:12,x:12,y:0),panelIndex:'34980d70-46e3-4b1f-a75a-9b5224f496b1',type:lens),(embeddableConfig:(attributes:(description:'',references:!((id:fbbdc689-be23-4b3d-8057-aa402e9ed0c5,name:indexpattern-datasource-layer-d20ec2fa-0df6-443f-bf5d-5cc3cc23f06b,type:index-pattern)),state:(adHocDataViews:(),datasourceStates:(formBased:(layers:(d20ec2fa-0df6-443f-bf5d-5cc3cc23f06b:(columnOrder:!('3e642804-8b6c-4b14-82df-83005e80073e'),columns:('3e642804-8b6c-4b14-82df-83005e80073e':(customLabel:!t,dataType:number,isBucketed:!f,label:'Build%20Executions',operationType:unique_count,params:(emptyAsNull:!t),scale:ratio,sourceField:build.scanId.keyword)),ignoreGlobalFilters:!f,incompleteColumns:(),sampling:1))),indexpattern:(layers:()),textBased:(layers:())),filters:!(),internalReferences:!(),query:(language:kuery,query:''),visualization:(layerId:d20ec2fa-0df6-443f-bf5d-5cc3cc23f06b,layerType:data,metricAccessor:'3e642804-8b6c-4b14-82df-83005e80073e')),title:'',type:lens,visualizationType:lnsMetric),enhancements:()),gridData:(h:4,i:'1163896b-34b5-47b5-9c6f-b6a532848e66',w:12,x:24,y:0),panelIndex:'1163896b-34b5-47b5-9c6f-b6a532848e66',type:lens),(embeddableConfig:(attributes:(description:'',references:!((id:fbbdc689-be23-4b3d-8057-aa402e9ed0c5,name:indexpattern-datasource-layer-6f78bf93-e05f-411e-b4a1-9a71d3d1b1bd,type:index-pattern)),state:(adHocDataViews:(),datasourceStates:(formBased:(currentIndexPatternId:fbbdc689-be23-4b3d-8057-aa402e9ed0c5,layers:('6f78bf93-e05f-411e-b4a1-9a71d3d1b1bd':(columnOrder:!('91a2bb54-0196-4223-8ed6-9244aafc40d2','79273e54-80cf-4438-8144-a15cafd15e72','79273e54-80cf-4438-8144-a15cafd15e72X0','79273e54-80cf-4438-8144-a15cafd15e72X3','79273e54-80cf-4438-8144-a15cafd15e72X2','79273e54-80cf-4438-8144-a15cafd15e72X1'),columns:('79273e54-80cf-4438-8144-a15cafd15e72':(customLabel:!f,dataType:number,isBucketed:!f,label:'count()%20%2F%20overall_max(count())',operationType:formula,params:(format:(id:percent,params:(decimals:0)),formula:'count()%20%2F%20overall_max(count())',isFormulaBroken:!f),references:!('79273e54-80cf-4438-8144-a15cafd15e72X3'),scale:ratio),'79273e54-80cf-4438-8144-a15cafd15e72X0':(customLabel:!t,dataType:number,isBucketed:!f,label:'Part%20of%20count()%20%2F%20overall_max(count())',operationType:count,params:(emptyAsNull:!f),scale:ratio,sourceField:records),'79273e54-80cf-4438-8144-a15cafd15e72X1':(customLabel:!t,dataType:number,isBucketed:!f,label:'Part%20of%20count()%20%2F%20overall_max(count())',operationType:count,params:(emptyAsNull:!f),scale:ratio,sourceField:records),'79273e54-80cf-4438-8144-a15cafd15e72X2':(customLabel:!t,dataType:number,isBucketed:!f,label:'Part%20of%20count()%20%2F%20overall_max(count())',operationType:overall_max,references:!('79273e54-80cf-4438-8144-a15cafd15e72X1'),scale:ratio),'79273e54-80cf-4438-8144-a15cafd15e72X3':(customLabel:!t,dataType:number,isBucketed:!f,label:'Part%20of%20count()%20%2F%20overall_max(count())',operationType:math,params:(tinymathAst:(args:!('79273e54-80cf-4438-8144-a15cafd15e72X0','79273e54-80cf-4438-8144-a15cafd15e72X2'),location:(max:30,min:0),name:divide,text:'count()%20%2F%20overall_max(count())',type:function)),references:!('79273e54-80cf-4438-8144-a15cafd15e72X0','79273e54-80cf-4438-8144-a15cafd15e72X2'),scale:ratio),'91a2bb54-0196-4223-8ed6-9244aafc40d2':(dataType:string,isBucketed:!t,label:'Top%2010%20values%20of%20build.tags.keyword',operationType:terms,params:(exclude:!('(CI%7Cmain%7C%5Cd%2B.%5Cd%2B)'),excludeIsRegex:!t,include:!(),includeIsRegex:!f,missingBucket:!f,orderAgg:(dataType:number,isBucketed:!f,label:'Count%20of%20records',operationType:count,params:(emptyAsNull:!t),scale:ratio,sourceField:records),orderBy:(type:custom),orderDirection:desc,otherBucket:!f,parentFormat:(id:terms),size:10),scale:ordinal,sourceField:build.tags.keyword)),ignoreGlobalFilters:!f,incompleteColumns:(),indexPatternId:fbbdc689-be23-4b3d-8057-aa402e9ed0c5,sampling:1))),indexpattern:(layers:()),textBased:(layers:())),filters:!(('$state':(store:appState),meta:(alias:!n,disabled:!f,field:failed,index:c3b65f7a-b534-4b2f-a8f4-a79c70c2b59c,key:failed,negate:!f,params:(query:!t),type:phrase),query:(match_phrase:(failed:!t))),('$state':(store:appState),meta:(alias:!n,disabled:!f,field:suite,index:'7e1054a6-3acb-4a8c-8ac9-0ddab627d9ce',key:suite,negate:!f,params:(query:!f),type:phrase),query:(match_phrase:(suite:!f)))),internalReferences:!(),query:(language:kuery,query:''),visualization:(layers:!((categoryDisplay:default,colorMapping:(assignmentMode:auto,assignments:!(),colorMode:(sort:asc,steps:!((colorIndex:0,paletteId:eui_amsterdam_color_blind,touched:!f,type:categorical)),type:gradient),paletteId:eui_amsterdam_color_blind,specialAssignments:!((color:(colorIndex:1,paletteId:neutral,type:categorical),rule:(type:other),touched:!f))),layerId:'6f78bf93-e05f-411e-b4a1-9a71d3d1b1bd',layerType:data,legendDisplay:default,metrics:!('79273e54-80cf-4438-8144-a15cafd15e72'),nestedLegend:!f,numberDisplay:value,percentDecimals:2,primaryGroups:!('91a2bb54-0196-4223-8ed6-9244aafc40d2'))),shape:treemap)),title:'',type:lens,visualizationType:lnsPie),enhancements:(),hidePanelTitles:!f),gridData:(h:16,i:d35dbbf9-8bc9-498d-b3a3-b560e8ead693,w:12,x:0,y:4),panelIndex:d35dbbf9-8bc9-498d-b3a3-b560e8ead693,title:'Top%20Failure%20Tags',type:lens),(embeddableConfig:(attributes:(description:'',references:!((id:fbbdc689-be23-4b3d-8057-aa402e9ed0c5,name:indexpattern-datasource-layer-6d8f3e80-f6b4-4cfc-8afe-a77ddfa58dc6,type:index-pattern)),state:(adHocDataViews:(),datasourceStates:(formBased:(currentIndexPatternId:fbbdc689-be23-4b3d-8057-aa402e9ed0c5,layers:('6d8f3e80-f6b4-4cfc-8afe-a77ddfa58dc6':(columnOrder:!(e456e0ba-6886-4c85-9628-e28eb32b02c7),columns:(e456e0ba-6886-4c85-9628-e28eb32b02c7:(customLabel:!t,dataType:number,filter:(language:kuery,query:'failed%20:%20true'),isBucketed:!f,label:'Failed%20Builds',operationType:unique_count,params:(emptyAsNull:!f),scale:ratio,sourceField:build.scanId.keyword)),ignoreGlobalFilters:!f,incompleteColumns:(),indexPatternId:fbbdc689-be23-4b3d-8057-aa402e9ed0c5,sampling:1))),indexpattern:(layers:()),textBased:(layers:())),filters:!(),internalReferences:!(),query:(language:kuery,query:''),visualization:(color:%23e48673,layerId:'6d8f3e80-f6b4-4cfc-8afe-a77ddfa58dc6',layerType:data,metricAccessor:e456e0ba-6886-4c85-9628-e28eb32b02c7,showBar:!f)),title:'',type:lens,visualizationType:lnsMetric),enhancements:()),gridData:(h:4,i:c8202da2-f726-40b8-90ac-cb92e467022e,w:12,x:36,y:0),panelIndex:c8202da2-f726-40b8-90ac-cb92e467022e,type:lens),(embeddableConfig:(attributes:(description:'',references:!((id:fbbdc689-be23-4b3d-8057-aa402e9ed0c5,name:indexpattern-datasource-layer-27fdb8d9-1c3b-4028-8c0b-7cda5064801e,type:index-pattern)),state:(adHocDataViews:(),datasourceStates:(formBased:(currentIndexPatternId:fbbdc689-be23-4b3d-8057-aa402e9ed0c5,layers:('27fdb8d9-1c3b-4028-8c0b-7cda5064801e':(columnOrder:!('0c6336a5-0069-4677-95bf-106bfadb94ae',b996f933-7b55-47d0-a1c6-68aa8f42bbbe,'8908dfa8-6b37-4dff-a218-d253a2a1e4d0'),columns:('0c6336a5-0069-4677-95bf-106bfadb94ae':(dataType:string,isBucketed:!t,label:'Top%204%20values%20of%20build.values.Git%20Branch.keyword',operationType:terms,params:(exclude:!(),excludeIsRegex:!f,include:!('(main%7C%5Cd%2B.%5Cd%2B)'),includeIsRegex:!t,missingBucket:!f,orderBy:(columnId:'8908dfa8-6b37-4dff-a218-d253a2a1e4d0',type:column),orderDirection:desc,otherBucket:!f,parentFormat:(id:terms),size:4),scale:ordinal,sourceField:'build.values.Git%20Branch.keyword'),'8908dfa8-6b37-4dff-a218-d253a2a1e4d0':(customLabel:!t,dataType:number,isBucketed:!f,label:Failures,operationType:count,params:(emptyAsNull:!t),scale:ratio,sourceField:records),b996f933-7b55-47d0-a1c6-68aa8f42bbbe:(dataType:date,isBucketed:!t,label:'@timestamp',operationType:date_histogram,params:(dropPartials:!f,includeEmptyRows:!t,interval:d),scale:interval,sourceField:'@timestamp')),incompleteColumns:(),indexPatternId:fbbdc689-be23-4b3d-8057-aa402e9ed0c5,sampling:1))),indexpattern:(layers:()),textBased:(layers:())),filters:!(('$state':(store:appState),meta:(alias:!n,disabled:!f,field:failed,index:'7734cf97-d0b2-4f99-a820-c7a0224e8cd3',key:failed,negate:!f,params:(query:!t),type:phrase),query:(match_phrase:(failed:!t))),('$state':(store:appState),meta:(alias:!n,disabled:!f,field:suite,index:a40b41a1-bb68-44b7-ba67-b11f4354fdfd,key:suite,negate:!f,params:(query:!f),type:phrase),query:(match_phrase:(suite:!f)))),internalReferences:!(),query:(language:kuery,query:'failed%20:%20true'),visualization:(axisTitlesVisibilitySettings:(x:!f,yLeft:!t,yRight:!t),fittingFunction:None,gridlinesVisibilitySettings:(x:!t,yLeft:!t,yRight:!t),hideEndzones:!t,labelsOrientation:(x:0,yLeft:0,yRight:0),layers:!((accessors:!('8908dfa8-6b37-4dff-a218-d253a2a1e4d0'),layerId:'27fdb8d9-1c3b-4028-8c0b-7cda5064801e',layerType:data,palette:(name:default,type:palette),position:top,seriesType:bar_stacked,showGridlines:!f,splitAccessor:'0c6336a5-0069-4677-95bf-106bfadb94ae',xAccessor:b996f933-7b55-47d0-a1c6-68aa8f42bbbe)),legend:(isInside:!f,isVisible:!t,position:top),preferredSeriesType:bar_percentage_stacked,tickLabelsVisibilitySettings:(x:!t,yLeft:!t,yRight:!t),valueLabels:hide,valuesInLegend:!f)),title:'',type:lens,visualizationType:lnsXY),enhancements:(dynamicActions:(events:!())),hidePanelTitles:!f),gridData:(h:16,i:d1b26fe0-1a87-469e-a445-1d0baa028707,w:18,x:30,y:4),panelIndex:d1b26fe0-1a87-469e-a445-1d0baa028707,title:'Failures%20By%20Branch',type:lens),(embeddableConfig:(attributes:(description:'',references:!((id:fbbdc689-be23-4b3d-8057-aa402e9ed0c5,name:indexpattern-datasource-layer-fc28e04a-9d1e-41d4-aefd-c7c86e2ba8c7,type:index-pattern)),state:(adHocDataViews:(),datasourceStates:(formBased:(currentIndexPatternId:fbbdc689-be23-4b3d-8057-aa402e9ed0c5,layers:(fc28e04a-9d1e-41d4-aefd-c7c86e2ba8c7:(columnOrder:!(c284db41-8b0a-4cc2-a184-d8cb3e41fce4,'274fe047-621b-4fbc-bccf-13361bbe8448',fc915caa-c5ea-48f4-a067-e607632f5266),columns:('274fe047-621b-4fbc-bccf-13361bbe8448':(dataType:string,isBucketed:!t,label:Filters,operationType:filters,params:(filters:!((input:(language:kuery,query:'failed:%20false%20and%20skipped:%20false'),label:Success),(input:(language:kuery,query:'skipped:%20true'),label:Skipped),(input:(language:kuery,query:'failed:%20true%20'),label:Failed))),scale:ordinal),c284db41-8b0a-4cc2-a184-d8cb3e41fce4:(dataType:date,isBucketed:!t,label:'@timestamp',operationType:date_histogram,params:(dropPartials:!f,includeEmptyRows:!t,interval:d),scale:interval,sourceField:'@timestamp'),fc915caa-c5ea-48f4-a067-e607632f5266:(customLabel:!t,dataType:number,isBucketed:!f,label:Tests,operationType:count,params:(emptyAsNull:!t),scale:ratio,sourceField:records)),ignoreGlobalFilters:!f,incompleteColumns:(),indexPatternId:fbbdc689-be23-4b3d-8057-aa402e9ed0c5,sampling:1))),indexpattern:(layers:()),textBased:(layers:())),filters:!(('$state':(store:appState),meta:(alias:!n,disabled:!f,field:suite,index:'8e220a72-8586-4840-b7c9-18182cce040d',key:suite,negate:!f,params:(query:!f),type:phrase),query:(match_phrase:(suite:!f)))),internalReferences:!(),query:(language:kuery,query:''),visualization:(axisTitlesVisibilitySettings:(x:!f,yLeft:!t,yRight:!t),fittingFunction:None,gridlinesVisibilitySettings:(x:!t,yLeft:!t,yRight:!t),hideEndzones:!t,labelsOrientation:(x:0,yLeft:0,yRight:0),layers:!((accessors:!(fc915caa-c5ea-48f4-a067-e607632f5266),colorMapping:(assignmentMode:manual,assignments:!((color:(colorCode:%2300cb9c,type:colorCode),rule:(type:matchExactly,values:!(Success)),touched:!t),(color:(colorIndex:1,paletteId:neutral,type:categorical),rule:(type:matchExactly,values:!(Skipped)),touched:!t),(color:(colorIndex:9,paletteId:eui_amsterdam_color_blind,type:categorical),rule:(type:matchExactly,values:!(Failed)),touched:!t)),colorMode:(type:categorical),paletteId:eui_amsterdam_color_blind,specialAssignments:!((color:(colorIndex:1,paletteId:neutral,type:categorical),rule:(type:other),touched:!f))),layerId:fc28e04a-9d1e-41d4-aefd-c7c86e2ba8c7,layerType:data,position:top,seriesType:bar_stacked,showGridlines:!f,splitAccessor:'274fe047-621b-4fbc-bccf-13361bbe8448',xAccessor:c284db41-8b0a-4cc2-a184-d8cb3e41fce4)),legend:(horizontalAlignment:left,isInside:!f,isVisible:!t,position:top,verticalAlignment:bottom),preferredSeriesType:bar_stacked,tickLabelsVisibilitySettings:(x:!t,yLeft:!t,yRight:!t),valueLabels:hide)),title:'',type:lens,visualizationType:lnsXY),enhancements:(),hidePanelTitles:!f),gridData:(h:16,i:'221ee73c-ddc3-4b33-b744-532f1d8f9ff8',w:18,x:12,y:4),panelIndex:'221ee73c-ddc3-4b33-b744-532f1d8f9ff8',title:'Test%20Results',type:lens),(embeddableConfig:(attributes:(description:'',references:!((id:fbbdc689-be23-4b3d-8057-aa402e9ed0c5,name:indexpattern-datasource-layer-7d02ffda-ba9c-49c9-92d1-f732438fa2b1,type:index-pattern),(id:fbbdc689-be23-4b3d-8057-aa402e9ed0c5,name:'10ebf466-c9fc-42e1-a77f-d1e64dda7530',type:index-pattern),(id:fbbdc689-be23-4b3d-8057-aa402e9ed0c5,name:'9074483e-b602-4235-9405-7bbdd1976d13',type:index-pattern)),state:(adHocDataViews:(),datasourceStates:(formBased:(layers:('7d02ffda-ba9c-49c9-92d1-f732438fa2b1':(columnOrder:!('3148ddf0-8fdf-43df-8ab5-7f5859a8bf05',b890eb79-8854-4935-925b-5c442caf50dc),columns:('3148ddf0-8fdf-43df-8ab5-7f5859a8bf05':(customLabel:!t,dataType:string,isBucketed:!t,label:Test,operationType:terms,params:(accuracyMode:!f,exclude:!(),excludeIsRegex:!f,include:!(),includeIsRegex:!f,missingBucket:!f,orderBy:(columnId:b890eb79-8854-4935-925b-5c442caf50dc,type:column),orderDirection:desc,otherBucket:!f,parentFormat:(id:multi_terms),secondaryFields:!(className.keyword,name.keyword),size:10),scale:ordinal,sourceField:task.keyword),b890eb79-8854-4935-925b-5c442caf50dc:(customLabel:!t,dataType:number,isBucketed:!f,label:Failures,operationType:count,params:(emptyAsNull:!t),scale:ratio,sourceField:records)),incompleteColumns:(),sampling:1))),indexpattern:(layers:()),textBased:(layers:())),filters:!(('$state':(store:appState),meta:(alias:!n,disabled:!f,field:failed,index:'10ebf466-c9fc-42e1-a77f-d1e64dda7530',key:failed,negate:!f,params:(query:!t),type:phrase),query:(match_phrase:(failed:!t))),('$state':(store:appState),meta:(alias:!n,disabled:!f,field:suite,index:'9074483e-b602-4235-9405-7bbdd1976d13',key:suite,negate:!f,params:(query:!f),type:phrase),query:(match_phrase:(suite:!f)))),internalReferences:!(),query:(language:kuery,query:''),visualization:(columns:!((alignment:left,columnId:'3148ddf0-8fdf-43df-8ab5-7f5859a8bf05',isTransposed:!f,width:832),(columnId:b890eb79-8854-4935-925b-5c442caf50dc,hidden:!f,isTransposed:!f)),layerId:'7d02ffda-ba9c-49c9-92d1-f732438fa2b1',layerType:data)),title:'',type:lens,visualizationType:lnsDatatable),enhancements:(dynamicActions:(events:!())),hidePanelTitles:!f),gridData:(h:16,i:'2208b623-0c03-434a-982f-be95f6ec967c',w:24,x:0,y:20),panelIndex:'2208b623-0c03-434a-982f-be95f6ec967c',title:'Top%20Failures',type:lens),(embeddableConfig:(attributes:(description:'',references:!((id:fbbdc689-be23-4b3d-8057-aa402e9ed0c5,name:indexpattern-datasource-layer-b8536ffe-162e-4f73-86b1-8be7ed896749,type:index-pattern)),state:(adHocDataViews:(),datasourceStates:(formBased:(currentIndexPatternId:fbbdc689-be23-4b3d-8057-aa402e9ed0c5,layers:(b8536ffe-162e-4f73-86b1-8be7ed896749:(columnOrder:!(b839284a-259b-40a3-aef1-0866ec8eaee6,'28b704e9-f885-4f08-bfdc-8aaf74de1fc5'),columns:('28b704e9-f885-4f08-bfdc-8aaf74de1fc5':(customLabel:!t,dataType:number,isBucketed:!f,label:Failures,operationType:count,params:(emptyAsNull:!t),scale:ratio,sourceField:records),b839284a-259b-40a3-aef1-0866ec8eaee6:(customLabel:!t,dataType:string,isBucketed:!t,label:'Failure%20message',operationType:terms,params:(exclude:!(),excludeIsRegex:!f,include:!(),includeIsRegex:!f,missingBucket:!f,orderBy:(columnId:'28b704e9-f885-4f08-bfdc-8aaf74de1fc5',type:column),orderDirection:desc,otherBucket:!t,parentFormat:(id:terms),size:10),scale:ordinal,sourceField:failureMessage.keyword)),incompleteColumns:(),indexPatternId:fbbdc689-be23-4b3d-8057-aa402e9ed0c5,sampling:1))),indexpattern:(layers:()),textBased:(layers:())),filters:!(),internalReferences:!(),query:(language:kuery,query:''),visualization:(columns:!((columnId:b839284a-259b-40a3-aef1-0866ec8eaee6,isTransposed:!f,width:1002.5),(columnId:'28b704e9-f885-4f08-bfdc-8aaf74de1fc5',isTransposed:!f)),layerId:b8536ffe-162e-4f73-86b1-8be7ed896749,layerType:data,rowHeight:auto)),title:'',type:lens,visualizationType:lnsDatatable),enhancements:(),hidePanelTitles:!f),gridData:(h:16,i:'449c1f21-f509-4fff-b487-c9c3c24a5c60',w:24,x:24,y:20),panelIndex:'449c1f21-f509-4fff-b487-c9c3c24a5c60',title:'Top%20Failure%20Exceptions',type:lens))))
Failure excerpt: