[CI] EnterpriseGeoIpDownloaderIT testEnterpriseDownloaderTask failing

elasticsearchmachine commented 1 month ago

Build Scans:

Reproduction Line:

./gradlew ':modules:ingest-geoip:internalClusterTest' --tests "org.elasticsearch.ingest.geoip.EnterpriseGeoIpDownloaderIT.testEnterpriseDownloaderTask" -Dtests.seed=DABFA3621C4F496C -Dtests.jvm.argline="-Dgeoip_use_service=true" -Dtests.locale=hi -Dtests.timezone=US/Central -Druntime.java=22

Applicable branches: 8.15

Reproduces locally?: N/A

Failure History: See dashboard&_a=(controlGroupInput:(chainingSystem:HIERARCHICAL,controlStyle:twoLine,ignoreParentSettings:(ignoreFilters:!f,ignoreQuery:!f,ignoreTimerange:!f,ignoreValidations:!t),panels:('0c0c9cb8-ccd2-45c6-9b13-96bac4abc542':(explicitInput:(dataViewId:fbbdc689-be23-4b3d-8057-aa402e9ed0c5,enhancements:(),fieldName:task.keyword,grow:!t,id:'0c0c9cb8-ccd2-45c6-9b13-96bac4abc542',searchTechnique:wildcard,selectedOptions:!(),singleSelect:!t,title:'Gradle%20Task',width:medium),grow:!t,order:0,type:optionsListControl,width:small),'144933da-5c1b-4257-a969-7f43455a7901':(explicitInput:(dataViewId:fbbdc689-be23-4b3d-8057-aa402e9ed0c5,enhancements:(),fieldName:name.keyword,grow:!t,id:'144933da-5c1b-4257-a969-7f43455a7901',searchTechnique:wildcard,selectedOptions:!('testEnterpriseDownloaderTask'),title:Test,width:medium),grow:!t,order:2,type:optionsListControl,width:medium),'4e6ad9d6-6fdc-4fcc-bf1a-aa6ca79e0850':(explicitInput:(dataViewId:fbbdc689-be23-4b3d-8057-aa402e9ed0c5,enhancements:(),fieldName:className.keyword,grow:!t,id:'4e6ad9d6-6fdc-4fcc-bf1a-aa6ca79e0850',searchTechnique:wildcard,selectedOptions:!('org.elasticsearch.ingest.geoip.EnterpriseGeoIpDownloaderIT'),title:Suite,width:medium),grow:!t,order:1,type:optionsListControl,width:medium)))))

Failure Message:

java.lang.AssertionError: timed out waiting for green state

Issue Reasons:

[8.15] 12 consecutive failures in step third-party_geoip
[8.15] 13 failures in test testEnterpriseDownloaderTask (3.8% fail rate in 345 executions)
[8.15] 12 failures in step third-party_geoip (100.0% fail rate in 12 executions)
[8.15] 12 failures in pipeline elasticsearch-periodic (100.0% fail rate in 12 executions)

Note: This issue was created using new test triage automation. Please report issues or feedback to es-delivery.

elasticsearchmachine commented 1 month ago

Pinging @elastic/es-data-management (Team:Data Management)

elasticsearchmachine commented 1 month ago

This has been muted on branch main

Mute Reasons:

[main] 2 consecutive failures in step third-party_geoip
[main] 4 failures in test testEnterpriseDownloaderTask (5.9% fail rate in 68 executions)
[main] 2 failures in step third-party_geoip (100.0% fail rate in 2 executions)
[main] 2 failures in pipeline elasticsearch-periodic (100.0% fail rate in 2 executions)

Build Scans:

joegallo commented 1 month ago

I back ported the original test to 8.15, so it's likely to cause the same problems there -- whatever fix we end up merging should also be merged to 8.15.

I did not backport the muting of the test (I'm not sure what the process is there and I don't want to interfere with the robots), but I figure if it comes up on that branch then the automation will mute it there automatically, too, so I'm not sweating it right now.

elasticsearchmachine commented 1 month ago

This has been muted on branch main

Mute Reasons:

[main] 3 consecutive failures in step third-party_geoip
[main] 5 failures in test testEnterpriseDownloaderTask (4.5% fail rate in 112 executions)
[main] 3 failures in step third-party_geoip (100.0% fail rate in 3 executions)
[main] 3 failures in pipeline elasticsearch-periodic (100.0% fail rate in 3 executions)

Build Scans:

elastic / elasticsearch

[CI] EnterpriseGeoIpDownloaderIT testEnterpriseDownloaderTask failing #111002