search

elastic / elasticsearch

Free and Open Source, Distributed, RESTful Search Engine
https://www.elastic.co/products/elasticsearch
Other
1.49k stars 24.88k forks source link

[CI] OldRepositoryAccessIT testOldRepoAccess failing #115631

Open elasticsearchmachine opened 1 month ago

elasticsearchmachine commented 1 month ago

Build Scans:

Reproduction Line:

./gradlew ":x-pack:qa:repository-old-versions:javaRestTestBeforeRestart#5_0_0" --tests "org.elasticsearch.oldrepos.OldRepositoryAccessIT.testOldRepoAccess" -Dtests.seed=360E9472706BB500 -Dtests.locale=de-DE -Dtests.timezone=SystemV/PST8 -Druntime.java=22

Applicable branches: main

Reproduces locally?: N/A

Failure History: See dashboard&_a=(controlGroupState:(initialChildControlState:('0c0c9cb8-ccd2-45c6-9b13-96bac4abc542':(dataViewId:fbbdc689-be23-4b3d-8057-aa402e9ed0c5,fieldName:task.keyword,order:0,selectedOptions:!(),title:'GradleTask',type:optionsListControl),'4e6ad9d6-6fdc-4fcc-bf1a-aa6ca79e0850':(dataViewId:fbbdc689-be23-4b3d-8057-aa402e9ed0c5,fieldName:className.keyword,order:1,selectedOptions:!(org.elasticsearch.oldrepos.OldRepositoryAccessIT),title:'Suite',type:optionsListControl),'144933da-5c1b-4257-a969-7f43455a7901':(dataViewId:fbbdc689-be23-4b3d-8057-aa402e9ed0c5,fieldName:name.keyword,order:2,selectedOptions:!(testOldRepoAccess),title:'Test',type:optionsListControl)))))

Failure Message:

java.lang.AssertionError: expected:<1> but was:<0>

Issue Reasons:

Note: This issue was created using new test triage automation. Please report issues or feedback to es-delivery.

elasticsearchmachine commented 1 month ago

This has been muted on branch main

Mute Reasons:

Build Scans:

elasticsearchmachine commented 1 month ago

Pinging @elastic/es-search-foundations (Team:Search Foundations)

elasticsearchmachine commented 1 month ago

Pinging @elastic/es-distributed (Team:Distributed)

henningandersen commented 1 month ago

Looks like this line is the failure:

                assertEquals(numberOfShards, searchResponse.getSkippedShards());

marking as low risk since this only concerns shard skipping and it seems we should expect other tests to catch this.

Assigning search team as well, to get their eyes on this.

andreidan commented 1 month ago

++ thanks @henningandersen Apologies for the noise

andreidan commented 1 month ago

I'm unable to run this test:

./gradlew ":x-pack:qa:repository-old-versions:javaRestTestBeforeRestart#5_0_0" --tests "org.elasticsearch.oldrepos.OldRepositoryAccessIT.testOldRepoAccess" -Dtests.seed=360E9472706BB500 -Dtests.locale=de-DE -Dtests.timezone=SystemV/PST8 -Druntime.java=22

due to 

  [log]
    Running [/Users/andrei/.gradle/caches/8.10.2/transforms/ccbbe6228fc0f642d5c0a44eb8738e9a/transformed/elasticsearch-5.0.0.zip/elasticsearch-5.0.0/bin/elasticsearch, -p, pid]
    port not found

@elastic/es-delivery is this something you can help with?

mark-vieira commented 1 month ago

@elastic/es-delivery is this something you can help with?

I have the same problem locally, and I believe the issue is that you cannot run these tests on an aarch64 system since old versions of Elasticsearch are x64 only.

nik9000 commented 1 week ago

Another: https://gradle-enterprise.elastic.co/s/gedpv26kjtgks/tests/overview?outcome=FAILED

elasticsearchmachine commented 1 week ago

This has been muted on branch 8.x

Mute Reasons:

Build Scans:

breskeby commented 1 week ago

I noticed yesterday that this hasn't been working for a while on osx due to an issue with the jdk download plugin and how it handles the Zulu jdk 8 download / artifact transform. This bit should be solved via https://github.com/elastic/elasticsearch/pull/117120

cbuescher commented 1 week ago

At least in one of the build-scans above I can see

OldRepositoryAccessIT > testOldRepoAccess FAILED
    java.lang.AssertionError: expected:<1> but was:<0>
        at __randomizedtesting.SeedInfo.seed([A86299CB7C4BA6A0:5A15EC8D39D25895]:0)
[...]]
        at org.elasticsearch.oldrepos.OldRepositoryAccessIT.assertDocs(OldRepositoryAccessIT.java:487)
        at org.elasticsearch.oldrepos.OldRepositoryAccessIT.restoreMountAndVerify(OldRepositoryAccessIT.java:329)
        at org.elasticsearch.oldrepos.OldRepositoryAccessIT.beforeRestart(OldRepositoryAccessIT.java:253)
        at org.elasticsearch.oldrepos.OldRepositoryAccessIT.runTest(OldRepositoryAccessIT.java:109)
        at org.elasticsearch.oldrepos.OldRepositoryAccessIT.testOldRepoAccess(OldRepositoryAccessIT.java:77)

That line in OldRepositoryAccessIT was changed recently (25/10) by https://github.com/elastic/elasticsearch/pull/115314/files#diff-089e10b8f461c2e6a342b3fdb4d430eeea6284cf807472471244b532193a5678R487, so at least that would coincide with the first ocurrences of this failure. I'll do some digging if this reproduces or if not, why...

elasticsearchmachine commented 1 week ago

This has been muted on branch 8.17

Mute Reasons:

Build Scans:

cbuescher commented 3 days ago

The test fails reproducibly when we only use one shard here. I'm not sure if that is an intended effect of the changes in https://github.com/elastic/elasticsearch/pull/115314 as I don't see any aggregations in the request we are sending later, but I will open a PR that adjusts expectations conditionally and ping @piergm for his opinion when he returns.