search

elastic / elasticsearch

Free and Open, Distributed, RESTful Search Engine
https://www.elastic.co/products/elasticsearch
Other
68.66k stars 24.38k forks source link

[CI] DockerTests test021InstallPlugin failing #110343

Open elasticsearchmachine opened 2 weeks ago

elasticsearchmachine commented 2 weeks ago

Build Scans:

Reproduction Line:

null

Applicable branches: main

Reproduces locally?: N/A

Failure History: See dashboard&_a=(controlGroupInput:(chainingSystem:HIERARCHICAL,controlStyle:twoLine,ignoreParentSettings:(ignoreFilters:!f,ignoreQuery:!f,ignoreTimerange:!f,ignoreValidations:!t),panels:('0c0c9cb8-ccd2-45c6-9b13-96bac4abc542':(explicitInput:(dataViewId:fbbdc689-be23-4b3d-8057-aa402e9ed0c5,enhancements:(),fieldName:task.keyword,grow:!t,id:'0c0c9cb8-ccd2-45c6-9b13-96bac4abc542',searchTechnique:wildcard,selectedOptions:!(),singleSelect:!t,title:'Gradle%20Task',width:medium),grow:!t,order:0,type:optionsListControl,width:small),'144933da-5c1b-4257-a969-7f43455a7901':(explicitInput:(dataViewId:fbbdc689-be23-4b3d-8057-aa402e9ed0c5,enhancements:(),fieldName:name.keyword,grow:!t,id:'144933da-5c1b-4257-a969-7f43455a7901',searchTechnique:wildcard,selectedOptions:!('test021InstallPlugin'),title:Test,width:medium),grow:!t,order:2,type:optionsListControl,width:medium),'4e6ad9d6-6fdc-4fcc-bf1a-aa6ca79e0850':(explicitInput:(dataViewId:fbbdc689-be23-4b3d-8057-aa402e9ed0c5,enhancements:(),fieldName:className.keyword,grow:!t,id:'4e6ad9d6-6fdc-4fcc-bf1a-aa6ca79e0850',searchTechnique:wildcard,selectedOptions:!('org.elasticsearch.packaging.test.DockerTests'),title:Suite,width:medium),grow:!t,order:1,type:optionsListControl,width:medium)))))

Failure Message:

org.elasticsearch.packaging.util.Shell$ShellException: Command was not successful: [bash -c docker exec --tty f1ebc38563e039c39295485aef963a4b6cc817b8cc819a9ab6f773fe22cf02e3 /usr/share/elasticsearch/bin/elasticsearch-plugin list]
   result: exitCode = [137] stdout = [analysis-icu] stderr = []

Issue Reasons:

Note: This issue was created using new, in-development muting automation. Please mind any issues.

elasticsearchmachine commented 2 weeks ago

Pinging @elastic/es-delivery (Team:Delivery)

mark-vieira commented 2 weeks ago

The underlying error is the following:

java.lang.IllegalStateException: failed to load plugin analysis-icu while checking for jar hell
    at org.elasticsearch.server@8.15.0-SNAPSHOT/org.elasticsearch.plugins.PluginsUtils.checkBundleJarHell(PluginsUtils.java:353)
    at org.elasticsearch.server@8.15.0-SNAPSHOT/org.elasticsearch.plugins.PluginsService.loadBundles(PluginsService.java:294)
    at org.elasticsearch.server@8.15.0-SNAPSHOT/org.elasticsearch.plugins.PluginsService.<init>(PluginsService.java:165)
    at org.elasticsearch.server@8.15.0-SNAPSHOT/org.elasticsearch.node.NodeServiceProvider.newPluginService(NodeServiceProvider.java:55)
    at org.elasticsearch.server@8.15.0-SNAPSHOT/org.elasticsearch.node.NodeConstruction.createEnvironment(NodeConstruction.java:454)
    at org.elasticsearch.server@8.15.0-SNAPSHOT/org.elasticsearch.node.NodeConstruction.prepareConstruction(NodeConstruction.java:255)
    at org.elasticsearch.server@8.15.0-SNAPSHOT/org.elasticsearch.node.Node.<init>(Node.java:192)
    at org.elasticsearch.server@8.15.0-SNAPSHOT/org.elasticsearch.bootstrap.Elasticsearch$2.<init>(Elasticsearch.java:240)
    at org.elasticsearch.server@8.15.0-SNAPSHOT/org.elasticsearch.bootstrap.Elasticsearch.initPhase3(Elasticsearch.java:240)
    at org.elasticsearch.server@8.15.0-SNAPSHOT/org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:75)
Caused by: java.nio.file.NoSuchFileException: /usr/share/elasticsearch/plugins/.installing-10801102274982928062/analysis-icu-8.15.0-SNAPSHOT.jar
    at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:92)
    at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:106)
    at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)
    at java.base/sun.nio.fs.UnixFileAttributeViews$Basic.readAttributes(UnixFileAttributeViews.java:56)
    at java.base/sun.nio.fs.UnixFileSystemProvider.readAttributes(UnixFileSystemProvider.java:171)
    at java.base/sun.nio.fs.LinuxFileSystemProvider.readAttributes(LinuxFileSystemProvider.java:99)
    at java.base/java.nio.file.Files.readAttributes(Files.java:1858)
    at java.base/java.util.zip.ZipFile$Source.get(ZipFile.java:1491)
    at java.base/java.util.zip.ZipFile$CleanableResource.<init>(ZipFile.java:724)
    at java.base/java.util.zip.ZipFile.<init>(ZipFile.java:251)
    at java.base/java.util.zip.ZipFile.<init>(ZipFile.java:180)
    at java.base/java.util.jar.JarFile.<init>(JarFile.java:345)
    at java.base/java.util.jar.JarFile.<init>(JarFile.java:316)
    at java.base/java.util.jar.JarFile.<init>(JarFile.java:255)
    at org.elasticsearch.base@8.15.0-SNAPSHOT/org.elasticsearch.jdk.JarHell.checkJarHell(JarHell.java:221)
    at org.elasticsearch.server@8.15.0-SNAPSHOT/org.elasticsearch.plugins.PluginsUtils.checkBundleJarHell(PluginsUtils.java:349)
    ... 9 more

@elastic/es-core-infra I'm not familiar with this .installing file and why it would be missing. As far as the logs show, the plugin was successfully installed. Has anything changes regarding plugin installation recently?

elasticsearchmachine commented 2 weeks ago

Pinging @elastic/es-core-infra (Team:Core/Infra)

rjernst commented 1 week ago

When a plugin is installed the zip is extracted into a temporary .installing directory. We check jarhell on the plugin there before moving it to the final plugin directory.

This test failed because when we tried to access a jar from the plugin the file system said it doesn't exist. Yet the way we know about the jar in the first place is we listed the extracted plugin dir. So something is messed up with the filesystem such that the file was there, and then it wasn't. From the plugin installer point of view, there is nothing we can do about that.

mark-vieira commented 6 days ago

I have a suspicion this has something to do with some disk space issues in CI we're seeing. Let's get those sorted out and see if these failures continue.

DaveCTurner commented 2 days ago

Failed again for me at https://gradle-enterprise.elastic.co/s/hywviexz5z5xw