google / guava

Google core libraries for Java
Apache License 2.0
50.25k stars 10.91k forks source link

Toolchain downloads frequently failing #7492

Open cpovirk opened 1 month ago

cpovirk commented 1 month ago

I've had multiple PRs recently that have failed testing because of trouble downloading a JDK. Here's the most recent example, which downloaded 11 but failed on 23, but note that I think I've seen failures for 11 and have definitely seen a failure for 8:

[INFO] --- toolchains:4.5.0:toolchain (download-11) @ guava ---
[INFO] Required toolchain: jdk [ vendor='zulu' version='11' ]
[INFO] Begin to install JDK 11
[INFO] Download zulu11.76.21-ca-jdk11.0.25-linux_x64.tar.gz from https://cdn.azul.com/zulu/bin/zulu11.76.21-ca-jdk11.0.25-linux_x64.tar.gz
[INFO] Extract zulu11.76.21-ca-jdk11.0.25-linux_x64.tar.gz
[INFO] JDK installed: /home/runner/.m2/jdks/zulu11.76.21-ca-jdk11.0.25-linux_x64
[INFO] 
[INFO] --- toolchains:4.5.0:toolchain (download-23-and-surefire-version) @ guava ---
[INFO] Required toolchain: jdk [ vendor='zulu' version='23' ]
[INFO] Begin to install JDK 23
[INFO] Download zulu23.30.13-ca-jdk23.0.1-linux_x64.tar.gz from https://cdn.azul.com/zulu/bin/zulu23.30.13-ca-jdk23.0.1-linux_x64.tar.gz
[INFO] Extract zulu23.30.13-ca-jdk23.0.1-linux_x64.tar.gz
Error:  Failed to download and install JDK
org.codehaus.plexus.archiver.ArchiverException: Error while expanding /home/runner/.m2/jdks/zulu23.30.13-ca-jdk23.0.1-linux_x64.tar.gz
    at org.codehaus.plexus.archiver.tar.TarUnArchiver.execute (TarUnArchiver.java:124)
    at org.codehaus.plexus.archiver.tar.TarUnArchiver.execute (TarUnArchiver.java:88)
    at org.codehaus.plexus.archiver.AbstractUnArchiver.extract (AbstractUnArchiver.java:151)
    at org.apache.maven.plugins.toolchain.FoojayService.extractArchiveFile (FoojayService.java:206)
    at org.apache.maven.plugins.toolchain.FoojayService.downloadAndExtract (FoojayService.java:166)
    at org.apache.maven.plugins.toolchain.FoojayService.downloadAndExtractJdk (FoojayService.java:94)
    at org.apache.maven.plugins.toolchain.ToolchainMojo.autoInstallJdk (ToolchainMojo.java:233)
    at org.apache.maven.plugins.toolchain.ToolchainMojo.selectToolchain (ToolchainMojo.java:195)
    at org.apache.maven.plugins.toolchain.ToolchainMojo.execute (ToolchainMojo.java:111)
    at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo (DefaultBuildPluginManager.java:126)
    at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute2 (MojoExecutor.java:328)
    at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute (MojoExecutor.java:316)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:212)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:174)
    at org.apache.maven.lifecycle.internal.MojoExecutor.access$000 (MojoExecutor.java:75)
    at org.apache.maven.lifecycle.internal.MojoExecutor$1.run (MojoExecutor.java:162)
    at org.apache.maven.plugin.DefaultMojosExecutionStrategy.execute (DefaultMojosExecutionStrategy.java:39)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:159)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:105)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:73)
    at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build (SingleThreadedBuilder.java:53)
    at org.apache.maven.lifecycle.internal.LifecycleStarter.execute (LifecycleStarter.java:118)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:261)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:173)
    at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:101)
    at org.apache.maven.cli.MavenCli.execute (MavenCli.java:906)
    at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:283)
    at org.apache.maven.cli.MavenCli.main (MavenCli.java:206)
    at jdk.internal.reflect.DirectMethodHandleAccessor.invoke (DirectMethodHandleAccessor.java:103)
    at java.lang.reflect.Method.invoke (Method.java:580)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced (Launcher.java:255)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launch (Launcher.java:201)
    at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode (Launcher.java:361)
    at org.codehaus.plexus.classworlds.launcher.Launcher.main (Launcher.java:314)
    at jdk.internal.reflect.DirectMethodHandleAccessor.invoke (DirectMethodHandleAccessor.java:103)
    at java.lang.reflect.Method.invoke (Method.java:580)
    at org.apache.maven.wrapper.BootstrapMainStarter.start (BootstrapMainStarter.java:52)
    at org.apache.maven.wrapper.WrapperExecutor.execute (WrapperExecutor.java:161)
    at org.apache.maven.wrapper.MavenWrapperMain.main (MavenWrapperMain.java:73)
Caused by: java.io.EOFException: Unexpected end of ZLIB input stream
    at java.util.zip.InflaterInputStream.fill (InflaterInputStream.java:266)
    at java.util.zip.InflaterInputStream.read (InflaterInputStream.java:175)
    at java.util.zip.GZIPInputStream.read (GZIPInputStream.java:128)
    at java.io.BufferedInputStream.fill (BufferedInputStream.java:291)
    at java.io.BufferedInputStream.read1 (BufferedInputStream.java:347)
    at java.io.BufferedInputStream.implRead (BufferedInputStream.java:420)
    at java.io.BufferedInputStream.read (BufferedInputStream.java:399)
    at org.apache.commons.compress.archivers.tar.TarArchiveInputStream.read (TarArchiveInputStream.java:873)
    at java.io.InputStream.read (InputStream.java:220)
    at org.codehaus.plexus.util.IOUtil.copy (IOUtil.java:189)
    at org.codehaus.plexus.util.IOUtil.copy (IOUtil.java:175)
    at org.codehaus.plexus.archiver.AbstractUnArchiver.extractFile (AbstractUnArchiver.java:372)
    at org.codehaus.plexus.archiver.tar.TarUnArchiver.execute (TarUnArchiver.java:114)
    at org.codehaus.plexus.archiver.tar.TarUnArchiver.execute (TarUnArchiver.java:88)
    at org.codehaus.plexus.archiver.AbstractUnArchiver.extract (AbstractUnArchiver.java:151)
    at org.apache.maven.plugins.toolchain.FoojayService.extractArchiveFile (FoojayService.java:206)
    at org.apache.maven.plugins.toolchain.FoojayService.downloadAndExtract (FoojayService.java:166)
    at org.apache.maven.plugins.toolchain.FoojayService.downloadAndExtractJdk (FoojayService.java:94)
    at org.apache.maven.plugins.toolchain.ToolchainMojo.autoInstallJdk (ToolchainMojo.java:233)
    at org.apache.maven.plugins.toolchain.ToolchainMojo.selectToolchain (ToolchainMojo.java:195)
    at org.apache.maven.plugins.toolchain.ToolchainMojo.execute (ToolchainMojo.java:111)
    at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo (DefaultBuildPluginManager.java:126)
    at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute2 (MojoExecutor.java:328)
    at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute (MojoExecutor.java:316)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:212)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:174)
    at org.apache.maven.lifecycle.internal.MojoExecutor.access$000 (MojoExecutor.java:75)
    at org.apache.maven.lifecycle.internal.MojoExecutor$1.run (MojoExecutor.java:162)
    at org.apache.maven.plugin.DefaultMojosExecutionStrategy.execute (DefaultMojosExecutionStrategy.java:39)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:159)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:105)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:73)
    at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build (SingleThreadedBuilder.java:53)
    at org.apache.maven.lifecycle.internal.LifecycleStarter.execute (LifecycleStarter.java:118)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:261)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:173)
    at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:101)
    at org.apache.maven.cli.MavenCli.execute (MavenCli.java:906)
    at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:283)
    at org.apache.maven.cli.MavenCli.main (MavenCli.java:206)
    at jdk.internal.reflect.DirectMethodHandleAccessor.invoke (DirectMethodHandleAccessor.java:103)
    at java.lang.reflect.Method.invoke (Method.java:580)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced (Launcher.java:255)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launch (Launcher.java:201)
    at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode (Launcher.java:361)
    at org.codehaus.plexus.classworlds.launcher.Launcher.main (Launcher.java:314)
    at jdk.internal.reflect.DirectMethodHandleAccessor.invoke (DirectMethodHandleAccessor.java:103)
    at java.lang.reflect.Method.invoke (Method.java:580)
    at org.apache.maven.wrapper.BootstrapMainStarter.start (BootstrapMainStarter.java:52)
    at org.apache.maven.wrapper.WrapperExecutor.execute (WrapperExecutor.java:161)
    at org.apache.maven.wrapper.MavenWrapperMain.main (MavenWrapperMain.java:73)
Error:  No toolchain matched from 2 found for toolchainType jdk
[INFO] Required toolchain: jdk [ vendor='zulu' version='21' ]
[INFO] Found matching toolchain for toolchainType jdk: JDK[/opt/hostedtoolcache/Java_Zulu_jdk/21.0.5-11/x64]
Error:  Cannot find matching toolchain definitions for the following toolchain types:
jdk [ vendor='zulu' version='23' ]

(Aside: GitHub is inserting Markdown when I copy and paste directly from the Actions results, possibly to link to anchors in that page? Not that that would work inside the code block that I'm pasting into. I ended up pasting into a plain-text window first and then copying and pasting again.)

The automatic toolchain downloads are nice, but it would probably be more reliable for CI to download all the toolchains we need with setup-java. (That would likely also be faster, as noted in internal bug 334979149.) The trick would be ensuring that we still use the toolchain that we want for (a) testing and (b, possibly less important) running Maven itself. Error Prone has this worked out by being careful about the order in which they list their downloaded toolchains and then (I'm guessing) by configuring maven-surefire-plugin more directly than we do.

(We could still keep the toolchain plugin for local builds. It's much nicer for users than if they have to download various toolchains themselves.)

ben-manes commented 1 month ago

Also consider trying to ensure that you are leveraging the hosted tool cache which avoids downloading if you use one of the pre-installed jdks (temurin 8, 11, 17, 21). You can also use composite github actions for reuse, optionally across repositories, if you find a common setup is helpful (e.g. I wrangle all of my jdk versions in this action)