search

devonfw / ide

Tool to automate setup and update of development environment (esp. for Java projects).
Apache License 2.0
33 stars 101 forks source link

gradle download broken #1218

Closed hohwille closed 1 year ago

hohwille commented 1 year ago

It seems that gradle downloads got broken - maybe the URLs have been changed by the tool vendor. Our github action already noted that the downloads broken immediately after the metadata was created as you can see in the status.json files. It seems we have to fix the GradleUrlUpdater and also delete the entire gradle folder from ide-urls so it will be recreated in the night.

p.s.: I hope such effects will not recur often as this sucks and causes us maintenance effort that we actually do not want to have. The idea of #941 was actually to reduce the maintenance effort as much as possible.

hohwille commented 1 year ago

Example for current download link leading to error: https://services.gradle.org/distributions/gradle-9.95-bin.zip

hohwille commented 1 year ago

It seems to be different. The download link structure is OK. https://github.com/devonfw/ide-urls/blob/master/gradle/gradle/8.1.1/urls https://services.gradle.org/distributions/gradle-8.1.1-bin.zip works

As you can see on the download website of gradle there are no 9.x release versions listed. So just tracking git tags does not seem to be the proper approach here. Still I guess that deleting the gradle folder and rebuilding it over night will not rebuild the broken releases. My assumption is that the broken versions have been created from the very old code of url-updater that did not follow redirects. So as a first step, I will simply delete gradle folder.

hohwille commented 1 year ago

Even the latest official version failed:

$ devon gradle setup
Updating gradle from version 8.0.2 to version 8.1.1...
Success: run command mkdir
Starting installation of gradle in version 8.1.1 to /d/projects/test/software/gradle
Deleting old installation as it is a symlink: /d/projects/test/software/gradle
Trying to download gradle-8.1.1.zip from https://services.gradle.org/distributions/gradle-8.1.1-bin.zip
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
 18  118M   18 22.3M    0     0   188k      0  0:10:44  0:02:01  0:08:43     0
curl: (56) Recv failure: Connection was reset
Failed to download https://services.gradle.org/distributions/gradle-8.1.1-bin.zip with exit code 56
hohwille commented 1 year ago

This is nothing we can change. The vendor of gradle has to provide a reliable service for downloading its tool.

jan-vcapgemini commented 1 year ago

When I try to generate the checksums locally, I get following exception:

14:16:34.915 [main] [C:] - ERROR - c.d.t.i.u.updater.AbstractUrlUpdater - For tool gradle we failed to add version 6.9.1. java.lang.IllegalStateException: Failed to read body of download https://services.gradle.org/distributions/gradle-6.9.1-bin.zip at com.devonfw.tools.ide.url.updater.AbstractUrlUpdater.doGenerateChecksum(AbstractUrlUpdater.java:324) at com.devonfw.tools.ide.url.updater.AbstractUrlUpdater.checkDownloadUrl(AbstractUrlUpdater.java:280) at com.devonfw.tools.ide.url.updater.AbstractUrlUpdater.doAddVersion(AbstractUrlUpdater.java:190) at com.devonfw.tools.ide.url.updater.AbstractUrlUpdater.doAddVersion(AbstractUrlUpdater.java:164) at com.devonfw.tools.ide.url.updater.AbstractUrlUpdater.doAddVersion(AbstractUrlUpdater.java:149) at com.devonfw.tools.ide.url.updater.AbstractUrlUpdater.doAddVersion(AbstractUrlUpdater.java:136) at com.devonfw.tools.ide.url.updater.gradle.GradleUrlUpdater.addVersion(GradleUrlUpdater.java:33) at com.devonfw.tools.ide.url.updater.AbstractUrlUpdater.update(AbstractUrlUpdater.java:452) at com.devonfw.tools.ide.integrationtests.JsonUrlUpdaterIT.testGradle(JsonUrlUpdaterIT.java:78) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:566) at org.junit.platform.commons.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:727) at org.junit.jupiter.engine.execution.MethodInvocation.proceed(MethodInvocation.java:60) at org.junit.jupiter.engine.execution.InvocationInterceptorChain$ValidatingInvocation.proceed(InvocationInterceptorChain.java:131) at org.junit.jupiter.engine.extension.TimeoutExtension.intercept(TimeoutExtension.java:156) at org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestableMethod(TimeoutExtension.java:147) at org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestMethod(TimeoutExtension.java:86) at org.junit.jupiter.engine.execution.InterceptingExecutableInvoker$ReflectiveInterceptorCall.lambda$ofVoidMethod$0(InterceptingExecutableInvoker.java:103) at org.junit.jupiter.engine.execution.InterceptingExecutableInvoker.lambda$invoke$0(InterceptingExecutableInvoker.java:93) at org.junit.jupiter.engine.execution.InvocationInterceptorChain$InterceptedInvocation.proceed(InvocationInterceptorChain.java:106) at org.junit.jupiter.engine.execution.InvocationInterceptorChain.proceed(InvocationInterceptorChain.java:64) at org.junit.jupiter.engine.execution.InvocationInterceptorChain.chainAndInvoke(InvocationInterceptorChain.java:45) at org.junit.jupiter.engine.execution.InvocationInterceptorChain.invoke(InvocationInterceptorChain.java:37) at org.junit.jupiter.engine.execution.InterceptingExecutableInvoker.invoke(InterceptingExecutableInvoker.java:92) at org.junit.jupiter.engine.execution.InterceptingExecutableInvoker.invoke(InterceptingExecutableInvoker.java:86) at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.lambda$invokeTestMethod$7(TestMethodTestDescriptor.java:217) at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73) at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.invokeTestMethod(TestMethodTestDescriptor.java:213) at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:138) at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:68) at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:151) at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73) at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:141) at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137) at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$9(NodeTestTask.java:139) at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73) at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:138) at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:95) at java.base/java.util.ArrayList.forEach(ArrayList.java:1541) at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.invokeAll(SameThreadHierarchicalTestExecutorService.java:41) at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:155) at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73) at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:141) at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137) at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$9(NodeTestTask.java:139) at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73) at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:138) at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:95) at java.base/java.util.ArrayList.forEach(ArrayList.java:1541) at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.invokeAll(SameThreadHierarchicalTestExecutorService.java:41) at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:155) at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73) at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:141) at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137) at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$9(NodeTestTask.java:139) at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73) at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:138) at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:95) at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.submit(SameThreadHierarchicalTestExecutorService.java:35) at org.junit.platform.engine.support.hierarchical.HierarchicalTestExecutor.execute(HierarchicalTestExecutor.java:57) at org.junit.platform.engine.support.hierarchical.HierarchicalTestEngine.execute(HierarchicalTestEngine.java:54) at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:147) at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:127) at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:90) at org.junit.platform.launcher.core.EngineExecutionOrchestrator.lambda$execute$0(EngineExecutionOrchestrator.java:55) at org.junit.platform.launcher.core.EngineExecutionOrchestrator.withInterceptedStreams(EngineExecutionOrchestrator.java:102) at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:54) at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:114) at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:86) at org.junit.platform.launcher.core.DefaultLauncherSession$DelegatingLauncher.execute(DefaultLauncherSession.java:86) at org.junit.platform.launcher.core.SessionPerRequestLauncher.execute(SessionPerRequestLauncher.java:53) at com.intellij.junit5.JUnit5IdeaTestRunner.startRunnerWithArgs(JUnit5IdeaTestRunner.java:57) at com.intellij.rt.junit.IdeaTestRunner$Repeater$1.execute(IdeaTestRunner.java:38) at com.intellij.rt.execution.junit.TestsRepeater.repeat(TestsRepeater.java:11) at com.intellij.rt.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:35) at com.intellij.rt.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:232) at com.intellij.rt.junit.JUnitStarter.main(JUnitStarter.java:55) Caused by: java.io.IOException: closed at java.net.http/jdk.internal.net.http.ResponseSubscribers$HttpResponseInputStream.current(ResponseSubscribers.java:368) at java.net.http/jdk.internal.net.http.ResponseSubscribers$HttpResponseInputStream.read(ResponseSubscribers.java:403) at java.base/java.io.InputStream.read(InputStream.java:205) at com.devonfw.tools.ide.url.updater.AbstractUrlUpdater.doGenerateChecksum(AbstractUrlUpdater.java:310) ... 78 common frames omitted Caused by: java.io.IOException: fixed content-length: 107669361, bytes received: 26966177 at java.net.http/jdk.internal.net.http.common.Utils.wrapWithExtraDetail(Utils.java:330) at java.net.http/jdk.internal.net.http.Http1Response$BodyReader.onReadError(Http1Response.java:758) at java.net.http/jdk.internal.net.http.Http1AsyncReceiver.checkForErrors(Http1AsyncReceiver.java:297) at java.net.http/jdk.internal.net.http.Http1AsyncReceiver.flush(Http1AsyncReceiver.java:263) at java.net.http/jdk.internal.net.http.common.SequentialScheduler$SynchronizedRestartableTask.run(SequentialScheduler.java:175) at java.net.http/jdk.internal.net.http.common.SequentialScheduler$CompleteRestartableTask.run(SequentialScheduler.java:147) at java.net.http/jdk.internal.net.http.common.SequentialScheduler$SchedulableTask.run(SequentialScheduler.java:198) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) Caused by: java.io.IOException: connection closed locally at java.net.http/jdk.internal.net.http.SocketTube.signalClosed(SocketTube.java:157) at java.net.http/jdk.internal.net.http.PlainHttpConnection.close(PlainHttpConnection.java:277) at java.net.http/jdk.internal.net.http.AsyncSSLConnection.close(AsyncSSLConnection.java:111) at java.net.http/jdk.internal.net.http.Http1Exchange.cancelImpl(Http1Exchange.java:490) at java.net.http/jdk.internal.net.http.Http1Exchange.cancel(Http1Exchange.java:426) at java.net.http/jdk.internal.net.http.Exchange.cancel(Exchange.java:228) at java.net.http/jdk.internal.net.http.MultiExchange.cancel(MultiExchange.java:240) at java.net.http/jdk.internal.net.http.ResponseTimerEvent.handle(ResponseTimerEvent.java:71) at java.net.http/jdk.internal.net.http.HttpClientImpl.purgeTimeoutsAndReturnNextDeadline(HttpClientImpl.java:1248) at java.net.http/jdk.internal.net.http.HttpClientImpl$SelectorManager.run(HttpClientImpl.java:877)

jan-vcapgemini commented 1 year ago

When I manually download the zip files with my browser, every download succeeds.

hohwille commented 1 year ago

Interesting: It seems to be unstable, but not entirely broken:

$ devon gradle setup
Success: run command mkdir
Starting installation of gradle in version 8.1.1 to /d/projects/devonfw/software/gradle
Trying to download gradle-8.1.1-gradle.zip from https://services.gradle.org/distributions/gradle-8.1.1-bin.zip
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
 41  118M   41 49.7M    0     0   411k      0  0:04:56  0:02:03  0:02:53     0
curl: (56) Recv failure: Connection was reset
Failed to download https://services.gradle.org/distributions/gradle-8.1.1-bin.zip with exit code 56

******** ATTENTION ********
Download of gradle in version 8.1.1 failed after trying 1 url(s).
We are sorry for the inconvenience. Please check the above errors, resolve them and try again.

hohwille@CE49454 MSYS /d/projects/devonfw
$ devon gradle setup
Success: run command mkdir
Starting installation of gradle in version 8.1.1 to /d/projects/devonfw/software/gradle
Trying to download gradle-8.1.1-gradle.zip from https://services.gradle.org/distributions/gradle-8.1.1-bin.zip
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100  118M  100  118M    0     0  10.5M      0  0:00:11  0:00:11 --:--:-- 10.6M
Download of gradle-8.1.1-gradle.zip from https://services.gradle.org/distributions/gradle-8.1.1-bin.zip succeeded.
No checksum found at /d/projects/devonfw/urls/gradle/gradle/8.1.1/urls.sha256
Success: run command unzip
Successfully extracted archive gradle-8.1.1-gradle.zip to /d/projects/devonfw/updates/extracted/gradle
Success: run command mv
Successfully installed gradle
The software gradle has been added. You need to rerun 'devon' command without arguments or restart your terminal to update your PATH so the newly installed software will be found.

Welcome to Gradle 8.1.1!

Here are the highlights of this release:
 - Stable configuration cache
 - Experimental Kotlin DSL assignment syntax
 - Building with Java 20

For more details see https://docs.gradle.org/8.1.1/release-notes.html

------------------------------------------------------------
Gradle 8.1.1
------------------------------------------------------------

Build time:   2023-04-21 12:31:26 UTC
Revision:     1cf537a851c635c364a4214885f8b9798051175b

Kotlin:       1.8.10
Groovy:       3.0.15
Ant:          Apache Ant(TM) version 1.10.11 compiled on July 10 2021
JVM:          20 (Eclipse Adoptium 20+36)
OS:           Windows 10 10.0 amd64

The good thing is that devonfw-ide does download caching so once the download succeeded further requests to the same tool and version will reuse the already downloaded file. This can also be used as a workaround if CURL cannot download. You can download with your browser and manually put the file under the right name in the download cache (~/Downloads/devonfw-ide/«filename»). You can get the proper «filename» from the log-message before the download starts ("Trying to download «filename» from «URL»").

hohwille commented 1 year ago

Was auto-closed from PR but not entirely solved.

hohwille commented 1 year ago

So there were two issues: