Closed wilkinsona closed 4 months ago
melix
commented
5 months ago Are you using a custom repository URL? The default should be to pick the release in Maven Central. I'm not sure that increasing the connect timeout would be helpful in general, but retries with exponential backoff may be.
wilkinsona
commented
5 months ago Thanks for taking a look, @melix. We're using the default repository.
melix
commented
5 months ago Right, it's quite curious, because in this case the repository should be resolved as a Maven artifact. I don't think we'd have the ability to retry in this case. Also it would be strange that this particular dependency fails but not the regular dependency resolution. Unless if you are using mirrors?
wilkinsona
commented
5 months ago No mirrors involved. The builds are running on GitHub Actions and use Maven Central for the vast majority of their dependencies, with a little bit of https://repo.spring.io mixed in when we're testing snapshots.
wilkinsona
commented
5 months ago Sorry, I may not have been entirely accurate in my earlier comment about the repository URL. Looking at the code in computeMetadataRepositoryUri, I think I've learned that a custom metadata version will result in a custom repository URL that points to github.com being used. We're setting the version to 0.3.7.
melix
commented
5 months ago Ah thanks, that makes more sense to me, then you're probably hitting rate limiting from GitHub APIs, we've seen this (in another context) for Micronaut builds too.
Something you might want to consider is actually to download the repository zip file and include it in your sources, then configure the extension to point to it. Not sure how much this is an option for you.
wilkinsona
commented
5 months ago AFAIK, GitHub doesn't rate-limit download links. When it does rate limit, from what I have seen it's done with a 403 response rather than by refusing a connection.
Experimenting with curl, https://github.com/oracle/graalvm-reachability-metadata/releases/download/0.3.7/graalvm-reachability-metadata-0.3.7.zip redirects to https://objects.githubusercontent.com which, judging by the query parameters such as ?X-Amz-Algorithm=AWS4-HMAC-SHA256, is an S3 bucket. I don't know for certain as it's not apparent from the failure, but this feels like a regular connection timeout when connecting to S3 for which retries (or a longer timeout) could help.
Not sure how much this is an option for you.
It's certainly an option. Thanks for the suggestion. That said, I think an improvement here would be better as it would have broader benefit. I am happy to contribute something if we can agree on the details (number of retries, back off approach, etc).
melix
commented
5 months ago Sure, we will implement retries in case we don't download from Central, my suggestion was a workaround in the meantime.
Is your feature request related to a problem? Please describe.
We build 10s of native images on a nightly basis and roughly one night in three (at the time of writing) we'll see a failure due to a connection timeout when downloading reachability metadata. I would like our builds to be more robust.
Describe the solution you'd like Builds could be made more robust by configuring a longer connection timeout or by retrying failures that should be transient.
Describe alternatives you've considered I considered some configuration on the Gradle side to retry the task that's failing. Some research suggests that this isn't something that Gradle offers, at least not without resorting to hacks. It also wouldn't benefit other users of
GraalVMReachabilityMetadataService.Additional context
Connection timeout stack trace
``` Caused by: java.net.ConnectException: Connection timed out at org.graalvm.buildtools.gradle.internal.GraalVMReachabilityMetadataService.newRepository(GraalVMReachabilityMetadataService.java:128) at org.graalvm.buildtools.gradle.internal.GraalVMReachabilityMetadataService.