awslabs / aws-glue-libs

AWS Glue Libraries are additions and enhancements to Spark for ETL operations.
Other
635 stars 299 forks source link

S3 Maven repository availability is flaky #199

Open borgoat opened 8 months ago

borgoat commented 8 months ago

Hi,

In a repo where I use this library, I counted at least 3 pipelines failures in the last week, due to some network issues with the Glue maven repository. I do not believe it was something on the GitHub Actions side, since other connections were unaffected.

[error] (sharedFeatureGlue / update) lmcoursier.internal.shaded.coursier.error.FetchError$DownloadingArtifacts: Error fetching artifacts:
[error] https://aws-glue-etl-artifacts.s3.amazonaws.com/release/com/amazonaws/AWSGlueETL/4.0.0/AWSGlueETL-4.0.0.jar: download error: Caught java.net.SocketException (Connection reset) while downloading https://aws-glue-etl-artifacts.s3.amazonaws.com/release/com/amazonaws/AWSGlueETL/4.0.0/AWSGlueETL-4.0.0.jar
[error] Total time: 36 s, completed Dec 27, 2023, 9:13:35 AM
Warning: run-commands command "sbt shared-feature-glue:assembly" exited with non-zero status code

Would it make sense to publish this package in the maven central repository or another well-known repo?^1

It's just 1 line, but it would also remove the need for

resolvers += "AWS Glue ETL" at "https://aws-glue-etl-artifacts.s3.amazonaws.com/release"

or other similar configurations in other package managers...