Closed alexrashed closed 3 months ago
I created a reproducer which allows you to easily reproduce this issue: https://github.com/alexrashed/reproducer-aws-glue-etl-4-incompatibility
It just defines com.amazonaws.AWSGlueETL-4.0.0
as dependency in maven and the main function runs into a NoSuchMethodError
.
My ask is to update the latest version of either AWSGlueETL-4.0.0
or AWSGlueDynamicSchema-0.9.0
such that they are compatible with each other agin.
I have the same problem, as @alexrashed describes there is an incompatibility between versions. I obtained a later jar which is compatible (due to the number of parameters and works correctly), suddenly the necessary update has been made? Another question that arises is because this problem only occurs locally, when I upload the job to AWS it works fine. I have created other jobs with glue 4, does AWS keep a type of cache for the jars and that is why when I upload to that account it takes the jar with the correct versions?
Thanks for the pointer, @anthonybastidas49! I can approve that the maven repository has been updated and that my test cases are working again with the latest jar files currently available in the maven repo. I'll close this issue for now, but the question from @anthonybastidas49 is still open...
Issue
We have recently seen issues where we had an unexpected
NoSuchMethodError
when testing Glue ETL jobs:This issue started appearing around the 28th of March. After quite some investigation I think the issue is caused by incompatible updates of the JAR files in the maven repository: https://aws-glue-etl-artifacts.s3.amazonaws.com/
Hypothesis
My hypothesis is the the following events caused this issue (based on certain observations and the timestamps in the maven repo index at https://aws-glue-etl-artifacts.s3.amazonaws.com/):
AWSGlueDynamicSchema-0.9.0.jar
has been updated.com.amazonaws.services.glue.schema.types.Field
accepting aString comment
.2024-03-21T23:42:55.000Z
AWSGlueETL-4.0.0.jar
has been updated.com.amazonaws.services.glue.util.DataCatalogWrapperUtils#getFieldsFromColumns
.2024-03-28T15:34:03.000Z
AWSGlueDynamicSchema-0.9.0.jar
has been updated again.Field
.Reproducer
Slight modification of the POM file of this repo to fix another incompatibility (
log4j
upgrade to2.17.2
):Main to reproduce the issue:
Output: