search

GoogleCloudDataproc / hadoop-connectors

Libraries and tools for interoperability between Hadoop-related open-source software and Google Cloud Platform.
Apache License 2.0
280 stars 238 forks source link

Maven profile for test to try Protobuf 4.x runtime #1265

Open suztomo opened 4 weeks ago

suztomo commented 4 weeks ago

I'm exploring how we can confirm whether the libraries in this repository are compatible with Protobuf 4.x runtime. This pull request is an option to specify a special test runtime dependencies through surefire plugin's <additionalClasspathDependencies> element. This setup generates the NoSuchMethodError for the method in the Protobuf class.

How to run a problematic test:

~/hadoop-connectors$ mvn -pl gcsio install -Pprotobuf4-test -Dtest=com.google.cloud.hadoop.gcsio.BatchHelperTest
...

[ERROR] Tests run: 7, Failures: 0, Errors: 3, Skipped: 0, Time elapsed: 0.416 s <<< FAILURE! -- in com.google.cloud.hadoop.gcsio.BatchHelperTest
[ERROR] com.google.cloud.hadoop.gcsio.BatchHelperTest.allRequestsAreSentInSingleBatch_withOneMaxThreads -- Time elapsed: 0.206 s <<< ERROR!
java.lang.NoSuchMethodError: 'java.lang.Object com.google.protobuf.DescriptorProtos$ServiceOptions.getExtension(com.google.protobuf.GeneratedMessage$GeneratedExtension)'
    at com.google.cloud.hadoop.gcsio.GoogleCloudStorageOptions.builder(GoogleCloudStorageOptions.java:58)
    at com.google.cloud.hadoop.gcsio.GoogleCloudStorageOptions.<clinit>(GoogleCloudStorageOptions.java:44)
    at com.google.cloud.hadoop.gcsio.GoogleCloudStorageTest.<clinit>(GoogleCloudStorageTest.java:134)
    at com.google.cloud.hadoop.gcsio.BatchHelperTest.allRequestsAreSentInSingleBatch_withOneMaxThreads(BatchHelperTest.java:171)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

Analysis

@blakeli0 analyzed the binary incompatibility:

For this specific issue, we found that the root cause is changes to ServiceOptions.getExtension() is not binary compatible and hadoop-connectors need to recompile and release with the new protobuf 4.28.2. In 3.x, ServiceOptions extends from GeneratedMessageV3.ExtendableMessage which contains a method getExtention(GeneratedExtension extension). In 4.x, ServiceOptions extends from GeneratedMessage.ExtendableMessage which contains a method getExtention(ExtensionLite extension). ExtensionLite is a super class of GeneratedExtension, hence it is source compatible but not binary compatible.

Solution?

The simplest solution is just compile the libraries with Protobuf 4.x. However doing so might drop the compatibility with Protobuf 3 library, requiring us to wait for a new major version release of this repository. How might we provide a good compatibility with both Protobuf 4 and 3 libraries?

suztomo commented 3 weeks ago

@arunkumarchacko In this (normal) Maven setup, the JUnit tests run before packaging (shading). If you explore the shading Protobuf dependencies, the tests may have to run as integration tests.