search

GoogleCloudPlatform / cloud-opensource-java

Tools for detecting and avoiding linkage errors in GCP open source projects
Apache License 2.0
156 stars 74 forks source link

Linkage Errors from word-count-beam with Cloud Dataflow #1025

Open suztomo opened 4 years ago

suztomo commented 4 years ago

Finished https://cloud.google.com/dataflow/docs/quickstarts . It worked.

I added libraries-bom to the project.

When I applied linkage checker enforcer rule to the project (with reportOnlyReahcable) it showed following errors:

[INFO] --- maven-enforcer-plugin:3.0.0-M2:enforce (enforce-linkage-checker) @ word-count-beam ---
[ERROR] Linkage Checker rule found 8 reachable errors. Linkage error report:
Class org.brotli.dec.BrotliInputStream is not found;
  referenced by 1 class file
    org.apache.beam.repackaged.core.org.apache.commons.compress.compressors.brotli.BrotliCompressorInputStream (beam-sdks-java-core-2.16.0.jar)
Class com.github.luben.zstd.ZstdInputStream is not found;
  referenced by 1 class file
    org.apache.beam.repackaged.core.org.apache.commons.compress.compressors.zstandard.ZstdCompressorInputStream (beam-sdks-java-core-2.16.0.jar)
Class com.github.luben.zstd.ZstdOutputStream is not found;
  referenced by 1 class file
    org.apache.beam.repackaged.core.org.apache.commons.compress.compressors.zstandard.ZstdCompressorOutputStream (beam-sdks-java-core-2.16.0.jar)
Class org.apache.beam.vendor.bytebuddy.v1_9_3.net.bytebuddy.jar.asm.commons.ModuleHashesAttribute is not found;
  referenced by 1 class file
    org.apache.beam.vendor.bytebuddy.v1_9_3.net.bytebuddy.jar.asm.commons.ClassRemapper (beam-vendor-bytebuddy-1_9_3-0.1.jar)
(opencensus-api-0.15.0.jar) io.opencensus.tags.TagContextBuilder's method putLocal(io.opencensus.tags.TagKey arg1, io.opencensus.tags.TagValue arg2) is not found;
  referenced by 1 class file
    io.grpc.internal.CensusStatsModule (grpc-core-1.24.1.jar)
(opencensus-api-0.15.0.jar) io.opencensus.tags.unsafe.ContextUtils's method withValue(io.grpc.Context arg1, io.opencensus.tags.TagContext arg2) is not found;
  referenced by 1 class file
    io.grpc.internal.CensusStatsModule (grpc-core-1.24.1.jar)
(opencensus-api-0.15.0.jar) io.opencensus.trace.unsafe.ContextUtils's method getValue(io.grpc.Context arg1) is not found;
  referenced by 1 class file
    io.grpc.internal.CensusTracingModule (grpc-core-1.24.1.jar)
(opencensus-api-0.15.0.jar) io.opencensus.trace.unsafe.ContextUtils's method withValue(io.grpc.Context arg1, io.opencensus.trace.Span arg2) is not found;
  referenced by 1 class file
    io.grpc.internal.CensusTracingModule (grpc-core-1.24.1.jar)

Are they valid?

Dependency Tree https://gist.github.com/suztomo/ee2266cc4b686e34e114c5c4159891e4

suztomo commented 4 years ago

First Guess

org.apache.beam.repackaged.core.org.apache.commons.compress.compressors are repackaged Apache commons-compress. Certain compression algorithms (Brotli, Zstd) are implemented in its dependencies. Apache Beam does not use it but repackaging all of classes in commons-compress.

ModuleHashesAttribute was missing in old bytebuddy. Bytebuddy fixed it.

Opencensus is just mismatch between opencensus-api-0.15.0 and grpc 1.24.1. Then what OpenCensus should it use?

suztomo commented 4 years ago

Comparison with/without libraries-bom

Interestingly the project has fewer number of linkage errors reported if libraries-bom is not used:

[ERROR] Linkage Checker rule found 4 reachable errors. Linkage error report:
Class org.brotli.dec.BrotliInputStream is not found;
  referenced by 1 class file
    org.apache.beam.repackaged.core.org.apache.commons.compress.compressors.brotli.BrotliCompressorInputStream (beam-sdks-java-core-2.16.0.jar)
Class com.github.luben.zstd.ZstdInputStream is not found;
  referenced by 1 class file
    org.apache.beam.repackaged.core.org.apache.commons.compress.compressors.zstandard.ZstdCompressorInputStream (beam-sdks-java-core-2.16.0.jar)
Class com.github.luben.zstd.ZstdOutputStream is not found;
  referenced by 1 class file
    org.apache.beam.repackaged.core.org.apache.commons.compress.compressors.zstandard.ZstdCompressorOutputStream (beam-sdks-java-core-2.16.0.jar)
Class org.apache.beam.vendor.bytebuddy.v1_9_3.net.bytebuddy.jar.asm.commons.ModuleHashesAttribute is not found;
  referenced by 1 class file
    org.apache.beam.vendor.bytebuddy.v1_9_3.net.bytebuddy.jar.asm.commons.ClassRemapper (beam-vendor-bytebuddy-1_9_3-0.1.jar)

(This does not show OpenCensus and grpc incompatibility)

suztomo commented 4 years ago

Where is OpenCensus 15 coming from?

bigtable-client-core:1.8.0, which has grandparent bigtable-client:1.8.0, has BOM google-cloud-clients:0.73.0-alpha, which specifies opencensus 0.15.0.

org.example:word-count-beam:jar:0.1
\- org.apache.beam:beam-sdks-java-io-google-cloud-platform:jar:2.16.0:compile
    \- com.google.cloud.bigtable:bigtable-client-core:jar:1.8.0:compile
      +- io.opencensus:opencensus-api:jar:0.15.0:compile
      \- io.opencensus:opencensus-contrib-grpc-util:jar:0.15.0:compile
         \- (io.opencensus:opencensus-api:jar:0.15.0:compile - omitted for duplicate)
suztomo commented 4 years ago

Questions

Should bigtable-client-core be part of libraries-bom?

The source repository is at https://github.com/googleapis/java-bigtable-hbase

Should org.apache.beam:beam-sdks-java-io-google-cloud-platform use higher version bigtable-client-core?

bigtable-client-core uses OpenCensus 0.23.0 and gRPC 1.23.0.

suztomo commented 4 years ago

Next step: try upgrading com.google.cloud.bigtable:bigtable-client-core:jar:1.8.0 to see what happens.