GoogleCloudPlatform / cloud-opensource-java

Tools for detecting and avoiding linkage errors in GCP open source projects
Apache License 2.0
155 stars 74 forks source link

spring-cloud-gcp has many linkage errors. Check the validity. #592

Closed suztomo closed 3 years ago

suztomo commented 5 years ago

There are many false positives when the enforcer rule checks spring-cloud-gcp as below. This is because of the class path missing optional/provided libraries. A design doc is created to tackle this issue (http://go/jdd-supplemental-classpath)

Reproduce

git clone https://github.com/spring-cloud/spring-cloud-gcp.git
cd spring-cloud-gcp
git checkout linkage-checker
mvn verify

With <fail>true</fail> at pom.xml, mvn verify fails with many errors. Investigate the validity of them.

[ERROR] Linkage Checker rule found errors. Linkage error report:
spring-core-5.2.0.M1.jar (98 errors):
  reactor.core.publisher.BaseSubscriber is not found, referenced from org.springframework.core.io.buffer.DataBufferUtils$WriteCompletionHandler
  reactor.core.publisher.FluxSink is not found, referenced from org.springframework.core.io.buffer.DataBufferUtils$WriteCompletionHandler
  reactor.core.publisher.SynchronousSink is not found, referenced from org.springframework.core.io.buffer.DataBufferUtils$ReadableByteChannelGenerator
  reactor.core.publisher.FluxSink is not found, referenced from org.springframework.core.io.buffer.DataBufferUtils$ReadCompletionHandler
  reactor.core.publisher.BaseSubscriber is not found, referenced from org.springframework.core.io.buffer.DataBufferUtils$WritableByteChannelSubscriber
  reactor.core.publisher.FluxSink is not found, referenced from org.springframework.core.io.buffer.DataBufferUtils$WritableByteChannelSubscriber
  io.netty.buffer.CompositeByteBuf is not found, referenced from org.springframework.core.io.buffer.NettyDataBufferFactory
  io.netty.buffer.ByteBufAllocator is not found, referenced from org.springframework.core.io.buffer.NettyDataBufferFactory
  io.netty.buffer.Unpooled is not found, referenced from org.springframework.core.io.buffer.NettyDataBufferFactory
  io.netty.buffer.ByteBuf is not found, referenced from org.springframework.core.io.buffer.NettyDataBufferFactory
...
  kotlin.jvm.internal.Intrinsics.checkParameterIsNotNull is not found, referenced from org.springframework.core.env.PropertyResolverExtensionsKt
  kotlin.jvm.internal.Intrinsics.reifiedOperationMarker is not found, referenced from org.springframework.core.env.PropertyResolverExtensionsKt
  kotlin.jvm.internal.Intrinsics.checkExpressionValueIsNotNull is not found, referenced from org.springframework.core.env.PropertyResolverExtensionsKt

...
spring-context-5.2.0.M1.jar (158 errors):
  org.springframework.instrument.InstrumentationSavingAgent is not found, referenced from org.springframework.instrument.classloading.InstrumentationLoadTimeWeaver$InstrumentationAccessor
  javax.enterprise.concurrent.ManagedExecutors is not found, referenced from org.springframework.scheduling.concurrent.ConcurrentTaskExecutor$ManagedTaskBuilder
  javax.enterprise.concurrent.ManagedScheduledExecutorService is not found, referenced from org.springframework.scheduling.concurrent.ConcurrentTaskScheduler$EnterpriseConcurrentTriggerScheduler
  javax.enterprise.concurrent.Trigger is not found, referenced from org.springframework.scheduling.concurrent.ConcurrentTaskScheduler$EnterpriseConcurrentTriggerScheduler$1
  javax.enterprise.concurrent.LastExecution is not found, referenced from org.springframework.scheduling.concurrent.ConcurrentTaskScheduler$EnterpriseConcurrentTriggerScheduler$1
  groovy.lang.GroovyObject is not found, referenced from org.springframework.context.support.GenericGroovyApplicationContext
  groovy.lang.GroovySystem is not found, referenced from org.springframework.context.support.GenericGroovyApplicationContext
  groovy.lang.MetaClassRegistry is not found, referenced from org.springframework.context.support.GenericGroovyApplicationContext
...
slf4j-api-1.7.26.jar (2 errors):
  org.slf4j.impl.StaticMarkerBinder.getSingleton is not found, referenced from org.slf4j.MarkerFactory
  org.slf4j.impl.StaticMDCBinder.getSingleton is not found, referenced from org.slf4j.MDC

mockito-core-2.25.1.jar (2 errors):
  org.mockito.internal.creation.bytebuddy.inject.MockMethodDispatcher is not found, referenced from org.mockito.internal.creation.bytebuddy.MockMethodAdvice$ForReadObject
  org.mockito.internal.creation.bytebuddy.inject.MockMethodDispatcher is not found, referenced from org.mockito.internal.creation.bytebuddy.MockMethodAdvice$ForEquals
  org.mockito.internal.creation.bytebuddy.inject.MockMethodDispatcher is not found, referenced from org.mockito.internal.creation.bytebuddy.MockMethodAdvice$SerializableRealMethodCall
  org.mockito.internal.creation.bytebuddy.inject.MockMethodDispatcher is not found, referenced from org.mockito.internal.creation.bytebuddy.InlineBytecodeGenerator
...
google-http-client-1.28.0.jar (5 errors):
  org.apache.http.impl.client.DefaultHttpClient is not found, referenced from com.google.api.client.testing.http.apache.MockHttpClient
  org.apache.http.message.BasicHttpResponse is not found, referenced from com.google.api.client.testing.http.apache.MockHttpClient$1
  org.apache.http.client.RequestDirector is not found, referenced from com.google.api.client.testing.http.apache.MockHttpClient$1
  org.apache.http.HttpException is not found, referenced from com.google.api.client.testing.http.apache.MockHttpClient$1
  org.apache.http.HttpVersion is not found, referenced from com.google.api.client.testing.http.apache.MockHttpClient$1
  org.apache.http.impl.client.DefaultHttpClient.<init> is not found, referenced from com.google.api.client.testing.http.apache.MockHttpClient
  org.apache.http.message.BasicHttpResponse.<init> is not found, referenced from com.google.api.client.testing.http.apache.MockHttpClient$1
  org.apache.http.HttpVersion.HTTP_1_1 is not found, referenced from com.google.api.client.testing.http.apache.MockHttpClient$1

...

byte-buddy-agent-1.9.12.jar (2 errors):
  org.newsclub.net.unix.AFUNIXSocket is not found, referenced from net.bytebuddy.agent.VirtualMachine$ForHotSpot$OnUnix
  org.newsclub.net.unix.AFUNIXSocketAddress is not found, referenced from net.bytebuddy.agent.VirtualMachine$ForHotSpot$OnUnix
  org.newsclub.net.unix.AFUNIXSocket.isSupported is not found, referenced from net.bytebuddy.agent.VirtualMachine$ForHotSpot$OnUnix
  org.newsclub.net.unix.AFUNIXSocket.newInstance is not found, referenced from net.bytebuddy.agent.VirtualMachine$ForHotSpot$OnUnix
  org.newsclub.net.unix.AFUNIXSocket.setSoTimeout is not found, referenced from net.bytebuddy.agent.VirtualMachine$ForHotSpot$OnUnix
...
logback-core-1.2.3.jar (3 errors):
  org.codehaus.janino.ScriptEvaluator is not found, referenced from ch.qos.logback.core.boolex.JaninoEventEvaluatorBase
  org.codehaus.janino.ClassBodyEvaluator is not found, referenced from ch.qos.logback.core.joran.conditional.PropertyEvalScriptBuilder
  org.codehaus.commons.compiler.CompileException is not found, referenced from ch.qos.logback.core.joran.conditional.PropertyEvalScriptBuilder
  org.codehaus.janino.ScriptEvaluator.<init> is not found, referenced from ch.qos.logback.core.boolex.JaninoEventEvaluatorBase
  org.codehaus.janino.ScriptEvaluator.evaluate is not found, referenced from ch.qos.logback.core.boolex.JaninoEventEvaluatorBase

...
suztomo commented 5 years ago

Result

reactor.core package

Spring-core artifact has optional dependency to reactor-core:

    <dependency>
      <groupId>io.projectreactor</groupId>
      <artifactId>reactor-core</artifactId>
      <version>3.2.8.RELEASE</version>
      <scope>compile</scope>
      <optional>true</optional>
    </dependency>

Because reactor-core is not a direct dependency, the dependency was omitted.

io.netty package

Spring-core's NettyDataBufferFactory uses netty. Spring-core artifact has optional dependency to netty-buffer.

    <dependency>
      <groupId>io.netty</groupId>
      <artifactId>netty-buffer</artifactId>
      <version>4.1.34.Final</version>
      <scope>compile</scope>
      <optional>true</optional>
    </dependency>

Because netty-buffer is not a direct dependency, the dependency was omitted.

kotlin.jvm package

Spring-core has optional dependency to Kotlin libraries:

    <dependency>
      <groupId>org.jetbrains.kotlin</groupId>
      <artifactId>kotlin-reflect</artifactId>
      <version>1.2.71</version>
      <scope>compile</scope>
      <optional>true</optional>
    </dependency>
    <dependency>
      <groupId>org.jetbrains.kotlin</groupId>
      <artifactId>kotlin-stdlib</artifactId>
      <version>1.2.71</version>
      <scope>compile</scope>
      <optional>true</optional>
    </dependency>

For example Intrinsics class is in kotlin-stdlib

$ jar tf /usr/local/google/home/suztomo/.m2/repository/org/jetbrains/kotlin/kotlin-stdlib/1.2.71/kotlin-stdlib-1.2.71.jar |grep internal/Intrinsics.class
kotlin/jvm/internal/Intrinsics.class

org.springframework.instrument package

Spring-context's InstrumentationLoadTimeWeaver uses spring-instrument's org.springframework.instrument.InstrumentationSavingAgent

Spring-context has optional dependency to spring-instrument

    <dependency>
      <groupId>org.springframework</groupId>
      <artifactId>spring-instrument</artifactId>
      <version>5.1.6.RELEASE</version>
      <scope>compile</scope>
      <optional>true</optional>
    </dependency>

javax.enterprise.concurrent package

Spring-context's ConcurrentTaskExecutor uses ManagedExecutors via optional dependency.

    <dependency>
      <groupId>javax.enterprise.concurrent</groupId>
      <artifactId>javax.enterprise.concurrent-api</artifactId>
      <version>1.0</version>
      <scope>compile</scope>
      <optional>true</optional>
    </dependency>

The dependency is optional and thus not fetched.

groovy.lang package

Spring-context's GenericGroovyApplicationContext uses groovy package. Spring-context has optional dependency to groovy.

    <dependency>
      <groupId>org.codehaus.groovy</groupId>
      <artifactId>groovy</artifactId>
      <version>2.5.6</version>
      <scope>compile</scope>
      <optional>true</optional>
    </dependency>

org.slf4j package

This is false positive because slf4j-api's MarkerFactory catches NoSuchMethodError and falls back to accessing the factory through field.

image

In this case, logback-classic:1.2.3 is the binding that does not have the methods. We may need to extend https://github.com/GoogleCloudPlatform/cloud-opensource-java/pull/402 to classes that check NoSuchMethodError.

org.mockito.internal.creation.bytebuddy.inject.MockMethodDispatcher class

This is the same issue as the unresolved https://github.com/GoogleCloudPlatform/cloud-opensource-java/issues/407. However with Mockito 2.25.1 the package name has changed a bit.

$ jar tf /usr/local/google/home/suztomo/.m2/repository/org/mockito/mockito-core/2.25.1/mockito-core-2.25.1.jar |grep MockMethodDis
org/mockito/internal/creation/bytebuddy/inject/MockMethodDispatcher.raw

org.apache.http package

This was true positive. com.google.http-client:google-http-client:1.28.0 was missing httpclient dependency. 1.29.0 has fixed it:

    <dependency>
      <groupId>org.apache.httpcomponents</groupId>
      <artifactId>httpclient</artifactId>
    </dependency>

org.newsclub.net.unix package

byte-buddy-agent:1.9.12 depends on junixsocket-native-common with scope:provided. A provided dependency of non-direct dependencies (in this case test scope) is omitted.

junixsocket is a Java/JNI library that allows the use of Unix Domain Sockets (AF_UNIX sockets) from Java. https://github.com/kohlschutter/junixsocket

        <dependency>
            <groupId>com.kohlschutter.junixsocket</groupId>
            <artifactId>junixsocket-native-common</artifactId>
            <version>${version.unixsocket}</version>
            <scope>provided</scope>
        </dependency>

These methods are available in AFUNIXSocket class in the artifact.

org.codehaus.janino package

Logback-core has optional dependency to janino.

    <dependency>
       <groupId>org.codehaus.janino</groupId>
      <artifactId>janino</artifactId>
      <scope>compile</scope>
      <optional>true</optional>
    </dependency>

Janino can not only compile a set of source files to a set of class files like JAVAC, but also compile a Java expression, block, class body or source file in memory, load the bytecode and execute it directly in the same JVM. http://janino-compiler.github.io/janino/

org.codehaus.commons.compiler package

This package is part of org.codehaus.janino:commons-compiler:3.0.6. org.codehaus.janino:janino:3.0.6 has dependency to it. They are in the same repository in Github.

    <dependency>
      <groupId>org.codehaus.janino</groupId>
      <artifactId>commons-compiler</artifactId>
      <version>${project.parent.version}</version>
    </dependency>
chingor13 commented 5 years ago

This is the same issue I'm seeing with our transitive dependency upon commons-logging:commons-logging:1.2. It declares optional dependencies upon log4j, logkit, and avalon-framework.

suztomo commented 5 years ago

@chingor13 Thank you for sharing observation. I'll update you once these issues are resolved.

elharo commented 3 years ago

@suztomo Is this a dupe of #1983?

suztomo commented 3 years ago

Yes.