search

USCDataScience / sparkler

Spark-Crawler: Apache Nutch-like crawler that runs on Apache Spark.
http://irds.usc.edu/sparkler/
Apache License 2.0
411 stars 143 forks source link

Maven build: overlapping classes found by shade plugin #121

Closed thammegowda closed 3 years ago

thammegowda commented 7 years ago

Issue Description

Sparkler has duplicate imports leading to overlap. Although it is not a blocking issue, it willl be nice to resolve it and get rid of WARNING message.

How to reproduce it

mvn clean package

[WARNING] xml-apis-1.3.04.jar, xmlParserAPIs-2.6.2.jar define 193 overlappping classes: 
[WARNING]   - org.w3c.dom.css.CSSValueList
[WARNING]   - org.w3c.dom.ProcessingInstruction
[WARNING]   - org.w3c.dom.html.HTMLBodyElement
[WARNING]   - org.w3c.dom.Entity
[WARNING]   - org.w3c.dom.html.HTMLMetaElement
[WARNING]   - org.xml.sax.helpers.SecuritySupport12$4
[WARNING]   - javax.xml.transform.dom.DOMResult
[WARNING]   - javax.xml.transform.OutputKeys
[WARNING]   - org.w3c.dom.css.ElementCSSInlineStyle
[WARNING]   - javax.xml.transform.FactoryFinder
[WARNING]   - 183 more...
[WARNING] jersey-core-1.9.jar, javax.ws.rs-api-2.0.1.jar define 55 overlappping classes: 
[WARNING]   - javax.ws.rs.core.HttpHeaders
[WARNING]   - javax.ws.rs.ext.RuntimeDelegate$HeaderDelegate
[WARNING]   - javax.ws.rs.DefaultValue
[WARNING]   - javax.ws.rs.core.StreamingOutput
[WARNING]   - javax.ws.rs.core.Request
[WARNING]   - javax.ws.rs.HEAD
[WARNING]   - javax.ws.rs.ext.Providers
[WARNING]   - javax.ws.rs.core.NewCookie
[WARNING]   - javax.ws.rs.core.UriBuilderException
[WARNING]   - javax.ws.rs.ext.ContextResolver
[WARNING]   - 45 more...
[WARNING] asm-3.1.jar, asm-5.1.jar define 21 overlappping classes: 
[WARNING]   - org.objectweb.asm.Type
[WARNING]   - org.objectweb.asm.AnnotationVisitor
[WARNING]   - org.objectweb.asm.MethodVisitor
[WARNING]   - org.objectweb.asm.Attribute
[WARNING]   - org.objectweb.asm.FieldWriter
[WARNING]   - org.objectweb.asm.signature.SignatureWriter
[WARNING]   - org.objectweb.asm.Edge
[WARNING]   - org.objectweb.asm.MethodWriter
[WARNING]   - org.objectweb.asm.Handler
[WARNING]   - org.objectweb.asm.ByteVector
[WARNING]   - 11 more...
[WARNING] guava-16.0.1.jar, spark-network-common_2.11-1.6.1.jar define 7 overlappping classes: 
[WARNING]   - com.google.common.base.Absent
[WARNING]   - com.google.common.base.Function
[WARNING]   - com.google.common.base.Supplier
[WARNING]   - com.google.common.base.Optional$1$1
[WARNING]   - com.google.common.base.Optional$1
[WARNING]   - com.google.common.base.Present
[WARNING]   - com.google.common.base.Optional
[WARNING] kryo-2.21.jar, reflectasm-1.07-shaded.jar define 23 overlappping classes: 
[WARNING]   - com.esotericsoftware.reflectasm.shaded.org.objectweb.asm.Opcodes
[WARNING]   - com.esotericsoftware.reflectasm.AccessClassLoader
[WARNING]   - com.esotericsoftware.reflectasm.shaded.org.objectweb.asm.Label
[WARNING]   - com.esotericsoftware.reflectasm.shaded.org.objectweb.asm.ClassWriter
[WARNING]   - com.esotericsoftware.reflectasm.shaded.org.objectweb.asm.AnnotationVisitor
[WARNING]   - com.esotericsoftware.reflectasm.shaded.org.objectweb.asm.Type
[WARNING]   - com.esotericsoftware.reflectasm.FieldAccess
[WARNING]   - com.esotericsoftware.reflectasm.ConstructorAccess
[WARNING]   - com.esotericsoftware.reflectasm.shaded.org.objectweb.asm.Edge
[WARNING]   - com.esotericsoftware.reflectasm.shaded.org.objectweb.asm.ClassVisitor
[WARNING]   - 13 more...
[WARNING] commons-logging-api-1.1.jar, commons-logging-1.2.jar, jcl-over-slf4j-1.7.10.jar define 6 overlappping classes: 
[WARNING]   - org.apache.commons.logging.impl.SimpleLog$1
[WARNING]   - org.apache.commons.logging.Log
[WARNING]   - org.apache.commons.logging.impl.SimpleLog
[WARNING]   - org.apache.commons.logging.LogConfigurationException
[WARNING]   - org.apache.commons.logging.impl.NoOpLog
[WARNING]   - org.apache.commons.logging.LogFactory
[WARNING] commons-logging-api-1.1.jar, commons-logging-1.2.jar define 13 overlappping classes: 
[WARNING]   - org.apache.commons.logging.LogSource
[WARNING]   - org.apache.commons.logging.LogFactory$4
[WARNING]   - org.apache.commons.logging.impl.WeakHashtable$1
[WARNING]   - org.apache.commons.logging.LogFactory$3
[WARNING]   - org.apache.commons.logging.LogFactory$2
[WARNING]   - org.apache.commons.logging.impl.WeakHashtable$Entry
[WARNING]   - org.apache.commons.logging.impl.Jdk14Logger
[WARNING]   - org.apache.commons.logging.LogFactory$1
[WARNING]   - org.apache.commons.logging.impl.WeakHashtable$Referenced
[WARNING]   - org.apache.commons.logging.impl.WeakHashtable$WeakKey
[WARNING]   - 3 more...
[WARNING] xml-apis-1.3.04.jar, stax-api-1.0.1.jar define 3 overlappping classes: 
[WARNING]   - javax.xml.namespace.NamespaceContext
[WARNING]   - javax.xml.namespace.QName
[WARNING]   - javax.xml.XMLConstants
[WARNING] javax.servlet-3.0.0.v201112011016.jar, javax.servlet-3.1.jar, javax.servlet-api-3.1.0.jar define 74 overlappping classes: 
[WARNING]   - javax.servlet.http.Cookie
[WARNING]   - javax.servlet.ServletContext
[WARNING]   - javax.servlet.Registration
[WARNING]   - javax.servlet.http.HttpSessionListener
[WARNING]   - javax.servlet.http.HttpSessionContext
[WARNING]   - javax.servlet.FilterChain
[WARNING]   - javax.servlet.http.HttpServletRequestWrapper
[WARNING]   - javax.servlet.http.HttpSessionAttributeListener
[WARNING]   - javax.servlet.annotation.HandlesTypes
[WARNING]   - javax.servlet.http.HttpSessionBindingListener
[WARNING]   - 64 more...
[WARNING] kryo-2.21.jar, minlog-1.2.jar define 2 overlappping classes: 
[WARNING]   - com.esotericsoftware.minlog.Log
[WARNING]   - com.esotericsoftware.minlog.Log$Logger
[WARNING] kryo-2.21.jar, objenesis-1.2.jar define 32 overlappping classes: 
[WARNING]   - org.objenesis.ObjenesisBase
[WARNING]   - org.objenesis.instantiator.gcj.GCJInstantiator
[WARNING]   - org.objenesis.ObjenesisHelper
[WARNING]   - org.objenesis.instantiator.jrockit.JRockitLegacyInstantiator
[WARNING]   - org.objenesis.instantiator.sun.SunReflectionFactoryInstantiator
[WARNING]   - org.objenesis.instantiator.ObjectInstantiator
[WARNING]   - org.objenesis.instantiator.gcj.GCJInstantiatorBase$DummyStream
[WARNING]   - org.objenesis.instantiator.basic.ObjectStreamClassInstantiator
[WARNING]   - org.objenesis.ObjenesisException
[WARNING]   - org.objenesis.Objenesis
[WARNING]   - 22 more...
[WARNING] org.apache.felix.main-5.4.0.jar, org.apache.felix.framework-5.4.0.jar, org.osgi.core-6.0.0.jar define 143 overlappping classes: 
[WARNING]   - org.osgi.framework.hooks.bundle.CollisionHook
[WARNING]   - org.osgi.framework.namespace.PackageNamespace
[WARNING]   - org.osgi.framework.FrameworkUtil$FilterImpl$Parser
[WARNING]   - org.osgi.resource.dto.package-info
[WARNING]   - org.osgi.framework.Bundle
[WARNING]   - org.osgi.framework.FrameworkUtil$DNChainMatching
[WARNING]   - org.osgi.framework.launch.FrameworkFactory
[WARNING]   - org.osgi.framework.PackagePermission
[WARNING]   - org.osgi.framework.hooks.service.EventHook
[WARNING]   - org.osgi.framework.hooks.bundle.package-info
[WARNING]   - 133 more...
[WARNING] jna-4.1.0.jar, jna-3.0.9.jar define 76 overlappping classes: 
[WARNING]   - com.sun.jna.NativeString
[WARNING]   - com.sun.jna.TypeConverter
[WARNING]   - com.sun.jna.FunctionResultContext
[WARNING]   - com.sun.jna.Pointer$Opaque
[WARNING]   - com.sun.jna.MethodParameterContext
[WARNING]   - com.sun.jna.ptr.ShortByReference
[WARNING]   - com.sun.jna.CallbackProxy
[WARNING]   - com.sun.jna.Pointer
[WARNING]   - com.sun.jna.win32.StdCallLibrary
[WARNING]   - com.sun.jna.Structure$FFIType$size_t
[WARNING]   - 66 more...
[WARNING] org.apache.felix.main-5.4.0.jar, org.apache.felix.framework-5.4.0.jar define 199 overlappping classes: 
[WARNING]   - org.apache.felix.resolver.util.OpenHashMap$FastEntryIterator
[WARNING]   - org.apache.felix.resolver.ResolverImpl$UseConstraintError
[WARNING]   - org.apache.felix.resolver.Candidates$DynamicImportFailed
[WARNING]   - org.apache.felix.resolver.ResolverImpl$7
[WARNING]   - org.apache.felix.resolver.ResolverImpl$ResolveSession
[WARNING]   - org.apache.felix.framework.cache.JarContent$EntriesEnumeration
[WARNING]   - org.apache.felix.framework.ExportedPackageImpl
[WARNING]   - org.apache.felix.framework.cache.JarContent
[WARNING]   - org.apache.felix.framework.StatefulResolver$ResolverHookRecord$1$1
[WARNING]   - org.apache.felix.framework.util.SecureAction
[WARNING]   - 189 more...
[WARNING] spark-core_2.11-1.6.1.jar, spark-launcher_2.11-1.6.1.jar, spark-unsafe_2.11-1.6.1.jar, spark-network-common_2.11-1.6.1.jar, unused-1.0.0.jar, spark-network-shuffle_2.11-1.6.1.jar define 1 overlappping classes: 
[WARNING]   - org.apache.spark.unused.UnusedStubClass
[WARNING] hadoop-yarn-api-2.2.0.jar, hadoop-yarn-common-2.2.0.jar define 3 overlappping classes: 
[WARNING]   - org.apache.hadoop.yarn.util.package-info
[WARNING]   - org.apache.hadoop.yarn.factories.package-info
[WARNING]   - org.apache.hadoop.yarn.factory.providers.package-info
[WARNING] maven-shade-plugin has detected that some .class files
[WARNING] are present in two or more JARs. When this happens, only
[WARNING] one single version of the class is copied in the uberjar.
[WARNING] Usually this is not harmful and you can skeep these
[WARNING] warnings, otherwise try to manually exclude artifacts
[WARNING] based on mvn dependency:tree -Ddetail=true and the above
[WARNING] output
[WARNING] See http://docs.codehaus.org/display/MAVENUSER/Shade+Plugin
[INFO] Replacing original artifact with shaded artifact.

Environment and Version Information

Please indicate relevant versions, including, if relevant:

An external links for reference

https://stackoverflow.com/questions/9802736/maven-build-warning-we-have-a-duplicate-class