Closed randyzwitch closed 8 years ago
These warnings aren't related to another version of Spark, but instead are caused by Maven Shade plugin, responsible for creating single "fat" jar (a.k.a. ubjerjar). Uberjar unpacks all dependency jars, combines their content and packs to a single big file. The problem is that some dependencies may themselves depend on different versions of the same library, e.g. our project A may depend on libraries B and C, where B also depends on D of version 1.1 while C depends on D of version 1.2:
A
| <- B <- D v1.1
| <- C <- D v1.2
To pack them all into a single jar Shade plugin has to select only one version of D. Normally, it chooses highest version (in this case D v1.2), but also gives a warning about duplicate content.
Creating an uberjar is something very usual and even recommended in Spark, so I don't think we need to change it. Though, there may be a way to suppress warnings to avoid confusion.
Sounds good, as long as it's known that it's occurring.
This isn't an issue per se, I'm just including this to help with refinement of the build process (and possibly by me). Here's the full log when you build on OSX, where homebrew is already used to install Spark. You get warnings, but it does report success at the end.