Packages build is often broken - should we support it?

[ruebot]

Since we've moved to a newer version of Tika, we've had on-and-off trouble (mostly on!) with aut working with --packages. The current state yields this:

        ::::::::::::::::::::::::::::::::::::::::::::::

        ::              FAILED DOWNLOADS            ::

        :: ^ see resolution messages for details  ^ ::

        ::::::::::::::::::::::::::::::::::::::::::::::

        :: javax.activation#activation;1.1!activation.jar

        :: com.google.guava#guava;28.0-jre!guava.jar(bundle)

        :: com.google.guava#failureaccess;1.0.1!failureaccess.jar(bundle)

        :: com.google.guava#listenablefuture;9999.0-empty-to-avoid-conflict-with-guava!listenablefuture.jar

        :: com.google.code.findbugs#jsr305;3.0.2!jsr305.jar

        :: org.checkerframework#checker-qual;2.8.1!checker-qual.jar

        :: com.google.errorprone#error_prone_annotations;2.3.2!error_prone_annotations.jar

        :: com.google.j2objc#j2objc-annotations;1.3!j2objc-annotations.jar

        :: org.codehaus.mojo#animal-sniffer-annotations;1.17!animal-sniffer-annotations.jar

        :: com.google.protobuf#protobuf-java;3.9.0!protobuf-java.jar(bundle)

        :: joda-time#joda-time;2.10.6!joda-time.jar

        ::::::::::::::::::::::::::::::::::::::::::::::

:: USE VERBOSE OR DEBUG MESSAGE LEVEL FOR MORE DETAILS
Exception in thread "main" java.lang.RuntimeException: [download failed: javax.activation#activation;1.1!activation.jar, download failed: com.google.guava#guava;28.0-jre!guava.jar(bundle), download failed: com.google.guava#failureaccess;1.0.1!failureaccess.jar(bundle), download failed: com.google.guava#listenablefuture;9999.0-empty-to-avoid-conflict-with-guava!listenablefuture.jar, download failed: com.google.code.findbugs#jsr305;3.0.2!jsr305.jar, download failed: org.checkerframework#checker-qual;2.8.1!checker-qual.jar, download failed: com.google.errorprone#error_prone_annotations;2.3.2!error_prone_annotations.jar, download failed: com.google.j2objc#j2objc-annotations;1.3!j2objc-annotations.jar, download failed: org.codehaus.mojo#animal-sniffer-annotations;1.17!animal-sniffer-annotations.jar, download failed: com.google.protobuf#protobuf-java;3.9.0!protobuf-java.jar(bundle), download failed: joda-time#joda-time;2.10.6!joda-time.jar]
    at org.apache.spark.deploy.SparkSubmitUtils$.resolveMavenCoordinates(SparkSubmit.scala:1389)
    at org.apache.spark.deploy.DependencyUtils$.resolveMavenDependencies(DependencyUtils.scala:54)
    at org.apache.spark.deploy.SparkSubmit.prepareSubmitEnvironment(SparkSubmit.scala:308)
    at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:871)
    at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:180)
    at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203)
    at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90)
    at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1007)
    at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1016)
    at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

I see two options:

1. We support it.

I can dive back into dependency tree, and do a lot of pom.xml surgery again to see if I can get us to a state to run --packages again. Last time I tried this with the two 3.0.0 previews, the results were not fruitful, and I'm highly doubtful I can get it work again.

2. We don't support it.

   aut can be loaded by Spark to driver and executor classpaths with --jars.

If we go with option 2 (which is my preference), I'll work on getting the documentation updated.

[ianmilligan1]

My preference is also for option 2 - I have no issues using the jar command, it works well, and I’ve always found it more straightforward than packages even when both worked. So consider this a strong vote in favour of loving towards a 1.0.0 without packages.

[SamFritz]

Similar to Ian's response, I think most of my work with AUT has used jars - which has been straightforward to work with. I can't recall a significant number of users who've identified using --packages over --jars, so it makes complete sense to move forward with option 2 (+1 from me)!

[lintool]

+1 for (2) sgtm

[ruebot]

Cool. I'll leave this open for a week, and just to make sure there isn't a solid argument to keep it from any community members.

[ruebot]

Marking as resolved; won't support.