spark v3.2 RoaringBitMap dependency is causing build failure

Aggarwal-Raghav commented 1 year ago

Apache Iceberg version

1.3.0 (latest release)

Query engine

None

Please describe the bug 🐞

While building iceberg with the build fails while building spark module: :iceberg-spark:iceberg-spark-runtime-3.2_2.12:compileIntegrationJava > Resolve dependencies of :iceberg-spark:iceberg-spark-runtime-3.2_2.12:integrationCompileClasspath > RoaringBitmap-0.9.0.module

Gradle command used: ./gradlew build -x test -x integrationTest -DhiveVersions=3 -DsparkVersions=3.2 -DscalaVersion=2.12 In spark module it is trying to download RoaringBitmap-0.9.0.module which is not present in open source maven repository.

This issue won't occur for spark v3.4 because in pom.xml of spark v3.4.0 the RoaringBitMap version is 0.9.38 and the corresponding 0.9.38.module is present in artifactory.

With -Pquick=true this issue is not happening.

Aggarwal-Raghav commented 1 year ago

The same will happen in spark v3.1

RussellSpitzer commented 1 year ago

I could not reproduce this in on master with spark 3.2 using ./gradlew clean jar or ./gradlew build -x test -x integrationTest -DhiveVersions=3 -DsparkVersions=3.2 -DscalaVersion=2.12

Do you have more specific repro instructions?

Aggarwal-Raghav commented 1 year ago

@RussellSpitzer, this was the error I received when I tried building. In the https://repo1.maven.org/maven2/org/roaringbitmap/RoaringBitmap/0.9.0, .module is missing as it is present in https://repo1.maven.org/maven2/org/roaringbitmap/RoaringBitmap/0.9.44

❯ make build
gradle clean

Deprecated Gradle features were used in this build, making it incompatible with Gradle 9.0.

You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins.

See https://docs.gradle.org/8.1.1/userguide/command_line_interface.html#sec:command_line_warnings

BUILD SUCCESSFUL in 2s
24 actionable tasks: 24 executed
gradle build -x test -x integrationTest -DhiveVersions=3 -DsparkVersions=3.2 -DscalaVersion=2.12

<other module building>

> Task :iceberg-spark:iceberg-spark-extensions-3.2_2.12:compileTestJava
Note: Some input files use or override a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
Note: Some input files use unchecked or unsafe operations.
Note: Recompile with -Xlint:unchecked for details.

> Task :iceberg-spark:iceberg-spark-runtime-3.2_2.12:compileIntegrationJava FAILED

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':iceberg-spark:iceberg-spark-runtime-3.2_2.12:compileIntegrationJava'.
> Could not resolve all files for configuration ':iceberg-spark:iceberg-spark-runtime-3.2_2.12:integrationCompileClasspath'.
   > Could not resolve org.roaringbitmap:RoaringBitmap:0.9.0.
     Required by:
         project :iceberg-spark:iceberg-spark-runtime-3.2_2.12 > org.apache.spark:spark-hive_2.12:3.2.2 > org.apache.spark:spark-core_2.12:3.2.2
         project :iceberg-spark:iceberg-spark-runtime-3.2_2.12 > org.apache.spark:spark-hive_2.12:3.2.2 > org.apache.spark:spark-core_2.12:3.2.2 > org.apache.spark:spark-network-common_2.12:3.2.2
         project :iceberg-spark:iceberg-spark-runtime-3.2_2.12 > org.apache.spark:spark-hive_2.12:3.2.2 > org.apache.spark:spark-core_2.12:3.2.2 > org.apache.spark:spark-network-shuffle_2.12:3.2.2
      > Could not resolve org.roaringbitmap:RoaringBitmap:0.9.0.
         > Could not get resource 'https://<personal-repo>/org/roaringbitmap/RoaringBitmap/0.9.0/RoaringBitmap-0.9.0.module'.
            > Could not GET 'https://<personal-repo>/org/roaringbitmap/RoaringBitmap/0.9.0/RoaringBitmap-0.9.0.module'.
               > Read timed out

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.

* Get more help at https://help.gradle.org

Deprecated Gradle features were used in this build, making it incompatible with Gradle 9.0.

You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins.

See https://docs.gradle.org/8.1.1/userguide/command_line_interface.html#sec:command_line_warnings

BUILD FAILED in 4m 59s
313 actionable tasks: 288 executed, 25 up-to-date

birdawade88 commented 3 months ago

Hi all, Could not resolve org.roaringbitmap:RoaringBitmap:0.9.0. Required by: project :iceberg-spark:iceberg-spark-runtime-3.2_2.12 > org.apache.spark:spark-hive_2.12:3.2.2 > org.apache.spark:spark-core_2.12:3.2.2 project :iceberg-spark:iceberg-spark-runtime-3.2_2.12 > org.apache.spark:spark-hive_2.12:3.2.2 > org.apache.spark:spark-core_2.12:3.2.2 > org.apache.spark:spark-network-common_2.12:3.2.2 project :iceberg-spark:iceberg-spark-runtime-3.2_2.12 > org.apache.spark:spark-hive_2.12:3.2.2 > org.apache.spark:spark-core_2.12:3.2.2 > org.apache.spark:spark-network-shuffle_2.12:3.2.2

Could not resolve org.roaringbitmap:RoaringBitmap:0.9.0 I am getting this issue during build. Can you please help on this?

Aggarwal-Raghav commented 3 months ago

@birdawade88, thanks for reaching out. IIRC, my org maintained a personal artifactory and there I was facing similar issue. I think I upgraded roaringbit version (minor version) at that moment and (upgraded same version in spark component as well) then it worked but few thing you can try out:

Clean gradle cache and then run build
Try running the gradle with debug mode to see stacktrace.
Ensure gradle version align with the one used by gradlew in iceberg

apache / iceberg