search

Kotlin / kotlin-spark-api

This projects gives Kotlin bindings and several extensions for Apache Spark. We are looking to have this as a part of Apache Spark 3.x
Apache License 2.0
464 stars 35 forks source link

Questions regarding RoaringBitmap processing fail #211

Closed Kimakjun closed 1 year ago

Kimakjun commented 1 year ago

The code that worked normally in Scala Spark does not work in Kotlin Spark, but I do not know the exact cause. Is RoaringBitmap processing not supported in Kotlin Spark?

       implementation("org.roaringbitmap:RoaringBitmap:0.9.39")
       implementation("org.jetbrains.kotlinx.spark:kotlin-spark-api_3.3.2_2.13:1.2.4")
       compileOnly("org.apache.spark:spark-sql_2.13:3.3.2")

        val bitmaps2 = ds.map {  
           RoaringBitmap.bitmapOf(20) 
        }

        Exception in thread "main" java.lang.IndexOutOfBoundsException: Index: 0, Size: 0
    at java.util.ArrayList.rangeCheck(ArrayList.java:659)
    at java.util.ArrayList.get(ArrayList.java:435)
    at org.jetbrains.kotlinx.spark.api.EncodingKt.schema(Encoding.kt:225)
    at org.jetbrains.kotlinx.spark.api.EncodingKt.schema$default(Encoding.kt:186)
    at org.jetbrains.kotlinx.spark.api.EncodingKt$memoizedSchema$1.invoke(Encoding.kt:368)
    at org.jetbrains.kotlinx.spark.api.EncodingKt$memoizedSchema$1.invoke(Encoding.kt:367)
    at org.jetbrains.kotlinx.spark.api.Memoize1.invoke(Encoding.kt:377)
    at org.jetbrains.kotlinx.spark.api.EncodingKt.generateEncoder(Encoding.kt:146)

Thanks!