NVIDIA / spark-rapids-jni

RAPIDS Accelerator JNI For Apache Spark
Apache License 2.0
36 stars 64 forks source link

[FEA] Implement a good serialization approach for transfering host data between Java and JNI #2199

Open ttnghia opened 3 months ago

ttnghia commented 3 months ago

Currently, we don't have a good way to transfer host data between Java and JNI. As such, most of the time we have to manually do reflection to get access to the Java object in JNI code. That is very inefficient and not suitable for transferring large amounts of objects.

We should have a better way for data transfer between Java and JNI. Maybe leveraging protocol buffer, or implementing our propriety data serializer/deserializer.