search

apache / datafusion-comet

Apache DataFusion Comet Spark Accelerator
https://datafusion.apache.org/comet
Apache License 2.0
612 stars 113 forks source link

feat: Support spark base64 function #420

Open leoluan2009 opened 1 month ago

leoluan2009 commented 1 month ago

Which issue does this PR close?

Closes #419 .

Rationale for this change

What changes are included in this PR?

How are these changes tested?

leoluan2009 commented 1 month ago

@viirya Help to start CI, thanks

leoluan2009 commented 1 month ago

@viirya help to start CI and review code, thanks

viirya commented 1 month ago

Looks like there are incompatible results compared with Spark.

leoluan2009 commented 1 month ago

Looks like there are incompatible results compared with Spark.

datafusion use Basic type of base64 scheme while spark use MIME type. After search the source of https://github.com/marshallpierce/rust-base64 which is used by datafusion for base64 function, I found this lib do not support MIME type of base64 scheme, so it may difficult to add Spark-compatible base64/unbase64 function. @viirya do you have some good ideas？

viirya commented 1 month ago

Could we find another library supporting that? We can implement custom base64 function in Comet.

leoluan2009 commented 1 month ago

Could we find another library supporting that? We can implement custom base64 function in Comet.

I will do some research and have a try, thanks