search

apache / datafusion-ray

Apache DataFusion Ray
https://datafusion.apache.org/ray
Apache License 2.0
74 stars 4 forks source link

Ray shuffle not working #10

Open austin362667 opened 2 days ago

austin362667 commented 2 days ago

I got an error on Mac: Memory pointer from external source (e.g, FFI) is not aligned with the specified scalar type. Before importing buffer through FFI, please make sure the allocation is aligned. while using ray shuffle on Datafusion Ray against TPC-H.

After that the Ray worker died.

@andygrove saw this error when trying to run the example on Linux, but it worked on Mac.

@franklsf95 Any suggestion would be helpful! I appreciate it!

andygrove commented 3 hours ago

Possible workaround in https://github.com/apache/arrow/issues/43552

andygrove commented 3 hours ago

arrow-rs already has a fix for this in https://github.com/apache/arrow-rs/pull/6472 and is included in 53.1.0, so this should be fixed when we upgrade to a version of DataFusion that uses this Arrow version