filesystem module should be built in install_arrow

[xumingming]

Problem description

We rely on arrow's filesystem module: https://github.com/apache/incubator-gluten/blob/43d0ff987fb526a885264f0efc252654c450a09f/cpp/core/shuffle/Utils.h#L21

But we didn't build it: https://github.com/facebookincubator/velox/blob/c0fa8f23123cfacc8807b010341303e9870cea38/scripts/setup-centos9.sh#L162-L187

To include filesystem module, we need to specify ARROW_FILESYSTEM: https://github.com/apache/arrow/blob/51d50d750012d3ca04127f6723c4f1e69ff4f5dc/cpp/src/arrow/CMakeLists.txt#L1254-L1256

System information

N/A

CMake log

N/A

[FelixYBW]

@PHILO-HE

I didn't see the issue when using vcpkg

[PHILO-HE]

It's strange that this issue is not uncovered in CI and local test. @xumingming, do you know the reason?

[xumingming]

It's strange that this issue is not uncovered in CI and local test. @xumingming, do you know the reason?

@PHILO-HE Don't know why. But gluten build fails on Alinux3, and after build arrow with ARROW_FILESYSTEM, it succeed.

/Code/gluten/cpp/core/shuffle/Utils.h:21:10: fatal error: arrow/filesystem/filesystem.h: No such file or directory
   21 | #include <arrow/filesystem/filesystem.h>
      |          ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
compilation terminated.
make[2]: *** [core/CMakeFiles/gluten.dir/build.make:346: core/CMakeFiles/gluten.dir/compute/ResultIterator.cc.o] Error 1
In file included from /Code/gluten/cpp/core/shuffle/Payload.h:25,
                 from /Code/gluten/cpp/core/shuffle/PartitionWriter.h:23,
                 from /Code/gluten/cpp/core/shuffle/ShuffleWriter.h:28,
                 from /Code/gluten/cpp/core/compute/Runtime.h:32,
                 from /Code/gluten/cpp/core/jni/JniCommon.h:26,
                 from /Code/gluten/cpp/core/jni/JniError.h:22,
                 from /Code/gluten/cpp/core/config/GlutenConfig.cc:22:

[xumingming]

And also we should build parquet module:

-- Found Arrow library: /usr/local/lib64/libarrow.a
CMake Error at CMake/ConfigArrow.cmake:40 (message):
  Arrow library Not Found: libparquet.a
Call Stack (most recent call first):
  core/CMakeLists.txt:240 (find_arrow_lib)

but the corresponding option is OFF in install_arrow:

    cmake_install \
      -DARROW_PARQUET=OFF \
      -DARROW_WITH_THRIFT=ON \

[xumingming]

In file included from /Code/gluten/cpp/core/memory/ColumnarBatch.cc:21:
/usr/local/include/arrow/c/helpers.h: In function ‘void ArrowSchemaMove(ArrowSchema*, ArrowSchema*)’:
/usr/local/include/arrow/c/helpers.h:53:3: error: ‘assert’ was not declared in this scope
   53 |   assert(dest != src);
      |   ^~~~~~

[PHILO-HE]

@xumingming, I just recalled something.

ARROW_PARQUET & ARROW_FILESYSTEM are set to ON in two places:

1. Gluten's build_arrow.sh. It will install arrow libs into system path. And we also remove install_arrow from velox's setup script to make sure those arrow libs built out with expected options are installed, see link.
2. Velox's modify_velox.patch. The code change by this patch will really take affect when building arrow from source. This build will be triggered only if system arrow libs are not found, assuming Arrow_SOURCE=AUTO.

This looks a bit tricky. If you have any advice, please kindly let me know.

I guess your system has installed arrow libs from Velox's script.

[xumingming]

@PHILO-HE build_arrow.sh is invoked manually, not included in other shell scripts like buildbundle-veloxbe.sh right?

[PHILO-HE]

@PHILO-HE build_arrow.sh is invoked manually, not included in other shell scripts like buildbundle-veloxbe.sh right?

@xumingming, no, if we execute buildbundle-veloxbe.sh or builddeps-veloxbe.sh, the script will build and install arrow by default. See https://github.com/apache/incubator-gluten/blob/main/dev/builddeps-veloxbe.sh#L218.