Open z-anderson opened 1 week ago
do you recommend that we should cherry-pick that commit (and a few more commits that it depends on)?
@z-anderson, I guess that commit will not fix this issue.
With VCPKG_TRIPLET=x64-linux-avx
, vcpkg will use a custom triplet file we created, see https://github.com/apache/incubator-gluten/blob/main/dev/vcpkg/triplets/x64-linux-avx.cmake. You can see some flags are set specifically for x86_64.
To support arm64, I recommend you to create another triplet file with proper flags set, e.g. set(VCPKG_TARGET_ARCHITECTURE arm64)
(see reference link). Then, in Gluten script, set VCPKG_TRIPLET
to this new custom triple instead of x64-linux-avx
when target architecture is arm64 (we may make target architecture a configurable build option).
Thank you!
@z-anderson, could you create a pr to fix it?
@z-anderson would you like to create the PR and maintain it? We currently don't have any ARM CI tests actually. There are several guys submitting PRs on ARM. Maybe you all can work together and contribute.
Sounds good. Yes, I'll put up a PR with my changes to https://github.com/apache/incubator-gluten/blob/main/dev/vcpkg/init.sh for this.
I put up a draft https://github.com/apache/incubator-gluten/pull/7687 . This PR gets past that error, but I'm still working on getting it to build completely (at least for my set up).
I don't have permission to create a new github runner (with ARM). Could someone else create the new runner, please? Then I'll use it in https://github.com/apache/incubator-gluten/blob/main/.github/workflows/build_bundle_package.yml
@z-anderson, could you first validate your fix in your arm64 environment? Currently, our CI resources are near to the limitation. We can add new jobs on arm in the future when some resources are freed up.
In my arm64 environment, it actually finds the community triplet (so it doesn't use the new triplet file). Using the community triplet does correctly get it to start building for arm64 instead of amd.
I don't have permission to create a new github runner (with ARM). Could someone else create the new runner, please? Then I'll use it in https://github.com/apache/incubator-gluten/blob/main/.github/workflows/build_bundle_package.yml
You may track the UT internally firstly. We may plan to add the ARM runner in future.
So will we continue the PR, but not do CI tests with an ARM runner for now?
Problem description
I'm trying to build gluten and velox on aarch64 architecture. I'm running the script
./dev/builddeps-veloxbe.sh --enable_vcpkg=ON --spark_version=3.5
. I get the error below. I see that these options-mavx2 -mfma -mavx -mf16c -mlzcnt -mbmi2
are only for x86_64, so they won't work on aarch64. Do you know how I can build gluten for aarch64 please? (In version 1.1.x.x., we were able to build for both aarch64 and x86_64).I see vcpkg
init.sh
hasVCPKG_TRIPLET=x64-linux-avx
https://github.com/apache/incubator-gluten/blob/c82af60a635fb52bf1314cd831f5915dd37d910d/dev/vcpkg/init.sh#L10. It makes sense that thex64
and theavx
parts of this triplet will cause errors running on aarch64. I see it got changed in https://github.com/apache/incubator-gluten/commit/f00e4536bc7ccc56a2eee5800084319130cbf806#diff-4a8a09cebf7ed92cb7321136b9ef223c4653757270e77e942fd6fe2dca30793e - do you recommend that we should cherry-pick that commit (and a few more commits that it depends on)?System information
Velox System Info v0.0.2 Commit: 69a6bb9602c97050cc3459a09c846b84182ecb9c CMake Version: 3.28.3 System: Linux-5.15.0-1070-aws Arch: aarch64 CPU Name: Model name: Neoverse-N1 C++ Compiler: /usr/bin/c++ C++ Compiler Version: 9.4.0 C Compiler: /usr/bin/cc C Compiler Version: 9.4.0 CMake Prefix Path: /usr/local;/usr;/;/usr/local;/usr/local;/usr/X11R6;/usr/pkg;/opt
\nThe results will be copied to your clipboard if xclip is installed.
CMake log
No response