[arrow] cuda build failure

[mattpulver]

Host Environment

• OS: all
• Compiler: all

To Reproduce Compile code with the line

#include <arrow/gpu/cuda_api.h>

Failure logs

fatal error C1083: Cannot open include file: 'arrow/gpu/cuda_api.h': No such file or directory

Additional context https://github.com/microsoft/vcpkg/pull/24570#issuecomment-1124246043

[Adela0814]

This header file is not installed in the upstream source， I think we should confirm with the upstream if it is necessary to install this.

[Adela0814]

https://github.com/apache/arrow/issues/13145

[mattpulver]

This header file is not installed in the upstream source， I think we should confirm with the upstream if it is necessary to install this.

Hi @Adela0814,

As far as I can tell, the upstream source has the cuda_api.h file available here:

https://github.com/apache/arrow/blob/master/cpp/src/arrow/gpu/cuda_api.h

Please correct me if I am missing something, but my expectation is that it is the responsibility of vcpkg to take this file from the apache package and install it with the arrow vcpkg.

[Adela0814]

For files that are not installed upstream, we need to confirm whether them are private, some files cannot be installed and used outside.

[dg0yt]

my expectation is that it is the responsibility of vcpkg to take this file from the apache package and install it with the arrow vcpkg.

I disagree: It is upstream's responsibility to put installation commands for all relevant files into the build system. However, there seems to be such a command: https://github.com/apache/arrow/blob/cf414db758a58320250b0129ba1f8e8b139d2513/cpp/src/arrow/gpu/CMakeLists.txt#L79

[Adela0814]

I checked on my local, this file is already installed. I can find it in VS. Could you please check if you have cuda_api.h file in \vcpkg\installed\x64-windows\include\arrow\gpu? (Note that you have to open cude feature)

[mattpulver]

Running .\vcpkg.exe install --recurse arrow:x64-windows does not install the gpu directory under \vcpkg\installed\x64-windows\include\arrow (and thus no cuda_api.h. I've also searched for this file w/ find . -name cuda_api.h and it is nowhere under the top vcpkg directory.) I would expect (correct me if I'm wrong) for that to be installed with .\vcpkg.exe install --recurse arrow[cuda]:x64-windows. However when I run that, I encounter an error:

PS C:\Users\User\Documents\heavyai\heavydb-deps\vcpkg> .\vcpkg.exe install --recurse arrow[cuda]:x64-windows
Computing installation plan...
The following packages will be built and installed:
    arrow[core,csv,cuda,filesystem,json,parquet]:x64-windows -> 8.0.0
Detecting compiler hash for triplet x64-windows...
Restored 0 packages from C:\Users\User\AppData\Local\vcpkg\archives in 172 us. Use --debug to see more details.
Installing 1/1 arrow:x64-windows...
Building arrow[core,csv,cuda,filesystem,json,parquet]:x64-windows...
-- Using cached apache-arrow-apache-arrow-8.0.0.tar.gz.
-- Cleaning sources at C:/Users/User/Documents/heavyai/heavydb-deps/vcpkg/buildtrees/arrow/src/rrow-8.0.0-e240b88362.clean. Use --editable to skip cleaning for the packages you specify.
-- Extracting source C:/Users/User/Documents/heavyai/heavydb-deps/vcpkg/downloads/apache-arrow-apache-arrow-8.0.0.tar.gz
-- Applying patch vs-2022-fixes.patch
-- Applying patch all.patch
-- Applying patch fix-ThirdPartyToolchain.patch
-- Using source at C:/Users/User/Documents/heavyai/heavydb-deps/vcpkg/buildtrees/arrow/src/rrow-8.0.0-e240b88362.clean
-- Found external ninja('1.10.2').
-- Configuring x64-windows
-- Building x64-windows-dbg
CMake Error at scripts/cmake/vcpkg_execute_build_process.cmake:158 (message):
    Command failed: C:/Users/User/Documents/heavyai/heavydb-deps/vcpkg/downloads/tools/cmake-3.22.2-windows/cmake-3.22.2-windows-i386/bin/cmake.exe --build . --config Debug --target install -- -v -j13
    Working Directory: C:/Users/User/Documents/heavyai/heavydb-deps/vcpkg/buildtrees/arrow/x64-windows-dbg
    See logs for more information:
      C:\Users\User\Documents\heavyai\heavydb-deps\vcpkg\buildtrees\arrow\install-x64-windows-dbg-out.log

Call Stack (most recent call first):
  installed/x64-windows/share/vcpkg-cmake/vcpkg_cmake_build.cmake:112 (vcpkg_execute_build_process)
  installed/x64-windows/share/vcpkg-cmake/vcpkg_cmake_install.cmake:41 (vcpkg_cmake_build)
  ports/arrow/portfile.cmake:89 (vcpkg_cmake_install)
  scripts/ports.cmake:146 (include)

error: building arrow:x64-windows failed with: BUILD_FAILED
Please ensure you're using the latest port files with `git pull` and `vcpkg update`.
Then check for known issues at:
    https://github.com/microsoft/vcpkg/issues?q=is%3Aissue+is%3Aopen+in%3Atitle+arrow
You can submit a new issue at:
    https://github.com/microsoft/vcpkg/issues/new?template=report-package-build-failure.md&title=[arrow]+Build+error
Include '[arrow] Build error' in your bug report title, the following version information in your bug description, and attach any relevant failure logs from above.
    vcpkg-tool version: 2022-05-05-67e17c1782801cf481be9ac0b3765dff3e4bdeb8
    vcpkg-scripts version: 38bb87c55 2022-06-02 (17 hours ago)

The log file install-x64-windows-dbg-out.log shows no indication of an error. Its last lines are:

--   Outputting build configuration summary to C:/Users/User/Documents/heavyai/heavydb-deps/vcpkg/buildtrees/arrow/x64-windows-dbg/cmake_summary.json
-- Configuring done
-- Generating done
CMake Warning:
  Manually-specified variables were not used by the project:

    ZSTD_MSVC_LIB_PREFIX
    _VCPKG_ROOT_DIR

-- Build files have been written to: C:/Users/User/Documents/heavyai/heavydb-deps/vcpkg/buildtrees/arrow/x64-windows-dbg

Is there a Windows 10 CI server that is successfully building/installing arrow[cuda]:x64-windows?

Lastly maybe I am not understanding what you mean by

(Note that you have to open cuda feature)

What do you mean by "open"? I do have the vcpkg cuda installed, as well as the NVidia CUDA Toolkit directly from NVidia, if that's what you mean. Otherwise can you give an explicit command to run?

[Adela0814]

arrow[cuda] will open cuda feature. I tested it on linuxand it works fine.

[mattpulver]

Is there a Windows 10 CI server that is successfully building/installing arrow[cuda]:x64-windows?

Do you have any confirmation that arrow[cuda]:x64-windows builds on any Windows 10 platform?

[Adela0814]

I had other issues with windows 10, will try again when I have time.

[Cheney-W]

I could install arrow[cuda]:x64-windows successfully, and I could find the arrow/gpu/cuda_api.h file at the vcpkg\installed\{triplets}\include folder. My test environments as below:

1. Windows 11 22H2 22621.1265.
2. vcpkg 2023-01-24-8a88d63f241d391772fbde69af9cab96c3c64c75
3. CUDA v12.0

Thanks for posting this issue. Please reopen this issue if this is still a problem for you.