search

duckdb / duckdb_azure

Azure extension for DuckDB
MIT License
50 stars 17 forks source link

Support R windows_amd64_rtools build #60

Open natbprice opened 6 months ago

natbprice commented 6 months ago

It appears that the azure extension does not currently work with R on Windows because it requires a special build (GCC 12/MinGW-w64). See https://github.com/duckdb/duckdb/pull/10204 and https://github.com/duckdb/duckdb-r/issues/100.

There is a workflow for this build, but it appears that it is being excluded in https://github.com/duckdb/duckdb_azure/blob/main/.github/workflows/MainDistributionPipeline.yml. I assume this is because it does not work as-is, but it is a bit beyond my understanding to determine what might be required to fix this issue.

natbprice commented 6 months ago

I forked the repo and tested the build after removing exclusion: https://github.com/natbprice/duckdb_azure/actions/runs/9177755461/job/25236890148

mkdir -p build/release && \
cmake  -DEXTENSION_STATIC_BUILD=1 -DDUCKDB_EXTENSION_CONFIGS='/d/a/duckdb_azure/duckdb_azure/extension_config.cmake'  -DOSX_BUILD_ARCH=  -DVCPKG_MANIFEST_DIR='/d/a/duckdb_azure/duckdb_azure/' -DVCPKG_BUILD=1 -DCMAKE_TOOLCHAIN_FILE='D:\a\duckdb_azure\duckdb_azure/vcpkg/scripts/buildsystems/vcpkg.cmake' -DVCPKG_TARGET_TRIPLET='x64-mingw-static' -DDUCKDB_EXPLICIT_PLATFORM='windows_amd64_rtools'  -DCMAKE_BUILD_TYPE=Release -S ./duckdb/ -B build/release && \
cmake --build build/release --config Release
-- Building for: Unix Makefiles
-- Found Python3: C:/hostedtoolcache/windows/Python/3.11.9/x64/python3.exe (found version "3.11.9") found components: Interpreter 
-- Running vcpkg install
Detecting compiler hash for triplet x64-windows...
Detecting compiler hash for triplet x64-mingw-static...
The following packages will be built and installed:
  * azure-core-cpp[core,curl,http,winhttp]:x64-mingw-static@1.11.0
    azure-identity-cpp:x64-mingw-static@1.6.0
    azure-storage-blobs-cpp:x64-mingw-static@12.10.0
  * azure-storage-common-cpp:x64-mingw-static@12.5.0
    azure-storage-files-datalake-cpp:x64-mingw-static@12.9.0
  * curl[core,non-http,schannel,ssl,sspi]:x64-mingw-static@8.5.0#1
  * vcpkg-cmake:x64-windows@2023-05-04
  * vcpkg-cmake-config:x64-windows@2022-02-06#1
  * wil:x64-mingw-static@2023-10-28
  * zlib:x64-mingw-static@1.3
Additional packages (*) will be modified to complete this operation.
A suitable version of 7zip was not found (required v23.1.0) Downloading portable 7zip 23.1.0...
Downloading 7zip...
https://www.7-zip.org/a/7z2301-extra.7z->D:\a\duckdb_azure\duckdb_azure\vcpkg\downloads\7z2301-extra.7z
Downloading https://www.7-zip.org/a/7z2301-extra.7z
Extracting 7zip...
Restored 2 package(s) from GitHub Actions Cache in 944 ms. Use --debug to see more details.
Installing 1/10 vcpkg-cmake-config:x64-windows@2022-02-06#1...
Elapsed time to handle vcpkg-cmake-config:x64-windows: 2.[31](https://github.com/natbprice/duckdb_azure/actions/runs/9177755461/job/25236890148#step:10:32) ms
vcpkg-cmake-config:x64-windows package ABI: ba4cffff032243d4c0473a0eeabddf5f75327cd59c0bc3c2e8a4c2b76e9dc811
Installing 2/10 vcpkg-cmake:x64-windows@2023-05-04...
Elapsed time to handle vcpkg-cmake:x64-windows: 2.28 ms
vcpkg-cmake:x64-windows package ABI: b5f858c3bb197b1a07f510b5333562c0c98fe176119806206e3a4bc256ac2d03
Installing 3/10 wil:x64-mingw-static@2023-10-28...
Building wil:x64-mingw-static@2023-10-28...
warning: -- Using community triplet x64-mingw-static. This triplet configuration is not guaranteed to succeed.
-- [COMMUNITY] Loading triplet configuration from: D:\a\duckdb_azure\duckdb_azure\vcpkg\triplets\community\x64-mingw-static.cmake
-- Downloading https://github.com/Microsoft/wil/archive/v1.0.231028.1.tar.gz -> Microsoft-wil-v1.0.231028.1.tar.gz...
-- Extracting source D:/a/duckdb_azure/duckdb_azure/vcpkg/downloads/Microsoft-wil-v1.0.231028.1.tar.gz
-- Using source at D:/a/duckdb_azure/duckdb_azure/vcpkg/buildtrees/wil/src/0.231028.1-2698fce114.clean
-- Configuring x64-mingw-static
-- Building x64-mingw-static-rel
-- Up-to-date: D:/a/duckdb_azure/duckdb_azure/vcpkg/packages/wil_x64-mingw-static/share/wil/natvis
-- Installing: D:/a/duckdb_azure/duckdb_azure/vcpkg/packages/wil_x64-mingw-static/share/wil/natvis/wil.natvis
-- Installing: D:/a/duckdb_azure/duckdb_azure/vcpkg/packages/wil_x64-mingw-static/share/wil/copyright
-- Performing post-build validation
Stored binaries in 1 destinations in 334 ms.
Elapsed time to handle wil:x64-mingw-static: 5.1 s
wil:x64-mingw-static package ABI: be2d8cd484e0927fc22fd6cd1ea86c2d28278c2455d08c5f1543cb294e90c05b
Installing 4/10 zlib:x64-mingw-static@1.3...
Building zlib:x64-mingw-static@1.3...
warning: -- Using community triplet x64-mingw-static. This triplet configuration is not guaranteed to succeed.
-- [COMMUNITY] Loading triplet configuration from: D:\a\duckdb_azure\duckdb_azure\vcpkg\triplets\community\x64-mingw-static.cmake
-- Downloading https://github.com/madler/zlib/archive/v1.3.tar.gz -> madler-zlib-v1.3.tar.gz...
-- Extracting source D:/a/duckdb_azure/duckdb_azure/vcpkg/downloads/madler-zlib-v1.3.tar.gz
-- Applying patch 0001-Prevent-invalid-inclusions-when-HAVE_-is-set-to-0.patch
-- Applying patch 0002-skip-building-examples.patch
-- Applying patch 0003-build-static-or-shared-not-both.patch
-- Applying patch 0004-android-and-mingw-fixes.patch
-- Using source at D:/a/duckdb_azure/duckdb_azure/vcpkg/buildtrees/zlib/src/v1.3-8825ee792f.clean
-- Configuring x64-mingw-static
-- Building x64-mingw-static-dbg
-- Building x64-mingw-static-rel
-- Installing: D:/a/duckdb_azure/duckdb_azure/vcpkg/packages/zlib_x64-mingw-static/share/zlib/vcpkg-cmake-wrapper.cmake
-- Fixing pkgconfig file: D:/a/duckdb_azure/duckdb_azure/vcpkg/packages/zlib_x64-mingw-static/lib/pkgconfig/zlib.pc
-- Downloading https://mirror.msys2.org/mingw/mingw64/mingw-w64-x86_64-pkgconf-1~2.1.0-1-any.pkg.tar.zst;https://repo.msys2.org/mingw/mingw64/mingw-w64-x86_64-pkgconf-1~2.1.0-1-any.pkg.tar.zst;https://mirror.yandex.ru/mirrors/msys2/mingw/mingw64/mingw-w64-x86_64-pkgconf-1~2.1.0-1-any.pkg.tar.zst;https://mirrors.tuna.tsinghua.edu.cn/msys2/mingw/mingw64/mingw-w64-x86_64-pkgconf-1~2.1.0-1-any.pkg.tar.zst;https://mirrors.ustc.edu.cn/msys2/mingw/mingw64/mingw-w64-x86_64-pkgconf-1~2.1.0-1-any.pkg.tar.zst;https://mirror.selfnet.de/msys2/mingw/mingw64/mingw-w64-x86_64-pkgconf-1~2.1.0-1-any.pkg.tar.zst -> mingw-w64-x86_64-pkgconf-1~2.1.0-1-any.pkg.tar.zst...
-- Downloading https://mirror.msys2.org/msys/x86_64/msys2-runtime-3.4.9-3-x86_64.pkg.tar.zst;https://repo.msys2.org/msys/x86_64/msys2-runtime-3.4.9-3-x86_64.pkg.tar.zst;https://mirror.yandex.ru/mirrors/msys2/msys/x86_64/msys2-runtime-3.4.9-3-x86_64.pkg.tar.zst;https://mirrors.tuna.tsinghua.edu.cn/msys2/msys/x86_64/msys2-runtime-3.4.9-3-x86_64.pkg.tar.zst;https://mirrors.ustc.edu.cn/msys2/msys/x86_64/msys2-runtime-3.4.9-3-x86_64.pkg.tar.zst;https://mirror.selfnet.de/msys2/msys/x86_64/msys2-runtime-3.4.9-3-x86_64.pkg.tar.zst -> msys2-msys2-runtime-3.4.9-3-x86_64.pkg.tar.zst...
-- Using msys root at D:/a/duckdb_azure/duckdb_azure/vcpkg/downloads/tools/msys2/023cdb3ca06f77f2
-- Fixing pkgconfig file: D:/a/duckdb_azure/duckdb_azure/vcpkg/packages/zlib_x64-mingw-static/debug/lib/pkgconfig/zlib.pc
-- Installing: D:/a/duckdb_azure/duckdb_azure/vcpkg/packages/zlib_x64-mingw-static/share/zlib/copyright
-- Performing post-build validation
Stored binaries in 1 destinations in 291 ms.
Elapsed time to handle zlib:x64-mingw-static: 15 s
zlib:x64-mingw-static package ABI: b056b95a19ea9feb83d0070b702bd6f630096bddf7b557ebc27f89559b61e346
Installing 5/10 curl[core,non-http,schannel,ssl,sspi]:x64-mingw-static@8.5.0#1...
Building curl[core,non-http,schannel,ssl,sspi]:x64-mingw-static@8.5.0#1...
warning: -- Using community triplet x64-mingw-static. This triplet configuration is not guaranteed to succeed.
-- [COMMUNITY] Loading triplet configuration from: D:\a\duckdb_azure\duckdb_azure\vcpkg\triplets\community\x64-mingw-static.cmake
-- Downloading https://github.com/curl/curl/archive/curl-8_5_0.tar.gz -> curl-curl-curl-8_5_0.tar.gz...
-- Extracting source D:/a/duckdb_azure/duckdb_azure/vcpkg/downloads/curl-curl-curl-8_5_0.tar.gz
-- Applying patch 0002_fix_uwp.patch
-- Applying patch 0005_remove_imp_suffix.patch
-- Applying patch 0012-fix-dependency-idn2.patch
-- Applying patch 0020-fix-pc-file.patch
-- Applying patch 0022-deduplicate-libs.patch
-- Applying patch mbedtls-ws2_[32](https://github.com/natbprice/duckdb_azure/actions/runs/9177755461/job/25236890148#step:10:33).patch
-- Applying patch export-components.patch
-- Applying patch dependencies.patch
-- Applying patch cmake-config.patch
-- Applying patch cmake-config-ios.patch
-- Using source at D:/a/duckdb_azure/duckdb_azure/vcpkg/buildtrees/curl/src/curl-8_5_0-7491aadb0f.clean
-- Configuring x64-mingw-static
-- Building x64-mingw-static-dbg
-- Building x64-mingw-static-rel
-- Fixing pkgconfig file: D:/a/duckdb_azure/duckdb_azure/vcpkg/packages/curl_x64-mingw-static/lib/pkgconfig/libcurl.pc
-- Using cached mingw-w64-x86_64-pkgconf-1~2.1.0-1-any.pkg.tar.zst.
-- Using cached msys2-msys2-runtime-3.4.9-3-x86_64.pkg.tar.zst.
-- Using msys root at D:/a/duckdb_azure/duckdb_azure/vcpkg/downloads/tools/msys2/023cdb3ca06f77f2
-- Fixing pkgconfig file: D:/a/duckdb_azure/duckdb_azure/vcpkg/packages/curl_x64-mingw-static/debug/lib/pkgconfig/libcurl.pc
-- Installing: D:/a/duckdb_azure/duckdb_azure/vcpkg/packages/curl_x64-mingw-static/share/curl/vcpkg-cmake-wrapper.cmake
-- Installing: D:/a/duckdb_azure/duckdb_azure/vcpkg/packages/curl_x64-mingw-static/share/curl/usage
-- Performing post-build validation
Stored binaries in 1 destinations in 1.1 s.
Elapsed time to handle curl:x64-mingw-static: 1.9 min
curl:x64-mingw-static package ABI: fa82f0987e8da32724109f1802dc0864d289b58a52a02649fa1020a47be894e2
Installing 6/10 azure-core-cpp[core,curl,http,winhttp]:x64-mingw-static@1.11.0...
Building azure-core-cpp[core,curl,http,winhttp]:x64-mingw-static@1.11.0...
warning: -- Using community triplet x64-mingw-static. This triplet configuration is not guaranteed to succeed.
-- [COMMUNITY] Loading triplet configuration from: D:\a\duckdb_azure\duckdb_azure\vcpkg\triplets\community\x64-mingw-static.cmake
-- Downloading https://github.com/Azure/azure-sdk-for-cpp/archive/azure-core_1.11.0.tar.gz -> Azure-azure-sdk-for-cpp-azure-core_1.11.0.tar.gz...
-- Extracting source D:/a/duckdb_azure/duckdb_azure/vcpkg/downloads/Azure-azure-sdk-for-cpp-azure-core_1.11.0.tar.gz
-- Using source at D:/a/duckdb_azure/duckdb_azure/vcpkg/buildtrees/azure-core-cpp/src/ore_1.11.0-4959a5d799.clean
-- Configuring x64-mingw-static
-- Building x64-mingw-static-dbg
CMake Error at scripts/cmake/vcpkg_execute_build_process.cmake:1[34](https://github.com/natbprice/duckdb_azure/actions/runs/9177755461/job/25236890148#step:10:35) (message):
Error:     Command failed: "C:/Program Files/CMake/bin/cmake.exe" --build . --config Debug --target install -- -v -j5
    Working Directory: D:/a/duckdb_azure/duckdb_azure/vcpkg/buildtrees/azure-core-cpp/x64-mingw-static-dbg
    See logs for more information:
      D:\a\duckdb_azure\duckdb_azure\vcpkg\buildtrees\azure-core-cpp\install-x64-mingw-static-dbg-out.log

Call Stack (most recent call first):
  D:/a/duckdb_azure/duckdb_azure/build/release/vcpkg_installed/x64-windows/share/vcpkg-cmake/vcpkg_cmake_build.cmake:74 (vcpkg_execute_build_process)
  D:/a/duckdb_azure/duckdb_azure/build/release/vcpkg_installed/x64-windows/share/vcpkg-cmake/vcpkg_cmake_install.cmake:16 (vcpkg_cmake_build)
  ports/azure-core-cpp/portfile.cmake:[37](https://github.com/natbprice/duckdb_azure/actions/runs/9177755461/job/25236890148#step:10:38) (vcpkg_cmake_install)
  scripts/ports.cmake:170 (include)

error: building azure-core-cpp:x64-mingw-static failed with: BUILD_FAILED
Elapsed time to handle azure-core-cpp:x64-mingw-static: 21 s
Please ensure you're using the latest port files with `git pull` and `vcpkg update`.
Then check for known issues at:
  https://github.com/microsoft/vcpkg/issues?q=is%3Aissue+is%3Aopen+in%3Atitle+azure-core-cpp
You can submit a new issue at:
  https://github.com/microsoft/vcpkg/issues/new?title=[azure-core-cpp]+Build+error+on+x64-mingw-static&body=Copy+issue+body+from+D%3A%2Fa%2Fduckdb_azure%2Fduckdb_azure%2Fbuild%2Frelease%2Fvcpkg_installed%2Fvcpkg%2Fissue_body.md
You can also submit an issue by running (GitHub CLI must be installed):
  gh issue create -R microsoft/vcpkg --title "[azure-core-cpp] Build failure on x64-mingw-static" --body-file D:/a/duckdb_azure/duckdb_azure/build/release/vcpkg_installed/vcpkg/issue_body.md
-- Running vcpkg install - failed
CMake Error at D:/a/duckdb_azure/duckdb_azure/vcpkg/scripts/buildsystems/vcpkg.cmake:899 (message):
Error:   vcpkg install failed.  See logs for more information:
  D:/a/duckdb_azure/duckdb_azure/build/release/vcpkg-manifest-install.log
Call Stack (most recent call first):
  C:/rtools[42](https://github.com/natbprice/duckdb_azure/actions/runs/9177755461/job/25236890148#step:10:43)/x86_64-w64-mingw32.static.posix/share/cmake-3.24.1/Modules/CMakeDetermineSystem.cmake:124 (include)
  CMakeLists.txt:23 (project)

CMake Error: CMake was unable to find a build program corresponding to "Unix Makefiles".  CMAKE_MAKE_PROGRAM is not set.  You probably need to select a different build tool.
CMake Error: CMAKE_C_COMPILER not set, after EnableLanguage
CMake Error: CMAKE_CXX_COMPILER not set, after EnableLanguage
-- Configuring incomplete, errors occurred!
make: *** [extension-ci-tools/makefiles/duckdb_extension.Makefile:56: release] Error 1
Error: Process completed with exit code 1.
abfleishman commented 4 months ago

Any updates on this issue or next steps for addressing it?

eitsupi commented 1 month ago

This may be an upstream problem (azure cpp or mingw) that is unlikely to be resolved in the short term. microsoft/vcpkg#39554 Why not use WSL2?