search

microsoft / vcpkg

C++ Library Manager for Windows, Linux, and MacOS
MIT License
22.81k stars 6.3k forks source link

[curl] x64-windows build failure with features brotli,c-ares,core,http2,mbedtls,non-http,openssl,schannel,ssh,ssl,sspi,tool,winssl after curl Fixes pkgconfig configuration file (#14350) #14681

Closed StarGate-One closed 3 years ago

StarGate-One commented 3 years ago

Issue curl[brotli,c-ares,core,http2,mbedtls,non-http,openssl,schannel,ssh,ssl,sspi,tool,winssl] build failure after commit https://github.com/microsoft/vcpkg/commit/878d7ea830412dcda31a4d4c6589fe9e0015fe5a [curl] Fixes pkgconfig configuration file (#14350)

Host Environment

To Reproduce Steps to reproduce the behavior:

  1. git pull
  2. .\vcpkg.exe install --x-use-aria2 --recurse --keep-going boost[core,mpi]:x64-windows boost-iostreams[core]:x64-windows boost-mpi[core]:x64-windows boost-parameter-python[core]:x64-windows boost-python[core]:x64-windows ctbignum[core]:x64-windows curl[brotli,c-ares,core,http2,mbedtls,non-http,openssl,schannel,ssh,ssl,sspi,tool,winssl]:x64-windows curlpp[core]:x64-windows date[core,remote-api]:x64-windows devil[core,jasper,lcms,libjpeg,libpng,openexr,tiff]:x64-windows directxtex[core,dx12,openexr]:x64-windows forge[core]:x64-windows freeimage[core]:x64-windows libarchive[bzip2,core,libxml2,lz4,lzma,lzo,openssl,zstd]:x64-windows libharu[core,notiffsymbols]:x64-windows liblzma[core]:x64-windows libpq[core,icu,nls,openssl,xml,zlib]:x64-windows libpqxx[core]:x64-windows libwebp[all,anim,core,cwebp,dwebp,extras,gif2webp,img2webp,info,mux,nearlossless,simd,swap16bitcsp,unicode,vwebp,vwebp-sdl]:x64-windows libxml2[core]:x64-windows libzip[bzip2,commoncrypto,core,default-aes,liblzma,mbedtls,openssl,wincrypto]:x64-windows libzippp[core]:x64-windows magnum-plugins[core,ddsimporter,devilimageimporter,drflacaudioimporter,freetypefont,harfbuzzfont,icoimporter,jpegimageconverter,jpegimporter,miniexrimageconverter,openddl,opengeximporter,pngimageconverter,pngimporter,stanfordimporter,stanfordsceneconverter,stbimageconverter,stbimageimporter,stlimporter]:x64-windows ogre[core]:x64-windows openexr[core]:x64-windows poco[core,pdf,postgresql]:x64-windows pybind11[core]:x64-windows python3[core]:x64-windows sdl2-image[core,libjpeg-turbo,libwebp,tiff]:x64-windows sdl2pp[core]:x64-windows shiva[core]:x64-windows tiff[core,tool]:x64-windows tmx[core]:x64-windows wxwidgets[core]:x64-windows

Failure logs

Error: Building package curl:x64-windows failed with: BUILD_FAILED Elapsed time for package curl:x64-windows: 1.169 min

Additional context Add any other context about the problem here, such as what you have already tried to resolve the issue. @PhoebeHui @Matioupi cc: @strega-nil

strega-nil commented 3 years ago

Looking into this, thanks for reporting @StarGate-One

strega-nil commented 3 years ago

@StarGate-One this doesn't seem to be a real issue; I think it is caused by not updating the libraries after a git pull.

I recommend ./vcpkg x-set-installed to remove all installed libraries, and then re-running the install.

However, when looking at this issue, I did find an issue with libpq: #14685

StarGate-One commented 3 years ago

@StarGate-One this doesn't seem to be a real issue; I think it is caused by not updating the libraries after a git pull.

I recommend ./vcpkg x-set-installed to remove all installed libraries, and then re-running the install.

However, when looking at this issue, I did find an issue with libpq: #14685

I am trying on a fresh clone and see what happens. Normally I rename the buildtrees[port-name] directory(ies) when a port install fails (the port and features ports) and I try again. I failed do that this time as I was researching the libzippp issue at the same time. Oh no, not a libpq also 😒 Never a dull moment and job security 👍😊

StarGate-One commented 3 years ago

@strega-nil on this one, it still fails with the same issue, as fresh vanilla clone as of a few minutes ago, no previous ports installed. The only ports installed before port curl failed was: SYSTEM@ANDROMEDA 11/20/2020 19:16:55 D:\Git\vcpkg\vs2017-x64-v141 ->vcpkg list

7zip:x64-windows 19.00#2 Library for archiving file with a high compressi... brotli:x64-windows 1.0.9 a generic-purpose lossless compression algorithm... bzip2:x64-windows 1.0.8#1 bzip2 is a freely available, patent free, high-q... c-ares:x64-windows 2019-5-2-1 A C library for asynchronous DNS requests dbg-macro:x64-windows 2019-07-11 A dbg(...) macro for C++ gzip-hpp:x64-windows 0.1.0 Gzip header-only C++ library liblzma:x64-windows 5.2.5#2 Compression library with an API similar to that ... libssh2:x64-windows 1.9.0 The SSH library mbedtls:x64-windows 2.24.0#1 An open source, portable, easy to use, readable ... mbedtls[pthreads]:x64-windows Multi-threading support nghttp2:x64-windows 1.39.2-2 Implementation of the Hypertext Transfer Protoco... openssl:x64-windows 1.1.1h#2 OpenSSL is an open source project that provides ... pthreads:x64-windows 3.0.0-6 pthreads for windows yasm-tool:x86-windows 2020-10-12 A modular assembler. This port is intended to bu... zlib:x64-windows 1.2.11#9 A compression library

Console log about the same: Starting package 11/35: curl:x64-windows Building package curl[brotli,c-ares,core,http2,mbedtls,non-http,openssl,schannel,ssh,ssl,sspi,tool,winssl]:x64-windows... -- Downloading curl-curl-315ee3fe75dade912b48a21ceec9ccda0230d937.tar.gz... -- Extracting source D:/Git/vcpkg/vs2017-x64-v141/downloads/curl-curl-315ee3fe75dade912b48a21ceec9ccda0230d937.tar.gz -- Applying patch 0002_fix_uwp.patch -- Applying patch 0004_nghttp2_staticlib.patch -- Applying patch 0005_remove_imp_suffix.patch -- Applying patch 0006_fix_tool_depends.patch -- Applying patch 0007_disable_tool_export_curl_target.patch -- Applying patch 0009_fix_openssl_config.patch -- Applying patch 0010_fix_othertests_cmake.patch -- Applying patch 0011_fix_static_build.patch -- Using source at D:/Git/vcpkg/vs2017-x64-v141/buildtrees/curl/src/da0230d937-b280319101.clean -- Configuring x64-windows -- Building x64-windows-dbg -- Building x64-windows-rel -- Installing: D:/Git/vcpkg/vs2017-x64-v141/packages/curl_x64-windows/share/curl/curl-config -- Using msys root at D:/Git/vcpkg/vs2017-x64-v141/downloads/tools/msys2/c809757c94447846 -- Fixing pkgconfig file: D:/Git/vcpkg/vs2017-x64-v141/packages/curl_x64-windows/lib/pkgconfig/libcurl.pc -- CHECK_LIB_optimized D:/Git/vcpkg/vs2017-x64-v141/packages/curl_x64-windows/lib/pthreadVC3.lib_RELEASE:CHECK_LIB_optimized D:/Git/vcpkg/vs2017-x64-v141/packages/curl_x64-windows/lib/pthreadVC3.lib_RELEASE-NOTFOUND CMake Error at scripts/cmake/vcpkg_fixup_pkgconfig.cmake:233 (message): Library "optimized D:/Git/vcpkg/vs2017-x64-v141/packages/curl_x64-windows/lib/pthreadVC3.lib" was not found! If it is a system library use the SYSTEM_LIBRARIES parameter for the vcpkg_fixup_pkgconfig call! Otherwise, correct the *.pc file Call Stack (most recent call first): scripts/cmake/vcpkg_fixup_pkgconfig.cmake:299 (vcpkg_fixup_pkgconfig_check_files) ports/curl/portfile.cmake:170 (vcpkg_fixup_pkgconfig) scripts/ports.cmake:136 (include)

Error: Building package curl:x64-windows failed with: BUILD_FAILED Elapsed time for package curl:x64-windows: 1.199 min Starting package 12/35: dbg-macro:x64-windows Building package dbg-macro[core]:x64-windows...

strega-nil commented 3 years ago

@StarGate-One that's really odd... it works completely fine on my machine. Can you try the clean install anyways?

Also, don't worry about renaming buildtrees, that gets cleaned, the concern is installed.

StarGate-One commented 3 years ago
StarGate-One commented 3 years ago

@strega-nil FYI

StarGate-One commented 3 years ago

@PhoebeHui @Matioupi cc: @strega-nil

  1. I do not know how to fix, but I figured out the problem is related to mbedtls with feature pthreads
  2. If you install port mbedtls
    • with only feature core curl builds fine with all features
    • with both features core, pthreads curl fails as described above
  3. Either method of install fails with same issue:
    • .\vcpkg.exe x-set-installed c-ares curl[brotli,c-ares,http2,mbedtls,non-http,openssl,ssh,ssl,sspi,tool,winssl] libssh2 mbedtls[core,pthreads] nghttp2 openssl pthreads zlib
    • .\vcpkg.exe install --x-use-aria2 --recurse --keep-going -- c-ares curl[brotli,c-ares,http2,mbedtls,non-http,openssl,ssh,ssl,sspi,tool,winssl] libssh2 mbedtls[core,pthreads] nghttp2 openssl pthreads zlib
  4. Error for both as follows:

    -- Fixing pkgconfig file: D:/Git/vcpkg/vs2019-x64-v142/packages/curl_x64-windows/lib/pkgconfig/libcurl.pc -- CHECK_LIB_optimized D:/Git/vcpkg/vs2019-x64-v142/packages/curl_x64-windows/lib/pthreadVC3.lib_RELEASE:CHECK_LIB_optimized D:/Git/vcpkg/vs2019-x64-v142/packages/curl_x64-windows/lib/pthreadVC3.lib_RELEASE-NOTFOUND CMake Error at scripts/cmake/vcpkg_fixup_pkgconfig.cmake:233 (message): Library "optimized D:/Git/vcpkg/vs2019-x64-v142/packages/curl_x64-windows/lib/pthreadVC3.lib" was not found! If it is a system library use the SYSTEM_LIBRARIES parameter for the vcpkg_fixup_pkgconfig call! Otherwise, correct the *.pc file Call Stack (most recent call first): scripts/cmake/vcpkg_fixup_pkgconfig.cmake:299 (vcpkg_fixup_pkgconfig_check_files) ports/curl/portfile.cmake:170 (vcpkg_fixup_pkgconfig) scripts/ports.cmake:136 (include)

    Error: Building package curl:x64-windows failed with: BUILD_FAILED Please ensure you're using the latest portfiles with .\vcpkg update, then submit an issue at https://github.com/Microsoft/vcpkg/issues including: Package: curl:x64-windows Vcpkg version: 2020.11.12-nohash

Matioupi commented 3 years ago

Hello, this looks like the same kind or error message I had in adding "fix" cases for the libcurl.pc file. Still I missed testinf those features and adding the same kind of search & replace fix. Will take care of it today. (and maybe refactor a little bit to make it more generic)

Matioupi commented 3 years ago

Hello,

This commit : https://github.com/microsoft/vcpkg/issues/14681

Fixes the issue (only handle and tested for windows case at time, more work needed for other platforms).

I decided to remove the -loptimized / -ldebug that were in the package stage libcurl.pc. Maybe a better fix would be to add those in the vcpkg_fixup_pkgconfig calls at the end of the portfile.cmake

It also seems that the release/debug libcurl.pc at the package stage contains both the debug/release versions of the pthread libraries which is weird.

StarGate-One commented 3 years ago

OK - I got the fix from commit https://github.com/Matioupi/vcpkg/commit/236dae55a2736be5fc1e7db7cba7c61ce4a9d77c PR First fix attempts for microsoft#14681 and got a new error😢 with zlib with brotlidec.lib and brotlicommon lib. Unfortunately I would guess we end up fixing all the features.

vcpkg install curl[brotli,c-ares,core,http2,mbedtls,non-http,openssl,schannel,ssh,ssl,sspi,tool,winssl]

-- Building x64-windows-dbg -- Building x64-windows-rel -- Installing: D:/vcpkg_x64/vs2019/prod/packages/curl_x64-windows/share/curl/curl-config -- Using msys root at D:/vcpkg_x64/vs2019/prod/downloads/tools/msys2/17e91aa90d284835 -- Fixing pkgconfig file: D:/vcpkg_x64/vs2019/prod/packages/curl_x64-windows/lib/pkgconfig/libcurl.pc >-- CHECK_LIB_zlib D:/vcpkg_x64/vs2019/prod/packages/curl_x64-windows/lib/brotlicommon.lib D:/vcpkg_x64/vs2019/prod/packages/curl_x64-windows/lib/brotlidec.lib_RELEASE:CHECK_LIB_zlib D:/vcpkg_x64/vs2019/prod/packages/curl_x64-windows/lib/brotlicommon.lib D:/vcpkg_x64/vs2019/prod/packages/curl_x64-windows/lib/brotlidec.lib_RELEASE-NOTFOUND CMake Error at scripts/cmake/vcpkg_fixup_pkgconfig.cmake:233 (message): Library "zlib D:/vcpkg_x64/vs2019/prod/packages/curl_x64-windows/lib/brotlicommon.lib D:/vcpkg_x64/vs2019/prod/packages/curl_x64-windows/lib/brotlidec.lib" was not found! If it is a system library use the SYSTEM_LIBRARIES parameter for the vcpkg_fixup_pkgconfig call! Otherwise, correct the *.pc file Call Stack (most recent call first): scripts/cmake/vcpkg_fixup_pkgconfig.cmake:299 (vcpkg_fixup_pkgconfig_check_files) ports/curl/portfile.cmake:178 (vcpkg_fixup_pkgconfig) scripts/ports.cmake:136 (include)

Error: Building package curl:x64-windows failed with: BUILD_FAILED Please ensure you're using the latest portfiles with .\vcpkg update, then submit an issue at https://github.com/Microsoft/vcpkg/issues including: Package: curl:x64-windows

Matioupi commented 3 years ago

More fix here : https://github.com/Matioupi/vcpkg/commit/fe90a39bf93404084a02a2888ed812eb9f183c78

Matioupi commented 3 years ago

I just tested a linux build and get no errors with the specific features and build order that were used. Still the result libcurl.pc do not contain references to brotli libs even when they are present and the feature was activated. Not sure if this requires a specific fix or not.

Matioupi commented 3 years ago

Also, shouldn't the CONTROL section about brotli feature be changed from

Feature: brotli
Description: brotli support (brotli)

to

Feature: brotli
Build-Depends: brotli
Description: brotli support (brotli)

in order to force brotli build when the feature is activated ? is there a rationale for not having it ?

StarGate-One commented 3 years ago

@Matioupi

Thank you for the hard work👍👍👍😊😊😊

PhoebeHui commented 3 years ago

@StarGate-One, thanks for reporting this issue and figure out the repro steps!

@Matioupi, could you help summit the fix?

Matioupi commented 3 years ago

@PhoebeHui : I opened a new PR with the proposed fix : Phoe https://github.com/microsoft/vcpkg/pull/14715