microsoft / vcpkg

C++ Library Manager for Windows, Linux, and MacOS
MIT License
23.37k stars 6.46k forks source link

Binary caching cannot cross host #30716

Open huangqinjin opened 1 year ago

huangqinjin commented 1 year ago

The triplet arm64-android-dynamic for testing.

set(VCPKG_CMAKE_SYSTEM_NAME Android)
set(VCPKG_CMAKE_SYSTEM_VERSION 24)
set(VCPKG_TARGET_ARCHITECTURE arm64)
set(VCPKG_CRT_LINKAGE dynamic)
set(VCPKG_LIBRARY_LINKAGE dynamic)
set(VCPKG_BUILD_TYPE release)
set(VCPKG_DISABLE_COMPILER_TRACKING ON)
set(VCPKG_C_FLAGS -s)
set(VCPKG_CXX_FLAGS -s)

It produces exactly same binary libsqlite3.so (same sha1) on Windows and Linux. But the two vcpkg_abi_info.txt are not same.

sqlite3 vcpkg_abi_info.txt on Linux ``` CMakeLists.txt c04c434a9550cc4f172e7a27500006cdef77b3d785b6c2a6e9b4d534293dd5ee add-config-include.patch 11bd0e576cd75a7b87051a6af453f4fb32f1137d69b0710b9023158d366087b4 cmake 3.26.2 features core fix-arm-uwp.patch f4397f84d931cccb44a038194c895e28c749f1825ab45db4b5dccd6e3b6b5967 portfile.cmake 898df7e2dee3446b3369c098e91d01a672a6e410fd731fd5d408cfc3a2d7d3ce ports.cmake 47a3510fcec56fae26f4fb082afd972a70a04e26efa73e2de69123139500f02d post_build_checks 2 sqlite3-config.in.cmake 7a44f5805ced746c4dcb497055bce06e45eb3ecb3506465fcde4d442a2179d96 sqlite3-vcpkg-config.h.in 2e5fd3bd5cf891e948516f911cc20412c414839598c9c19fbf9ffb65fd2a1643 sqlite3.pc.in 7d0eef3b49bc8c7fbb6a06b449855060ab5d9b4ab37d1781a0269dda8421a391 triplet arm64-android-dynamic triplet_abi 54c15dc848f2118cb2e6ae9a5de541ab89f1bc26089412c06c410e243edb342e-4e1baa1a21b183d2ac6a852af84a3b06964cba5ea439fc3efdf9c47faf774189 usage 1cd1a4fe57d9caf51321f363f1d395074ac7d0c14b7035a5d2c218670be4b5b3 vcpkg-cmake d665dc8bb5dada261b51133b5f577f64ecedbd6dda5ab6881c00111e2ef0072f vcpkg-cmake-config 051b0c262bcca4636d54b92bbeb63ec9d03a41295c3909cd6241425e052ede6a vcpkg.json 6396727956f98b0a1c0dae40c6b7df9def7858e21975656cd4e0e2c7ed3fa371 vcpkg_check_features 3cdc889b7e6965ad5ddab803060aed2019c0177b6f314c7a5a947c01efa4db60 vcpkg_copy_pdbs d57e4f196c82dc562a9968c6155073094513c31e2de475694143d3aa47954b1c vcpkg_download_distfile 2d57e1336ae9daab2f04c0623217094b026080d19bdca3a92396d68ea3e729c7 vcpkg_extract_source_archive a25e0132b36c30e9675f934042fb1164128864eeefa064ab8eabb1eb347aa381 vcpkg_fixup_pkgconfig 588d833ff057d3ca99c14616c7ecfb5948b5e2a9e4fc02517dceb8b803473457 ```
sqlite3 vcpkg_abi_info.txt on Windows ``` CMakeLists.txt c04c434a9550cc4f172e7a27500006cdef77b3d785b6c2a6e9b4d534293dd5ee add-config-include.patch 11bd0e576cd75a7b87051a6af453f4fb32f1137d69b0710b9023158d366087b4 cmake 3.26.2 features core fix-arm-uwp.patch f4397f84d931cccb44a038194c895e28c749f1825ab45db4b5dccd6e3b6b5967 portfile.cmake 898df7e2dee3446b3369c098e91d01a672a6e410fd731fd5d408cfc3a2d7d3ce ports.cmake 47a3510fcec56fae26f4fb082afd972a70a04e26efa73e2de69123139500f02d post_build_checks 2 powershell 7.3.3 sqlite3-config.in.cmake 7a44f5805ced746c4dcb497055bce06e45eb3ecb3506465fcde4d442a2179d96 sqlite3-vcpkg-config.h.in 2e5fd3bd5cf891e948516f911cc20412c414839598c9c19fbf9ffb65fd2a1643 sqlite3.pc.in 7d0eef3b49bc8c7fbb6a06b449855060ab5d9b4ab37d1781a0269dda8421a391 triplet arm64-android-dynamic triplet_abi 54c15dc848f2118cb2e6ae9a5de541ab89f1bc26089412c06c410e243edb342e-4e1baa1a21b183d2ac6a852af84a3b06964cba5ea439fc3efdf9c47faf774189 usage 1cd1a4fe57d9caf51321f363f1d395074ac7d0c14b7035a5d2c218670be4b5b3 vcpkg-cmake 458529b8fa343f7cc05966d710a660abe75f6bb5a8af0b8f017ccb8caac33339 vcpkg-cmake-config 3c2c9d3921efa04d262efa1ea137c4de1d404d994f1ad70e00d910a93ecf37e6 vcpkg.json 6396727956f98b0a1c0dae40c6b7df9def7858e21975656cd4e0e2c7ed3fa371 vcpkg_check_features 3cdc889b7e6965ad5ddab803060aed2019c0177b6f314c7a5a947c01efa4db60 vcpkg_copy_pdbs d57e4f196c82dc562a9968c6155073094513c31e2de475694143d3aa47954b1c vcpkg_download_distfile 2d57e1336ae9daab2f04c0623217094b026080d19bdca3a92396d68ea3e729c7 vcpkg_extract_source_archive a25e0132b36c30e9675f934042fb1164128864eeefa064ab8eabb1eb347aa381 vcpkg_fixup_pkgconfig 588d833ff057d3ca99c14616c7ecfb5948b5e2a9e4fc02517dceb8b803473457 ```

The first issue is that vcpkg_abi_info.txt on Windows contains powershell version.

The second issue is that vcpkg-cmake is a host dependency and it's vcpkg_abi_info.txt contains the host triplet information. This means that all package ABIs indirectly contain host infomation even we are cross-compiling.

vcpkg-cmake vcpkg_abi_info.txt on Linux ``` cmake 3.26.2 features core portfile.cmake a711531b7f13b7da16fa1f25d7c5737a423d4a126465dc9e6689a0f043fcc1aa ports.cmake 47a3510fcec56fae26f4fb082afd972a70a04e26efa73e2de69123139500f02d post_build_checks 2 triplet x64-linux-dynamic triplet_abi c94625e2b7091a6dfd70d5f606db688b255b19a34b70a4bd7a12bb5d230af8eb-8483608216601f5eb6d401f81f364008853b8e98a0ea3bebd6dc1e55e1e3edbb vcpkg-port-config.cmake f0a30f77c8f5e3ac40436fe2518a61ad067f2955c7ef3be6d6a0ca4b81cd2a45 vcpkg.json df51210b477280bd0d98d51977bf94e3a2d539a9d7153cbab797b5dc03af48e3 vcpkg_add_to_path 5f5ae75cf37b2a58d1a8561ca96496b64cd91ec9a0afab0b976c3e5d59030bfe vcpkg_cmake_build.cmake 6d1c27080fe3e768b5e7b968d6a28a37db154ebcb214297de25f10b6713511e1 vcpkg_cmake_configure.cmake 7175c1125b9ab851623861f1867d5c266a242fa3dd735f88dea76ecfa50ac0f2 vcpkg_cmake_install.cmake 3ae7886dc8434fac6f1e61190cc355fdec5fbd4f60758e2de20423cf49c91369 vcpkg_configure_cmake ac8c9920c3ea9613615144c771fe797013d5dc8cc03a25d97c92e167a692af24 vcpkg_execute_build_process 4145b3b23fcc2dd94a0fe3c818b63fcea40854e5aa94d08dc83ddcc09727cedf vcpkg_execute_required_process 0c6f52ca848715efd88acfec9bdb2ed3b5f38a650baa6a614bf23ccc3eec150a vcpkg_find_acquire_program fdd8d8e0bf5217df8f40239e826c6f37c0c052ee2024c265a5916e8f7eef4107 vcpkg_list f5de3ebcbc40a4db90622ade9aca918e2cf404dc0d91342fcde457d730e6fa29 ```
vcpkg-cmake vcpkg_abi_info.txt on Windows ``` cmake 3.26.2 features core portfile.cmake a711531b7f13b7da16fa1f25d7c5737a423d4a126465dc9e6689a0f043fcc1aa ports.cmake 47a3510fcec56fae26f4fb082afd972a70a04e26efa73e2de69123139500f02d post_build_checks 2 powershell 7.3.3 triplet x64-windows triplet_abi 4556164a2cd3dd6f4742101eabb46def7e71b6e5856faa88e5d005aac12a803c-2b9f5a18de013332e7068fafe850651fd91434ba2a08aae552b8565a2236f81b vcpkg-port-config.cmake f0a30f77c8f5e3ac40436fe2518a61ad067f2955c7ef3be6d6a0ca4b81cd2a45 vcpkg.json df51210b477280bd0d98d51977bf94e3a2d539a9d7153cbab797b5dc03af48e3 vcpkg_add_to_path 5f5ae75cf37b2a58d1a8561ca96496b64cd91ec9a0afab0b976c3e5d59030bfe vcpkg_cmake_build.cmake 6d1c27080fe3e768b5e7b968d6a28a37db154ebcb214297de25f10b6713511e1 vcpkg_cmake_configure.cmake 7175c1125b9ab851623861f1867d5c266a242fa3dd735f88dea76ecfa50ac0f2 vcpkg_cmake_install.cmake 3ae7886dc8434fac6f1e61190cc355fdec5fbd4f60758e2de20423cf49c91369 vcpkg_configure_cmake ac8c9920c3ea9613615144c771fe797013d5dc8cc03a25d97c92e167a692af24 vcpkg_execute_build_process 4145b3b23fcc2dd94a0fe3c818b63fcea40854e5aa94d08dc83ddcc09727cedf vcpkg_execute_required_process 0c6f52ca848715efd88acfec9bdb2ed3b5f38a650baa6a614bf23ccc3eec150a vcpkg_find_acquire_program fdd8d8e0bf5217df8f40239e826c6f37c0c052ee2024c265a5916e8f7eef4107 vcpkg_list f5de3ebcbc40a4db90622ade9aca918e2cf404dc0d91342fcde457d730e6fa29 ```
russelltg commented 1 year ago

I've employed a slightly hacky way to get around this: pass --only-binarycaching --host-triplet <triplet of original build that you want to re-use>

Of course, this only works if you know for sure all your packages are built and cached

Probably only works if you disable compiler tracking as well.

huangqinjin commented 1 year ago

@russelltg It does not work if the <triplet of original build that you want to re-use> is x64-windows.

On Linux, --host-triplet x64-windows outputs

error: in triplet x64-windows: Use of Visual Studio's Developer Prompt is unsupported on non-Windows hosts.
Define 'VCPKG_CMAKE_SYSTEM_NAME' or 'VCPKG_CHAINLOAD_TOOLCHAIN_FILE' in the triplet file.

And no way to bypass the different hash calculation between Linux and Windows.

The first issue is that vcpkg_abi_info.txt on Windows contains powershell version.

JonLiu1993 commented 1 year ago

This issue hasn’t been updated in 3 month, if it is still an issue, please reopen this issue.

huangqinjin commented 11 months ago

@JonLiu1993 it is still an issue, kindly reopen.

github-actions[bot] commented 5 months ago

This is an automated message. Per our repo policy, stale issues get closed if there has been no activity in the past 180 days. The issue will be automatically closed in 14 days. If you wish to keep this issue open, please add a new comment.

russelltg commented 5 months ago

This is still an issue