Closed vikman90 closed 1 month ago
Started looking into this issue. Although in previous investigations we have concluded that Vcpkg is fit to handle precompiled binaries I decided to put it to test so we can compare it to Conan to the full extent of our requirements. Yet I've bumped into a steep learning curve to get specific versions of our dependencies. So far we have tried using dependencies without stating specific versions, in order to lock one for each of them, the process is proving to be somewhat cumbersome. It's necessary to define an additional configuration file.
For a vcpkg.json
file such as:
{
"name": "wazuh-agent-mvp",
"version": "5.0.0",
"dependencies": [
"boost-asio",
"boost-beast",
"bzip2",
"gtest",
"cjson"
],
"overrides": [
{
"name": "boost-asio",
"version": "1.84.0"
},
{
"name": "boost-beast",
"version": "1.84.0"
},
{
"name": "bzip2",
"version": "1.0.8"
},
{
"name": "cjson",
"version": "1.7.12"
},
{
"name": "gtest",
"version": "1.14.0"
}
]
}
...it's necessary to also define a vcpkg-configuration.json
with the following values:
{
"default-registry": {
"kind": "git",
"repository": "https://github.com/Microsoft/vcpkg",
"baseline": "f7423ee180c4b7f40d43402c2feb3859161ef625"
},
"registries": [
{
"kind": "git",
"repository": "https://github.com/Microsoft/vcpkg",
"baseline": "8ccb84df727bdf83045e53c319af05c554838b80",
"packages": [ "boost*", "boost-*"]
}
]
}
These baselines are commits in the vcpkg repository where the port is defined for the specific version needed. Finding out this can be done by running this command in the vcpkg repository: git log "--format=%H %cd %s" --date=short --left-only -- versions/b-/boost-asio.json
. Commits that update the port info will be listed.
Despite CPython not being necessary for building the old Agent, it still needs to be pointed out that the version required by the Wazuh Manager is not available. Moreover I struggled to build the following version and not yet succeeded.
It's also important to note that we don't just use dependencies as they come publicly packaged, we configure and build them using specific options. We need to also evaluate if this is possible using either Vcpkg or Conan. It looks like this can be achived with Vcpkg by creating our own ports. But that would mean that we need to create a fork of vcpkg, since apparently there is where they are expected to be found (at least, according to my current understanding).
These seem to be the available dependencies versions on Vcpkg:
Dependency | Required Version | Available in Vcpkg | Lowest Closest Available Version |
---|---|---|---|
bzip2 | 1.0.8 | Yes | 1.0.8#5 |
cJSON | 1.7.12 | Yes | 1.7.12 |
cPython* | 3.10.13 | No | 3.10.7#7 |
cURL | 8.5.0 | Yes | 8.5.0 |
Flatbuffers | 23.5.26 | Yes | 23.5.26 |
GoogleTest | 1.11.0 | Yes | 1.11.0 |
jemalloc | 5.2.1 | Yes | 5.3.0#0 |
Lua | 5.3.6 | No | 5.3.5#6 |
libarchive | 3.7.2 | Yes | 3.7.2#0 |
libdb | 18.1.40 | No | - |
libffi | 3.2.1 | Yes | 3.2.1#0 |
libpcre2 | 10.42.0 | Yes | 10.42.0#0 |
libplist | 2.2.0 | No | 2.0.1.197-2#0 |
libYAML | 0.1.7 | Yes | 0.2.1-1#0 |
liblzma | 5.4.2 | No | 5.4.1#1 |
Linux Audit userspace | 2.8.4 | No | 4.0.1#0 |
msgpack | 3.1.1 | Yes | 3.1.1#0 |
nlohmann | 3.7.3 | Yes | 3.7.3#0 |
OpenSSL | 3.0.12 | No | 3.0.8#2 |
pacman | 5.2.2 | No | - |
popt | 1.16 | Yes | 1.16#17 |
procps | 2.8.3 | No | - |
RocksDB | 8.3.2 | Yes | 8.3.2#0 |
rpm | 4.18.2 | No | - |
sqlite | 3.45.0 | Yes | 3.45.0#0 |
zlib | 1.3.1 | Yes | 1.3.1#0 |
There are a number of libraries or specific versions not available on Vcpkg, we need to look into a) how to create our own ports of the missing libraries, b) how to either create ports of the versions we need or how much work would it be to upgrade/downgrade the versions we are currently using, c) how to specify exactly how we want to build each dependency (in some cases we introduce options to configure them).
Sources:
Setting custom configuration
options via vcpkg_configure_make. This requires modifying or creating a new port file portfile.cmake
in the folder for the dependency (ie: vcpkg/ports/jemalloc). This block needs to be added or modified:
vcpkg_configure_make(
...
OPTIONS --host=${MINGW_HOST} --enable-static=yes
...
)
Provided we need to create a custom triplet, this is an example configuration (ie vcpkg/triplets/x64-mingw.cmake):
# x64-mingw.cmake
set(VCPKG_TARGET_ARCHITECTURE x64)
set(VCPKG_CRT_LINKAGE dynamic)
set(VCPKG_LIBRARY_LINKAGE static)
set(VCPKG_CMAKE_SYSTEM_NAME MinGW)
set(VCPKG_C_FLAGS "-fPIC")
set(VCPKG_CXX_FLAGS "-fPIC")
set(VCPKG_LINKER_FLAGS "")
set(VCPKG_C_COMPILER ${MING_BASE}${CC})
Then on the cmake the options needs to be set (see VCPKG_TARGET_TRIPLET)
set(VCPKG_TARGET_TRIPLET "x64-mingw" CACHE STRING "")
Binary caching with vcpkg
by setting:
set(ENV{VCPKG_BINARY_SOURCES} "clear;files,${PATH_TO_BINARIES},readwrite")
.
Provided we move forward with Vcpkg, once we generate all precompiled binaries we could upload them to our own storage and download them to the PATH_TO_BINARIES. There's also an option to use a http url and there's experimental support for AWS.
Dependency availability:
Software | Version | Available on Conan | Remarks |
---|---|---|---|
bzip2 | 1.0.8 | Yes | |
cJSON | 1.7.12 | Yes | |
cPython | 3.10.13 | No | 3.10.14 is available |
cURL | 8.5.0 | Yes | |
Flatbuffers | 23.5.26 | Yes | |
GoogleTest | 1.11.0 | Yes | |
jemalloc | 5.2.1 | Yes | |
Lua | 5.3.6 | Yes | |
libarchive | 3.7.2 | Yes | |
libdb | 18.1.40 | No | |
libffi | 3.2.1 | Yes | |
libpcre2 | 10.42.0 | Yes | |
libplist | 2.2.0 | No | |
libYAML | 0.1.7 | No | Version 0.2.2 is available |
liblzma | 5.4.2 | Yes | Listed as XZ |
Linux Audit userspace | 2.8.4 | No | |
msgpack | 3.1.1 | No | Version 3.2.1 available |
nlohmann | 3.7.3 | Yes | |
OpenSSL | 3.0.12 | Yes | |
pacman | 5.2.2 | No | |
popt | 1.16 | No | |
procps | 2.8.3 | No | |
RocksDB | 8.3.2 | No | 8.8.1 and higher available |
rpm | 4.18.2 | No | |
sqlite | 3.45.0 | Yes | |
zlib | 1.3.1 | Yes |
Conan needs to be installed as a Python package (this requires both python and pip). Once it is installed and a profile is set (conan profile detect
, sets one, which can be modified manually later on, to set different platforms/architectures), dependencies can be installed with a conanfile.txt
such as:
[requires]
boost/1.84.0
toml11/4.0.3
gtest/1.14.0
bzip2/1.0.8
cjson/1.7.12
libcurl/8.5.0
flatbuffers/23.5.26
lua/5.3.5
libarchive/3.7.2
[generators]
CMakeDeps
CMakeToolchain
[options]
boost/*:header_only=True
libcurl/*:with_openssl=True
libcurl/*:with_ssl=openssl
libcurl/*:with_zlib=True
And running the command:
conan install <path/to/conanfile.txt> --output-folder=<path/to/buildfolder> --build=missing
This will generate among other things a Conan tool chain cmake file that needs to be included in the CMakeLists files:
include(${CMAKE_BINARY_DIR}/conan_toolchain.cmake)
In Conan's documentation for each package, there's information about how to include and link against the dependencies' targets. See for example toml11
These are the main declared targets:
CMake package name(s): toml11
CMake target name(s): toml11::toml11
pkg-config file name(s): toml11.pc
A simple use case using the CMake file name and the global target:
# ...
find_package(toml11 REQUIRED)
# ...
target_link_libraries(YOUR_TARGET toml11::toml11)
This part seems to be more straightforward than using Vcpkg. Nonetheless I've been facing some problems making google test work in a small project. And I still have to test setting custom options (though they seem to be documented in each recipe's page (see for example the Packages tab, on the lib curl recipe) and binary caching (precompiled binaries).
There are some linking errors with GoogleTest using Conan, generating precompiled binaries doesn't seem to be possible locally, or at least is not as straight forward as it was with Vcpkg. I'm investigating these two issues.
Linking errors against GoogleTest were solved by correctly including the Conan cmake toolchain at the very top of the CMakeLists file:
include(${CMAKE_BINARY_DIR}/conan_toolchain.cmake)
Note that Conan needs a profile configuration, this will set a build type (ie: Debug, Release). If this doesn't match the build type of the CMake project, some linking problems will arise.
Also it was found that some Boost libraries are header only and, in contrast with Vcpkg, a different approach is necessary for them to be added, using Boost::headers
target to link and include directories. This also needs to be linked and included publicly (using PUBLIC
keyword).
Regarding the precompiled binaries see the documentation. It is possible to achieve the same as with Vcpkg, to speed up building time for both development and CI. Creating a Conan repository is necessary, but locally these can be achieved with
(pyenv) src/build$ conan cache save "boost/1.84.0"
Saving boost/1.84.0: boost8a8cc0ecec7a1
Local Cache
boost
boost/1.84.0
revisions
85fab16ba3b744074ca9733d2ed5fe5a (2024-07-04 08:10:19 UTC)
recipe_folder: boost8a8cc0ecec7a1
This creates a tar.gz
file that can be restored with:
(pyenv) src/build$ conan cache restore conan_cache_save.tgz
Restore: boost/1.84.0 in boost8a8cc0ecec7a1
Local Cache
boost
boost/1.84.0
revisions
85fab16ba3b744074ca9733d2ed5fe5a (2024-07-26 21:45:32 UTC)
recipe_folder: boost8a8cc0ecec7a1
Both Vcpkg and Conan2 have meet all the requirements. The availability of certain packages can be overcome be creating our own ports/recipes.
Since both vcpkg
and conan
are suitable options, we will continue to use vcpkg
, as we have already started using it and it covers everything we need. If any issues arise in the future, we will re-evaluate this decision taking into account that conan
also covers all our needs.
Description
We need to research and identify suitable tools and systems for managing dependencies in Wazuh. The goal is to find a solution that allows us to define, modify, and build dependencies individually, supports multiple platforms, and handles transitive dependencies effectively.
Tasks
Investigate Dependency Management Tools:
List Available Dependencies:
Tool Evaluation:
Documentation:
Acceptance Criteria
Related Issues