search

microsoft / vcpkg

C++ Library Manager for Windows, Linux, and MacOS
MIT License
23.18k stars 6.39k forks source link

"default-features":false seems to just be ignored. Feature selection always adds default-features #35694

Open Arech opened 10 months ago

Arech commented 10 months ago

Describe the bug I might misunderstand something in vcpkg, but I though that according to the docs, "default-features": false should totally disable default feature set defined for a port and install only explicitly selected features. But it's not the case. Default-features are always selected and installed.

I have a private repository containing two ports:

I was able to replicate that with publicly available ports: opencv4 port contains a spectra of default-enabled features.

vcpkg depend-info opencv4
...
opencv4[default-features, webp, jpeg, dnn, png, quirc, tiff]: libjpeg-turbo, libpng, libwebp, ....

opencv4 has ffmpeg feature defined in a similar manner with restricted feature set:

 "ffmpeg": {
      "description": "ffmpeg support for opencv",
      "dependencies": [
        {
          "name": "ffmpeg",
          "default-features": false,
          "features": [
            "avcodec",
            "avformat",
            "swresample",
            "swscale"
          ]
        }
      ]
    },

By default ffmpeg port has these default features:

 "default-features": [
    "avcodec",
    "avdevice",
    "avfilter",
    "avformat",
    "swresample",
    "swscale"
  ],

So I assume that when I install opencv4[ffmpeg] only 4 specifically selected by opencv4 manifest's features must be present. However, it's not true:

vcpkg depend-info opencv4[ffmpeg]
...
ffmpeg[avfilter, avdevice, avcodec, avformat, swresample, swscale]: ......
...
opencv4[default-features, jpeg, dnn, ffmpeg, png, quirc, tiff, webp]: ffmpeg,  libjpeg-turbo, libpng, libwebp, ......

i.e. vcpkg fetches all default features for ffmpeg plus those additionally specified, and for opencv4 it also selects all default features plus ffmpeg feature as an addition, instead of using it exclusively. (depend-info command is used to demonstrate the issue, the same is happening when vcpkg really install ports)

This seems like a bug to me. Documentation clearly states:

If the user wants to explicitly disable the default features, they can do so by adding "default-features": false to the dependency:

(see the bottom of the page)

Do I misunderstand something?

Environment The same results in Windows 10:

>vcpkg --version
vcpkg package management program version 2023-11-16-4c1df40a3c5c5e18de299a99e9accb03c2a82e1e

>git log -1
commit b051c80d93fb023ce959622271e1183bff291fd5 (HEAD -> master, origin/master, origin/HEAD)
Author: ...
Date:   Wed Dec 6 03:41:09 2023 +0800

as well as on Ubuntu20.04 (in WSL with own vcpkg checkout, which hardly matters..)

$ ./vcpkg --version
vcpkg package management program version 2023-11-16-4c1df40a3c5c5e18de299a99e9accb03c2a82e1e

$ git log -1
commit e85cafa30def190eca0dc1413bf8c04e434af567 (HEAD -> master, origin/master, origin/HEAD)
Author: ...
Date:   Thu Nov 30 13:58:50 2023 -0800
Neumann-A commented 10 months ago

"default-features": false just means the port does not depend on the default features it does not say that you do not want default features. If you do not want default features you need to explicitly say so on the command line or within or projects manifest. If a port is transitively pulled in you also need to mention that port and say you don't want the default features of that.

Arech commented 10 months ago

Thanks for the reply, Alexander

"default-features": false just means the port does not depend on the default features it does not say that you do not want default features.

I'm not an English language expert and not a native speaker, but to me if that's correct, then the docs with "If the user wants to explicitly disable the default features, they can do so by adding "default-features": false" is misleading. Or at least semantic of "default-feautures" is different for app manifest file (where it disables them indeed), and for ports (where it means something else). Can that be documented?

If a port is transitively pulled in you also need to mention that port and say you don't want the default features of that.

Do I get you right, that if I want opencv4[ffmpeg] to install only 4 features it needs (that are explicitly listed in opencv4 port manifest) instead of 6 default, I should manually specify dependency on ffmpeg in my app's manifest with these 4 features? So I basically should depend on opencv4 port internal implementation and must also freeze it's version?

Neumann-A commented 10 months ago

, I should manually specify dependency on ffmpeg in my app's manifest with these 4 features?

It is enough to say you want ffmpeg with "default-features": false . The other features will be pulled in by opencv4 since they are required by it.

Also there is nothing more to document. The docs already say:

Default features are a set of features to be automatically activated if the top-level project does not explicitly request a build without them

FrankXie05 commented 10 months ago

@Arech In actual testing, this function is normal and it does disable the build of the default features. But there is currently a problem: the installation log incorrectly displays the default features. 290780636-3344a5a0-aef1-4f65-bd5d-bf1661d4e9e3

image

FrankXie05 commented 10 months ago

https://github.com/microsoft/vcpkg/blob/50bffcc62d7f6571eb32bc1a0b1807e77af1166c/ports/tinyorm/vcpkg.json#L31

{ORM} was not passed to install the feature orm.

dg0yt commented 10 months ago

at least semantic of "default-features" is different for app manifest file (where it disables them indeed), and for ports (where it means something else).

This is right. In port manifests, it declares a possibility and enables user choice. The actual control is only in user project manifests (or at the command line using pseudo feature core).

Arech commented 10 months ago

That's a bit confusing... I think I haven't seen that difference mentioned in ports creation documentation, but now I understand it better. Probably should be said somewhere more explicitly...

Anyway, thanks a lot everyone involved and have a great weekend!

Arech commented 10 months ago

(and a personal thanks to @FrankXie05 for spotting the issue)

FrankXie05 commented 9 months ago

Currently, I can confirm that this feature is indeed broken. (either from the configuration parameters passed or the actual installed dependencies).

When default-features is false, the installation dependencies of default-features will not be prohibited at all.

@JavierMatosD @BillyONeal Could you please help take a look?

dg0yt commented 9 months ago

Currently, I can confirm that this feature is indeed broken. (either from the configuration parameters passed or the actual installed dependencies).

When default-features is false, the installation dependencies of default-features will not be prohibited at all.

@FrankXie05 please explain what is broken, given the expected behavior presented in
https://github.com/microsoft/vcpkg/issues/35694#issuecomment-1856695326,
https://github.com/microsoft/vcpkg/issues/35694#issuecomment-1857543880,
https://github.com/microsoft/vcpkg/issues/35694#issuecomment-1857861606.

Arech commented 9 months ago

Erm... Here's another test that shows that default-features: false works in a strange way even in an application manifest:

App manifest file explicitly prohibits default-features of opencv4 port and explicitly lists only jpeg feature that depends only on libjpeg-turbo.

{
  "$schema": "https://raw.githubusercontent.com/microsoft/vcpkg-tool/main/docs/vcpkg.schema.json",
  "name": "myapp",
  "version": "1",
  "vcpkg-configuration": {
    "$schema": "https://raw.githubusercontent.com/microsoft/vcpkg-tool/main/docs/vcpkg-configuration.schema.json",
    "default-registry": {
      "kind": "git",
      "repository": "https://github.com/microsoft/vcpkg",
      "baseline": "000d1bda1ffa95a73e0b40334fa4103d6f4d3d48"
    }
  },
  "features": {
    "common-opencv-port": {
      "description": "A single common for the whole repo port of OpenCV",
      "dependencies": [
        {
          "name": "opencv4",
          "default-features": false,
          "features": [ "jpeg" ]
        }
      ]
    },

    "irrelevant-feature": {
      "description": "heavy ports just for testing, they shouldn't be used",
      "dependencies": [ "opencv3", "opencv2" ]
    },

    "myfeature": {
      "description": "will use this feature only. common-opencv-port might be a dependency of many other features of this app",
      "dependencies": [
        {
          "name": "myapp",
          "default-features": false,
          "features": [ "common-opencv-port" ]
        }
      ]
    }
  },
  "dependencies": []
}

CMakeLists.txt to choose which feature to use depending on build settings

cmake_minimum_required(VERSION 3.16)
# choosing feature list (depending on some CMake vars in a real project)..
set(VCPKG_MANIFEST_FEATURES "myfeature")
# trigger vcpkg
project(MyTestProj)

I expect it to use only `"jpeg" feature of OpenCV4 port.

Now run and observe what vcpkg starts to build:

$ rm -rf ./build;cmake -B ./build -G Ninja -DCMAKE_TOOLCHAIN_FILE=$VCPKG_ROOT/scripts/buildsystems/vcpkg.cmake
-- Running vcpkg install
warning: Embedding `vcpkg-configuration` in a manifest file is an EXPERIMENTAL feature.
Fetching registry information from https://github.com/microsoft/vcpkg (HEAD)...
Detecting compiler hash for triplet x64-windows...
The following packages will be built and installed:
  * flatbuffers:x64-windows@23.5.26#1 -- ...\vcpkg\registries\git-trees\2f827d9a5f37614af7cdb44c15075dbaac88d740
  * libjpeg-turbo:x64-windows@3.0.1 -- ...\vcpkg\registries\git-trees\481b08127d4002ba7441f144df259e03271e7592
  * liblzma:x64-windows@5.4.4 -- ...\vcpkg\registries\git-trees\f285b7c4ffa2cc065c7c6fec4b61006f7fa2714e
  * libpng:x64-windows@1.6.40 -- ...\vcpkg\registries\git-trees\1a2a7f489e57c9e55e049b7c5f0a28c8ff4464ed
  * libwebp[core,libwebpmux,nearlossless,simd,unicode]:x64-windows@1.3.2 -- ...\vcpkg\registries\git-trees\0b981028589375097039d5e39e7d87659cdfa824
  * opencv4[core,default-features,dnn,jpeg,png,quirc,tiff,webp]:x64-windows@4.8.0#12 -- ...\vcpkg\registries\git-trees\c3d46b66df37511cf7b49ceffe96f5ff62acf6ea
  * protobuf:x64-windows@3.21.12 -- ...\vcpkg\registries\git-trees\3b145508ba614fe26989b23f6317f15bf6467be4
  * quirc:x64-windows@1.1#3 -- ...\vcpkg\registries\git-trees\85b17d675bbbb05448d16af79d434bc5fca51f7b
  * tiff[core,jpeg,lzma,zip]:x64-windows@4.6.0#1 -- ...\vcpkg\registries\git-trees\1c0fc716f916833234f0fb5e7061d52e9063e7c6
  * vcpkg-cmake:x64-windows@2023-05-04 -- ...\vcpkg\registries\git-trees\88a7058fc7fa73a9c4c99cfcae9d79e2abf87a5a
  * vcpkg-cmake-config:x64-windows@2022-02-06#1 -- ...\vcpkg\registries\git-trees\8d54cc4f487d51b655abec5f9c9c3f86ca83311f
  * vcpkg-get-python-packages:x64-windows@2023-07-28 -- ...\vcpkg\registries\git-trees\80e0cf9d38d72126b34eb1ee9b33f936c778a696
  * zlib:x64-windows@1.3 -- ...\vcpkg\registries\git-trees\5ac18c6e6e3e2bf5a9e3d0bc8a845f198e4c4e05
Additional packages (*) will be modified to complete this operation.
......

vcpkg ignores requirement to use only explicitly specified features and use all defaults instead. And it's not that it shows it wrongly, it really starts to build every other damn feature I don't need.

$ $VCPKG_ROOT/vcpkg.exe --version
vcpkg package management program version 2023-12-12-1c9ec1978a6b0c2b39c9e9554a96e3e275f7556e

$ cmake --version
cmake version 3.27.2-msvc1

The same behavior I see on Ubuntu 20.04 @ WSL

$ $VCPKG_ROOT/vcpkg --version
vcpkg package management program version 2023-12-12-1c9ec1978a6b0c2b39c9e9554a96e3e275f7556e

$ cmake --version
cmake version 3.25.1

You can clone https://github.com/Arech/vcpkg_default_features repo with that code

If I'm doing something wrong here, please advice me on how to modify the code to make it only use explicitly specified features? Current behavior brings enormous toll on our CI and on local devs...

dg0yt commented 9 months ago

If I'm doing something wrong here, please advice me on how to modify the code to make it only use explicitly specified features?

No, I think you provided a small reproducible example to prove the bug. 💣

I didn't test with CMake but with vcpkg install --dry-run and file modifications. Please verify my observations with CMake:
Explicitly adding common-opencv-port to VCPKG_MANIFEST_FEATURES will give the desired minimal installation.

Arech commented 9 months ago

Please verify my observations with CMake: Explicitly adding common-opencv-port to VCPKG_MANIFEST_FEATURES will give the desired minimal installation.

I don't need to run it to confirm your guess, as I see it regularly in the projects I deal with. Yes, referencing a feature directly will make vcpkg to respect default-features: false and make a minimal installation.

Assuming that the same code in vcpkg handles feature aggregation for top-level manifests and for ports, it looks like handling of features is somewhat broken for anything but directly and externally requested features. Features of a port dependency (such as I described in the first message) are never requested directly, hence default-features setting is ignored for them too.

To me the explanation of default-features behaviour in ports given in the thread ("In port manifests, it declares a possibility and enables user choice. The actual control is only in user project manifests" and so on) looks much more like a rationalization of existing bug->feature p.o.v. than a concious design choice, b/c as a user I absolutely don't need that "declares a possibility and enables user choice". There's absolutely no value in that for me, only headaches how to make ensure that 100s of dependencies are used optimally. I need the fastest runtime and hence the least amount of code to be built, i.e. minimal dependencies by default. Besides anything, that "a possibility that enables choice" just breaks encapsulation of information: a user just shouldn't have to know and do anything to ensure the ports they are using are behaving in an optimal way, so port creators strive to define minimal dependencies and the fact that vcpkg ignores that is a clear bug to me.

dg0yt commented 9 months ago

I need the fastest runtime and hence the least amount of code to be built, i.e. minimal dependencies by default.

It is a fair requirement for some use cases, and there is much support for adding an option to enable such a minimal-dependencies mode. And there is https://github.com/microsoft/vcpkg-tool/pull/538 which still doesn't seem to get enough interest from Microsoft.

Besides anything, that "a possibility that enables choice" just breaks encapsulation of information: a user just shouldn't have to know and do anything to ensure the ports they are using are behaving in an optimal way.

Doesn't installing minimal features just have the effect of forcing users to know and request all the features which they would expect to be generally included?
Can we really make the general assumption that minimal is optimal for every user when the set of features may vary between "barely usable" and "overkill"?
Nobody claims that the default size fits all. But IMO the concept of default features seems a reasonable approach to avoid forcing users to acquire detailed knowledge about the features of each single port. Until they are really interested in customization.

Arech commented 9 months ago

Doesn't installing minimal features just have the effect of forcing users to know and request all the features which they would expect to be generally included? Can we really make the general assumption that minimal is optimal for every user when the set of features may vary between "barely usable" and "overkill"?

There are several things to unpack here.

(1) I don't propose to force always installing minimal features. I think that current system that have some features enabled by default and a machinery to fully customize features are ideal and probably can't even be improved. I just want vcpkg to respect it's own settings, because actually the current approach (ignoring default-features:false for all non primary dependencies) is really detrimental and actually doesn't bring value even to those to whom it's supposed to help - I'll talk on that further.

(2)

Doesn't installing minimal features just have the effect of forcing users to know and request all the features which they would expect to be generally included?

I think the answer is definitely no and here's why:

(3) The last but not the least, why ignoring default-features:false is a serious problem that cost lots of money:

Many ports depend on dependencies that are fully external to vcpkg. Be it a platform SDK, system installed packages, or some specific user installed software, - most, if not all, non-trivial ports depend on it. These external dependencies gets updated on it's own schedule and vcpkg does great job on tracking dependent files change, and forcing recompilation of ports that depend on changed files. This is not very frequent on Windows, though even there a compiler might be updated a couple of times per month (requiring to recompile everything), but on Linux... it's a real pain on Linux, because the amount of system dependencies in form of source code is huge, and a system might get several updates per day. Not all these updates are in a dependency chain of some used port, but some of them are. We have quite a modestly sized project depending on boost and opencv ports and due to this, vcpkg has to rebuild many ports several times a week. This takes up to a dozen of minutes locally (multiply that by the number of developers affected) and then there are Azure CI agents that are less powerful than local workstations and depending on amount of changes, sometimes CI jobs even timeouts of an allocated hour of working time. This time is also wasted, because oftentimes developers depend on their PR to get merged before they can proceed further. So all of this stack up to a huge amount of wasted time for the company. I'm active vcpkg advocate in my company. I know what the life was before vcpkg in C++ and I really feel vcpkg is a great tool to solve a huge pain. But people got really annoyed by that frequent rebuilds and I constantly have to address their concerns of wasted time. My main argument is that "it rebuilds dependent ports to integrate the latest updates made to the system and that allows us to release a safer software." But now I must also add this: "Unfortunately vcpkg just ignores the dependency specification and adds a lot of absolutely irrelevant ports to the build plan. Most of rebuilds are caused by a change in some dependency of a port or a feature that we just don't need at all, so indeed most of that person-hours per week taken by rebuilds are just wasted".

Arech commented 9 months ago

Ah, and there's another major independent reason why ignoring default-features:false in ports is really bad idea: vcpkg's CI doesn't have 100% coverage for all supported platforms. For example, it doesn't cover QNX operating system at all and I repeatedly face challenges and obstacles while building my company's software for it. The latest issue I struggle with is with building some dependency of HDF5 file format support port. I had to totally drop support for QNX for the piece of software in question, because I had no time to address the port building issue. Given that default-features:false is ignored I'm not even sure that we are blocked because the failing port is important, not that vcpkg is trying to build the damn port just for the sake of some ephemeral completeness that we don't even need. What could possibly be dumber and worse than that situation?

dg0yt commented 9 months ago
  • second, but it is the most important: port dependencies (at least theoretically) must always remain private to the port; It must be only the port's concern on which other ports it depend on. Users mustn't rely on any knowledge of internal implementation of ports they use. Port implementation should be able to change its own dependencies without any fear of breaking user's code. For example, the fact that opencv4 port currently depend on libjpeg-turbo and installs it, doesn't mean, that I can rely on this dependency at all. If I want to use libjpeg-turbo in my code, I must list it as my code dependency directly in my manifest. Otherwise, should opencv4 abandon this dependency for any reason - my code will just break. That's just an engineering basics of managing dependencies and vcpkg should help people to follow it, not actively prevent that!

Understood. But the interpretation of default-features doesn't change isolation. Dependencies simply are not "private" here. (gn and alike do something like this IIUC.) In vcpkg, as soon as one package ("privately") depends on opencv4[jpeg], the libjpeg-turbo dependency of this feature will be installed before opencv4 and found by every other consumer.

There's no way to "expect something installed along with a port".

Agreed, with regard to "installed". But AFAIU the main purposes of "default-features" is not to install dependencies but to provide typical (but optional) capabilities. Like the most common compression modes in tiff. Or the most common raster and vector formats in gdal. Or GUI support in qtbase. Without default-features, the default installation would provide packages which would frequently be perceived as unusable, or broken, e.g. because they can't read, process or display a "typical" TIFF file.

Maybe that's the key problem: Most of us want minimal dependencies. But what is offered for control are features.

Most of us also want less rebuilds. But the immediate control is to pin versions, not the interpretation of default-features.

dg0yt commented 9 months ago

The latest issue I struggle with is with building some dependency of HDF5 file format support port.

HDF5 is challenging in all cross-builds. Upstream wants you to detect properties from your target runtime and feed them in at build time. And still, vcpkg cross-builds gdal and opencv4 in CI. With hdf5 kept out of (unconditional?) default features.

Arech commented 9 months ago

But AFAIU the main purposes of "default-features" is not to install dependencies but to provide typical (but optional) capabilities.

Yes, it was a bad phrasing on my side. By "installed" I meant not just installing binary tools, but primarily making additional headers available for inclusion and libraries/entry points into libraries to linking to for the port affected. These additional capabilities usually in the essence are just some additional code a vcpkg user could rely on in a transparent manner.

Anyway, we seems to agree on the definition of the problem.

How can I ping people, who make the decisions that we need here? Who are they and how to reach them? This problem is really huge damn pain in the a*s for me personally and for my company...

dg0yt commented 9 months ago
  • check and fix the handling of transitive dependencies in the user's manifest,

https://github.com/microsoft/vcpkg-tool/pull/1331

BillyONeal commented 6 months ago

The 'top level manifest' parts were fixed by @dg0yt and landed in https://github.com/microsoft/vcpkg/pull/38339

patrikhuber commented 5 months ago

I just did a git pull on vcpkg and .\bootstrap-vcpkg.bat, which updated vcpkg:

Downloading https://github.com/microsoft/vcpkg-tool/releases/download/2024-04-23/vcpkg.exe
[...]
vcpkg package management program version 2024-04-23-d6945642ee5c3076addd1a42c331bbf4cfc97457

I've got a package that I want to depend on opencv4 but not on its default features (because it essentially just needs cv::Mat, nothing else). vcpkg-custom-overlay/my-test-package/vcpkg.json:

{
  "$schema": "https://raw.githubusercontent.com/microsoft/vcpkg-tool/main/docs/vcpkg.schema.json",
  "name": "my-test-project",
  "version": "1.0.0",
  "homepage": "https://gitlab.com/private/private",
  "dependencies": [
    {
      "name" : "vcpkg-cmake",
      "host" : true
    },
    {
      "name" : "vcpkg-cmake-config",
      "host" : true
    },
    {
      "name": "opencv4",
      "default-features": false
    }
  ]
}

As far as I understood this thread, there should be a way of installing my-test-package with its opencv4 dependency without default features? How do I do that? If I do on the command-line: .\vcpkg.exe install my-test-package[core] --overlay-ports=C:\Users\MyUser\vcpkg-custom-overlay --dry-run, it still pulls in opencv4[core,default-features,dnn,jpeg,png,quirc,tiff,webp]:x64-windows@4.8.0#19.

Am I misunderstanding this thread, or is this still not possible, or am I doing something wrong?

(The difference in build-time, dependency size etc. between opencv4[core] and opencv4 (with default-features) is factor 3-5x, this probably warrants a separate issue specific for OpenCV, but in general for vcpkg there needs to be a way for a project that only depends on a dependency's minimal configuration to actually do kick off that build with the dependency being compiled without default-features.)

dg0yt commented 5 months ago

As far as I understood this thread, there should be a way of installing my-test-package with its opencv4 dependency without default features? How do I do that? If I do on the command-line: .\vcpkg.exe install my-test-package[core] --overlay-ports=C:\Users\MyUser\vcpkg-custom-overlay --dry-run, it still pulls in opencv4[core,default-features,dnn,jpeg,png,quirc,tiff,webp]:x64-windows@4.8.0#19.

You ("user") must ask for it in the command line or in the your "top-level" manifest. (The port manifest is just an enabler.) Command line example:

.\vcpkg.exe install  my-test-package  opencv4[core]  --overlay-ports=C:/Users/MyUser/vcpkg-custom-overlay