dub fetch --annotate doesn't generate complete dub.selections.lock file

TomaSajt commented 1 week ago

System information

dub version: dub >=1.37.0
OS Platform and distribution: Linux (NixOS)
compiler version N/A

Bug Description

Note: all of the examples below also work with dub upgrade instead of dub fetch

Running dub fetch --annotate used to generate a complete dub.selections.json file. After dub 1.37.0 it only fetches some but misses some deeper subprojects deeper in the dependency tree.

Just running dub fetch without --annotate will still work correctly, but it actually downloads the deps, which I do not want.

How to reproduce?

clean your DUB_HOME: rm -r ~/.dub
remove dub.selections.json if it exists: rm dub.selections.json
depend on a subproject
- e.g. add dependency "dcv:core" version="~>0.3.0" to dub.sdl
run dub fetch --annotate --verbose

see logs:

Using fixed selection dcv 0.3.0
Subpackage dcv:core@0.3.0 not found.

get only this as the selection:

{
"fileVersion": 1,
"versions": {
"dcv": "0.3.0"
}
}

Expected Behavior

run dub fetch --annotate --verbose
- note: here I just copied the result from dub fetch --verbose without --annotate

see logs similar to this, but without the actual fetching:

Using fixed selection dcv 0.3.0
Subpackage dcv:core@0.3.0 not found.
Dependency resolution result:
dcv: 0.3.0
Fetching dcv 0.3.0 (getting selected version)
Placing to User...
Found dependency dcv:core 0.3.0
Version selection for dependency bcaa (bcaa) of dcv:core is missing.
Missing dependency bcaa 0.0.8 of dcv:core
Version selection for dependency mir-algorithm (mir-algorithm) of dcv:core is missing.
Missing dependency mir-algorithm >=3.15.3 of dcv:core
Version selection for dependency mir-random (mir-random) of dcv:core is missing.
Missing dependency mir-random 2.2.19 of dcv:core
Using fixed selection dcv 0.3.0

get selections

{
"fileVersion": 1,
"versions": {
"bcaa": "0.0.8",
"dcv": "0.3.0",
"mir-algorithm": "3.22.1",
"mir-core": "1.7.1",
"mir-linux-kernel": "1.2.1",
"mir-random": "2.2.19",
"silly": "1.1.1"
}
}

Note: after having the dependencies downloaded into your DUB_HOME, the running the command with --annotate works as expected

Geod24 commented 1 week ago

I didn't consider --annotate when making changes to dub fetch, so there's definitely an issue here.

However, there's a few questions here: 1) If there's a selections file, it can easily print what it would fetch at the top level; 2) Because we don't know a dependency's dependency until we fetch the recipe file for the dependency, dub fetch --annotate will not be able to generate a selections file without doing network requests, but I think this is going against the spirit of --annotate; 3) dub upgrade --annotate should IMO be the preferred option here because you're asking for dependency resolution;

TomaSajt commented 1 week ago

dub upgrade --annotate should IMO be the preferred option here because you're asking for dependency resolution;

I discovered this issue while using dub upgrade, but it seems like somewhere in the middle of testing this for other versions, I started using dub fetch accidentally (it seemed to do the same thing when there's no selection file).

To clarify, I am using dub upgrade --annotate as a suggestion command to run to generate the missing selections file that will be parsed by a helper CLI tool. I can just edit it to recommend running dub upgrade instead, so it's no big deal. I just didn't want people to download files twice (since the CLI tool downloads them in a special way).

Geod24 commented 1 week ago

I just didn't want people to download files twice (since the CLI tool downloads them in a special way).

Then dub fetch is really not what you are looking for! For the record dub fetch used to not work, only the dub fetch <package_name> variant was working before the feature was overhauled in v1.37.0.

Regarding the CLI tool, my understanding is that it will become unnecessary once #1723 is implemented ?

TomaSajt commented 1 week ago

Regarding the CLI tool, my understanding is that it will become unnecessary once #1723 is implemented ?

Yes, if there's a proper hash based lockfile we will be able to mostly deprecate our custom lockfile format.

TomaSajt commented 1 week ago

Then dub fetch is really not what you are looking for! For the record dub fetch used to not work, only the dub fetch <package_name> variant was working before the feature was overhauled in v1.37.0.

Yes, when trying older dub versions, was when I realized I mixed up the two commands

dlang / dub