'pkg upgrade' erroneously removes non-automatic packages

[yurivict]

Command that is supposed to upgrade just one library is going to remove the top-level packages like blender-2.78c_1 and deluge-1.3.15,1.

net-p2p/deluge and graphics/blender are installed as non-automatic, so they should be kept during all upgrades, and it just deletes them.

# pkg upgrade boost-libs
Updating FreeBSD repository catalogue...
FreeBSD repository is up to date.
All repositories are up to date.
The following 17 package(s) will be affected (of 0 checked):

Installed packages to be REMOVED:
    kig-4.14.3_2
    blender-2.78c_1
    openimageio-1.6.12_7
    deluge-1.3.15,1
    deluge-cli-1.3.15

Installed packages to be UPGRADED:
    boost-libs: 1.63.0_1 -> 1.64.0
    libkolabxml: 1.1.6_2 -> 1.1.6_3
    libkolab: 0.6.3_3 -> 0.6.3_5
    kdepim-runtime: 4.14.10_2 -> 4.14.10_5
    bitcoin-daemon: 0.12.1_9 -> 0.12.1_11
    libtorrent-rasterbar: 1.1.2 -> 1.1.3_1
    py27-libtorrent-rasterbar: 1.1.2_2 -> 1.1.3_1
    boost-python-libs: 1.63.0 -> 1.64.0
    py27-pyopencl: 2015.1_3 -> 2017.1.1
    akonadi: 1.13.0_2 -> 1.13.0_3
    leveldb: 1.20 -> 1.20_1

Installed packages to be REINSTALLED:
    cadabra2-2.1.4 (needed shared library changed)

Number of packages to be removed: 5
Number of packages to be upgraded: 11
Number of packages to be reinstalled: 1

The operation will free 148 MiB.
20 MiB to be downloaded.

Proceed with this action? [y/N]: y

pkg-1.10.1

[bdrewery]

This is ancient but I just ran into the same problem on 1.17.2. I have 3 non-automatic packages and upgrade insists on removing one of them for no reason. -ddd gives no indication why either.

It seems like the problem is that it wants to delete but really should have chosen to ignore. The local package version matches the remote but a (reverse?) dependency of the to-delete package managed to insert the local package into the problem, but because the remote version still matched it ended up falling back on a delete rather than removing it from the job.

I had a local-meta package that depended on mysql57-server. For some reason it decided that mysql57-server should be deleted, so it chained down to local-meta too. I did pkg create mysql57-server; pkg delete -f mysql57-server; pkg add -A mysql57-server-5.7.35.pkg and now upgrade does NOT want to delete these. Strange. Luckily the vm is zfs snapshotted so I'll rollback and compare the db to the "fixed" one.

[bdrewery]

Adding non-automatic packages into the candidates list fixes my case. I have not yet run the full test suite or added a test case for this one.

diff --git a/libpkg/pkg_jobs.c b/libpkg/pkg_jobs.c
index 80a23fd0..2e1e6c3a 100644
--- a/libpkg/pkg_jobs.c
+++ b/libpkg/pkg_jobs.c
@@ -1564,6 +1576,8 @@ pkg_jobs_find_install_candidates(struct pkg_jobs *j, size_t *count)

        while (pkgdb_it_next(it, &pkg, PKG_LOAD_BASIC) == EPKG_OK) {
                if ((j->flags & PKG_FLAG_FORCE) ||
+                                               !pkg->automatic ||
+                                               pkg->vital ||
                                                pkg_jobs_check_remote_candidate(j, pkg)) {
                        c = pkg_jobs_new_candidate(pkg);
                        LL_PREPEND(candidates, c);

edit: maybe not. When I added everything into the candidate list it was fine but this small thing doesn't seem good enough yet or I'm just tired.

[bdrewery]

I think the problem here specifically is because protobuf had its shlib updated in main, and dependencies chased such as mysql57-server, but not backported to the 2021Q3 branch I was using. When I switched to 2021Q4 the chase was present but mysql had been updated after the protobuf chase - and that update had made it into 2021Q3 as well. So there was never any obvious indication that mysql57-server needed to be rebuilt because its version matched in the top of 2021Q3 and 2021Q4.

protobuf now provides libprotobuf.so.28 while mysql57-server wants libprotobuf.so.25. Because I was using PKG_NO_VERSION_FOR_DEPS mysql57-server was not rebuilt by Poudriere and so had the same manifest as the older 2021Q3 libprotobuf.so.25 one.

That's all pretty specific but the weird part is that as long as the manifest for the remote mysql57-server does not match the local one then it is added to the candidates list and it remains installed just fine. So despite the special setup here I think there is an argument to be made that it should not remove the package just because its digest is unchanged. I mean all I did was use this patch without rebuilding mysql57-server or making any other changes.

@@ -1536,7 +1536,8 @@ pkg_jobs_check_remote_candidate(struct pkg_jobs *j, struct pkg *pkg)
                         * Check package with the same uid and explore whether digest
                         * has been changed
                         */
-                       if (strcmp(p->digest, pkg->digest) != 0)
+                       if (strcmp(p->digest, pkg->digest) != 0 ||
+                               strcmp(pkg->name, "mysql57-server") == 0)
                                npkg ++;

                        pkg_free(p);

So the logic in the solver seems fine. Just this candidate check may benefit from adding in everything.

[bapt]

this is still strange because even if the version is not taken in account, the digest used to decide if something is a candidate includes the required_shlibs, to protobuf requirement going from .25 (quarterly and installed) to .28 (new remote) should have made mysql57-server a candidate anyway

[bdrewery]

Yeah! But the problem I have is that the mysql-server57 package was not rebuilt by poudriere because the quarterly branch switch missed an shlib revision chase. So the package remained the same. Assumption of mine: this poudriere branch isn’t auto rebuilding on dep updates anymore. This is the only remaining issue I have.

So the digest unchanging is not a pkg problem. The only problem in pkg is that non-automatic and vital don’t have protections in the solver.

I’m drafting a big RFC to ports asking for ideas on the poudriere side problem. I’ll explain the scenario more clearly in there.

Regards, Bryan Drewery

On Oct 26, 2021, at 02:04, Baptiste Daroussin @.***> wrote:

 this is still strange because even if the version is not taken in account, the digest used to decide if something is a candidate includes the required_shlibs, to protobuf requirement going from .25 (quarterly and installed) to .28 (new remote) should have made mysql57-server a candidate anyway

— You are receiving this because you commented. Reply to this email directly, view it on GitHub, or unsubscribe.

[bapt]

probably the right thing here is then to make provide to poudriere the digest compute pkg uses, pkg query -F mypkg "%X" should print it