`pkg rquery` returns duplicate results

[cgull]

On FreeBSD 13.2p1:

[root@jhood_lfs /zdata/home/jhood]# pkg -v
1.20.4
[root@jhood_lfs /zdata/home/jhood]# pkg rquery -a %n | wc -l
   56348
[root@jhood_lfs /zdata/home/jhood]# pkg rquery -a %n | sort -u | wc -l
   33908
[root@jhood_lfs /zdata/home/jhood]# pkg -vv | sed -n '/Repo/,$p'
Repositories:
  FreeBSD: { 
    url             : "pkg+http://pkg.FreeBSD.org/FreeBSD:13:amd64/latest",
    enabled         : yes,
    priority        : 0,
    mirror_type     : "SRV",
    signature_type  : "FINGERPRINTS",
    fingerprints    : "/usr/share/keys/pkg"
  }
[root@jhood_lfs /zdata/home/jhood]# 

pkg rquery %n firefox against a local repository also returns duplicate results. I haven't characterized this any further yet.

[cgull]

It seems to be a repo database query problem, not a repo database update problem:

On another machine, I did pkg update, the pkg rquery commands (both printed 33908), and then pkg upgrade --no-repo-update pkg (no scrollback to cut and paste, but it installed 1.20.4), and then

[root@lister /usr/home/jhood]# pkg rquery --no-repo-update -a %n | sort -u | wc -l
   33908
[root@lister /usr/home/jhood]# pkg rquery --no-repo-update -a %n | wc -l
   56348
[root@lister /usr/home/jhood]# 

[arrowd]

I wonder if this has something to do with pkgdb_all_search() function I added in ffdb473e704a48abca70446a5f025aba2eec8ca8

[bapt]

I think it does somehow, I haven't investigated it yet

[rcmcdonald91]

I just spent some time in gdb and yes it certainly has to do with the work from https://github.com/freebsd/pkg/commit/ffdb473e704a48abca70446a5f025aba2eec8ca8

More specifically, the pkgdb_it_next behavior has changed.

[rcmcdonald91]

May I ask why that work was done? It seems there are no consumers (yet) of the new function pkgdb_all_search, at least not in pkg proper. Perhaps something else linking libpkg maybe.

We might be better off backing out the offending commit and releasing an updated version. It might also be useful to include a test case at least to exercise the new function especially until we make use of it elsewhere

[arrowd]

May I ask why that work was done? It seems there are no consumers (yet) of the new function pkgdb_all_search, at least not in pkg proper. Perhaps something else linking libpkg maybe.

Yes, it is used for the PackageKit's backend, which isn't opensourced yet.

We might be better off backing out the offending commit and releasing an updated version.

This might be a bit too much. I'll look into this in near days and will try to fix it.

[arrowd]

Hum, I can't reproduce this. Do you have 2 remote repositories set up by any chance?

What I tried:

1. pkg install foo
2. pkg rquery %n foo
3. I get only one line in the output

[rcmcdonald91]

Yes, two remote repositories.

[arrowd]

Well, then it is an expected behavior, I guess? You query two remote repos and both contain firefox package.

[rcmcdonald91]

I should have qualified that the remote repositories contain different packages

pkg query "%n %o" reveals that the duplicated lines stem from the same remote

[bapt]

yes activating 2 repo results in duplicated from the same repo, this is the bug, I can reproduce but I don't have time to investigate yet

[cgull]

I have one remote repo on both of my machines in my original report. In /var/db/pkg there's only local.sqlite and repo-FreeBSD.sqlite. Both machines had their repo conf files edited from quarterly to latest long long ago, though.

[cgull]

@arrowd Not all packages are duplicated; for me bash isn't but firefox is. Package installation state doesn't seem to matter either, one of my machines is a desktop with firefox, the other is a server, but both show duplicates for firefox.

jhood@jhood_lfs:/home/jhood/src/hg/master 0 $ pkg rquery '%n %o %R' firefox
firefox www/firefox FreeBSD
firefox www/firefox FreeBSD

[arrowd]

I found that pkg_repo_binary_it_new creates an iterator that has ->local part initialized. It calls pkgdb_it_new_sqlite() instead of pkgdb_it_new_repo() to create the iterator. My understanding was that the former function is used to create a local DB iterator, while the latter is for remote ones. The struct pkgdb fakedb variable name also hints that something's being hacked around there.

In the rquery case the code also passes PKG_REMOTE as the int type parameter to pkgdb_it_new_sqlite. Maybe it should be taken into account somewhere.

[rcmcdonald91]

There are consumers today of rquery that don't expect to handle duplicated results.

We might be better off backing out the offending commit and releasing an updated version [ASAP]. It might also be useful to include a test case at least to exercise the new function especially until we make use of it elsewhere

[rcmcdonald91]

Still seeing duplicated results in my case.

PKG_DBDIR = "/var/db/pkg";
PKG_CACHEDIR = "/var/cache/pkg";
PORTSDIR = "/usr/ports";
INDEXDIR = "";
INDEXFILE = "INDEX-14";
HANDLE_RC_SCRIPTS = false;
DEFAULT_ALWAYS_YES = false;
ASSUME_ALWAYS_YES = false;
REPOS_DIR [
    "/etc/pkg/",
    "/usr/local/etc/pkg/repos/",
]
PLIST_KEYWORDS_DIR = "";
SYSLOG = true;
ABI = "FreeBSD:14:amd64";
ALTABI = "freebsd:14:x86:64";
DEVELOPER_MODE = false;
VULNXML_SITE = "http://vuxml.freebsd.org/freebsd/vuln.xml.xz";
FETCH_RETRY = 3;
PKG_PLUGINS_DIR = "/usr/local/lib/pkg/";
PKG_ENABLE_PLUGINS = true;
PLUGINS [
]
DEBUG_SCRIPTS = false;
PLUGINS_CONF_DIR = "/usr/local/etc/pkg/";
PERMISSIVE = false;
REPO_AUTOUPDATE = true;
NAMESERVER = "";
HTTP_USER_AGENT = "pkg/1.20.5";
EVENT_PIPE = "";
FETCH_TIMEOUT = 30;
UNSET_TIMESTAMP = false;
SSH_RESTRICT_DIR = "";
PKG_ENV {
    SSL_CA_CERT_FILE = "/etc/ssl/netgate-ca.pem";
    SSL_CLIENT_CERT_FILE = "/usr/local/etc/pfSense/pkg/repos/pfSense-repo-devel-cert.pem";
    SSL_CLIENT_KEY_FILE = "/usr/local/etc/pfSense/pkg/repos/pfSense-repo-devel-key.pem";
}
PKG_SSH_ARGS = "";
DEBUG_LEVEL = 0;
ALIAS {
}
CUDF_SOLVER = "";
SAT_SOLVER = "";
RUN_SCRIPTS = true;
CASE_SENSITIVE_MATCH = false;
LOCK_WAIT = 1;
LOCK_RETRIES = 5;
SQLITE_PROFILE = false;
WORKERS_COUNT = 0;
READ_LOCK = false;
IP_VERSION = 0;
AUTOMERGE = true;
VERSION_SOURCE = "";
CONSERVATIVE_UPGRADE = true;
PKG_CREATE_VERBOSE = false;
AUTOCLEAN = false;
DOT_FILE = "";
REPOSITORIES {
}
VALID_URL_SCHEME [
    "pkg+http",
    "pkg+https",
    "https",
    "http",
    "file",
    "ssh",
    "tcp",
]
ALLOW_BASE_SHLIBS = false;
WARN_SIZE_LIMIT = 1048576;
METALOG = "";
OSVERSION = 1400093;
IGNORE_OSVERSION = false;
BACKUP_LIBRARIES = false;
BACKUP_LIBRARY_PATH = "/usr/local/lib/compat/pkg";
PKG_TRIGGERS_DIR = "/usr/local/share/pkg/triggers";
PKG_TRIGGERS_ENABLE = true;
AUDIT_IGNORE_GLOB [
]
AUDIT_IGNORE_REGEX [
]
COMPRESSION_FORMAT = "";
COMPRESSION_LEVEL = -1;
ARCHIVE_SYMLINK = false;
REPO_ACCEPT_LEGACY_PKG = false;
FILES_IGNORE_GLOB [
]
FILES_IGNORE_REGEX [
]

Repositories:
  pfSense-core: {
    url             : "pkg+https://pfsense-plus-pkg-beta.netgate.com/pfSense_plus-master_amd64-core",
    enabled         : yes,
    priority        : 0,
    mirror_type     : "SRV",
    signature_type  : "FINGERPRINTS",
    fingerprints    : "/usr/local/share/pfSense/keys/pkg"
  }
  pfSense: {
    url             : "pkg+https://pfsense-plus-pkg-beta.netgate.com/pfSense_plus-master_amd64-pfSense_plus_devel",
    enabled         : yes,
    priority        : 0,
    mirror_type     : "SRV",
    signature_type  : "FINGERPRINTS",
    fingerprints    : "/usr/local/share/pfSense/keys/pkg"
  } 

[bapt]

can you pkg rquery "%n %R" or a case which has duplicated listing? second, can I use your repository to run my tests? if yes do you have an example of duplicated?

[cgull]

I think it's fixed for me.

jhood@jhood_lfs:/home/jhood/src/pkg 0 $ src/pkg -v
1.20.990.5-a9b88ac1
jhood@jhood_lfs:/home/jhood/src/pkg 0 $ src/pkg rquery -a %n | wc -l
   33918
jhood@jhood_lfs:/home/jhood/src/pkg 0 $ src/pkg rquery -a %n | sort -u | wc -l
   33918
jhood@jhood_lfs:/home/jhood/src/pkg 0 $ src/pkg -vv | sed -n '/Repo/,$p'
Repositories:
  FreeBSD: { 
    url             : "pkg+http://pkg.FreeBSD.org/FreeBSD:13:amd64/latest",
    enabled         : yes,
    priority        : 0,
    mirror_type     : "SRV",
    signature_type  : "FINGERPRINTS",
    fingerprints    : "/usr/share/keys/pkg"
  }
jhood@jhood_lfs:/home/jhood/src/pkg 0 $