swupd search results improvement

clearlinux / swupd-client

Software update client

Other

116 stars 61 forks source link

swupd search results improvement #807

Open eero-t opened 5 years ago

eero-t commented 5 years ago

When doing search for some file, swupd output lists that same file being in dozens of bundles.

If that means it's duplicated in all of those bundles, that seems a serious packaging problem.

If it means just that only one of those bundles contains the file, and others depend on that bundle, that output is very misleading and not really useful.

In latter case I would rather expect something like:

# swupd search libllvm
Searching for 'libllmv'

Bundle llvm [installed] (271 MB on system)
    /usr/lib64/libLLVM.so.7.0

"llvm" bundle is included by following installed bundles:
   foo1 (18534 MB)
   bar2 (234 MB)

If a bundle containing the searched file is included also by other bundles, but it's not installed on my system, I don't know why that information would be of interest. I search files to find the smallest installable bundle to get the file to my system, or do a search to find what caused it to be installed on my system.

eero-t commented 5 years ago

swupd search output verbosity has been seriously curbed. Now it looks like this:

# swupd search libpng12
No good bundles found, closest match is:
     games                              - Play games in Clear Linux  (396Mb)

This bundle can be installed with:
  swupd bundle-add  games

# swupd search libassimp
No good bundles found, closest match is:
     qt-basic                           - Run programs that use the Qt runtime  (257Mb)

This bundle can be installed with:
  swupd bundle-add  qt-basic

Alternative bundle options are
     devpkg-qt3d                        - All content for pkgconfig file qt3d.pc  (26Mb)
     desktop-dev                        - All packages required to build the desktop bundle.  (2511Mb)
     qt-basic-dev                       - All packages required to build the qt-basic bundle.  (2494Mb)

# swupd search libassimp3
Searching for 'libassimp3'

Downloading Clear Linux manifests
   46.12 MB total...

Completed manifests download.

Search term not found.
Displaying top 5 file results per bundle

# swupd search cpufreq
Bundle with the best search result:
     kernel-hyperv-lts                  - Run the hyperv specific LTS kernel  (39Mb)

This bundle can be installed with:
  swupd bundle-add  kernel-hyperv-lts

Alternative bundle options are
     kernel-hyperv                      - Run the hyperv specific kernel  (39Mb)
     kernel-gce                         - Run the kvm specific kernel  (29Mb)
     kernel-aws                         - Run the kvm specific kernel  (33Mb)

# swupd search cpufreq-info
Searching for 'cpufreq-info'

Search term not found.
Displaying top 5 file results per bundle

Issues:

What means "no good bundles found, closest match"? Did it find it or not?
If there's "closest match" e.g. for libassimp, why I need to guess what's the actual version of that?
If there were no matches, "Displaying top 5 file results per bundle" text is redundant
if swupd finds cpufreq, what it actually is? E.g. the tools found from Debian/Ubuntu cpufreq-utils package are missing. By default it could show few of the matches for best bundle and option to list all of them for it

(By accident I found that pressing ^C while swupd is searching, will show at least some of the matched files for the "best search result" cases. No, it wasn't the cpufreq tools that I was trying to locate.)

As to ongoing bundle splitting, that still needs work:

# swupd search glxinfo
Bundle with the best search result:
     os-testsuite-phoronix-desktop      - Run the Phoronix testsuite (desktop version)

This bundle can be installed with:
  swupd bundle-add  os-testsuite-phoronix-desktop

# swupd search gnuplot
Bundle with the best search result:
     os-testsuite-phoronix-desktop      - Run the Phoronix testsuite (desktop version)

This bundle can be installed with:
  swupd bundle-add  os-testsuite-phoronix-desktop

Alternative bundle options are
     os-testsuite-phoronix              - Run the Phoronix testsuite

=> Basic tools like glxinfo and gnuplot still require phoronix test suite (which I assume still to be huge, as swupd tool doesn't have an option to tell what's its size)

eero-t commented 5 years ago

Actually, swupd doesn't currently seem to be working properly, it returns weird results for basic desktop library and sometimes throws an exception:

# swupd search libGLESv2
Bundle with the best search result:
     kdiff3                             - Feature-Rich Code Comparison Utility  (1Mb)

This bundle can be installed with:

  swupd bundle-add  kdiff3

Alternative bundle options are
     lightdm                            - LightDM login manager with GTK greeter  (1Mb)
     gftp                               - Text mode and GUI FTP Client  (3Mb)
     v4l-utils                          - video4linux tools  (3Mb)

# swupd search libxcb-composite
Traceback (most recent call last):
  File "/usr/bin/swupd-search", line 378, in <module>
    main()
  File "/usr/bin/swupd-search", line 370, in main
    correct_scores_for_policy()
  File "/usr/bin/swupd-search", line 242, in correct_scores_for_policy
    bonus = 0.1 / size;
ZeroDivisionError: float division by zero
Searching for 'libxcb-composite'

Displaying top 5 file results per bundle

Bundle devpkg-libXau    (9 MB to install)
    /usr/lib64/libxcb-composite.so
    /usr/lib64/libxcb-composite.so.0
...

eero-t commented 5 years ago

Regarding searches for a specific library version, results seem bogus:

# swupd search libunwind
Bundle with the best search result:
     devpkg-libunwind                   - All content for pkgconfig file libunwind.pc

This bundle can be installed with:

  swupd bundle-add  devpkg-libunwind

Alternative bundle options are
     dev-utils-dev                      - All packages required to build the dev-utils bundle.
     os-clr-on-clr-dev                  - All packages required to build the os-clr-on-clr bundle.
     user-basic-dev                     - All packages required to build the user-basic bundle.

# swupd search libunwind.so.8 
Bundle with the best search result:
     libzmq                             - ZMQ tools and libs

This bundle can be installed with:

  swupd bundle-add  libzmq

Alternative bundle options are
     kdiff3                             - Feature-Rich Code Comparison Utility  (1Mb)
     lightdm                            - LightDM login manager with GTK greeter  (1Mb)
     gftp                               - Text mode and GUI FTP Client  (3Mb)

fenrus75 commented 5 years ago

libzmq is indeed the smallest bundle with that file in it....

fenrus75 commented 5 years ago

(overall, you are highlighting that "swupd search" and "swupd grep" are not the same thing, and for some things a "swupd grep" is desired...

fenrus75 commented 5 years ago

(fixing the division by zero bug now)

eero-t commented 5 years ago

libzmq is indeed the smallest bundle with that file in it....

Does the swupd decision about it being smallest take into consideration:

What other bundles that bundle pulls in?
What bundles are already installed?

I assume yes for first question, and no for second because I already had llibunwind.so.8 installed through some other dependency.

Same thing with libpng, it's already installed, but none of the suggested bundles is:

# swupd bundle-list | fgrep -e kdiff3 -e lightdm -e gftp -e v4l-utils
# ls -l /lib64/libpng16.so*
lrwxrwxrwx 6 root root     19 Dec  2 15:32 /lib64/libpng16.so -> libpng16.so.16.36.0
lrwxrwxrwx 6 root root     19 Dec  2 15:32 /lib64/libpng16.so.16 -> libpng16.so.16.36.0
-rwxr-xr-x 1 root root 296920 May 11  2018 /lib64/libpng16.so.16.34.0
-rwxr-xr-x 1 root root 292824 Feb  4 17:35 /lib64/libpng16.so.16.36.0
# swupd search libpng16.so
Bundle with the best search result:
     kdiff3                             - Feature-Rich Code Comparison Utility  (1Mb)

This bundle can be installed with:

  swupd bundle-add  kdiff3

Alternative bundle options are
     lightdm                            - LightDM login manager with GTK greeter  (1Mb)
     gftp                               - Text mode and GUI FTP Client  (3Mb)
     v4l-utils                          - video4linux tools  (3Mb)

eero-t commented 5 years ago

Regarding swupd search vs. grep issue, there's also something else missing. Working locate functionality. If searched file is included to an already installed bundle, maybe swupd could list which of the installed files match the search criteria, and which bundles brought them in.

Something like this:

# swupd search libpng16.so
"libpng16.so" already installed, and matches following files:
  /lib64/libpng16.so
  /lib64/libpng16.so.16
  /lib64/libpng16.so.34.0
  /lib64/libpng16.so.36.0

They were brought in by following bundles:
  bundle1 (X MB)
  bundle2 (Y MB)

Note: ClearLinux includes "locate" tool in "findutils" bundle, but it doesn't return any matches. I assume this is because there's nothing running updates on the locate database. For Cloud container that's sensible default, but IMHO not for a user desktop.

FYI: on Debian and its derivatives (like Ubuntu):

To search files installed by package management, one can use "dpkg -S" command
"locate" command (for searching anything on the system) comes from "mlocate" package and updates are done by: /etc/cron.daily/mlocate

fenrus75 commented 5 years ago

as usual, thanks for the feedback; need this kind of feedback to really improve things.

right now the assumption in search is that you would want to know what to additionally install, so search actively favors not installed bundles over installed ones; you're suggesting to do the opposite. I'll need to ponder how to reconcile that (not in the code sense, the code side of this is trivial) but in the 'what does the user want'. Sometimes the user wants to know what to install, and sometimes clearly on what is installed. Might end up printing both (and if nothing else, need to print an "(installed)" after already installed bundles)

file searches currently require root. one of the hard cases for me was "when is something a file search".... if it's ok to have a command line switch for that, makes it much much easier to build ... I'll poke at that shortly, won't be hard to make for the "run as root" case. the not-root case doesn't have access to the file archives so that's more complicated.

On Thu, Feb 28, 2019 at 5:07 AM eero-t notifications@github.com wrote:

Regarding swupd search vs. grep issue, there's also something else missing. Working locate functionality. If searched file is included to an already installed bundle, maybe swupd could list which of the installed files match the search criteria, and which bundles brought them in.

Something like this:

swupd search libpng16.so

"libpng16.so" already installed, and matches following files: /lib64/libpng16.so /lib64/libpng16.so.16 /lib64/libpng16.so.34.0 /lib64/libpng16.so.36.0

They were brought in by following bundles: bundle1 (X MB) bundle2 (Y MB)

Note: ClearLinux includes "locate" tool in "findutils" bundle, but it doesn't return any matches. I assume this is because there's nothing running updates on the locate database. For Cloud container that's sensible default, but IMHO not for a user desktop.

FYI: on Debian and its derivatives (like Ubuntu):

To search files installed by package management, one can use "dpkg -S" command

"locate" command (for searching anything on the system) comes from "mlocate" package and updates are done by: /etc/cron.daily/mlocate

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/clearlinux/swupd-client/issues/807#issuecomment-468266009, or mute the thread https://github.com/notifications/unsubscribe-auth/ABPeFQizoPuy7svclGzVOQPo-C0oDlTpks5vR9SugaJpZM4axAst .

eero-t commented 5 years ago

right now the assumption in search is that you would want to know what to additionally install, so search actively favors not installed bundles over installed ones; you're suggesting to do the opposite.

IMHO:

it's better to have separate commands for searching file names, and for searching RPM package & bundle names and descriptions, and be explicit about this distinction because search terms & amount of resulting matches differ a lot for these use-cases
in both cases user is interested to know whether the matches are already on the system or not, what are their names, and in case of bundles & packages, what are their sizes (size on system in case they want to remove them, and in the remote case, size if they want to install them)

Whether local & remote cases need a separate command, I personally care less. As to file name matches, they could either match full path names, just the last path component, or the end of file path string. Alternatively, it could support glob patterns.

otaviobp commented 5 years ago

right now the assumption in search is that you would want to know what to additionally install, so search actively favors not installed bundles over installed ones; you're suggesting to do the opposite. I'll need to ponder how to reconcile that (not in the code sense, the code side of this is trivial) but in the 'what does the user want'. Sometimes the user wants to know what to install, and sometimes clearly on what is installed.

What about having one command for searching what to be installed and another command with some grep-like features and several options to look for installed and not installed bundles? There's a suggestion on adding a bundle-info command that would cover that: https://github.com/clearlinux/swupd-client/issues/461

eero-t commented 5 years ago

Bundles make search results a mystery, so it would be good to have a bit more info about reasons for the search matches.

In Debian/Ubuntu its obvious whether the search results for a library name were what I was looking for because libraries are packaged separately:

$ apt search libpng16
Sorting... Done
Full Text Search... Done
libpng16-16/bionic-updates,now 1.6.34-1ubuntu0.18.04.1 amd64 [installed]
  PNG library - runtime (version 1.6)

$ apt search libpng
Sorting... Done
Full Text Search... Done
fp-units-gfx/bionic-updates 3.0.4+dfsg-18ubuntu2 amd64
  Free Pascal - graphics-library units dependency package

fp-units-gfx-3.0.4/bionic-updates 3.0.4+dfsg-18ubuntu2 amd64
  Free Pascal - graphics-library units

libpng++-dev/bionic,bionic 0.2.9-1 all
  C++ interface to the PNG (Portable Network Graphics) library

libpng-dev/bionic-updates 1.6.34-1ubuntu0.18.04.1 amd64
  PNG library - development (version 1.6)

libpng-sixlegs-java/bionic,bionic 2.0-1 all
  Sixlegs Java PNG Decoder

libpng-sixlegs-java-doc/bionic,bionic 2.0-1 all
  Documentation for Sixlegs Java PNG Decoder

libpng-tools/bionic-updates 1.6.34-1ubuntu0.18.04.1 amd64
  PNG library - tools (version 1.6)

libpng12-0/now 1.2.54-1ubuntu1 amd64 [installed,local]
  PNG library - runtime

libpng16-16/bionic-updates,now 1.6.34-1ubuntu0.18.04.1 amd64 [installed]
  PNG library - runtime (version 1.6)

libpnglite-dev/bionic 0.1.17-2 amd64
  lightweight C library for loading and writing PNG images

libpnglite0/bionic 0.1.17-2 amd64
  lightweight C library for loading and writing PNG images

pngquant/bionic 2.5.0-2 amd64
  PNG (Portable Network Graphics) image optimising utility

ruby-oily-png/bionic 1.2.1~dfsg-1build2 amd64
  native mixin to speed up ChunkyPNG

tkpng/bionic 0.9-1ubuntu2 amd64
  PNG photo image support to Tcl/Tk

Whereas in ClearLinux, more detailed search returns seemingly worse matches:

# swupd search libpng16
Bundle with the best search result:
     kdiff3                             - Feature-Rich Code Comparison Utility  (1Mb)

This bundle can be installed with:

  swupd bundle-add  kdiff3

Alternative bundle options are
     lightdm                            - LightDM login manager with GTK greeter  (1Mb)
     gftp                               - Text mode and GUI FTP Client  (3Mb)
     v4l-utils                          - video4linux tools  (3Mb)
# swupd search libpng
Bundle with the best search result:
     games                              - Play games in Clear Linux

This bundle can be installed with:

  swupd bundle-add  games

Alternative bundle options are
     libX11client                       - Grouping only bundle for use in X using bundles
     devpkg-libpng                      - All content for pkgconfig file libpng.pc
     games-dev                          - All packages required to build the games bundle.

devpkg-libpng is good because normally searches for libraries are because I want to build something (only use case for just a binary version of a library is when one wants to run on ClearLinux a binary built somewhere else).

But why search for libpng didn't match devpkg-libpng?

IMHO search results should list all the bundles which name includes the search term. Preferably also all bundles that are built from RPM packages which name matches the search term. And because latter matches aren't obvious with current bundle splits, output should tell what was actually matched. For example:

Bundles with matching names:
     devpkg-libpng                      - All content for pkgconfig file libpng.pc

(Additional) bundles directly including (RPM) packages with matching names:
     libX11client                       - Grouping only bundle for use in X using bundles

(Additional) bundles which descriptions match:
     games-dev                          - All packages required to build the games bundle.

eero-t commented 5 years ago

Does the swupd decision about it being smallest take into consideration:

What other bundles that bundle pulls in?

What bundles are already installed?

I assume yes for first question, and no for second because I already had llibunwind.so.8 installed through some other dependency.

Unfortunately answer is NO also to the the first (reasonable) assumption:

# swupd search beignet
Bundle with the best search result:
     os-testsuite-phoronix              - Run the Phoronix testsuite 

This bundle can be installed with:

  swupd bundle-add  os-testsuite-phoronix

Alternative bundle options are
     os-testsuite-phoronix-desktop      - Run the Phoronix testsuite (desktop version)  (376Mb) 
     machine-learning-basic             - Build machine learning applications  (618Mb) (installed)
     dev-utils-dev                      - All packages required to build the dev-utils bundle

Because according to this: https://github.com/clearlinux/clr-bundles/blob/master/bundles/os-testsuite-phoronix

"os-testsuite-phoronix" pulls in / includes also "os-testsuite-phoronix-desktop" and "machine-learning-basic".

(I think I would prefer just using RPMs directly instead of swupd/bundles because of better package searching, package content separation, package content and dependency information.)

eero-t commented 5 years ago

No search results for files installed from bundles:

$ sudo swupd clean 
819 files removed.
$ ls -l /usr/lib64/liblcms2.so.2 /usr/lib64/libcolord.so.2
lrwxrwxrwx 1 root root 18 Jan 13  2017 /usr/lib64/libcolord.so.2 -> libcolord.so.2.0.5
lrwxrwxrwx 4 root root 17 Apr  7  2018 /usr/lib64/liblcms2.so.2 -> liblcms2.so.2.0.8
$ swupd search libcolord.so.2
Search failed, no reasonable results found
$ swupd search liblcms2.so.2
Search failed, no reasonable results found

Although at least first one is found with simpler search term:

$ swupd search colord
Bundle with the best search result:
     devpkg-colord                      - All content for pkgconfig file colord.pc (installed)
...