Find_cat messes slackware repos

[mavrothal]

In Slackware64 database update when find_cat is called at line 619 of 0setup, the $DB_pkgname is changed from "$DB_nameonly-$DB_version" in /tmp/${ONE_PKGLISTS_COMPAT}temp to "$DBnameonly$DB_version" (with underscore instead of dash) in $ONE_PKGLISTS_COMPAT.

[mavrothal]

I replaced find_cat in slacko64 6.0.8.1 with the bacon version from 5.9.1 and now package names look OK. Furthermore find_cat input_file output_file produces the same result (of the missnamed package list) as the find_cat input_file. In contrast in the bacon version the first command produces the category and the second the the (correctly named) package list. Interestingly, the problem with the underscore in the names between name and version does not affect PPM function! The only problem is that it reports installed packages as failed (since they have a different name). We could hack around it but better have find_cat functioning properly.

[mavrothal]

It would appear is just a typo in the code. This patch fixes it

--- a/woof-code/support/find_cat.c
+++ b/woof-code/support/find_cat.c
@@ -256,7 +256,7 @@

 print:
        fputs(fields[0], stdout);
-       putc('_', stdout);
+       putc('-', stdout);
        fputs(fields[1], stdout);
        putc('|', stdout);

[mavrothal]

Fixed in 7e807c6

[mavrothal]

NO. The names are fixed but now package installation does not work. Reverted 7e807c6 More is needed

[mavrothal]

Looking at it a bit further, it would appear that the above patch in the find_cat results in adding at the end (last field) of each line in the package database: install_re#<NUMBER> <database_name> which then messes up installpgk.sh

[01micko]

This is purely cosmetic. Works fine as is.

However, just notice the pattern. The underscore is sneaking in in field[0] only when it delimits the generic name and the version... (an ubuntu/debian naming quirk).

[mavrothal]

I do not know about cosmetic but if you run PPM will report that all the packages that were downloaded and installed failed to download! Also if someone is looking for a package in the web will search with the wrong name. So there is more to it than looks. It would probably need a deb specific condition and a generic one

[dimkr]

I'll compare the output of both find_cat implementations - I did that when I rewrote it, maybe I missed something.

[dimkr]

Found some weird anomaly that also explains #622 - package categories are messed up, so package lookup picks the wrong packages.

[mavrothal]

Tried the latest that includes 3db80c7 and d4b265e in slacko64 and is actually worse than the original as it produces a non-functinal datbase both with the underscore in the name and the install_re#<NUMBER> <database_name> at the end.

[01micko]

it produces a non-functinal datbase both with the underscore in the name and the install_re# at the end.

I'm not sure I understand what you mean.

I'm just running a test with this patch:

+++ find_cat.c  2015-11-01 06:20:01.723372094 +1000
@@ -263,7 +263,13 @@ int main(int argc, char *argv[])

 print:
        fputs(fields[0], stdout);
-       putc('_', stdout);
+       if ((0 == strcmp(fields[9], "ubuntu")) ||
+           (0 == strcmp(fields[9], "trisquel")) ||
+           (0 == strcmp(fields[9], "debian")) ||
+           (0 == strcmp(fields[9], "raspbian")))
+           putc('_', stdout);
+       else
+           putc('-', stdout);
        fputs(fields[1], stdout);
        putc('|', stdout);

@@ -295,4 +301,4 @@ bail:
    free(iobuf);

    return EXIT_SUCCESS;

Database looks ok to me, but maybe I miss something.

BTW, I've no idea if this works for arch, mageia, scientific.

@dimkr , you missed 'devuan' in the test :wink:

[dimkr]

I think you missed something. The original debdb2pup does this unconditionally:

    DB_pkgname$=CONCAT$(DB_nameonly$,"_",DB_version$)

So, this must be the correct behavior for all distros (unless 0setup/1download/whatever replaces the underscores) and that's what both the new dbdb2pupdb and find_cat do.

@dimkr , you missed 'devuan' in the test :wink:

Thanks, I'll fix this.

[01micko]

debdb2pup isn't used by slacware packages. It's all done with sed and friends.

Anyway, after using my patched find_cat on a fresh (but a couple weeks old - so hasn't latest petget) slacko64 install here is my /root/.packages/user-installed-packages after installing QupZilla (which is in slacko64-official repo - but requires stuff from slackware64-official, slackware64-patches, and slackware64-salix)

QupZilla-1.8.6-x86_64_slk600|QupZilla|1.8.6-x86_64_slk600||Internet|7632||QupZilla-1.8.6-x86_64_slk600.pet|+qt|Open source qt-webkit based web browser|slackware64|14.1||
gst-plugins-base-0.10.36-x86_64-2.txz|gst-plugins-base|0.10.36-x86_64-2.txz|gst-plugins-base-0.10.36-x86_64-2|BuildingBlock|10970K|slackware64/l|gst-plugins-base-0.10.36-x86_64-2.txz|+gstreamer,+util-linux,+xz,+zlib,+cdparanoia,+harfbuzz,+libogg,+libpng,+libtheora,+libvisual,+libvorbis|base set of GStreamer plugins|slackware64|14.1|
gstreamer-0.10.36-x86_64-2.txz|gstreamer|0.10.36-x86_64-2.txz|gstreamer-0.10.36-x86_64-2|Multimedia|13300K|slackware64/l|gstreamer-0.10.36-x86_64-2.txz|+glib2,+libffi,+libxml2,+xz,+zlib|streaming multimedia framework|slackware64|14.1|
libaio-0.3.109-x86_64-1.txz|libaio|0.3.109-x86_64-1.txz|libaio-0.3.109-x86_64-1|BuildingBlock|60K|slackware64/l|libaio-0.3.109-x86_64-1.txz||asynchronous IO library|slackware64|14.1|
libiodbc-3.52.7-x86_64-2.txz|libiodbc|3.52.7-x86_64-2.txz|libiodbc-3.52.7-x86_64-2|Personal;database|1110K|slackware64/l|libiodbc-3.52.7-x86_64-2.txz|+atk,+bzip2,+cairo,+cxxlibs,+gcc-g++,+expat,+fontconfig,+freetype,+gcc,+gdk-pixbuf2,+glib2,+gtk+2,+harfbuzz,+icu4c,+libX11,+libXau,+libXcomposite,+libXcursor,+libXdamage,+libXdmcp,+libXext,+libXfixes,+libXi,+libXinerama,+libXrandr,+libXrender,+libffi,+libpng,+libxcb,+pango,+pixman,+zlib|Independent Open DataBase Connectivity|slackware64|14.1|
libvisual-0.4.0-x86_64-3.txz|libvisual|0.4.0-x86_64-3.txz|libvisual-0.4.0-x86_64-3|BuildingBlock|570K|slackware64/l|libvisual-0.4.0-x86_64-3.txz||audio visualization library|slackware64|14.1|
mariadb-5.5.32-x86_64-1.txz|mariadb|5.5.32-x86_64-1.txz|mariadb-5.5.32-x86_64-1|Network|158410K|slackware64/ap|mariadb-5.5.32-x86_64-1.txz|+cxxlibs,+gcc-g++,+gcc,+libaio,+ncurses,+openssl-solibs,+openssl,+zlib|Drop in replacement for the MySQL Database Server|slackware64|14.1|
qt-4.8.5-x86_64-2.txz|qt|4.8.5-x86_64-2.txz|qt-4.8.5-x86_64-2|Graphic|121330K|slackware64/l|qt-4.8.5-x86_64-2.txz|+alsa-lib,+bzip2,+cxxlibs,+gcc-g++,+expat,+fontconfig,+freetype,+gcc,+glib2,+gst-plugins-base,+gstreamer,+lcms,+libICE,+libSM,+libX11,+libXau,+libXdamage,+libXdmcp,+libXext,+libXfixes,+libXrender,+libXxf86vm,+libdrm,+libffi,+libiodbc,+libjpeg,+libmng,+libpng,+libtiff,+libxcb,+libxml2,+mariadb,+mesa,+openssl-solibs,+openssl,+sqlite,+util-linux,+xz,+zlib|a multi platform C++ graphical user interface toolkit|slackware64|14.1|
libiodbc-3.52.8-x86_64-1_slack14.1.txz|libiodbc|3.52.8-x86_64-1_slack14.1.txz|libiodbc-3.52.8-x86_64-1_slack14.1|Personal;database|1130K|patches/packages|libiodbc-3.52.8-x86_64-1_slack14.1.txz||Independent Open DataBase Connectivity|slackware64|14.1|
mariadb-5.5.43-x86_64-1_slack14.1.txz|mariadb|5.5.43-x86_64-1_slack14.1.txz|mariadb-5.5.43-x86_64-1_slack14.1|Network|160810K|patches/packages|mariadb-5.5.43-x86_64-1_slack14.1.txz||Drop in replacement for the MySQL Database Server|slackware64|14.1|
qt-4.8.7-x86_64-1_slack14.1.txz|qt|4.8.7-x86_64-1_slack14.1.txz|qt-4.8.7-x86_64-1_slack14.1|Graphic|122380K|patches/packages|qt-4.8.7-x86_64-1_slack14.1.txz||a multi platform C++ graphical user interface toolkit|slackware64|14.1|

[dimkr]

debdb2pup isn't used by slacware packages. It's all done with sed and friends.

Yes, but what I was trying to say is that this format (the underscore) is common to all distros, unless something (say, 0setup) behaves differently.

[01micko]

It isn't. Slackware delimits generic name and version with a hyphen.

[01micko]

Below is an snippet from PACKAGES.TXT , in this case *patches from slackware - note there is no dependency info. That is only in databases grabbed from Salix, which includes the main slackware PACKAGES.TXT dep info.

PACKAGE NAME:  btrfs-progs-20150213-x86_64-1.txz
PACKAGE LOCATION:  ./patches/packages
PACKAGE SIZE (compressed):  436 K
PACKAGE SIZE (uncompressed):  3120 K
PACKAGE DESCRIPTION:
btrfs-progs: btrfs-progs (Btrfs filesystem utilities)
btrfs-progs:
btrfs-progs: Btrfs is a new copy on write filesystem for Linux aimed at implementing
btrfs-progs: advanced features while focusing on fault tolerance, repair and easy
btrfs-progs: administration.  Initially developed by Oracle, Btrfs is licensed under
btrfs-progs: the GPL and open for contribution from anyone.
btrfs-progs:
btrfs-progs: Btrfs homepage:  http://btrfs.wiki.kernel.org
btrfs-progs:

PACKAGE NAME:  ca-certificates-20150426-noarch-2_slack14.1.txz
PACKAGE LOCATION:  ./patches/packages
PACKAGE SIZE (compressed):  164 K
PACKAGE SIZE (uncompressed):  460 K
PACKAGE DESCRIPTION:
ca-certificates: ca-certificates (PEM Files of CA Certificates)
ca-certificates:
ca-certificates: This package includes PEM files of CA certificates to allow SSL-based
ca-certificates: applications to check for the authenticity of SSL connections.
ca-certificates:
ca-certificates: Homepage: http://packages.qa.debian.org/c/ca-certificates.html
ca-certificates:

I will (at some point) attempt to write a slcakdb2pup utility so that we can avoid all the slow shell stuff, but it will be in due course. I'll have to read up how this crc32 thing works. :smile:

[dimkr]

Try b1b038e, should fix the problem.

It changes findcat so it doesn't assume the delimiter is - it just passes the one used in the temporary package list. I think that originally, the use of a hyphen was a mistake on BK's part, but now we have to deal with this, for the sake of backward compatibility :disappointed:

[01micko]

Close, but category appears to go to wrong field.

EG:

Original from pup_ro2

yptools-2.14-x86_64-3_slack14.1.txz|yptools|2.14-x86_64-3_slack14.1.txz|yptools-2.14-x86_64-3_slack14.1|BuildingBlock|850K|patches/packages|yptools-2.14-x86_64-3_slack14.1.txz||NIS servers and clients|slackware64|14.1||

with very latest find_cat

yptools-2.14-x86_64-3_slack14.1.txz|yptools|2.14-x86_64-3_slack14.1.txz|BuildingBlock||850K|patches/packages|yptools-2.14-x86_64-3_slack14.1.txz||NIS servers and clients|slackware64|14.1|

[01micko]

changing all fields[3] to fields[4] fixes it and you know why :wink:

[dimkr]

Embarrassing. Fixed.

[01micko]

Looks good. I'll build and commit the 64 version.

[mavrothal]

Not here, I'm afraid. tried both the compiled binary and compiled from source in slacko6.0.8.1 a) "Official" database is is severely truncated (42 entries) b) what ever is there fails to download because it tried to download (c) below. c) all database entries have as last field install_re#130126 Packages-slackware64-14.1-official (or patches). Do you guys see any of these or is only me.

[01micko]

I'm gonna try an 0setup now.

[01micko]

EDIT: no ... was using find_cat on my system.. so yes, I do see sever truncation but no garbage on the end.

Ok, there is one bug I see. Sometime BuildingBlock gets missed.

#  cat Packages-slackware64-14.1-official|cut -d '|' -f5|egrep '^[a-z]|^[A-Z]|^[0-9]'|wc -l
# 793
#  cat Packages-slackware64-14.1-official|wc -l
# 1244

So, if 451 BuildingBlocks goes missing we have a problem!

[01micko]

Maybe if feilds[4] is NULL we force it to BuildingBlock? I know @dimkr is probably sleeping now so I'll take a look until he can do a real fix.

I definitely am not seeing the garbage on the end that you are seeing :confused:

[01micko]

Chokes on audacious. IDK why, but .. https://github.com/puppylinux-woof-CE/woof-CE/blob/6eb8db85d9b4cd11ee1c2a571f340e7bfb6f7e82/woof-code/support/find_cat.c works with my stupid patch that filters deb based and if it's not deb based then uses hyphen.

Yes I know it is wrong a work around but I need to push out some builds so I'll commit it until it can be fixed properly.. BUT, I'll give it a test run in tahr or something to make sure it doesn't mess that up.

readv(3, [{"", 0}, {"_64-1||50K|slackware64/x|appres-"..., 1024}], 2) = 1024
readv(3, [{"", 0}, {"l2,+lzo,+nettle,+pcre,+phonon,+q"..., 1024}], 2) = 1024
writev(1, [{"|slackware64/a|apmd-3.2.2-x86_64"..., 941}, {"+acl,+attica,+attr,+bzip2,+cxxli"..., 400}], 2) = 1341
readv(3, [{"", 0}, {" bridge to ATK|slackware64|14.1|"..., 1024}], 2) = 1024
writev(1, [{"|KDE archiver tool|slackware64|1"..., 1023}, {"14.1|\n", 6}], 2) = 1029
readv(3, [{"", 0}, {"K|slackware64/xap|audacious-3.3."..., 1024}], 2) = 1024
writev(1, [{"at-spi2-core-2.8.0-x86_64-1.txz|"..., 1013}, {"audacious-3.3.4-x86_64-1", 24}], 2) = 1037
readv(3, [{"", 0}, {"n,+libnotify,+libogg,+libpng,+li"..., 1024}], 2) = 1024
writev(1, [{"|Multimedia;mediaplayer|1350K|sl"..., 695}, {"+alsa-lib,+at-spi2-atk,+at-spi2-"..., 598}], 2) = 1293
--- SIGSEGV {si_signo=SIGSEGV, si_code=SEGV_MAPERR, si_addr=0x7f5e331a6000} ---
+++ killed by SIGSEGV +++
Segmentation fault

[mavrothal]

Tried the latest and it works on a fresh run of slacko64. However, I noticed the database entries have one field less I do not remember what the last field is for right now but I hope it will not generate any issues in some PPM function. If you want to release something today you might also consider the bacon version of find_cat that still works fine. Is a pretty isolated change and should not affect anything else. We can go back to the C version when is ironed out.

[mavrothal]

It would also appear that the new version fails to get the categories right putting too many things as BuildingBlocks (amarok, amp and other sound related apps). This last field however suppose to be unused for the moment, so would be OK

[01micko]

Yeah, it's just temporary but if PPM works ok I might release some isos.

[01micko]

Maybe I will go back to the bacon version.. for now. You wanna commit it? I dont :grin:

[mavrothal]

I think that we can wait a day for Igu before committing anything. That part of the word is just waking up ;) BTW I run your new version in Tahr64 and now 2571 more packages (from the total of 8800 packages in main) show as BuildingBlock! So for an immediate release the bacon version looks safer.

[01micko]

I'll wait it out. I have other stuff already in place and will tweak that in the mean time.

[dimkr]

Grrr ... I hate parsers so much, especially they handle a silly format! It's amazing how sensitive and ugly the entire 0setup/findpkgs/find_cat combo really is.

I'll take yet another look at this crap when I get home.

[01micko]

@dimkr I feel your frustration! Please, don't rush. There is no imperative reason that I have to push some isos. I wish I knew C (programming in general) better to be of some use!

For the time being, we are stuck with 0setup/findpkgs/find_cat. In the future perhaps we can go a bit closer to freedesktop standards and avoid this mess completely.

[mavrothal]

Revisited things a bit. Before the current changes started, both "package name" (field 1) and "full file name" (field 8) had an underscore between name and version in Ubuntu. But in Slackware, field 1 had an underscore and field 8 a dash. This would make PPM to report failure while packages were downloaded fine. The little change in 7e807c6 fixed Slackware and now in both "package name" (field 1) and "full file name" (field 8) name and version are separated with a dash. However in Ubuntu, package name is correct (with a dash) but full file name still has an underscore, so now PPM reports failure in Ubuntu @:-{ If we can do something about the latter we would be OK. Hopefully these can help someone that understands C.

[dimkr]

The little change in 7e807c6 fixed Slackware and now in both "package name" (field 1) and "full file name" (field 8) name and version are separated with a dash

The right thing to do is the change I introduced in b1b038e71d64b4e92b29e80acbff78cf7897b69b - the delimiter used by 0setup (either - or _) should be preserved when the package list passes through find_cat.

[mavrothal]

b1b038e is fine when it comes to having fields 1 and 8 consistent. However, it chokes on autacious-plugings-3.3.4-x86_64-1. Maybe some tolerance or error reporting could be added.

[mavrothal]

Looking, where it chokes is usually in packages with a lot (60+ dependencies). After audacious plugins is calligra and then kde-runtime and then kde-worksapce and then... However I'm not sure is the number of dependencies since amarok which has 70+ is processed OK but nothing after that with more than 60 dependencies

[mavrothal]

Without really knowing what I'm doing (...) I took b1b038e and just double BUF_SIZE, NAME_SIZE, DESC_SIZE and MAX_KWDS in find_cat.c (lines 20-25). Now it goes through the Slackware database without any problem! You may want to adjust only the one really needed to go through the long dependency lists and commit.

[mavrothal]

Man is frustrating... Increasing sizes makes it segfault in Ubuntu in the databases with more than 1000 entries (main, universe. Not multiverse). I give up.

[dimkr]

Try now, I fixed an off-by-one error. This has nothing to do with buffer sizes, they're OK.

[mavrothal]

Pretty good. Works in slacko and tahr both 32 and 64bit. In slacko has a little problem in that the "Setup" category is empty and the 'Utilities" category now was language support files development files etc that should go in system or dev. So looks like that "Setup" is merged into "Utilities". This is not the case in Ubuntu.

[mavrothal]

Oooops forget the above. Slacko never had anything under setup.