error building 4.3.3 (bz3-related)

[dave-theunsub]

Got this using build.sh with 4.3.3:

*** SINGLE-THREADED TESTS ***

cd ./tests && env UGREP_ABS_PATH=/tmp/ugrep-4.3.3/./bin/ugrep CONFIGH_ABS_PATH=/tmp/ugrep-4.3.3/./config.h ./verify.sh -J1
ugrep 4.3.3 x86_64-pc-linux-gnu +avx2; -P:pcre2jit; -z:zlib,bzip2,lzma,lz4,zstd,brotli,bzip3
Have libpcre2?  yes (recommended)
Have libz?      yes (recommended)
Have libbz2?    yes (recommended)
Have liblzma?   yes (recommended)
Have liblz4?    yes (optional)
Have libzstd?   yes (optional)
Have libbrotli? yes (optional)
Have libbzip3?  yes (optional)
---- snip ----
ugrep: warning: cannot decompress archive.tar.bz3: an error was detected in the bzip3 compressed data
--- -   2023-11-16 02:44:03.778760222 -0600
+++ out/archive.tar.bz3.out     2023-11-15 11:18:04.000000000 -0600
@@ -1 +1,8 @@
-0
+archive.tar.bz3{Hello.bat}:1
+archive.tar.bz3{Hello.class}:1
+archive.tar.bz3{Hello.java}:2
+archive.tar.bz3{Hello.pdf}:1
+archive.tar.bz3{Hello.sh}:1
+archive.tar.bz3{Hello.txt}:1
+archive.tar.bz3{empty.txt}:0
+archive.tar.bz3{emptyline.txt}:0
Error: ugrep --sort -z -c Hello archive.tar.bz3 failed
make: *** [Makefile:978: test] Error 1
Testing failed, please open an issue at:
https://github.com/Genivia/ugrep/issues

Let me know if you need other information.

Dave M

[genivia-inc]

This is a bzip3 internal error, perhaps because the bzip library that ugrep linked with was old or had some other issue. I've tested the latest bzip3 which worked.

Including bzip3 in ugrep is entirely optional (only need-to-use basis) and is placed in the third tier, see the v4.3.3. release notes. This is decided to prevent a future proliferation of assumed/required dependencies.

I would strongly suggest ./build.sh --without-bzip3 to proceed.

[genivia-inc]

One thing I could do for the next update is perhaps make the default ./build.sh not link with bzip3 by default whenever a bzip3 library is found, but only do this on demand with the command:

./build.sh --with-bzip3

This makes the third tier placement of such dependencies more practical.

[genivia-inc]

I've committed an updated configure script to build ugrep with ./build.sh without bzip3 by default, and only include the bzip3 library when ./build.sh --with-bzip3 is specified, which runs ./configure --with-bzip3.

[kspalaiologos]

May we know the bz3 version?

[genivia-inc]

@dave-theunsub please let us know what bzip3 version is installed on your system. Perhaps we can include an installation version check in ugrep to avoid this problem in the future.

[dave-theunsub]

Sure. I'm on Fedora 38:

bzip3, bzip3-devel, bzip3-libs 1.3.1-1

[kspalaiologos]

ugrep: warning: cannot decompress archive.tar.bz3: an error was detected in the bzip3 compressed data

This is overall not very helpful. Why not use bz3_strerror to get a more descriptive error message?

[genivia-inc]

@kspalaiologos This appears to be a bzip3 problem, because the bzip3 library returns some error. So this is probably best to pick this up on your end to verify and update bzip3 if necessary. If this is somehow a ugrep error then I stand corrected and I'll be happy to fix it on my end.

Ugrep does not output specific compression library errors because it is not useful for ugrep users to deal with any of those. I'd rather keep the error messages simple.

[kspalaiologos]

Well, I personally can't reproduce and unless I am provided with more information I consider this issue as invalid. If the test archive made using bzip3 CLI decompresses using bzip3 CLI, then this can not be an issue with bzip3 itself.

[dave-theunsub]

4.3.4 builds just fine, thanks. But I am curious - am I the only one for whom 4.3.3 failed during the build? No one else had an issue? I mean, it made it through tests, right?

[kspalaiologos]

I am using bzip3 v1.3.2 from Debian Sid and it worked fine.

[dave-theunsub]

I upgraded to Fedora 39, which also has bzip3 v1.3.2, and it builds just fine.

[genivia-inc]

I'm just being overly cautious for the moment and will not yet build ugrep by default with bzip3 and instead people should use ./build.sh --with-bzip3. I like bzip3, but on the other hand, we really should avoid any build problems. I worry that people simply forget about installing and using ugrep if there is a build error.

Package maintainers are free to decide what dependencies they deem OK with ugrep. I have no control over that, besides offering suggestions in the release notes.

I do my best to make ugrep perfect. But there is always a nonzero chance I may have overlooked something despite source code reviewing and testing. I have to take a bit of risk to add something new that doesn't break something, which fortunately almost never happens with due diligence.

I hope that people keep sharing their experiences with ugrep by opening issues if something may seem amiss or if something could/should be improved. That's how everyone benefits.

[kspalaiologos]

Of course, I understand. Thanks!