lrz incompatible with --linebuffer --compress-program in GNU Parallel

[ole-tange]

Current git version fails on: parallel --linebuffer --compress-program /usr/local/bin/lrz seq ::: 100000

Version 89d7b33e6a6450eed326b40084b547d42bad333f works.

--linebuffer sets the file handle as non-blocking, so lrz somehow starts to get picky about that at version 4893e869e3fc36c65123ce8fedafeb82cba745a4.

It seems this is the culprit:

@@ -1632,6 +1632,10 @@ fill_another:
        c_len = le64toh(c_len);
        u_len = le64toh(u_len);
        last_head = le64toh(last_head);
+ if (unlikely(c_len < 1 || u_len < 1 || last_head < 0)) {
+         fatal_return(("Invalid data compressed len %lld uncompressed %lld last_head %lld\n",
+                      c_len, u_len, last_head), -1);
+ }
        print_maxverbose("Fill_buffer stream %d c_len %lld u_len %lld last_head %lld\n", streamno, c_len, u_len, last_head);

        padded_len = MAX(c_len, MIN_SIZE);

Remove that, and it works again.

[pete4abw]

See https://github.com/ckolivas/lrzip/issues/123 and https://github.com/ckolivas/lrzip/issues/122. I believe there is a problem when either multiple files are in queue or very small files or deeply compressed ones with multi thread. The issue with stream.c that you correctly point out has a proposed patch but I highly recommend not deleting the test all together. https://github.com/ckolivas/lrzip/files/3717564/smallfile.patch.gz. It also may turn out that there is a memory issue with no free after use with multiple files.

[ole-tange]

Seems to be fixed in 597be1ff.

[ckolivas]

It appears that https://github.com/ckolivas/lrzip/pull/157 was responsible for this bug and reverting it was the fix. Updated code to address that issue would be required to remerge that change.