Write error: Inappropriate ioctl for device when decompressing

[bczhc]

When the block size is 1MiB, this happens:

~ ❯ cat /dev/random | head -c1048576 | bzip3 -b1 | bzip3 -d | wc -c
Write error: Inappropriate ioctl for device
1048576

Exit status indicator: ✔ PIPE|0|0|1|0, meaning bzip3 -d exits with status code 1, with the Write error: Inappropriate ioctl for device error.

But after changing the block size to a higher one like 2MiB, it doesn't happen.

~ ❯ cat /dev/random | head -c1048576 | bzip3 -b2 | bzip3 -d | wc -c
1048576
✔ PIPE|0|0|0|0

Also, when using valgrind on bzip3 with the 1M (2M it's okay) block size, it reports so many invalid reads and writes. Are these memory errors a serious problem?

~ ❯ cat /dev/random | head -c1048576 | valgrind bzip3 -b1 > /dev/null                                       14:54:27
==1954741== Memcheck, a memory error detector
==1954741== Copyright (C) 2002-2022, and GNU GPL'd, by Julian Seward et al.
==1954741== Using Valgrind-3.19.0 and LibVEX; rerun with -h for copyright info
==1954741== Command: bzip3 -b1
==1954741== 
==1954741== Invalid write of size 4
==1954741==    at 0x4892ECE: bz3_encode_block (in /usr/lib/libbzip3.so.0.0.0)
==1954741==    by 0x10ADF0: ??? (in /usr/bin/bzip3)
==1954741==    by 0x10A7F0: ??? (in /usr/bin/bzip3)
==1954741==    by 0x48C528F: (below main) (libc_start_call_main.h:58)
==1954741==  Address 0x528c0bc is 4,194,428 bytes inside a block of size 4,198,304 in arena "client"
==1954741== 
==1954741== Invalid write of size 4
==1954741==    at 0x4892F05: bz3_encode_block (in /usr/lib/libbzip3.so.0.0.0)
==1954741==    by 0x10ADF0: ??? (in /usr/bin/bzip3)
==1954741==    by 0x10A7F0: ??? (in /usr/bin/bzip3)
==1954741==    by 0x48C528F: (below main) (libc_start_call_main.h:58)
==1954741==  Address 0x528c0bc is 4,194,428 bytes inside a block of size 4,198,304 in arena "client"
==1954741== 
==1954741== Invalid write of size 4
==1954741==    at 0x4892F3F: bz3_encode_block (in /usr/lib/libbzip3.so.0.0.0)
==1954741==    by 0x10ADF0: ??? (in /usr/bin/bzip3)
==1954741==    by 0x10A7F0: ??? (in /usr/bin/bzip3)
==1954741==    by 0x48C528F: (below main) (libc_start_call_main.h:58)
==1954741==  Address 0x528c0bc is 4,194,428 bytes inside a block of size 4,198,304 in arena "client"
...
==1954741== HEAP SUMMARY:
==1954741==     in use at exit: 0 bytes in 0 blocks
==1954741==   total heap usage: 9 allocs, 9 frees, 7,551,599 bytes allocated
==1954741== 
==1954741== All heap blocks were freed -- no leaks are possible
==1954741== 
==1954741== For lists of detected and suppressed errors, rerun with: -s
==1954741== ERROR SUMMARY: 932 errors from 155 contexts (suppressed: 0 from 0)

[kspalaiologos]

Hi! The first error is a regression introduced in #55 that is now fixed on master. To quote about the confusing error message:

In some cases, this message will occur even when no ioctl has been issued by the program. This is due to the way the isatty() library routine works. The error code errno is only set when a system call fails. One of the first system calls made by the C standard I/O library is in an isatty() call used to determine if the program is being run interactively by a human (in which case isatty() will succeed and the library will write its output a line at a time so the user sees a regular flow of text) or as part of a pipeline (in which case it writes a block at a time for efficiency). If a library routine fails for some reason unrelated to a system call (for example, because a user name wasn't found in the password file) and a naïve programmer blindly calls the normal error reporting routine perror() on every failure, the leftover ENOTTY will result in an utterly inappropriate "Not a typewriter" (or "Not a teletype", or "Inappropriate ioctl for device") being delivered to the user. For many years the UNIX mail program sendmail contained this bug: when mail was delivered from another system, the mail program was being run non-interactively. If the destination address was local, but referred to a user name not found in the local password file, the message sent back to the originator of the email was the announcement that the person they were attempting to communicate with was not a typewriter.

As for the memory errors, if you recompile with -g3 you discover that they are caused by libsais:

==54331== Invalid read of size 4
==54331==    at 0x10B86A: libsais_bwt_copy_8u (libsais.h:4037)
==54331==    by 0x115621: libsais_bwt (libsais.h:4117)
==54331==    by 0x115621: bz3_encode_block (libbz3.c:574)
==54331==    by 0x117696: process.constprop.0 (main.c:224)
==54331==    by 0x10956D: main (main.c:695)
==54331==  Address 0x524c0b4 is 4,194,420 bytes inside a block of size 4,198,304 in arena "client"
==54331==
==54331== Invalid read of size 4
==54331==    at 0x10B87C: libsais_bwt_copy_8u (libsais.h:4037)
==54331==    by 0x115621: libsais_bwt (libsais.h:4117)
==54331==    by 0x115621: bz3_encode_block (libbz3.c:574)
==54331==    by 0x117696: process.constprop.0 (main.c:224)
==54331==    by 0x10956D: main (main.c:695)
==54331==  Address 0x524c0b8 is 4,194,424 bytes inside a block of size 4,198,304 in arena "client"
==54331==
==54331== Invalid read of size 4
==54331==    at 0x10B88E: libsais_bwt_copy_8u (libsais.h:4037)
==54331==    by 0x115621: libsais_bwt (libsais.h:4117)
==54331==    by 0x115621: bz3_encode_block (libbz3.c:574)
==54331==    by 0x117696: process.constprop.0 (main.c:224)
==54331==    by 0x10956D: main (main.c:695)

There is nothing I can do about this at present, because the maintainer refuses to fix these memory errors even though I have an evidence that invalid input will lead to a segfault (see #59).

If you feel like trying to make libsais memory safe, your effort will be greatly appreciated.

[kspalaiologos]

Future problems related to libsais are to be discussed in #59.

[bczhc]

I see... Thanks for the info!