xz 5.2.1: builds but doesn't work

[gitfoxi]

A step I had a hard time with in making the Standalone was that xz builds but only does this:

xz: Error creating a pipe: Function not implemented

I was able to continue the process by using my system's xz. I couldn't see a way to blacklist the broken formula so I kind of had to manually delete it to keep other builders from using it.

[gitfoxi]

Regarding #290 I also see that the executable permission is not set

[cavtools@vm lib]$ ls -l
total 436
-r--r--r-- 1 cavtools cavtools 280220 Apr 13 20:11 liblzma.a
lrwxrwxrwx 1 cavtools cavtools     16 Apr 13 20:11 liblzma.so -> liblzma.so.5.2.1
lrwxrwxrwx 1 cavtools cavtools     16 Apr 13 20:11 liblzma.so.5 -> liblzma.so.5.2.1
-r--r--r-- 1 cavtools cavtools 151544 Apr 13 20:11 liblzma.so.5.2.1
drwxr-xr-x 2 cavtools cavtools   4096 Apr 13 20:11 pkgconfig

Setting it does not solve my problem though.

[gitfoxi]

Here's some strace

open("/home/cavtools/.linuxbrew/lib/libc.so.6", O_RDONLY|0x80000 /* O_??? */) = 3
read(3, "\177ELF\2\1\1\3\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\266\2\2\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0555, st_size=1856344, ...}) = 0
mmap(NULL, 3655392, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x2b1644554000
mprotect(0x2b16446c8000, 2093056, PROT_NONE) = 0
mmap(0x2b16448c7000, 24576, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x173000) = 0x2b16448c7000
mmap(0x2b16448cd000, 14048, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x2b16448cd000
close(3)                                = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x2b16448d1000
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x2b16448d2000
arch_prctl(ARCH_SET_FS, 0x2b16448d22c0) = 0
mprotect(0x2b16448c7000, 16384, PROT_READ) = 0
mprotect(0x2b164454e000, 4096, PROT_READ) = 0
mprotect(0x2b1644114000, 4096, PROT_READ) = 0
set_tid_address(0x2b16448d2590)         = 16698
set_robust_list(0x2b16448d25a0, 0x18)   = 0
futex(0x7fff319a23cc, FUTEX_WAKE_PRIVATE, 1) = 0
futex(0x7fff319a23cc, 0x189 /* FUTEX_??? */, 1, NULL, 2b16448d22c0) = -1 ENOSYS (Function not implemented)
rt_sigaction(SIGRTMIN, {0x2b164433cf24, [], SA_RESTORER|SA_SIGINFO, 0x2b1644345a40}, NULL, 8) = 0
rt_sigaction(SIGRT_1, {0x2b164433cf9e, [], SA_RESTORER|SA_RESTART|SA_SIGINFO, 0x2b1644345a40}, NULL, 8) = 0
rt_sigprocmask(SIG_UNBLOCK, [RTMIN RT_1], NULL, 8) = 0
getrlimit(RLIMIT_STACK, {rlim_cur=10240*1024, rlim_max=RLIM_INFINITY}) = 0
fcntl(0, F_GETFD)                       = 0
fcntl(1, F_GETFD)                       = 0
fcntl(2, F_GETFD)                       = 0
geteuid()                               = 1002
syscall_293(0x6133c8, 0x800, 0, 0x2b16441164b0, 0x4095d8, 0x2b1643f04cd4, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0) = -1 (errno 38)
write(2, "xz: ", 4xz: )                     = 4
write(2, "Error creating a pipe: Function "..., 47Error creating a pipe: Function not implemented) = 47
write(2, "\n", 1
)                       = 1
exit_group(1)                           = ?

You can see my system strace can't really decode the calls in the Standalone libc very well. Maybe tools like strace could be included in the Standalone too? Probably trying to do something my kernel doesn't support but hard to tell. xz's config tests should be able to figure out the system capabilities but a linuxbrew/centos5 frankenstein may be past the limits of what it expects.

[gitfoxi]

All I can say is that it's a call to pipe2 which was a very new feature when kernel 2.6.18 was released in September 2006 and has seen a lot of patches since.

https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/log/fs/pipe.c?ofs=50

[sjackman]

What version of the Linux kernel are you running? You can try brew edit glibc and adding a --enable-kernel=x.y.z to the configure options. http://www.gnu.org/software/libc/manual/html_node/Configuring-and-compiling.html

[gitfoxi]

I see. It doesn't actually do the check in configure. Needs some more things installed and probably to call one of the autotools I know so little about.

[gitfoxi]

Thanks for enable-kernel. I will try it. I'm surprised the glibc build isn't smart enough to detect the running kernel.

[gitfoxi]

enable-kernel seems like a great idea and should go in the wiki. However, in this case it did not help.

[gitfoxi]

Another idea is to install autoconf, automake and gettext so you can run ./autogen.sh. This also failed to help.

[gitfoxi]

My next idea is to savagely expunge the pipe2 check from the configure file. I don't think this kind of patch can integrate with a build system like homebrew.

It works!

The offending code in configure is:

# These are nice to have but not mandatory.
for ac_func in posix_fadvise pipe2
do :
  as_ac_var=`$as_echo "ac_cv_func_$ac_func" | $as_tr_sh`
ac_fn_c_check_func "$LINENO" "$ac_func" "$as_ac_var"
if eval test \"x\$"$as_ac_var"\" = x"yes"; then :
  cat >>confdefs.h <<_ACEOF
#define `$as_echo "HAVE_$ac_func" | $as_tr_cpp` 1
_ACEOF

fi
done

The bug is upstream. Either in xz or autotools or both.

[sjackman]

Cool! That's really good to know. Red Hat / CentOS / Fedora have been known to occasionally introduce non-binary compatible patches into the kernel. That may be a possibility here. You can try replacing the Linuxbrew version of linux-headers with your particular version.

[gitfoxi]

Replacing linux-headers turns out to not be what you want:

What version of the Linux kernel headers should be used?

The headers from the most recent Linux kernel should be used. The headers used while compiling the GNU C library and the kernel binary used when using the library do not need to match. The GNU C library runs without problems on kernels that are older than the kernel headers used. The other way round (compiling the GNU C library with old kernel headers and running on a recent kernel) does not necessarily work as expected. For example you can't use new kernel features if you used old kernel headers to compile the GNU C library.

Even if you are using an older kernel on your machine, we recommend you compile GNU libc with the most current kernel headers. That way you won't have to recompile libc if you ever upgrade to a newer kernel. To tell libc which headers to use, give configure the --with-headers switch (e.g. --with-headers=/usr/src/linux-3.3/include).

To install Linux kernel headers, run make header_install in the kernel source tree. This is described in the kernel documentation.

https://sourceware.org/glibc/wiki/FAQ#What_version_of_the_Linux_kernel_headers_should_be_used.3F

[gitfoxi]

Are you sure this is closed? Yes, the bug is upstream, but it needs to be reported (I'm not sure to whom) and meanwhile patched in homebrew.

[sjackman]

Which version of the Linux kernel are you using? Please report uname -a

This issue is closed in that I don't really have the time myself to troubleshoot it, but I'm glad that you found a workaround. If you find a solution, please open a pull request.

[gitfoxi]

Linux vm 2.6.18-404.el5 #1 SMP Tue Apr 7 12:42:54 EDT 2015 x86_64 x86_64 x86_64 GNU/Linux

I can patch the formula to fix it if you could suggest how I can ask homebrew to detect a specific kernel version. (I don't know ruby, but was able to muddle through the instructions on how to add a patch.)

[sjackman]

2.6.18 should work (though it seems not to). The servers I use at work are running CentOS 5.10

$ echo hello >hi
$ tar cf hi.tar hi
$ xz hi.tar
$ tar xfO hi.tar.xz
hello
$ uname -a
Linux xhost11 2.6.18-371.1.2.el5 #1 SMP Tue Oct 22 12:51:53 EDT 2013 x86_64 GNU/Linux
$ tar --version
tar (GNU tar) 1.28
$ xz --version
xz (XZ Utils) 5.2.0
liblzma 5.2.0

[gitfoxi]

Maybe it's a very recently introduced bug? I brewed xz 5.2.1.

We are really in the same boat having to live with old Linux until Redhat says it can die. Thanks for taking charge and doing something about it.

[sjackman]

Yes! You're right! Sorry that I didn't catch this issue earlier. It looks like an upstream xz bug then?

$ /gsc/btl/linuxbrew/Cellar/xz/5.2.1/bin/xz --version
/gsc/btl/linuxbrew/Cellar/xz/5.2.1/bin/xz: Error creating a pipe: Function not implemented

[sjackman]

Here's the workaround to install xz 5.2.0:

https://raw.githubusercontent.com/Homebrew/linuxbrew/a51741bc83fa5af3644ce3413b0cfcef76f91952/Library/Formula/xz.rb

[sjackman]

@gitfoxi Would you be able to open an issue with the upstream xz authors?

[gitfoxi]

I e-mailed the author and he says it will be fixed in subsequent xz.

Lasse Collin Apr 20 (7 days ago)

to me On 2015-04-16 Michael Fox wrote:

Recent releases of xz build under kernel 2.6.18 but fail to run because the kernel has not implemented pipe2 (even though the call exists it doesn't work).

Thanks for the bug report. I feel a bit stupid that I didn't think about this possible problem (libc has the function but kernel doesn't). Since pipe2() isn't important to xz, I reverted the commit that added it so the next release won't try to use it.

[sjackman]

Great! Thanks.