Build fails on Ubuntu 11.10 (changed ld default behavior)

[miguno]

I noticed that the build of hadoop-lzo fails on Ubuntu 11.10. The reason is that the default behavior of ld was changed in 11.10.

From Ubuntu 11.10 Release Notes:

The compiler passes by default two additional flags to the linker:

[...snipp...]

-Wl,--as-needed with this option the linker will only add a DT_NEEDED tag for a dynamic library mentioned on the command line if if the library is actually used.

This was apparently planned to be changed already back in 11.04 but was eventually reverted in the final release. From 11.04 Toolchain Transition:

Also in Natty, ld runs with the --as-needed option enabled by default. This means that, in the example above, if no symbols from libwheel were needed by racetrack, then libwheel would not be linked even if it was explicitly included in the command-line compiler flags. NOTE: The ld --as-needed default was reverted for the final natty release, and will be re-enabled in the o-series.

Now in hadoop-lzo this manifests itself in the script {{src/native/configure}}. The following check will fail because of the changed ld default behavior:

  echo 'int main(int argc, char **argv){return 0;}' > conftest.c
  if test -z "`${CC} ${CFLAGS} ${LDFLAGS} -o conftest conftest.c -llzo2 2>&1`"; then
        if test ! -z "`which otool | grep -v 'no otool'`"; then
      ac_cv_libname_lzo2=\"`otool -L conftest | grep lzo2 | sed -e 's/^  *//' -e 's/ .*//'`\";
    elif test ! -z "`which objdump | grep -v 'no objdump'`"; then
      ac_cv_libname_lzo2="`objdump -p conftest | grep NEEDED | grep lzo2 | sed 's/\W*NEEDED\W*\(.*\)\W*$/\"\1\"/'`"
    elif test ! -z "`which ldd | grep -v 'no ldd'`"; then
      ac_cv_libname_lzo2="`ldd conftest | grep lzo2 | sed 's/^[^A-Za-z0-9]*\([A-Za-z0-9\.]*\)[^A-Za-z0-9]*=>.*$/\"\1\"/'`"
    else
      as_fn_error $? "Can't find either 'objdump' or 'ldd' to compute the dynamic library for '-llzo2'" "$LINENO" 5
    fi
  else

This line compiles a dummy C script and tells gcc to link it to the liblzo library (the native LZO library).

${CC} ${CFLAGS} ${LDFLAGS} -o conftest conftest.c -llzo2

Because of the changed ld behavior in 11.10, however, conftest will not link liblzo2. Hence the subsequent check will fail; since I do not have otool installed on my local dev box, the following code is run in my case:

 ac_cv_libname_lzo2="`objdump -p conftest | grep NEEDED | grep lzo2 | sed 's/\W*NEEDED\W*\(.*\)\W*$/\"\1\"/'`"

This command will return an empty string and assign it to the variable ac_cv_libname_lzo2. The eventual result is that the symbol HADOOP_LZO_LIBRARY will be assigned an empty string, too.

cat >>confdefs.h <<_ACEOF
#define HADOOP_LZO_LIBRARY ${ac_cv_libname_lzo2}
_ACEOF

Without a proper value for HADOOP_LZO_LIBRARY, however, the build of hadoop-lzo will fail.

HOW TO FIX

In our build setup we are using LDFLAGS to pass this option when invoking ant like so:

$ env LDFLAGS="-Wl,--no-as-needed" ant ...

In general though I'd think it would be best to fix this in build.xml:

    <exec dir="${build.native}" executable="sh" failonerror="true">
       <env key="OS_NAME" value="${os.name}"/>
       <env key="OS_ARCH" value="${os.arch}"/>
       <env key="LDFLAGS" value="-Wl,--no-as-needed"/>    <== ADD THIS LINE
       ...
    </exec>

Best, Michael

[miguno]

PS: Alternatively, one could extend the dummy C script so that it actually makes use of liblzo2. Personally, I found the LDFLAGS approach easier though... cough C cough

[miguno]

If you are ok with fixing this issue by modifying build.xml, you can merge in my pull request: https://github.com/kevinweil/hadoop-lzo/pull/34

[miguno]

Many thanks for the quick integration of the patch, Raghu!