3.12 is failing to build on indigo ubuntu 64bit

[tfoote]

http://build.ros.org/view/Kbin_uX64/job/Kbin_uX64__parrot_arsdk__ubuntu_xenial_amd64__binary/35/

    dpkg-shlibdeps -Tdebian/ros-kinetic-parrot-arsdk.substvars -l/tmp/binarydeb/ros-kinetic-parrot-arsdk-3.12.6/debian/ros-kinetic-parrot-arsdk//opt/ros/kinetic/lib/ debian/ros-kinetic-parrot-arsdk/opt/ros/kinetic/lib/parrot_arsdk/libjson-c.so.2.0.2 debian/ros-kinetic-parrot-arsdk/opt/ros/kinetic/lib/parrot_arsdk/libmux.so debian/ros-kinetic-parrot-arsdk/opt/ros/kinetic/lib/parrot_arsdk/libarnetwork.so debian/ros-kinetic-parrot-arsdk/opt/ros/kinetic/lib/parrot_arsdk/libardatatransfer.so debian/ros-kinetic-parrot-arsdk/opt/ros/kinetic/lib/parrot_arsdk/libardiscovery.so debian/ros-kinetic-parrot-arsdk/opt/ros/kinetic/lib/parrot_arsdk/libarmavlink.so debian/ros-kinetic-parrot-arsdk/opt/ros/kinetic/lib/parrot_arsdk/libarupdater.so debian/ros-kinetic-parrot-arsdk/opt/ros/kinetic/lib/parrot_arsdk/libarsal.so debian/ros-kinetic-parrot-arsdk/opt/ros/kinetic/lib/parrot_arsdk/libcurl.so debian/ros-kinetic-parrot-arsdk/opt/ros/kinetic/lib/parrot_arsdk/libpomp.so debian/ros-kinetic-parrot-arsdk/opt/ros/kinetic/lib/parrot_arsdk/libarmedia.so debian/ros-kinetic-parrot-arsdk/opt/ros/kinetic/lib/parrot_arsdk/libarcommands.so debian/ros-kinetic-parrot-arsdk/opt/ros/kinetic/lib/parrot_arsdk/libarcontroller.so debian/ros-kinetic-parrot-arsdk/opt/ros/kinetic/lib/parrot_arsdk/libarutils.so debian/ros-kinetic-parrot-arsdk/opt/ros/kinetic/lib/parrot_arsdk/libarstream.so debian/ros-kinetic-parrot-arsdk/opt/ros/kinetic/lib/parrot_arsdk/libarstream2.so debian/ros-kinetic-parrot-arsdk/opt/ros/kinetic/lib/parrot_arsdk/libarnetworkal.so debian/ros-kinetic-parrot-arsdk/opt/ros/kinetic/lib/parrot_arsdk/x86_64-linux-gnu/libstdc\+\+.so.6.0.21
dpkg-shlibdeps: warning: couldn't find library libarsal.so needed by debian/ros-kinetic-parrot-arsdk/opt/ros/kinetic/lib/parrot_arsdk/libarcontroller.so (ELF format: 'elf64-x86-64'; RPATH: '')
dpkg-shlibdeps: warning: couldn't find library libardiscovery.so needed by debian/ros-kinetic-parrot-arsdk/opt/ros/kinetic/lib/parrot_arsdk/libarcontroller.so (ELF format: 'elf64-x86-64'; RPATH: '')
dpkg-shlibdeps: warning: couldn't find library libarcommands.so needed by debian/ros-kinetic-parrot-arsdk/opt/ros/kinetic/lib/parrot_arsdk/libarcontroller.so (ELF format: 'elf64-x86-64'; RPATH: '')
dpkg-shlibdeps: warning: couldn't find library libarstream.so needed by debian/ros-kinetic-parrot-arsdk/opt/ros/kinetic/lib/parrot_arsdk/libarcontroller.so (ELF format: 'elf64-x86-64'; RPATH: '')
dpkg-shlibdeps: warning: couldn't find library libarstream2.so needed by debian/ros-kinetic-parrot-arsdk/opt/ros/kinetic/lib/parrot_arsdk/libarcontroller.so (ELF format: 'elf64-x86-64'; RPATH: '')
dpkg-shlibdeps: warning: couldn't find library libarnetworkal.so needed by debian/ros-kinetic-parrot-arsdk/opt/ros/kinetic/lib/parrot_arsdk/libarcontroller.so (ELF format: 'elf64-x86-64'; RPATH: '')
dpkg-shlibdeps: warning: couldn't find library libarnetwork.so needed by debian/ros-kinetic-parrot-arsdk/opt/ros/kinetic/lib/parrot_arsdk/libarcontroller.so (ELF format: 'elf64-x86-64'; RPATH: '')
dpkg-shlibdeps: warning: couldn't find library libmux.so needed by debian/ros-kinetic-parrot-arsdk/opt/ros/kinetic/lib/parrot_arsdk/libarcontroller.so (ELF format: 'elf64-x86-64'; RPATH: '')
dpkg-shlibdeps: error: couldn't find library libjson-c.so.2 needed by debian/ros-kinetic-parrot-arsdk/opt/ros/kinetic/lib/parrot_arsdk/libarcontroller.so (ELF format: 'elf64-x86-64'; RPATH: '')
dpkg-shlibdeps: warning: couldn't find library libarsal.so needed by debian/ros-kinetic-parrot-arsdk/opt/ros/kinetic/lib/parrot_arsdk/libarupdater.so (ELF format: 'elf64-x86-64'; RPATH: '')
dpkg-shlibdeps: warning: couldn't find library libardiscovery.so needed by debian/ros-kinetic-parrot-arsdk/opt/ros/kinetic/lib/parrot_arsdk/libarupdater.so (ELF format: 'elf64-x86-64'; RPATH: '')
dpkg-shlibdeps: warning: couldn't find library libarutils.so needed by debian/ros-kinetic-parrot-arsdk/opt/ros/kinetic/lib/parrot_arsdk/libarupdater.so (ELF format: 'elf64-x86-64'; RPATH: '')
dpkg-shlibdeps: warning: couldn't find library libardatatransfer.so needed by debian/ros-kinetic-parrot-arsdk/opt/ros/kinetic/lib/parrot_arsdk/libarupdater.so (ELF format: 'elf64-x86-64'; RPATH: '')
dpkg-shlibdeps: warning: couldn't find library libmux.so needed by debian/ros-kinetic-parrot-arsdk/opt/ros/kinetic/lib/parrot_arsdk/libarupdater.so (ELF format: 'elf64-x86-64'; RPATH: '')
dpkg-shlibdeps: warning: couldn't find library libpomp.so needed by debian/ros-kinetic-parrot-arsdk/opt/ros/kinetic/lib/parrot_arsdk/libarupdater.so (ELF format: 'elf64-x86-64'; RPATH: '')
dpkg-shlibdeps: error: couldn't find library libjson-c.so.2 needed by debian/ros-kinetic-parrot-arsdk/opt/ros/kinetic/lib/parrot_arsdk/libarupdater.so (ELF format: 'elf64-x86-64'; RPATH: '')
dpkg-shlibdeps: warning: couldn't find library libarsal.so needed by debian/ros-kinetic-parrot-arsdk/opt/ros/kinetic/lib/parrot_arsdk/libardatatransfer.so (ELF format: 'elf64-x86-64'; RPATH: '')
dpkg-shlibdeps: warning: couldn't find library libardiscovery.so needed by debian/ros-kinetic-parrot-arsdk/opt/ros/kinetic/lib/parrot_arsdk/libardatatransfer.so (ELF format: 'elf64-x86-64'; RPATH: '')
dpkg-shlibdeps: warning: couldn't find library libarutils.so needed by debian/ros-kinetic-parrot-arsdk/opt/ros/kinetic/lib/parrot_arsdk/libardatatransfer.so (ELF format: 'elf64-x86-64'; RPATH: '')
dpkg-shlibdeps: warning: couldn't find library libarsal.so needed by debian/ros-kinetic-parrot-arsdk/opt/ros/kinetic/lib/parrot_arsdk/libarcommands.so (ELF format: 'elf64-x86-64'; RPATH: '')
dpkg-shlibdeps: warning: couldn't find library libarsal.so needed by debian/ros-kinetic-parrot-arsdk/opt/ros/kinetic/lib/parrot_arsdk/libarnetwork.so (ELF format: 'elf64-x86-64'; RPATH: '')
dpkg-shlibdeps: warning: couldn't find library libarnetworkal.so needed by debian/ros-kinetic-parrot-arsdk/opt/ros/kinetic/lib/parrot_arsdk/libarnetwork.so (ELF format: 'elf64-x86-64'; RPATH: '')
dpkg-shlibdeps: warning: couldn't find library libpomp.so needed by debian/ros-kinetic-parrot-arsdk/opt/ros/kinetic/lib/parrot_arsdk/libmux.so (ELF format: 'elf64-x86-64'; RPATH: '')
dpkg-shlibdeps: warning: couldn't find library libarsal.so needed by debian/ros-kinetic-parrot-arsdk/opt/ros/kinetic/lib/parrot_arsdk/libarutils.so (ELF format: 'elf64-x86-64'; RPATH: '')
dpkg-shlibdeps: warning: couldn't find library libardiscovery.so needed by debian/ros-kinetic-parrot-arsdk/opt/ros/kinetic/lib/parrot_arsdk/libarutils.so (ELF format: 'elf64-x86-64'; RPATH: '')
dpkg-shlibdeps: warning: couldn't find library libmux.so needed by debian/ros-kinetic-parrot-arsdk/opt/ros/kinetic/lib/parrot_arsdk/libarutils.so (ELF format: 'elf64-x86-64'; RPATH: '')
dpkg-shlibdeps: warning: /usr/lib/x86_64-linux-gnu/libcurl.so has an unexpected SONAME (libcurl.so.4)
dpkg-shlibdeps: warning: can't extract name and version from library name 'libcurl.so'
dpkg-shlibdeps: warning: can't extract name and version from library name 'libcurl.so'
dpkg-shlibdeps: warning: couldn't find library libarsal.so needed by debian/ros-kinetic-parrot-arsdk/opt/ros/kinetic/lib/parrot_arsdk/libarmedia.so (ELF format: 'elf64-x86-64'; RPATH: '')
dpkg-shlibdeps: warning: couldn't find library libardiscovery.so needed by debian/ros-kinetic-parrot-arsdk/opt/ros/kinetic/lib/parrot_arsdk/libarmedia.so (ELF format: 'elf64-x86-64'; RPATH: '')
dpkg-shlibdeps: error: couldn't find library libjson-c.so.2 needed by debian/ros-kinetic-parrot-arsdk/opt/ros/kinetic/lib/parrot_arsdk/libarmedia.so (ELF format: 'elf64-x86-64'; RPATH: '')
dpkg-shlibdeps: warning: couldn't find library libarsal.so needed by debian/ros-kinetic-parrot-arsdk/opt/ros/kinetic/lib/parrot_arsdk/libarstream2.so (ELF format: 'elf64-x86-64'; RPATH: '')
dpkg-shlibdeps: warning: couldn't find library libmux.so needed by debian/ros-kinetic-parrot-arsdk/opt/ros/kinetic/lib/parrot_arsdk/libarstream2.so (ELF format: 'elf64-x86-64'; RPATH: '')
dpkg-shlibdeps: warning: couldn't find library libpomp.so needed by debian/ros-kinetic-parrot-arsdk/opt/ros/kinetic/lib/parrot_arsdk/libarstream2.so (ELF format: 'elf64-x86-64'; RPATH: '')
dpkg-shlibdeps: warning: couldn't find library libarmedia.so needed by debian/ros-kinetic-parrot-arsdk/opt/ros/kinetic/lib/parrot_arsdk/libarstream2.so (ELF format: 'elf64-x86-64'; RPATH: '')
dpkg-shlibdeps: warning: couldn't find library libarsal.so needed by debian/ros-kinetic-parrot-arsdk/opt/ros/kinetic/lib/parrot_arsdk/libarnetworkal.so (ELF format: 'elf64-x86-64'; RPATH: '')
dpkg-shlibdeps: warning: couldn't find library libmux.so needed by debian/ros-kinetic-parrot-arsdk/opt/ros/kinetic/lib/parrot_arsdk/libarnetworkal.so (ELF format: 'elf64-x86-64'; RPATH: '')
dpkg-shlibdeps: warning: couldn't find library libpomp.so needed by debian/ros-kinetic-parrot-arsdk/opt/ros/kinetic/lib/parrot_arsdk/libarnetworkal.so (ELF format: 'elf64-x86-64'; RPATH: '')
dpkg-shlibdeps: warning: couldn't find library libarsal.so needed by debian/ros-kinetic-parrot-arsdk/opt/ros/kinetic/lib/parrot_arsdk/libardiscovery.so (ELF format: 'elf64-x86-64'; RPATH: '')
dpkg-shlibdeps: warning: couldn't find library libarnetworkal.so needed by debian/ros-kinetic-parrot-arsdk/opt/ros/kinetic/lib/parrot_arsdk/libardiscovery.so (ELF format: 'elf64-x86-64'; RPATH: '')
dpkg-shlibdeps: warning: couldn't find library libmux.so needed by debian/ros-kinetic-parrot-arsdk/opt/ros/kinetic/lib/parrot_arsdk/libardiscovery.so (ELF format: 'elf64-x86-64'; RPATH: '')
dpkg-shlibdeps: warning: couldn't find library libpomp.so needed by debian/ros-kinetic-parrot-arsdk/opt/ros/kinetic/lib/parrot_arsdk/libardiscovery.so (ELF format: 'elf64-x86-64'; RPATH: '')
dpkg-shlibdeps: error: couldn't find library libjson-c.so.2 needed by debian/ros-kinetic-parrot-arsdk/opt/ros/kinetic/lib/parrot_arsdk/libardiscovery.so (ELF format: 'elf64-x86-64'; RPATH: '')
dpkg-shlibdeps: warning: couldn't find library libarsal.so needed by debian/ros-kinetic-parrot-arsdk/opt/ros/kinetic/lib/parrot_arsdk/libarmavlink.so (ELF format: 'elf64-x86-64'; RPATH: '')
dpkg-shlibdeps: warning: couldn't find library libarsal.so needed by debian/ros-kinetic-parrot-arsdk/opt/ros/kinetic/lib/parrot_arsdk/libarstream.so (ELF format: 'elf64-x86-64'; RPATH: '')
dpkg-shlibdeps: warning: couldn't find library libarnetwork.so needed by debian/ros-kinetic-parrot-arsdk/opt/ros/kinetic/lib/parrot_arsdk/libarstream.so (ELF format: 'elf64-x86-64'; RPATH: '')
dpkg-shlibdeps: warning: package could avoid a useless dependency if debian/ros-kinetic-parrot-arsdk/opt/ros/kinetic/lib/parrot_arsdk/libarutils.so was not linked against libcurl.so (it uses none of the library's symbols)
dpkg-shlibdeps: error: cannot continue due to the errors listed above
Note: libraries are not searched in other binary packages that do not have any shlibs or symbols file.
To help dpkg-shlibdeps find private libraries, you might need to use -l.
dh_shlibdeps: dpkg-shlibdeps -Tdebian/ros-kinetic-parrot-arsdk.substvars -l/tmp/binarydeb/ros-kinetic-parrot-arsdk-3.12.6/debian/ros-kinetic-parrot-arsdk//opt/ros/kinetic/lib/ debian/ros-kinetic-parrot-arsdk/opt/ros/kinetic/lib/parrot_arsdk/libjson-c.so.2.0.2 debian/ros-kinetic-parrot-arsdk/opt/ros/kinetic/lib/parrot_arsdk/libmux.so debian/ros-kinetic-parrot-arsdk/opt/ros/kinetic/lib/parrot_arsdk/libarnetwork.so debian/ros-kinetic-parrot-arsdk/opt/ros/kinetic/lib/parrot_arsdk/libardatatransfer.so debian/ros-kinetic-parrot-arsdk/opt/ros/kinetic/lib/parrot_arsdk/libardiscovery.so debian/ros-kinetic-parrot-arsdk/opt/ros/kinetic/lib/parrot_arsdk/libarmavlink.so debian/ros-kinetic-parrot-arsdk/opt/ros/kinetic/lib/parrot_arsdk/libarupdater.so debian/ros-kinetic-parrot-arsdk/opt/ros/kinetic/lib/parrot_arsdk/libarsal.so debian/ros-kinetic-parrot-arsdk/opt/ros/kinetic/lib/parrot_arsdk/libcurl.so debian/ros-kinetic-parrot-arsdk/opt/ros/kinetic/lib/parrot_arsdk/libpomp.so debian/ros-kinetic-parrot-arsdk/opt/ros/kinetic/lib/parrot_arsdk/libarmedia.so debian/ros-kinetic-parrot-arsdk/opt/ros/kinetic/lib/parrot_arsdk/libarcommands.so debian/ros-kinetic-parrot-arsdk/opt/ros/kinetic/lib/parrot_arsdk/libarcontroller.so debian/ros-kinetic-parrot-arsdk/opt/ros/kinetic/lib/parrot_arsdk/libarutils.so debian/ros-kinetic-parrot-arsdk/opt/ros/kinetic/lib/parrot_arsdk/libarstream.so debian/ros-kinetic-parrot-arsdk/opt/ros/kinetic/lib/parrot_arsdk/libarstream2.so debian/ros-kinetic-parrot-arsdk/opt/ros/kinetic/lib/parrot_arsdk/libarnetworkal.so debian/ros-kinetic-parrot-arsdk/opt/ros/kinetic/lib/parrot_arsdk/x86_64-linux-gnu/libstdc++.so.6.0.21 returned exit code 2
debian/rules:51: recipe for target 'override_dh_shlibdeps' failed
make[1]: *** [override_dh_shlibdeps] Error 2
make[1]: Leaving directory '/tmp/binarydeb/ros-kinetic-parrot-arsdk-3.12.6'
make: *** [binary] Error 2
debian/rules:23: recipe for target 'binary' failed
dpkg-buildpackage: error: fakeroot debian/rules binary gave error exit status 2
E: Building failed
Traceback (most recent call last):
  File "/tmp/ros_buildfarm/ros_buildfarm/binarydeb_job.py", line 133, in build_binarydeb
    subprocess.check_call(cmd, cwd=source_dir)
  File "/usr/lib/python3.5/subprocess.py", line 581, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['apt-src', 'build', 'ros-kinetic-parrot-arsdk']' returned non-zero exit status 1

[mani-monaj]

Thanks @tfoote.

I've been trying to chase this issue down for a while. The pre-release tests all pass since the problem is at debian packaging level. I tried re-producing the problem locally using the instructions here [1, 2], but could not run the script successfully. I will do another pass later today.

I would really appreciate it if you share any insight about why dh_shlibdeps may fail for only a specific target platform (Xenial/64bit)?

[1] https://github.com/ros-infrastructure/ros_buildfarm/blob/master/doc/jobs/release_jobs.rst#run-the-release-job-locally [2] https://github.com/ros-infrastructure/ros_buildfarm/blob/master/doc/environment.rst

[tfoote]

It looks like potentially some dependencies are missing? The error about not finding libjson-c is likely the culprit.

For setting up your environment. You can now use the debian package python3-ros-buildfarm and go straight to the test. You will also need docker without sudo setup.

I was able to reproduce using this:

 2000  sudo apt-get install python3-ros-buildfarm
 2001  mkdir /tmp/release_job
 2002  generate_release_script.py https://raw.githubusercontent.com/ros-infrastructure/ros_buildfarm_config/production/index.yaml kinetic default parrot_arsdk ubuntu xenial amd64 > /tmp/release_job/test.sh
 2003  less test.sh
 2004  ls /tmp/release_job/test.sh 
 2005  cd /tmp/release_job/
 2006  cat test.sh 
 2007  sh test.sh 

[mani-monaj]

Thanks @tfoote for your instructions.

Unfortunately, I still could not run the release_job locally. The progress would advance, but I would receive ccache permission errors (during the cmake configuration step) within the container, similar to https://github.com/ros-infrastructure/buildfarm_deployment/issues/113 . (target folder: /home/buildfarm/.ccache/tmp).

Nevertheless, I think I found where the problem is coming from.

1. parrot_arsdk (from Parrot), comes with its own build and workspace management system which itself is fairly complex.
2. As part of its default build process, it builds libjson and libcurl from source. Those are used by many of its higher level libraries.
3. When we wrapped this process with catkin, I deliberatively made the install step to copy all the libraries to a folder within the install location (e.g. /opt/ros/kinetic/lib/parrot_arsdk/libar*.so instead of /opt/ros/kinetic/lib/libar*.so). This path is not added to LD_LIBRARY_PATH. Any downstream dependency (such as bebop_autonomy/bebop_driver), manually set their link_directories to a variable exported by this package during their cmake configuration step. The rational is to prevent custom version of libjson or libcurl to be installed into a shared library install location.
4. Neither the dpkg-shlibdeps step of the build script, nor ldconfig know about this folder when a release_job is being executed. As you can see, dpkg-shlibdeps is throwing a lot of warnings because it can not find any of the custom libraries that libar* libraries depend on.
5. According to man:dpkg-shlibdeps, it (4) is ok for private library dependencies (*.so), but not for public library dependencies (*.so.x.y.z). The former causes a warning, while the latter results in an error.
6. On some of the target platforms (such as Xenial x64), the SDK generated libraries are linked against libjson-c.so.2, which because of (4) can not be found by dpkg-shlibdeps. Since it looks like a public library dependency (5), dpkg-shlibdeps fails the build.

It might be difficult to trace down why this dependency happens on a certain platform only.

@tfoote In my opinion the easiest fix for this problem is to let the install step install everything (including libjson, libcurl and liblibressl) to /opt/ros/*/lib. Is this going to cause any troubles for any other package?

Another option is to provide an additional -l argument to dpkg-shlibdeps. Is there any way that we can control these options at the package level?

[tfoote]

Unfortunately installing those custom versions of libcurl and libjson will break any other user of those libraries which is expecting the system version.

If you want to install these things they should have all their symbols renamed as well as the libraries so as not to collide with the system libraries.

If you can find a way to pass the extra libraries to dpkg-shlibdeps that probably would be the best approach. The other thing to test would be if they really need the custom forks of those libraries or if they could be overridden to use the system versions. Often these things are integrated into the build when you want a newer version. And at some point the newer version becomes available generally but it's already in the build tree and no one wants to remove it.

[mani-monaj]

Thanks so much @tfoote for your suggestions.

Unfortunately installing those custom versions of libcurl and libjson will break any other user of those libraries which is expecting the system version.If you want to install these things they should have all their symbols renamed as well as the libraries so as not to collide with the system libraries.

I think that renaming these files and/or renaming their symbols will be a tedious task. Therefore, installing them to a common folder is not an option at all.

If you can find a way to pass the extra libraries to dpkg-shlibdeps that probably would be the best approach.

I agree. I figured that bloom adds these flags to the debian/rules file. I don't think the extra effort to change the logic there to make it more flexible worth it.

The other thing to test would be if they really need the custom forks of those libraries or if they could be overridden to use the system versions. Often these things are integrated into the build when you want a newer version. And at some point the newer version becomes available generally but it's already in the build tree and no one wants to remove it.

I agree. We will look into this option first.

[thomas-bamford]

PR #7 adds a patch that aims to temporarily fix this issue.

[thomas-bamford]

Similar issues encountered in release 3.14.0. Fixed in release 3.14.1.