Closed jlblancoc closed 6 years ago
You can use this library to generate a script which will run commands on your local machine just as our Jenkins jobs do.
You will need docker >= 17.05, git, and the vcs your package uses installed on your host system.
You'll need to install the master branch of ros_buildfarm locally. I recommend creating a python3 virtualenv, activating it, and then running pip install git+https://github.com/ros-infrastructure/ros_buildfarm
Create a temporary directory to test in.
mkdir /tmp/kinetic-mrpt1-x64
cd /tmp/kinetic-mrpt1-x64
With the virtualenv activated you can generate a release script.
generate_release_script.py https://raw.githubusercontent.com/ros-infrastructure/ros_buildfarm_config/production/index.yaml kinetic default mrpt1 ubuntu xenial amd64 > run.sh
Run the generated script.
run.sh
is a shell script which performs the same steps as the buildfarm sourcedeb and binarydeb jobs do. I usually recommend running it once without any modifications in order to verify that it fails the same way as on the buildfarm.
sh run.sh
If you can reproduce the failure, you can go back and remove --rm
from the docker run invocations in order to inspect or restart the containers. When you re-run the script, delete the source
and binary
directories which are created. You'll get a warning if you forget to and it should be heeded.
There shouldn't be a need to use the latest master branch. The latest release 2.0.1 installed from e.g. Debian packages should be sufficient.
The steps to run the release job locally are also described in the docs.
:+1: thanks a lot!
Now I could reproduce the error locally... that's good enough to allow further investigation.
Despite I could reproduce the error, it seems I can't make much progress due to my lack of docker or ros_buildfarm know-how... This is how I've tried (unsuccessfully) to debug this issue:
--rm
flags). I could reproduce the same error that happens in the farm builder in the local docker container. So far, so good.dpkg-shlibdeps: error: couldn't find library libmrpt-gui.so.1.5 needed by debian/ros-kinetic-mrpt1/opt/ros/kinetic/bin/ReactiveNavigationDemo (ELF format: 'elf64-x86-64'; RPATH: '')
...
make[1]: *** [override_dh_shlibdeps] Error 2
debian/rules:47: recipe for target 'override_dh_shlibdeps' failed
make[1]: Leaving directory '/tmp/binarydeb/ros-kinetic-mrpt1-1.5.7'
make: *** [binary] Error 2
debian/rules:22: recipe for target 'binary' failed
dpkg-buildpackage: error: fakeroot debian/rules binary gave error exit status 2
E: Building failed
Traceback (most recent call last):
File "/tmp/ros_buildfarm/ros_buildfarm/binarydeb_job.py", line 138, in build_binarydeb
subprocess.check_call(cmd, cwd=source_dir)
File "/usr/lib/python3.5/subprocess.py", line 581, in check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['apt-src', 'build', 'ros-kinetic-mrpt1']' returned non-zero exit status 1
# END SUBSECTION
--------------------------------------------------------------------------------------------------
`apt-src build ros-kinetic-mrpt1` failed.
This is usually because of an error building the package.
The traceback from this failure (just above) is printed for completeness, but you can ignore it.
You should look above `E: Building failed` in the build log for the actual cause of the failure.
--------------------------------------------------------------------------------------------------
$ docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
b0bcd500f3c0 binarydeb_build.kinetic_ubuntu_xenial_amd64_mrpt1 "sh -c 'PYTHONPATH=/…" 7 hours ago Exited (1) 41 minutes ago distracted_albattani
7556613ee99c binarydeb_task_generation.kinetic_ubuntu_xenial_amd64_mrpt1 "sh -c 'PYTHONPATH=/…" 7 hours ago Exited (0) 7 hours ago xenodochial_chatterjee
d41677786856 sourcedeb.kinetic_ubuntu_xenial_mrpt1 "sh -c 'PYTHONPATH=/…" 7 hours ago Exited (0) 7 hours ago condescending_thompson
docker commit b0bcd500f3c0 jlblancoc/test_img
docker run -ti --entrypoint=bash jlblancoc/test_img
I can browse around and I can see many files in the /tmp
directory that are left from the unit tests run as part of the package generation, but the important directory /tmp/binarydeb/
exists but it's empty.
Am I doing something wrong with docker? Or is there something else to change in the run.sh
to prevent it to clean up the debian build directory?
Thanks
I just took a brief look. The specified branch in the tracks of the release repo of mrpt1 is null
which implies the default branch. But as far as I can see the default branch is for version 2. Should the devel_branch be changed to mrpt-1.5
?
The specified branch in the tracks of the release repo of mrpt1 is null which implies the default branch. But as far as I can see the default branch is for version 2. Should the devel_branch be changed to mrpt-1.5?
Correct! What are the implications of this mismatch? AFAIK, it would make Jenkins to try to build the incorrect branch for the "dev" builds, right? Anyway, that wouldn't affect the builds for releases, which have an explicit tag
, right?
Anyway, what's the recommended way to fix it? Manual editing, gloom,...?
PS: Just found the reason I left it to "empty" (null). Bloom docs say:
Upstream Devel Branch:
Branch in upstream repository on which to search for the version. This is used only when version is set to ':{auto}'. [None]:
and since version was set to :{ask}
....
PS2: Will continue trying to figuring out what's going on inside the docker container...
Good news: I nailed down the error. It's related to the use of MultiArch path postfixes. For Kinetic, using u16.04, the following command fails:
dpkg-shlibdeps \
-Tdebian/ros-kinetic-mrpt1.substvars \
-ldebian/ros-kinetic-mrpt1/opt/ros/kinetic/lib/ \
debian/ros-kinetic-mrpt1/opt/ros/kinetic/lib/x86_64-linux-gnu/libmrpt-maps.so.1.5.7 \
...
fails, but if the search path is changed to include the multiarch part, it works: -ldebian/ros-kinetic-mrpt1/opt/ros/kinetic/lib/x86_64-linux-gnu
.
However, something like the original command line seems to work for opencv3 (log). Will checkout how opencv3 deals with multiarch at the cmake level just in case that's the key. In case I definitively found the fix, I'll come back to close this one.
Thanks again for the help debugging this!
For the records (this should explain why it builds in Bionic):
From debhelper 10.2.1 onwards CMAKE_INSTALL_LIBDIR is set by debhelper (to /usr/lib/
. This means that a fair number of packages (that install libraries to CMAKE_INSTALL_LIBDIR) will 'just work', so all you need to do is add: depends: debhelper (>=10.2.1) https://wiki.debian.org/Multiarch/Implementation#CMake
debhelper versions:
The problem was not related with ros_buildfarm, but with different behavior of different versions of debhelper.
Thanks for the support!
Hi guys,
Not sure if this is actually a bug of the buildfarm code, bloom, or something else, but after many days trying to debug it, I think I nailed it down to something wrong in the docker containers used in the buildfarm for some distros, so I'll share my findings in the hope someone can give me a hand.
Instead of repeating here the entire situation that leads to this error, let me link to my former posts in ROS answers:
The latest finding is that exactly the same upstream sources, released for melodic, build as a charm for Artful and Bionic; example log:
(this command is run without errors in this case!!)
So, in summary: 1) Building with ROS prerelease.sh: works (but this doesn't attempt to build the .deb which is the problematic part) 2) Manually building (from a native Xenial machine) the .deb using
dpkg-buildpackage
using exactly the same .dsc that is built inside the build farm: works (!), whereas it fails inside the buildfarm. Grep forerror:
in this log for example. 3) Letting the build farm to build the same upstream sources within Artful / Bionic docker images: works (!). Example log:I hope someone can give me some ideas on how to exactly setup a docker image with the same contents as inside the build farm so we can debug what's wrong with the case (2) above.
Obviously, my motivation is that the error prevent the package to be released for kinetic, whereas it'll be released for melodic without problems.