Possible loss of wet dependencies for dry packages in a variant

[wjwwood]

When generating rosinstall files from variants it seems that you can lose some of the dry -> wet dependencies.

For example, if I get the wet elements of the desktop_debug variant:

http://packages.ros.org/web/rosinstall/generate/raw/groovy/desktop_debug

Build it, then source it, and then get the dry elements of desktop_debug:

http://packages.ros.org/web/rosinstall/generate/dry/raw/groovy/desktop_debug

and try to build them tf2_ros (from geometry_experimental) will fail because it cannot find actionlib. actionlib is wet and should be in the wet list of packages for desktop_debug but it is absent.

P.S. I created the desktop_debug variant from the desktop variant so I could apply a work around.

[dirk-thomas]

The question is which dependencies are missing: either wet packages which dry packages directly depend on or wet packages which are only indirectly depended on by dry packages?

[wjwwood]

A new occurrence:

cv_bridge is not pulled in, even though rqt_image_view depends on it.

[wjwwood]

@eitanme any ideas on this? I tried to just read through the logic, but I didn't see anything obviously wrong with it.

[wjwwood]

@tfoote suggests merging the dependency_walker.py logic here (groovy-devel branch):

https://github.com/willowgarage/catkin-debs/blob/groovy-devel/src/buildfarm/dependency_walker.py

This is used on Jenkins to produce the dependency graph. It would be good to push this into the https://github.com/ros-infrastructure/rosdistro project for reuse in multiple places.

[wmeeusse]

The rosdistro project already has support for generating the full dependency tree for wet catkin packages. Does the dependency walker provide the same functionality?

[wjwwood]

New development:

http://packages.ros.org/web/rosinstall/generate/raw/groovy/ros_comm

Does not even pull in gencpp or genlisp, but rosdep can detect that it is a dependency, resulting in this:

∫ rosdep install --from-paths src --ignore-src --rosdistro groovy -y
executing command [brew install ros/groovy/gencpp]
Error: No available formula for gencpp 
Please tap it and then try again: brew tap ros/groovy
ERROR: the following rosdeps failed to install
  homebrew: command [brew install ros/groovy/gencpp] failed

[wjwwood]

@wmeeusse not sure, but it needs to be able to do dry and wet, and if the rosinstall generator is using the logic from the rosdistro repo, then the rosdistro implementation is likely not working correctly or at the least being miss used by the rosinstall generator.

[eitanme]

Sorry for not being a part of this conversation yet... it totally got lost in the noise for me until @wmeeusse just pointed it out. At a glance, I'm not sure what could be going wrong... the logic seems correct to me. I'll look into things in more detail though since you have some use cases I can test with. I hope to get to that later today, tomorrow morning at the latest.

[wjwwood]

Great, let me know if you need help testing it or need more information.

[eitanme]

@wjwwood Upon further examination, I can't actually reproduce this. Following the links that you have in your first post, I see an entry generated for actionlib in the wet part of the rosinstall that is as follows:

- tar: {local-name: actionlib, uri: 'https://github.com/ros-gbp/actionlib-release/archive/release/actionlib/1.9.11.tar.gz',
    version: actionlib-release-release-actionlib-1.9.11}

Furthermore, running the following command

rosinstall desktop_debug_wet "http://packages.ros.org/web/rosinstall/generate/raw/groovy/desktop_debug"

Results in a directory structure that includes actionlib. Are you sure that there is actually a problem with the generation scripts? Perhaps the problem lies in weirdness with overlays, catkin, and dry stuff when you're trying to build?

[wjwwood]

actionlib might have changed, because tf2 from geometry_experimental was recently catkinized.

[wjwwood]

Let me see if I can reproduce this in another way.

[eitanme]

OK, just let me know when you have something that I can test with. Hopefully, it shouldn't be too bad to track down once it's reproducible.

[wjwwood]

Not sure if this is a case of this failing or a problem with dependencies, but this url:

http://packages.ros.org/web/rosinstall/generate/raw/groovy/octomap_ros

Produces a rosinstall file which contains genpy but not gencpp or genlisp, which results in a build error:

==> Processing catkin package: 'std_msgs'
==> Building with env: '/tmp/octomap_ws/install/env_cached.sh'
==> cmake /tmp/octomap_ws/src/std_msgs -DCATKIN_STATIC_ENV=1 -DCATKIN_DEVEL_PREFIX=/tmp/octomap_ws/devel/std_msgs -DCMAKE_INSTALL_PREFIX=/tmp/octomap_ws/install
-- Using CATKIN_DEVEL_PREFIX: /tmp/octomap_ws/devel/std_msgs
-- Using CMAKE_PREFIX_PATH: /tmp/octomap_ws/install
-- This workspace overlays: /tmp/octomap_ws/install
-- Found gtest sources under '/usr/src/gtest': gtests will be built
-- catkin 0.5.57
-- Generating static environment
CMake Error at /tmp/octomap_ws/install/share/message_generation/cmake/message_generationConfig.cmake:141 (find_package):
  Could not find module Findgencpp.cmake or a configuration file for package
  gencpp.

  Adjust CMAKE_MODULE_PATH to find Findgencpp.cmake or set gencpp_DIR to the
  directory containing a CMake configuration file for gencpp.  The file will
  have one of the following names:

    gencppConfig.cmake
    gencpp-config.cmake

Call Stack (most recent call first):
  /tmp/octomap_ws/install/share/catkin/cmake/catkinConfig.cmake:71 (find_package)
  CMakeLists.txt:4 (find_package)

CMake Error at /tmp/octomap_ws/install/share/message_generation/cmake/message_generationConfig.cmake:141 (find_package):
  Could not find module Findgenlisp.cmake or a configuration file for package
  genlisp.

  Adjust CMAKE_MODULE_PATH to find Findgenlisp.cmake or set genlisp_DIR to
  the directory containing a CMake configuration file for genlisp.  The file
  will have one of the following names:

    genlispConfig.cmake
    genlisp-config.cmake

Call Stack (most recent call first):
  /tmp/octomap_ws/install/share/catkin/cmake/catkinConfig.cmake:71 (find_package)
  CMakeLists.txt:4 (find_package)

-- std_msgs: 32 messages, 0 services
WARNING: 'catkin' should be listed as a buildtool dependency in the package.xml (instead of build dependency)
-- Configuring incomplete, errors occurred!

<== Failed to process package 'std_msgs': 
  Command '/tmp/octomap_ws/install/env_cached.sh cmake /tmp/octomap_ws/src/std_msgs -DCATKIN_STATIC_ENV=1 -DCATKIN_DEVEL_PREFIX=/tmp/octomap_ws/devel/std_msgs -DCMAKE_INSTALL_PREFIX=/tmp/octomap_ws/install' returned non-zero exit status 1
Command failed, exiting.

[wjwwood]

The above example also does not pull in rosbag...

[wjwwood]

Oh wait, that was my bad, rosbag is in there.

[wjwwood]

Nope, it really isn't in there, I am just blind.

http://packages.ros.org/web/rosinstall/generate/raw/groovy/octomap_ros

Does not have rosbash.

[wjwwood]

So not having rosbash might make sense for this wet package, but not having the generators (gencpp and genlisp) definitely doesn't seem right.

[eitanme]

So, this isn't a problem with the website, this is an issue with the rosdistro library itself. From any machine with python-rosdistro installed:

from rosdistro import RosDistro
distro = RosDistro('groovy')
deps = distro.get_depends('octomap_ros')
print 'gencpp' in deps['build'] + deps['buildtool'] + deps['test'] + deps['run']

I can try to look into what Wim @wmeeusse is doing, but perhaps this bug should move to that project?

[wjwwood]

Sure, that might make more sense.

[eitanme]

I'm not going to go crazy deep into this since @wmeeusse will be much faster, but some more information:

deps_std = distro.get_depends('std_msgs')
print 'gencpp' in deps_std['build'] + deps_std['buildtool'] + deps_std['test'] + deps_std['run']
print 'message_generation' in deps_std['build'] + deps_std['buildtool'] + deps_std['test'] + deps_std['run']

deps_msg = distro.get_depends('message_generation')
print 'gencpp' in deps_msg['build'] + deps_msg['buildtool'] + deps_msg['test'] + deps_msg['run']

Prints:

False
True
True

So, something is not going correctly in whatever recursion is happening.

[eitanme]

Created https://github.com/ros-infrastructure/rosdistro/issues/1 to track this.