search

catkin / catkin_tools

Command line tools for working with catkin
Apache License 2.0
164 stars 146 forks source link

Symlink stage may fail when binary files collide #470

Open nickswalker opened 7 years ago

nickswalker commented 7 years ago

System Info

Build / Run Issue

Expected Behavior

catkin tools should emit a warning when a binary path needs to be symlinked from two different locations.

Actual Behavior

In the symlink stage, the code that checks if a symlink already exists will take an MD5 hash of colliding files. In the process of reading the file, it decodes the file and throws an exception the first time in encounters a non-ascii byte sequence. Here's an example stacktrace.

Steps to Reproduce the Issue

Inspect this piece of code. Create two packages that produce a .so with the same name, then run catkin build.

mikepurvis commented 7 years ago

I would have expected an issue like this with Python 3, but interesting that you're also getting it with Python 2.

asmodehn commented 7 years ago

I also had this issue while building a python2 package that I am currently developing. Everything works fine with catkin_make. But with catkin build I get : 

Errors     << pyros_msgs:symlink /opt/groot/bootstrap_ws/logs/pyros_msgs/build.symlink.000.log                                                                                                                                      
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/catkin_tools/execution/stages.py", line 174, in function_proxy
    return function(logger, event_queue, *args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/catkin_tools/jobs/catkin.py", line 291, in link_devel_products
    source_hash = md5(open(os.path.realpath(source_file)).read().encode('utf-8')).hexdigest()
UnicodeDecodeError: 'ascii' codec can't decode byte 0xf3 in position 1: ordinal not in range(128)

Stage `symlink` failed with arguments:
  package: {'package_format': 2, 'exports': [<catkin_pkg.package.Export object at 0x7f6331dd1950>], 'buildtool_export_depends': [], 'exec_depends': [<catkin_pkg.package.Dependency object at 0x7f6330426590>, <catkin_pkg.package.Dependency object at 0x7f63304265f0>, <catkin_pkg.package.Dependency object at 0x7f6330426650>, <catkin_pkg.package.Dependency object at 0x7f63304266b0>, <catkin_pkg.package.Dependency object at 0x7f6330426710>, <catkin_pkg.package.Dependency object at 0x7f6330426770>, <catkin_pkg.package.Dependency object at 0x7f63304267d0>, <catkin_pkg.package.Dependency object at 0x7f6330426830>], 'name': 'pyros_msgs', 'filename': '/opt/groot/bootstrap_ws/src/pyros_msgs/package.xml', 'replaces': [], 'version_abi': None, 'build_export_depends': [<catkin_pkg.package.Dependency object at 0x7f6330426890>, <catkin_pkg.package.Dependency object at 0x7f63304268f0>, <catkin_pkg.package.Dependency object at 0x7f6330426950>, <catkin_pkg.package.Dependency object at 0x7f63304269b0>, <catkin_pkg.package.Dependency object at 0x7f6330426a10>, <catkin_pkg.package.Dependency object at 0x7f6330426a70>, <catkin_pkg.package.Dependency object at 0x7f6330426ad0>], 'doc_depends': [<catkin_pkg.package.Dependency object at 0x7f6330426b30>], 'test_depends': [<catkin_pkg.package.Dependency object at 0x7f6330426b90>], 'maintainers': [<catkin_pkg.package.Person object at 0x7f6331b613d0>], 'version': '0.1.1', 'build_depends': [<catkin_pkg.package.Dependency object at 0x7f6330426bf0>, <catkin_pkg.package.Dependency object at 0x7f6330426c50>, <catkin_pkg.package.Dependency object at 0x7f6330426cb0>, <catkin_pkg.package.Dependency object at 0x7f6330426d10>, <catkin_pkg.package.Dependency object at 0x7f6330426d70>, <catkin_pkg.package.Dependency object at 0x7f6330426dd0>, <catkin_pkg.package.Dependency object at 0x7f6330426e30>, <catkin_pkg.package.Dependency object at 0x7f6330426e90>, <catkin_pkg.package.Dependency object at 0x7f6330426ef0>], 'urls': [<catkin_pkg.package.Url object at 0x7f6331b04e10>, <catkin_pkg.package.Url object at 0x7f6331b04510>, <catkin_pkg.package.Url object at 0x7f6331b04e90>], 'authors': [<catkin_pkg.package.Person object at 0x7f6331b04790>], 'licenses': ['BSD'], 'buildtool_depends': [<catkin_pkg.package.Dependency object at 0x7f6330426f50>, <catkin_pkg.package.Dependency object at 0x7f6330426fb0>], 'conflicts': [], 'description': u'Pyros messages and services definition'}
  metadata_path: /opt/groot/bootstrap_ws/.catkin_tools/profiles/default
  devel_manifest_path: /opt/groot/bootstrap_ws/.catkin_tools/profiles/default/packages/pyros_msgs
  prebuild: False
  package_path: pyros_msgs
  dest_devel_path: /opt/groot/bootstrap_ws/devel
  source_devel_path: /opt/groot/bootstrap_ws/devel/.private/pyros_msgs

I dont think it is related with binary .so files, as I am just building python code...

Log says : 

Symlinking /opt/groot/bootstrap_ws/devel/lib/pkgconfig/pyros_msgs.pc
Symlinking /opt/groot/bootstrap_ws/devel/lib/python2.7/dist-packages/pyros_msgs/__init__.py
Symlinking /opt/groot/bootstrap_ws/devel/lib/python2.7/dist-packages/pyros_msgs/msg/_OptionalFields.py
Symlinking /opt/groot/bootstrap_ws/devel/lib/python2.7/dist-packages/pyros_msgs/msg/__init__.py
Symlinking /opt/groot/bootstrap_ws/devel/lib/python2.7/site-packages/pyros-msgs-nspkg.pth
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/catkin_tools/execution/stages.py", line 174, in function_proxy
    return function(logger, event_queue, *args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/catkin_tools/jobs/catkin.py", line 291, in link_devel_products
    source_hash = md5(open(os.path.realpath(source_file)).read().encode('utf-8')).hexdigest()
UnicodeDecodeError: 'ascii' codec can't decode byte 0xf3 in position 1: ordinal not in range(128)

I checked the last file in the log just in case, but everything seems fine:

alexv@alexv-indigo-vbox:/opt/groot/bootstrap_ws$ cat devel/lib/python2.7/site-packages/pyros-msgs-nspkg.pth 
import sys, types, os;has_mfs = sys.version_info > (3, 5);p = os.path.join('/opt/groot/bootstrap_ws/src/pyros_msgs', *('pyros_msgs',));importlib = has_mfs and __import__('importlib.util');has_mfs and __import__('importlib.machinery');m = has_mfs and sys.modules.setdefault('pyros_msgs', importlib.util.module_from_spec(importlib.machinery.PathFinder.find_spec('pyros_msgs', [os.path.dirname(p)])));m = m or not has_mfs and sys.modules.setdefault('pyros_msgs', types.ModuleType('pyros_msgs'));mp = (m or []) and m.__dict__.setdefault('__path__',[]);(p not in mp) and mp.append(p)
alexv@alexv-indigo-vbox:/opt/groot/bootstrap_ws$ python -c "from hashlib import md5; import os; print(md5(open('devel/lib/python2.7/site-packages/pyros-msgs-nspkg.pth', 'rb').read()))"
<md5 HASH object @ 0x7f471bc21c60>

Let me know if I can do something to help track this down and fix it for good.

mantkiew commented 7 years ago

I hit this issue recently and spent many hours trying to solve it. Finally, finding this issue, I commented out a fragment of CMakeFiles for an executable target that had the same name as an executable from another package we had, which solved the issue. 

add_executable(interpolation
...omitted...
)

(there was the same executable target called "interpolation" in another package).

Catkin should definitely issue a warning about duplicate executable files.

RozDavid commented 6 years ago

Hi @mantkiew!

Could you help me in which CMakeFile did you find the second executable? I could not dig it out... Thanks a lot!

mantkiew commented 6 years ago

I think these issues happen when you have targets with the same name from different packages which end up in the same location. I cannot give the specific CMakeFile as it's internal but the tools should just double check that no target name conflicts arise.