search

start-jsk / jsk_apc

Other
36 stars 35 forks source link

[WIP] Fix errors in building grasp_fusion and instance_occlsegm #2697

Closed pazeshun closed 5 years ago

pazeshun commented 5 years ago

Add Cython to requirements for installing pycocotools

Problem & Solution

When building jsk_apc with .travis.rosinstall and .travis.rosinstall.kinetic, I encountered the following error at instance_occlsegm:

Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-install-W46fSl/pycocotools/

This is because Cython is not automatically installed before installing pycocotools:

$ pip install --user pycocotools
Collecting pycocotools
  Downloading https://files.pythonhosted.org/packages/96/84/9a07b1095fd8555ba3f3d519517c8743c2554a245f9476e5e39869f948d2/pycocotools-2.0.0.tar.gz (1.5MB)
    100% |████████████████████████████████| 1.5MB 308kB/s 
    Complete output from command python setup.py egg_info:
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-build-c36GAZ/pycocotools/setup.py", line 2, in <module>
        from Cython.Build import cythonize
    ImportError: No module named Cython.Build

    ----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-build-c36GAZ/pycocotools/

This is fixed in latest pycocotools but not released (see https://github.com/cocodataset/cocoapi/issues/327). So I tried to install pycocotools from source in catkin_virtualenv, but failed due to https://github.com/locusrobotics/catkin_virtualenv/issues/48. Therefore, I add Cython to requirements.txt of instance_occlsegm. Cython installation completes before pycocotools thanks to alphabetical order.

Why travis doesn't catch this problem?

Because docker container used in travis already includes Cython:

$ sudo docker ps -a
CONTAINER ID        IMAGE                                COMMAND                  CREATED             STATUS                          PORTS               NAMES
186863aa12b0        jskrobotics/jsk_apc:kinetic-latest   "/ros_entrypoint.sh …"   30 minutes ago      Exited (0) About a minute ago                       sharp_jackson
...
root@186863aa12b0:/# pip show cython
Name: Cython
Version: 0.23.4
Summary: The Cython compiler for writing C extensions for the Python language.
Home-page: http://cython.org/
Author: Robert Bradshaw, Stefan Behnel, Dag Seljebotn, Greg Ewing, et al.
Author-email: cython-devel@python.org
License: UNKNOWN
Location: /usr/lib/python2.7/dist-packages
Requires:

Avoid skipping installed python requirements which is incompatible

Problem

I encountered the following red lines at instance_occlsegm and grasp_fusion after solving the above issue:

opencv-python-headless 4.1.1.26 has requirement numpy>=1.11.1, but you'll have numpy 1.11.0 which is incompatible.
imgaug 0.3.0 has requirement numpy>=1.15, but you'll have numpy 1.11.0 which is incompatible.
imgaug 0.3.0 has requirement scikit-image>=0.14.2, but you'll have scikit-image 0.10.1 which is incompatible.
pandas 0.24.2 has requirement numpy>=1.12.0, but you'll have numpy 1.11.0 which is incompatible.
opencv-python-headless 4.1.1.26 has requirement numpy>=1.11.1, but you'll have numpy 1.11.0 which is incompatible.
imgaug 0.3.0 has requirement numpy>=1.15, but you'll have numpy 1.11.0 which is incompatible.
imgaug 0.3.0 has requirement scikit-image>=0.14.2, but you'll have scikit-image 0.10.1 which is incompatible.
pandas 0.24.2 has requirement numpy>=1.12.0, but you'll have numpy 1.11.0 which is incompatible.

(Related to https://medium.com/knerd/the-nine-circles-of-python-dependency-hell-481d53e3e025 ?)

Solution

numpy and scikit-image are installed via apt, so I add argument --upgrade --upgrade-strategy=eager to pip running in catkin_virtualenv. By this, latest numpy and scikit-image are installed in catkin_virtualenv. I wanted to use --upgrade-strategy=only-if-needed, but when I used that, I encountered another error which I comment in CMakeLists.txt.

pazeshun commented 5 years ago

Travis failed due to rosdep failure:

ERROR: Rosdep experienced an error: cannot import name OS_NIXOS

Please go to the rosdep page [1] and file a bug report with the stack trace below.

[1] : http://www.ros.org/wiki/rosdep

rosdep version: 0.16.1

Traceback (most recent call last):

  File "/usr/lib/python2.7/dist-packages/rosdep2/main.py", line 143, in rosdep_main

    exit_code = _rosdep_main(args)

  File "/usr/lib/python2.7/dist-packages/rosdep2/main.py", line 397, in _rosdep_main

    return _no_args_handler(command, parser, options, args)

  File "/usr/lib/python2.7/dist-packages/rosdep2/main.py", line 406, in _no_args_handler

    return command_handlers[command](options)

  File "/usr/lib/python2.7/dist-packages/rosdep2/main.py", line 617, in command_update

    skip_eol_distros=not options.include_eol_distros)

  File "/usr/lib/python2.7/dist-packages/rosdep2/sources_list.py", line 497, in update_sources_list

    rosdep_data = get_gbprepo_as_rosdep_data(dist_name)

  File "/usr/lib/python2.7/dist-packages/rosdep2/gbpdistro_support.py", line 150, in get_gbprepo_as_rosdep_data

    ctx = create_default_installer_context()

  File "/usr/lib/python2.7/dist-packages/rosdep2/__init__.py", line 63, in create_default_installer_context

    from .platforms import nix

  File "/usr/lib/python2.7/dist-packages/rosdep2/platforms/nix.py", line 31, in <module>

    from rospkg.os_detect import OS_NIXOS

ImportError: cannot import name OS_NIXOS

Related to https://www.gitmemory.com/kahkeng (?)

knorth55 commented 5 years ago

it seems that you need to install rospkg >= 1.1.10 https://github.com/ros-infrastructure/rosdep/pull/717

Shingo Kitagawa the University of Tokyo, JSK Lab s-kitagawa@jsk.imi.i.u-tokyo.ac.jp

2019年10月12日(土) 2:09 Shun Hasegawa notifications@github.com:

Travis failed due to rosdep failure:

ERROR: Rosdep experienced an error: cannot import name OS_NIXOS

Please go to the rosdep page [1] and file a bug report with the stack trace below.

[1] : http://www.ros.org/wiki/rosdep

rosdep version: 0.16.1

Traceback (most recent call last):

File "/usr/lib/python2.7/dist-packages/rosdep2/main.py", line 143, in rosdep_main

exit_code = _rosdep_main(args)

File "/usr/lib/python2.7/dist-packages/rosdep2/main.py", line 397, in _rosdep_main

return _no_args_handler(command, parser, options, args)

File "/usr/lib/python2.7/dist-packages/rosdep2/main.py", line 406, in _no_args_handler

return command_handlers[command](options)

File "/usr/lib/python2.7/dist-packages/rosdep2/main.py", line 617, in command_update

skip_eol_distros=not options.include_eol_distros)

File "/usr/lib/python2.7/dist-packages/rosdep2/sources_list.py", line 497, in update_sources_list

rosdep_data = get_gbprepo_as_rosdep_data(dist_name)

File "/usr/lib/python2.7/dist-packages/rosdep2/gbpdistro_support.py", line 150, in get_gbprepo_as_rosdep_data

ctx = create_default_installer_context()

File "/usr/lib/python2.7/dist-packages/rosdep2/init.py", line 63, in create_default_installer_context

from .platforms import nix

File "/usr/lib/python2.7/dist-packages/rosdep2/platforms/nix.py", line 31, in 

from rospkg.os_detect import OS_NIXOS

ImportError: cannot import name OS_NIXOS

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/start-jsk/jsk_apc/pull/2697?email_source=notifications&email_token=ACG6QX42HW4XMHB24YWSSA3QOCXNNA5CNFSM4I7MXOF2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEBAUCLI#issuecomment-541147437, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACG6QXY7TLQDVSXTHJDQQJ3QOCXNNANCNFSM4I7MXOFQ .

pazeshun commented 5 years ago

https://github.com/ros-infrastructure/rosdep/pull/717 is merged and released, so I restarted travis. Though the rosdep failure disappeared, travis test failed because log became too long:

The job exceeded the maximum log length, and has been terminated.

(the last line of https://api.travis-ci.org/v3/job/596105466/log.txt)

I'll try to fix this by suppressing the following warning messages from pip in catkin_virtualenv:

DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7. More details about Python 2 support in pip, can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support
You are using pip version 19.0.3, however version 19.3.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.

But this trial may fail, because the main causes of making log long seem the messages from catkin run_tests like the following:

[grasp_fusion:symlink] Linked: (/home/travis/ros/ws_jsk_apc/devel/.private/grasp_fusion/share/grasp_fusion/venv/bin/imageio_remove_bin, /home/travis/ros/ws_jsk_apc/devel/share/grasp_fusion/venv/bin/imageio_remove_bin)
pazeshun commented 5 years ago

I'll try to fix this by suppressing the following warning messages from pip in catkin_virtualenv

I added b665a65. Even with b665a65, errors from pip are shown:

$ catkin clean grasp_fusion
Starting  >>> grasp_fusion             
Finished  <<< grasp_fusion              [ 0.3 seconds ]
[clean] Summary: All 1 packages succeeded!
[clean]   Ignored:   None.
[clean]   Warnings:  None.
[clean]   Abandoned: None.
[clean]   Failed:    None.
[clean] Runtime: 0.7 seconds total.
[clean] Note: Parts of the workspace have been cleaned which will necessitate re-configuring CMake on the next build.
$ git diff
diff --git a/demos/grasp_fusion/ros/grasp_fusion/CMakeLists.txt b/demos/grasp_fusion/ros/grasp_fusion/CMakeLists.txt
index 556704b..6f4d2a6 100644
--- a/demos/grasp_fusion/ros/grasp_fusion/CMakeLists.txt
+++ b/demos/grasp_fusion/ros/grasp_fusion/CMakeLists.txt
@@ -28,6 +28,7 @@ catkin_package(

 catkin_generate_virtualenv(
   EXTRA_PIP_ARGS
+    -qqq  # Suppress log not to exceed maximum length in travis test
     --upgrade
     --upgrade-strategy=eager
     # '--upgrade-strategy=only-if-needed' does not work in following case:
$ catkin build grasp_fusion --no-deps
---------------------------------------------------------------
Profile:                     default
Extending:          [cached] /opt/ros/kinetic
Workspace:                   /home/ushun/ros/ws_jsk_apc
---------------------------------------------------------------
Build Space:        [exists] /home/ushun/ros/ws_jsk_apc/build
Devel Space:        [exists] /home/ushun/ros/ws_jsk_apc/devel
Install Space:      [unused] /home/ushun/ros/ws_jsk_apc/install
Log Space:          [exists] /home/ushun/ros/ws_jsk_apc/logs
Source Space:       [exists] /home/ushun/ros/ws_jsk_apc/src
DESTDIR:            [unused] None
---------------------------------------------------------------
Devel Space Layout:          linked
Install Space Layout:        None
---------------------------------------------------------------
Additional CMake Args:       None
Additional Make Args:        None
Additional catkin Make Args: None
Internal Make Job Server:    True
Cache Job Environments:      False
---------------------------------------------------------------
Whitelisted Packages:        None
Blacklisted Packages:        None
---------------------------------------------------------------
Workspace configuration appears valid.

NOTE: Forcing CMake to run for each package.
---------------------------------------------------------------
[build] Found '82' packages in 0.0 seconds.                                                                                                   
[build] Package table is up to date.                                                                                                          
Starting  >>> grasp_fusion                                                                                                                    
______________________________________________________________________________________________________________________________________________
Warnings   << grasp_fusion:make /home/ushun/ros/ws_jsk_apc/logs/grasp_fusion/build.make.008.log                                               
/home/ushun/ros/ws_jsk_apc/devel/.private/grasp_fusion/share/grasp_fusion/venv/local/lib/python2.7/site-packages/pip/_vendor/requests/__init__.py:83: RequestsDependencyWarning: Old version of cryptography ([1, 2, 3]) may cause slowdown.
  warnings.warn(warning, RequestsDependencyWarning)
/home/ushun/ros/ws_jsk_apc/devel/.private/grasp_fusion/share/grasp_fusion/venv/local/lib/python2.7/site-packages/pip/_vendor/requests/__init__.py:83: RequestsDependencyWarning: Old version of cryptography ([1, 2, 3]) may cause slowdown.
  warnings.warn(warning, RequestsDependencyWarning)
/home/ushun/ros/ws_jsk_apc/build/grasp_fusion/venv/local/lib/python2.7/site-packages/pip/_vendor/requests/__init__.py:83: RequestsDependencyWarning: Old version of cryptography ([1, 2, 3]) may cause slowdown.
  warnings.warn(warning, RequestsDependencyWarning)
/home/ushun/ros/ws_jsk_apc/build/grasp_fusion/venv/local/lib/python2.7/site-packages/pip/_vendor/requests/__init__.py:83: RequestsDependencyWarning: Old version of cryptography ([1, 2, 3]) may cause slowdown.
  warnings.warn(warning, RequestsDependencyWarning)
cd /home/ushun/ros/ws_jsk_apc/build/grasp_fusion; catkin build --get-env grasp_fusion | catkin env -si  /usr/bin/make --jobserver-fds=6,7 -j; cd -
..............................................................................................................................................
Finished  <<< grasp_fusion                [ 3 minutes and 23.9 seconds ]                                                                      
[build] Summary: All 1 packages succeeded!                                                                                                    
[build]   Ignored:   81 packages were skipped or are blacklisted.                                                                             
[build]   Warnings:  1 packages succeeded with warnings.                                                                                      
[build]   Abandoned: None.                                                                                                                    
[build]   Failed:    None.                                                                                                                    
[build] Runtime: 3 minutes and 24.6 seconds total.                                                                                            
[build] Note: Workspace packages have changed, please re-source setup files to use them.
$ vi CMakeLists.txt
$ git diff
diff --git a/demos/grasp_fusion/ros/grasp_fusion/CMakeLists.txt b/demos/grasp_fusion/ros/grasp_fusion/CMakeLists.txt
index 556704b..d6a55e3 100644
--- a/demos/grasp_fusion/ros/grasp_fusion/CMakeLists.txt
+++ b/demos/grasp_fusion/ros/grasp_fusion/CMakeLists.txt
@@ -28,8 +28,9 @@ catkin_package(

 catkin_generate_virtualenv(
   EXTRA_PIP_ARGS
-    --upgrade
-    --upgrade-strategy=eager
+    -qqq  # Suppress log not to exceed maximum length in travis test
+    #--upgrade
+    #--upgrade-strategy=eager
     # '--upgrade-strategy=only-if-needed' does not work in following case:
     # python-dateutil is required by matplotlib.
     # It is satisfied and skipped as python-dateutil 2.4.2 is already installed.
$ catkin clean grasp_fusion
Starting  >>> grasp_fusion             
Finished  <<< grasp_fusion              [ 0.9 seconds ]
[clean] Summary: All 1 packages succeeded!
[clean]   Ignored:   None.
[clean]   Warnings:  None.
[clean]   Abandoned: None.
[clean]   Failed:    None.
[clean] Runtime: 1.3 seconds total.
[clean] Note: Parts of the workspace have been cleaned which will necessitate re-configuring CMake on the next build.
$ catkin build grasp_fusion --no-deps
---------------------------------------------------------------
Profile:                     default
Extending:          [cached] /opt/ros/kinetic
Workspace:                   /home/ushun/ros/ws_jsk_apc
---------------------------------------------------------------
Build Space:        [exists] /home/ushun/ros/ws_jsk_apc/build
Devel Space:        [exists] /home/ushun/ros/ws_jsk_apc/devel
Install Space:      [unused] /home/ushun/ros/ws_jsk_apc/install
Log Space:          [exists] /home/ushun/ros/ws_jsk_apc/logs
Source Space:       [exists] /home/ushun/ros/ws_jsk_apc/src
DESTDIR:            [unused] None
---------------------------------------------------------------
Devel Space Layout:          linked
Install Space Layout:        None
---------------------------------------------------------------
Additional CMake Args:       None
Additional Make Args:        None
Additional catkin Make Args: None
Internal Make Job Server:    True
Cache Job Environments:      False
---------------------------------------------------------------
Whitelisted Packages:        None
Blacklisted Packages:        None
---------------------------------------------------------------
Workspace configuration appears valid.

NOTE: Forcing CMake to run for each package.
---------------------------------------------------------------
[build] Found '82' packages in 0.0 seconds.                                                                                                   
[build] Package table is up to date.                                                                                                          
Starting  >>> grasp_fusion                                                                                                                    
______________________________________________________________________________________________________________________________________________
Warnings   << grasp_fusion:make /home/ushun/ros/ws_jsk_apc/logs/grasp_fusion/build.make.009.log                                               
/home/ushun/ros/ws_jsk_apc/build/grasp_fusion/venv/local/lib/python2.7/site-packages/pip/_vendor/requests/__init__.py:83: RequestsDependencyWarning: Old version of cryptography ([1, 2, 3]) may cause slowdown.
  warnings.warn(warning, RequestsDependencyWarning)
/home/ushun/ros/ws_jsk_apc/devel/.private/grasp_fusion/share/grasp_fusion/venv/local/lib/python2.7/site-packages/pip/_vendor/requests/__init__.py:83: RequestsDependencyWarning: Old version of cryptography ([1, 2, 3]) may cause slowdown.
  warnings.warn(warning, RequestsDependencyWarning)
/home/ushun/ros/ws_jsk_apc/build/grasp_fusion/venv/local/lib/python2.7/site-packages/pip/_vendor/requests/__init__.py:83: RequestsDependencyWarning: Old version of cryptography ([1, 2, 3]) may cause slowdown.
  warnings.warn(warning, RequestsDependencyWarning)
/home/ushun/ros/ws_jsk_apc/devel/.private/grasp_fusion/share/grasp_fusion/venv/local/lib/python2.7/site-packages/pip/_vendor/requests/__init__.py:83: RequestsDependencyWarning: Old version of cryptography ([1, 2, 3]) may cause slowdown.
  warnings.warn(warning, RequestsDependencyWarning)
imgaug 0.3.0 has requirement numpy>=1.15, but you'll have numpy 1.11.0 which is incompatible.
imgaug 0.3.0 has requirement scikit-image>=0.14.2, but you'll have scikit-image 0.10.1 which is incompatible.
opencv-python-headless 4.1.1.26 has requirement numpy>=1.11.1, but you'll have numpy 1.11.0 which is incompatible.
imgaug 0.3.0 has requirement numpy>=1.15, but you'll have numpy 1.11.0 which is incompatible.
imgaug 0.3.0 has requirement scikit-image>=0.14.2, but you'll have scikit-image 0.10.1 which is incompatible.
opencv-python-headless 4.1.1.26 has requirement numpy>=1.11.1, but you'll have numpy 1.11.0 which is incompatible.
cd /home/ushun/ros/ws_jsk_apc/build/grasp_fusion; catkin build --get-env grasp_fusion | catkin env -si  /usr/bin/make --jobserver-fds=6,7 -j; cd -
..............................................................................................................................................
Finished  <<< grasp_fusion                [ 2 minutes and 30.8 seconds ]                                                                      
[build] Summary: All 1 packages succeeded!                                                                                                    
[build]   Ignored:   81 packages were skipped or are blacklisted.                                                                             
[build]   Warnings:  1 packages succeeded with warnings.                                                                                      
[build]   Abandoned: None.                                                                                                                    
[build]   Failed:    None.                                                                                                                    
[build] Runtime: 2 minutes and 31.5 seconds total.                                                                                            
[build] Note: Workspace packages have changed, please re-source setup files to use them.
pazeshun commented 5 years ago

But this trial may fail

As I thought, b665a65 is meaningless. https://api.travis-ci.org/v3/job/601132004/log.txt

pazeshun commented 5 years ago

Moved to #2699 . This PR is kept for backlog.