search

locusrobotics / catkin_virtualenv

Bundle python requirements in a catkin package via virtualenv
http://wiki.ros.org/catkin_virtualenv
83 stars 25 forks source link

Problems building a catkin package with catkin_virtualenv due to `requirements.in` #78

Closed rojas70 closed 2 years ago

rojas70 commented 2 years ago

I am currently trying to run your setup on https://github.com/NVlabs/Deep_Object_Pose.

I have updated packages.xml and CMakeLists.txt accordingly and run catkin_make on catkin_virtualenv.

I get the following error when running: catkin_make -DCATKIN_WHITELIST_PACKAGES="dope": and trying to use requiments.in.

>>$ catkin_make -DCATKIN_WHITELIST_PACKAGES="dope"
Base path: /media/juan/sdd/ros/noetic/ros_ws
Source space: /media/juan/sdd/ros/noetic/ros_ws/src
Build space: /media/juan/sdd/ros/noetic/ros_ws/build
Devel space: /media/juan/sdd/ros/noetic/ros_ws/devel
Install space: /media/juan/sdd/ros/noetic/ros_ws/install
####
#### Running command: "cmake /media/juan/sdd/ros/noetic/ros_ws/src -DCATKIN_WHITELIST_PACKAGES=dope -DCATKIN_DEVEL_PREFIX=/media/juan/sdd/ros/noetic/ros_ws/devel -DCMAKE_INSTALL_PREFIX=/media/juan/sdd/ros/noetic/ros_ws/install -G Unix Makefiles" in "/media/juan/sdd/ros/noetic/ros_ws/build"
####
-- Using CATKIN_DEVEL_PREFIX: /media/juan/sdd/ros/noetic/ros_ws/devel
-- Using CMAKE_PREFIX_PATH: /media/juan/sdd/ros/noetic/ros_ws/devel;/opt/ros/noetic
-- This workspace overlays: /media/juan/sdd/ros/noetic/ros_ws/devel;/opt/ros/noetic
-- Found PythonInterp: /usr/bin/python3 (found suitable version "3.8.10", minimum required is "3") 
-- Using PYTHON_EXECUTABLE: /usr/bin/python3
-- Using Debian Python package layout
-- Using empy: /usr/lib/python3/dist-packages/em.py
-- Using CATKIN_ENABLE_TESTING: ON
-- Call enable_testing()
-- Using CATKIN_TEST_RESULTS_DIR: /media/juan/sdd/ros/noetic/ros_ws/build/test_results
-- Forcing gtest/gmock from source, though one was otherwise available.
-- Found gtest sources under '/usr/src/googletest': gtests will be built
-- Found gmock sources under '/usr/src/googletest': gmock will be built
-- Found PythonInterp: /usr/bin/python3 (found version "3.8.10") 
-- Using Python nosetests: /usr/bin/nosetests3
-- catkin 0.8.10
-- BUILD_SHARED_LIBS is on
-- BUILD_SHARED_LIBS is on
-- Using CATKIN_WHITELIST_PACKAGES: dope
-- ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-- ~~  traversing 1 packages in topological order:
-- ~~  - dope
-- ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-- +++ processing catkin package: 'dope'
-- ==> add_subdirectory(vision/dope)
-- Using these message generators: gencpp;geneus;genlisp;gennodejs;genpy
-- Using system site packages
-- Using virtualenv to run Python nosetests: 
CMake Warning at python_tools/catkin_virtualenv/catkin_virtualenv/cmake/catkin_install_python.cmake:52 (message):
  Making /media/juan/sdd/ros/noetic/ros_ws/src/vision/dope/nodes/camera
  non-executable.  Otherwise 'rosrun dope camera' will not work as expected.
Call Stack (most recent call first):
  vision/dope/CMakeLists.txt:178 (catkin_install_python)

CMake Warning at python_tools/catkin_virtualenv/catkin_virtualenv/cmake/catkin_install_python.cmake:52 (message):
  Making /media/juan/sdd/ros/noetic/ros_ws/src/vision/dope/nodes/dope
  non-executable.  Otherwise 'rosrun dope dope' will not work as expected.
Call Stack (most recent call first):
  vision/dope/CMakeLists.txt:178 (catkin_install_python)

CMake Warning at python_tools/catkin_virtualenv/catkin_virtualenv/cmake/catkin_install_python.cmake:52 (message):
  Making /media/juan/sdd/ros/noetic/ros_ws/src/vision/dope/scripts/train.py
  non-executable.  Otherwise 'rosrun dope train.py' will not work as
  expected.
Call Stack (most recent call first):
  vision/dope/CMakeLists.txt:178 (catkin_install_python)

-- Configuring done
-- Generating done
-- Build files have been written to: /media/juan/sdd/ros/noetic/ros_ws/build
####
#### Running command: "make -j8 -l8" in "/media/juan/sdd/ros/noetic/ros_ws/build"
####
Scanning dependencies of target dope_generate_virtualenv
make[2]: *** No rule to make target '/media/juan/sdd/ros/noetic/ros_ws/src/requirements.in', needed by '/media/juan/sdd/ros/noetic/ros_ws/src/vision/dope/requirements.txt'.  Stop.
make[1]: *** [CMakeFiles/Makefile2:2088: vision/dope/CMakeFiles/dope_generate_virtualenv.dir/all] Error 2
make: *** [Makefile:141: all] Error 2
Invoking "make -j8 -l8" failed

I am not sure if this is failing due to the presence of .py files in nodes and scripts, or something else?

The contents of package.xml is:

<?xml version="1.0"?>
<package format="2">
  <name>dope</name>
  <version>0.0.0</version>
  <description>The DOPE package for deep object pose estimation</description>

  <!-- One maintainer tag required, multiple allowed, one person per tag -->
  <!-- Example:  -->
  <!-- <maintainer email="jane.doe@example.com">Jane Doe</maintainer> -->
  <maintainer email="jtremblay@nvidia.com">jtremblay</maintainer>

  <!-- One license tag required, multiple allowed, one license per tag -->
  <!-- Commonly used license strings: -->
  <!--   BSD, MIT, Boost Software License, GPLv2, GPLv3, LGPLv2.1, LGPLv3 -->
  <license>CC BY-NC-SA 4.0</license>

  <!-- Url tags are optional, but multiple are allowed, one per tag -->
  <!-- Optional attribute type can be: website, bugtracker, or repository -->
  <!-- Example: -->
  <url type="website">https://research.nvidia.com/publication/2018-09_Deep-Object-Pose</url>

  <!-- Author tags are optional, multiple are allowed, one per tag -->
  <!-- Authors do not have to be maintainers, but could be -->
  <!-- Example: -->
  <!-- <author email="jane.doe@example.com">Jane Doe</author> -->

  <!-- The *depend tags are used to specify dependencies -->
  <!-- Dependencies can be catkin packages or system dependencies -->
  <!-- Examples: -->
  <!-- Use depend as a shortcut for packages that are both build and exec dependencies -->
  <!--   <depend>roscpp</depend> -->
  <!--   Note that this is equivalent to the following: -->
  <!--   <build_depend>roscpp</build_depend> -->
  <!--   <exec_depend>roscpp</exec_depend> -->
  <!-- Use build_depend for packages you need at compile time: -->
  <!--   <build_depend>message_generation</build_depend> -->
  <!-- Use build_export_depend for packages you need in order to build against this package: -->
  <!--   <build_export_depend>message_generation</build_export_depend> -->
  <!-- Use buildtool_depend for build tool packages: -->
  <!--   <buildtool_depend>catkin</buildtool_depend> -->
  <!-- Use exec_depend for packages you need at runtime: -->
  <!--   <exec_depend>message_runtime</exec_depend> -->
  <!-- Use test_depend for packages you need only for testing: -->
  <!--   <test_depend>gtest</test_depend> -->
  <!-- Use doc_depend for packages you need only for building documentation: -->
  <!--   <doc_depend>doxygen</doc_depend> -->
  <buildtool_depend>catkin</buildtool_depend>

  <!-- virtualenv -->
  <build_depend>catkin_virtualenv</build_depend>

  <depend>camera_info_manager_py</depend>
  <depend>cv_bridge</depend>
  <depend>geometry_msgs</depend>
  <depend>message_filters</depend>
  <depend>python-argparse</depend>
  <depend>resource_retriever</depend>
  <depend>rospy</depend>
  <depend>sensor_msgs</depend>
  <depend>std_msgs</depend>
  <depend>tf</depend>
  <depend>vision_msgs</depend>
  <depend>visualization_msgs</depend>
  <depend>python3-pyrr-pip</depend>
  <depend>python-pytorch-pip</depend>
  <depend>python3-numpy</depend>
  <depend>python3-scipy</depend>
  <depend>python3-opencv</depend>
  <depend>python3-pil</depend>
  <depend>python-configparser</depend>

  <!-- The export tag contains other, unspecified, tags -->
  <export>
    <pip_requirements>requirements.txt</pip_requirements>
  </export>
</package>

And the CMakeLists.txt file is:

cmake_minimum_required(VERSION 2.8.3)
project(dope)

## Compile as C++11, supported in ROS Kinetic and newer
# add_compile_options(-std=c++11)

## Find catkin macros and libraries
## if COMPONENTS list like find_package(catkin REQUIRED COMPONENTS xyz)
## is used, also find other catkin packages
find_package(catkin REQUIRED COMPONENTS
  cv_bridge
  geometry_msgs
  message_filters
  resource_retriever
  rospy
  sensor_msgs
  std_msgs
  tf
  vision_msgs
  visualization_msgs
  catkin_virtualenv
)

# Generate the virtualenv
catkin_generate_virtualenv(
  # Specify the input requirements for this package that catkin_virtualenv will automatically lock.
  INPUT_REQUIREMENTS requirements.in
  )

## System dependencies are found with CMake's conventions
# find_package(Boost REQUIRED COMPONENTS system)

## Uncomment this if the package has a setup.py. This macro ensures
## modules and global scripts declared therein get installed
## See http://ros.org/doc/api/catkin/html/user_guide/setup_dot_py.html
catkin_python_setup()

################################################
## Declare ROS messages, services and actions ##
################################################

## To declare and build messages, services or actions from within this
## package, follow these steps:
## * Let MSG_DEP_SET be the set of packages whose message types you use in
##   your messages/services/actions (e.g. std_msgs, actionlib_msgs, ...).
## * In the file package.xml:
##   * add a build_depend tag for "message_generation"
##   * add a build_depend and a exec_depend tag for each package in MSG_DEP_SET
##   * If MSG_DEP_SET isn't empty the following dependency has been pulled in
##     but can be declared for certainty nonetheless:
##     * add a exec_depend tag for "message_runtime"
## * In this file (CMakeLists.txt):
##   * add "message_generation" and every package in MSG_DEP_SET to
##     find_package(catkin REQUIRED COMPONENTS ...)
##   * add "message_runtime" and every package in MSG_DEP_SET to
##     catkin_package(CATKIN_DEPENDS ...)
##   * uncomment the add_*_files sections below as needed
##     and list every .msg/.srv/.action file to be processed
##   * uncomment the generate_messages entry below
##   * add every package in MSG_DEP_SET to generate_messages(DEPENDENCIES ...)

## Generate messages in the 'msg' folder
# add_message_files(
#   FILES
#   Message1.msg
#   Message2.msg
# )

## Generate services in the 'srv' folder
# add_service_files(
#   FILES
#   Service1.srv
#   Service2.srv
# )

## Generate actions in the 'action' folder
# add_action_files(
#   FILES
#   Action1.action
#   Action2.action
# )

## Generate added messages and services with any dependencies listed here
# generate_messages(
#   DEPENDENCIES
#   std_msgs
# )

################################################
## Declare ROS dynamic reconfigure parameters ##
################################################

## To declare and build dynamic reconfigure parameters within this
## package, follow these steps:
## * In the file package.xml:
##   * add a build_depend and a exec_depend tag for "dynamic_reconfigure"
## * In this file (CMakeLists.txt):
##   * add "dynamic_reconfigure" to
##     find_package(catkin REQUIRED COMPONENTS ...)
##   * uncomment the "generate_dynamic_reconfigure_options" section below
##     and list every .cfg file to be processed

## Generate dynamic reconfigure parameters in the 'cfg' folder
# generate_dynamic_reconfigure_options(
#   cfg/DynReconf1.cfg
#   cfg/DynReconf2.cfg
# )

###################################
## catkin specific configuration ##
###################################
## The catkin_package macro generates cmake config files for your package
## Declare things to be passed to dependent projects
## INCLUDE_DIRS: uncomment this if your package contains header files
## LIBRARIES: libraries you create in this project that dependent projects also need
## CATKIN_DEPENDS: catkin_packages dependent projects also need
## DEPENDS: system dependencies of this project that dependent projects also need
catkin_package(
  CATKIN_DEPENDS geometry_msgs sensor_msgs std_msgs vision_msgs visualization_msgs
#  DEPENDS system_lib
)

###########
## Build ##
###########

## Specify additional locations of header files
## Your package locations should be listed before other locations
include_directories(
# include
  ${catkin_INCLUDE_DIRS}
)

## Declare a C++ library
# add_library(${PROJECT_NAME}
#   src/${PROJECT_NAME}/dope/dope.cpp
# )

## Add cmake target dependencies of the library
## as an example, code may need to be generated before libraries
## either from message generation or dynamic reconfigure
# add_dependencies(${PROJECT_NAME} ${${PROJECT_NAME}_EXPORTED_TARGETS} ${catkin_EXPORTED_TARGETS})

## Declare a C++ executable
## With catkin_make all packages are built within a single CMake context
## The recommended prefix ensures that target names across packages don't collide
# add_executable(${PROJECT_NAME}_node src/dope_vis_node.cpp)

## Rename C++ executable without prefix
## The above recommended prefix causes long target names, the following renames the
## target back to the shorter version for ease of user use
## e.g. "rosrun someones_pkg node" instead of "rosrun someones_pkg someones_pkg_node"
# set_target_properties(${PROJECT_NAME}_node PROPERTIES OUTPUT_NAME node PREFIX "")

## Add cmake target dependencies of the executable
## same as for the library above
# add_dependencies(${PROJECT_NAME}_node ${${PROJECT_NAME}_EXPORTED_TARGETS} ${catkin_EXPORTED_TARGETS})

## Specify libraries to link a library or executable target against
# target_link_libraries(${PROJECT_NAME}_node
#   ${catkin_LIBRARIES}
# )

#############
## Install ##
#############

# all install targets should use catkin DESTINATION variables
# See http://ros.org/doc/api/catkin/html/adv_user_guide/variables.html

## Mark executable scripts (Python etc.) for installation
## in contrast to setup.py, you can choose the destination
install(FILES requirements.txt
  DESTINATION ${CATKIN_PACKAGE_SHARE_DESTINATION}
)

catkin_install_python(PROGRAMS
  nodes/camera
  nodes/dope
  scripts/train.py
  DESTINATION ${CATKIN_PACKAGE_BIN_DESTINATION})

## Mark executables and/or libraries for installation
# install(TARGETS ${PROJECT_NAME} ${PROJECT_NAME}_node
#   ARCHIVE DESTINATION ${CATKIN_PACKAGE_LIB_DESTINATION}
#   LIBRARY DESTINATION ${CATKIN_PACKAGE_LIB_DESTINATION}
#   RUNTIME DESTINATION ${CATKIN_PACKAGE_BIN_DESTINATION}
# )

## Mark cpp header files for installation
# install(DIRECTORY include/${PROJECT_NAME}/
#   DESTINATION ${CATKIN_PACKAGE_INCLUDE_DESTINATION}
#   FILES_MATCHING PATTERN "*.h"
#   PATTERN ".svn" EXCLUDE
# )

## Mark other files for installation (e.g. launch and bag files, etc.)
# install(FILES
#   # myfile1
#   # myfile2
#   DESTINATION ${CATKIN_PACKAGE_SHARE_DESTINATION}
# )

#############
## Testing ##
#############

## Add gtest based cpp test target and link libraries
# catkin_add_gtest(${PROJECT_NAME}-test test/test_dope_vis.cpp)
# if(TARGET ${PROJECT_NAME}-test)
#   target_link_libraries(${PROJECT_NAME}-test ${PROJECT_NAME})
# endif()

## Add folders to be run by python nosetests
# catkin_add_nosetests(test)

This is my current file structure:

── CMakeLists.txt
├── config
│   ├── camera_info.yaml
│   ├── config_pose.yaml
│   ├── config_pose.yaml~
│   ├── config_pose.yaml.orig
│   └── config_pose.yaml.orig~
├── doc
│   └── camera_tutorial.md
├── docker
│   ├── Dockerfile.noetic
│   ├── init_workspace.sh
│   ├── readme.md
│   └── run_dope_docker.sh
├── dope_objects.png
├── launch
│   ├── camera.launch
│   └── dope.launch
├── license.md
├── nodes
│   ├── camera
│   └── dope
├── package.xml
├── readme.md
├── requirements.in
├── requirements.txt
├── scripts
│   └── train.py
├── setup.py
├── src
│   └── dope
├── weights

If I comment requirement.in, then I get another set of incompatibility errors with dependencies as shown below:

####
#### Running command: "make -j8 -l8" in "/media/juan/sdd/ros/noetic/ros_ws/build"
####
[ 25%] Generate virtualenv in /media/juan/sdd/ros/noetic/ros_ws/build/venv
True
ERROR: py-deps 1.0.1 has requirement setuptools>=46.1, but you'll have setuptools 44.0.0 which is incompatible.
ERROR: pgraph 1.0.3 has requirement setuptools<47.0,>=46.0, but you'll have setuptools 44.0.0 which is incompatible.
[ 50%] Install requirements to /media/juan/sdd/ros/noetic/ros_ws/build/venv
ERROR: torchtext 0.6.0 requires torch, which is not installed.
ERROR: tensorflow 2.3.1 requires absl-py>=0.7.0, which is not installed.
ERROR: tensorboard 2.4.0 requires absl-py>=0.4, which is not installed.
ERROR: sphinx 3.5.3 requires alabaster<0.8,>=0.7, which is not installed.
ERROR: pytorch-lightning 1.1.0 requires torch>=1.3, which is not installed.
ERROR: hydra-core 1.0.4 requires antlr4-python3-runtime==4.8, which is not installed.
ERROR: tensorflow 2.3.1 has requirement h5py<2.11.0,>=2.10.0, but you'll have h5py 3.1.0 which is incompatible.
ERROR: awscli 1.19.64 has requirement docutils<0.16,>=0.10, but you'll have docutils 0.16 which is incompatible.
ERROR: roboticstoolbox-python 0.6.2 requires ansitable, which is not installed.
ERROR: spatialmath-python 0.8.7 requires ansitable, which is not installed.
ERROR: robosuite 1.2.2 has requirement mujoco-py==2.0.2.13, but you'll have mujoco-py 2.0.2.2 which is incompatible.
[ 75%] Prepare relocated virtualenvs for develspace and installspace
[100%] Per-package virtualenv target
[100%] Built target dope_generate_virtualenv

My current requirements.txt looks as follows:

pyrr==0.10.3
torch==1.6.0
torchvision==0.7.0
numpy==1.17.4
scipy==1.5.2
opencv_python==4.4.0.44
Pillow==8.1.1
configparser==5.0.0

Are these ERROR springing up from dependencies detected with other packages in my PYTHON_PATH?

paulbovbel commented 2 years ago

Can you try using catkin build? I don't test against catkin_make since it does some funny things with cmake.