search

MIT-TESSE / goseek-challenge

Instructions for competing in the GOSEEK challenge at ICRA 2020
67 stars 16 forks source link

Need help in running eval.py on aws #22

Closed akshay-sharma1995 closed 4 years ago

akshay-sharma1995 commented 4 years ago

Hey, I was trying to setup the goseek-challenge using the given instructions. I am able to set it up on my local machine, but when I tried to set it up on aws I am getting an error on the 5th step. When I run this command: python eval.py --agent-config baselines/config/random-agent.yaml --episode-config config/check-ground-truth.yaml I get the following error:


  Major opcode of failed request:  151 (GLX)
  Minor opcode of failed request:  3 (X_GLXCreateContext)
  Value in failed request:  0x0
  Serial number of failed request:  98
  Current serial number in output stream:  99
Evaluation episode on episode 0, scene 3
Traceback (most recent call last):
  File "eval.py", line 85, in <module>
    results = main(episode_cfg, agent_args)
  File "eval.py", line 66, in main
    return benchmark.evaluate(agent)
  File "/home/ubuntu/tesse-gym/src/tesse_gym/tasks/goseek/goseek_benchmark.py", line 97, in evaluate
    scene_id=self.scenes[episode], random_seed=self.random_seeds[episode]
  File "/home/ubuntu/tesse-gym/src/tesse_gym/tasks/goseek/goseek.py", line 138, in reset
    super().reset(scene_id, random_seed)
  File "/home/ubuntu/tesse-gym/src/tesse_gym/core/tesse_gym.py", line 247, in reset
    observation = self.get_synced_observation()
  File "/home/ubuntu/tesse-gym/src/tesse_gym/core/tesse_gym.py", line 279, in get_synced_observation
    response = self.observe()
  File "/home/ubuntu/tesse-gym/src/tesse_gym/tasks/goseek/goseek_full_perception.py", line 95, in observe
    return self._data_request(DataRequest(metadata=True, cameras=cameras))
  File "/home/ubuntu/tesse-gym/src/tesse_gym/core/tesse_gym.py", line 383, in _data_request
    raise TesseConnectionError()
tesse_gym.core.utils.TesseConnectionError: Cannot receive data from the simulator. The connection is blocked or the simulator is not running.```
ZacRavichandran commented 4 years ago

Hello,

it looks like an issue with the virtual display (required in lieu of a physical display when running headless). We've posted some instructions for setting this up here. Are they of any help?

akshay-sharma1995 commented 4 years ago

Hey @ZacRavichandran, thanks for the reply. The earlier error got resolved by following the instructions you posted. But now I am facing another error on step 7 part b) of the installation instructions. How do I set the correct path for "FindGTSAM.cmake" for kimera-vio? When I run the command: `docker build --rm -t goseek-kimera . I get the following output:

Sending build context to Docker daemon  1.754GB
Step 1/52 : FROM nvidia/cuda:10.0-cudnn7-devel-ubuntu18.04
 ---> a387f44414ed
Step 2/52 : RUN apt-get clean && apt-get update && apt-get install -y locales
 ---> Using cache
 ---> 42990b72ff12
Step 3/52 : RUN echo "en_US.UTF-8 UTF-8" > /etc/locale.gen &&     locale-gen
 ---> Using cache
 ---> ca5a1b97c8bd
Step 4/52 : ENV LC_ALL en_US.UTF-8
 ---> Using cache
 ---> 1ffca8fe332c
Step 5/52 : ENV LANG en_US.UTF-8
 ---> Using cache
 ---> c46f8ed13ff6
Step 6/52 : ENV LANGUAGE en_US.UTF-8
 ---> Using cache
 ---> 14b8a62dba62
Step 7/52 : ENV SHELL /bin/bash
 ---> Using cache
 ---> 4f2e588d7e95
Step 8/52 : ENV DEBIAN_FRONTEND noninteractive
 ---> Using cache
 ---> 7e4001591f6e
Step 9/52 : SHELL ["/bin/bash", "-c"]
 ---> Using cache
 ---> f2c4dee5f028
Step 10/52 : RUN apt-get update &&     apt-get install -y curl bzip2 wget vim ffmpeg git tmux unzip
 ---> Using cache
 ---> 21df12b4fee8
Step 11/52 : RUN apt-get update && apt-get install -y libsm6 libxext6 libxrender-dev
 ---> Using cache
 ---> 08f8fb4b5ab4
Step 12/52 : RUN apt-get update &&     apt-get install -y --no-install-recommends apt-utils
 ---> Using cache
 ---> 6665e9a4552a
Step 13/52 : ENV ROS_DISTRO melodic
 ---> Using cache
 ---> 5d9a621cc549
Step 14/52 : RUN apt-get update &&     apt-get install -y     lsb-release     gnupg
 ---> Using cache
 ---> 23c1c6b3a5df
Step 15/52 : RUN sh -c 'echo "deb http://packages.ros.org/ros/ubuntu $(lsb_release -sc) main" > /etc/apt/sources.list.d/ros-latest.list'
 ---> Using cache
 ---> 92f70905925d
Step 16/52 : RUN mkdir ~/.gnupg
 ---> Using cache
 ---> d07836a692fb
Step 17/52 : RUN echo "disable-ipv6" >> ~/.gnupg/dirmngr.conf
 ---> Using cache
 ---> 53b6b6d00a04
Step 18/52 : RUN curl -sSL 'http://keyserver.ubuntu.com/pks/lookup?op=get&search=0xC1CF6E31E6BADE8868B172B4F42ED6FBAB17C654' | apt-key add -
 ---> Using cache
 ---> 5a78c71e97e2
Step 19/52 : RUN apt-get update &&     apt-get install -y ros-melodic-desktop-full
 ---> Using cache
 ---> 9961d5ed9b82
Step 20/52 : RUN apt-get install -y     python     python-pip     python-dev
 ---> Using cache
 ---> 24ac0042db28
Step 21/52 : RUN apt-get install python-rosdep
 ---> Using cache
 ---> e1cc5088b1c6
Step 22/52 : RUN rosdep init
 ---> Using cache
 ---> 8148aacc4039
Step 23/52 : RUN rosdep update
 ---> Using cache
 ---> c8c0e5bd06a7
Step 24/52 : RUN source /opt/ros/melodic/setup.bash &&     apt install -y     python-rosinstall     python-rosinstall-generator     python-wstool build-essential     python-catkin-tools
 ---> Using cache
 ---> ac6e11dccc51
Step 25/52 : RUN source /opt/ros/melodic/setup.bash &&     mkdir -p /catkin_ws/src &&     cd /catkin_ws/ &&     catkin init &&     catkin config --extend /opt/ros/melodic &&     catkin config --cmake-args -DCMAKE_BUILD_TYPE=Release &&     catkin config --merge-devel
 ---> Using cache
 ---> 0377f7b8ae12
Step 26/52 : WORKDIR /catkin_ws/src
 ---> Using cache
 ---> 3373cdf5343c
Step 27/52 : RUN pip install scipy
 ---> Using cache
 ---> d1d49566580a
Step 28/52 : RUN git clone https://github.com/MIT-TESSE/tesse-interface.git -b 0.1.3-SNAPSHOT &&     cd tesse-interface &&     python setup.py install &&     cd ..
 ---> Using cache
 ---> 3eeba3e09702
Step 29/52 : WORKDIR /catkin_ws/src
 ---> Using cache
 ---> d4ee785cb6f8
Step 30/52 : RUN source /opt/ros/melodic/setup.bash
 ---> Using cache
 ---> 26a01b6e91cd
Step 31/52 : RUN wstool init
 ---> Using cache
 ---> cf1f2d69c3b2
Step 32/52 : RUN git clone https://github.com/MIT-TESSE/tesse-ros-bridge.git &&     wstool merge -y tesse-ros-bridge/install/tesse_ros_bridge.rosinstall
 ---> Using cache
 ---> a5f41c2a18fc
Step 33/52 : RUN git clone https://github.com/MIT-TESSE/tesse-segmentation-ros.git &&     wstool merge -y tesse-segmentation-ros/install/tesse_segmentation_ros.rosinstall
 ---> Using cache
 ---> 8841dd0f2c64
Step 34/52 : RUN git clone https://github.com/MIT-TESSE/tesse-gym-bridge.git &&     wstool merge -y tesse-gym-bridge/install/tesse_gym_bridge.rosinstall
 ---> Using cache
 ---> 71d9758c0ae2
Step 35/52 : RUN wstool update
 ---> Using cache
 ---> 243d46ee3f2f
Step 36/52 : RUN catkin build tesse_ros_bridge tesse_segmentation_ros tesse_gym_bridge
 ---> Using cache
 ---> d45d32ba56d0
Step 37/52 : RUN apt-get update &&      apt-get install -y     libgoogle-glog-dev     doxygen     cmake build-essential pkg-config autoconf     libboost-all-dev     libjpeg-dev libpng-dev libtiff-dev     libvtk6-dev libgtk-3-dev     libatlas-base-dev gfortran     libparmetis-dev     libtbb-dev
 ---> Using cache
 ---> 783ef09f29a1
Step 38/52 : ENV PYTHONIOENCODING=UTF-8
 ---> Using cache
 ---> ad63bef62627
Step 39/52 : WORKDIR /catkin_ws/src
 ---> Using cache
 ---> 85f1cedd71a5
Step 40/52 : RUN git clone https://github.com/MIT-SPARK/Kimera-VIO-ROS.git &&       cd Kimera-VIO-ROS &&       git checkout db6afe795f3bf6b88842d2c45cb2306926505051 &&       cd .. &&       wstool merge -y Kimera-VIO-ROS/install/kimera_vio_ros_https.rosinstall &&       wstool update
 ---> Using cache
 ---> c5320ebcc7de
Step 41/52 : RUN git clone https://github.com/MIT-SPARK/Kimera-Semantics.git &&       cd Kimera-Semantics &&       git checkout 0ae759f24cce9e9ffc28d0495327a3d2b37d2c99 &&       cd .. &&       wstool merge -y Kimera-Semantics/kimera/install/kimera_semantics_https.rosinstall &&       wstool update
 ---> Using cache
 ---> 01b19fe0d4df
Step 42/52 : RUN   cd Kimera-VIO-ROS &&       git checkout db6afe795f3bf6b88842d2c45cb2306926505051 &&       cd .. &&       cd Kimera-VIO &&       git fetch && git checkout 13a52d8772305be143ebdf470c8714759e411f59 &&       cd ..
 ---> Using cache
 ---> 4640056ab661
Step 43/52 : RUN cd gtsam &&     git fetch &&     git checkout develop &&     git pull
 ---> Using cache
 ---> ed1a0b368bb1
Step 44/52 : RUN catkin build
 ---> Running in 93b45e68c8b8
NOTICE: Could not determine the width of the terminal. A default width of 80 will be used. This warning will only be printed once.
-------------------------------------------------------
Profile:                     default
Extending:        [explicit] /opt/ros/melodic
Workspace:                   /catkin_ws
-------------------------------------------------------
Build Space:        [exists] /catkin_ws/build
Devel Space:        [exists] /catkin_ws/devel
Install Space:      [unused] /catkin_ws/install
Log Space:          [exists] /catkin_ws/logs
Source Space:       [exists] /catkin_ws/src
DESTDIR:            [unused] None
-------------------------------------------------------
Devel Space Layout:          merged
Install Space Layout:        None
-------------------------------------------------------
Additional CMake Args:       -DCMAKE_BUILD_TYPE=Release
Additional Make Args:        None
Additional catkin Make Args: None
Internal Make Job Server:    True
Cache Job Environments:      False
-------------------------------------------------------
Whitelisted Packages:        None
Blacklisted Packages:        None
-------------------------------------------------------
Workspace configuration appears valid.
-------------------------------------------------------
[build] Found '51' packages in 0.0 seconds.                                    
[build] Updating package table.                                                
Starting >>> catkin_boost_python_buildtool                                     
Starting >>> catkin_simple                                                     
Starting >>> disparity_image_proc                                              
Starting >>> gtsam                                                             
Finished <<< catkin_simple                                     [ 0.3 seconds ] 
Starting >>> kimera_vio                                                        
_______________________________________________________________________________
Errors << kimera_vio:cmake /catkin_ws/logs/kimera_vio/build.cmake.000.log      
CMake Error at /catkin_ws/src/Kimera-VIO/CMakeLists.txt:34 (find_package):
  By not providing "FindGTSAM.cmake" in CMAKE_MODULE_PATH this project has
  asked CMake to find a package configuration file provided by "GTSAM", but
  CMake did not find one.

  Could not find a package configuration file provided by "GTSAM" with any of
  the following names:

    GTSAMConfig.cmake
    gtsam-config.cmake

  Add the installation prefix of "GTSAM" to CMAKE_PREFIX_PATH or set
  "GTSAM_DIR" to a directory containing one of the above files.  If "GTSAM"
  provides a separate development package or SDK, be sure it has been
  installed.

cd /catkin_ws/build/kimera_vio; catkin build --get-env kimera_vio | catkin env -si  /usr/bin/cmake /catkin_ws/src/Kimera-VIO --no-warn-unused-cli -DCMAKE_INSTALL_PREFIX=/catkin_ws/devel -DCMAKE_BUILD_TYPE=Release; cd -
...............................................................................
Failed << kimera_vio:cmake                                   [ Exited with code 1 ]
Failed <<< kimera_vio                                        [ 2.1 seconds ]   
Abandoned <<< python_module                                     [ Unrelated job failed ]
Abandoned <<< sm_doc                                            [ Unrelated job failed ]
Abandoned <<< cmake_external_project_catkin                     [ Unrelated job failed ]
Abandoned <<< dbow2_catkin                                      [ Unrelated job failed ]
Abandoned <<< eigen_catkin                                      [ Unrelated job failed ]
Abandoned <<< gflags_catkin                                     [ Unrelated job failed ]
Abandoned <<< kimera_rviz_markers                               [ Unrelated job failed ]
Abandoned <<< mesh_rviz_plugins                                 [ Unrelated job failed ]
Abandoned <<< opencv3_catkin                                    [ Unrelated job failed ]
Abandoned <<< pose_graph_tools                                  [ Unrelated job failed ]
Abandoned <<< protobuf_catkin                                   [ Unrelated job failed ]
Abandoned <<< sm_common                                         [ Unrelated job failed ]
Abandoned <<< sm_deprecation                                    [ Unrelated job failed ]
Abandoned <<< sm_logging                                        [ Unrelated job failed ]
Abandoned <<< sm_random                                         [ Unrelated job failed ]
Abandoned <<< tesse_gym_bridge                                  [ Unrelated job failed ]
Abandoned <<< tesse_ros_bridge                                  [ Unrelated job failed ]
Abandoned <<< tesse_segmentation_ros                            [ Unrelated job failed ]
Abandoned <<< voxblox_msgs                                      [ Unrelated job failed ]
Abandoned <<< catkin_boost_python_buildtool_test                [ Unrelated job failed ]
Abandoned <<< glog_catkin                                       [ Unrelated job failed ]
Abandoned <<< eigen_checks                                      [ Unrelated job failed ]
Abandoned <<< kimera_rpgo                                       [ Unrelated job failed ]
Abandoned <<< minkindr                                          [ Unrelated job failed ]
Abandoned <<< minkindr_conversions                              [ Unrelated job failed ]
Abandoned <<< opengv_catkin                                     [ Unrelated job failed ]
Abandoned <<< pointcloud_recolor                                [ Unrelated job failed ]
Abandoned <<< kimera_vio_ros                                    [ Unrelated job failed ]
Abandoned <<< numpy_eigen                                       [ Unrelated job failed ]
Abandoned <<< minkindr_python                                   [ Unrelated job failed ]
Abandoned <<< sm_boost                                          [ Unrelated job failed ]
Abandoned <<< sm_matrix_archive                                 [ Unrelated job failed ]
Abandoned <<< sm_opencv                                         [ Unrelated job failed ]
Abandoned <<< sm_property_tree                                  [ Unrelated job failed ]
Abandoned <<< sm_eigen                                          [ Unrelated job failed ]
Abandoned <<< sm_kinematics                                     [ Unrelated job failed ]
Abandoned <<< sm_matlab                                         [ Unrelated job failed ]
Abandoned <<< sm_timing                                         [ Unrelated job failed ]
Abandoned <<< sm_value_store                                    [ Unrelated job failed ]
Abandoned <<< sm_python                                         [ Unrelated job failed ]
Abandoned <<< voxblox                                           [ Unrelated job failed ]
Abandoned <<< kimera_semantics                                  [ Unrelated job failed ]
Abandoned <<< voxblox_rviz_plugin                               [ Unrelated job failed ]
Abandoned <<< voxblox_ros                                       [ Unrelated job failed ]
Abandoned <<< kimera_semantics_ros                              [ Unrelated job failed ]
Finished <<< catkin_boost_python_buildtool                     [ 3.4 seconds ] 
Finished <<< disparity_image_proc                              [ 15.7 seconds ]
_______________________________________________________________________________
Warnings << gtsam:make /catkin_ws/logs/gtsam/build.make.000.log                
/catkin_ws/src/gtsam/gtsam/3rdparty/CCOLAMD/Source/ccolamd.c: In function ‘ccolamd2’:
/catkin_ws/src/gtsam/gtsam/3rdparty/CCOLAMD/Source/ccolamd.c:1563:9: warning: variable ‘cs’ set but not used [-Wunused-but-set-variable]
     Int cs ;
         ^~
In file included from /catkin_ws/src/gtsam/gtsam/3rdparty/Eigen/Eigen/Core:440:0,
                 from /catkin_ws/src/gtsam/gtsam/3rdparty/Eigen/Eigen/Dense:1,
                 from /catkin_ws/src/gtsam/gtsam/base/OptionalJacobian.h:22,
                 from /catkin_ws/src/gtsam/gtsam/base/Matrix.h:26,
                 from /catkin_ws/src/gtsam/gtsam/base/Manifold.h:22,
                 from /catkin_ws/src/gtsam/gtsam/geometry/BearingRange.h:21,
                 from /catkin_ws/src/gtsam/gtsam/geometry/Pose3.h:22,
                 from /catkin_ws/src/gtsam/examples/SFMdata.h:29,
                 from /catkin_ws/src/gtsam/examples/SFMExample.cpp:19:
/catkin_ws/src/gtsam/gtsam/3rdparty/Eigen/Eigen/src/Core/CoreEvaluators.h: In function ‘int main(int, char**)’:
/catkin_ws/src/gtsam/gtsam/3rdparty/Eigen/Eigen/src/Core/CoreEvaluators.h:713:74: warning: ‘*((void*)&<anonymous> +16)’ may be used uninitialized in this function [-Wmaybe-uninitialized]
     return m_functor(m_lhsImpl.coeff(row, col), m_rhsImpl.coeff(row, col));
                                                                          ^
cd /catkin_ws/build/gtsam; catkin build --get-env gtsam | catkin env -si  /usr/bin/make --jobserver-fds=6,7 -j; cd -
...............................................................................
Finished <<< gtsam                                             [ 12 minutes and 34.0 seconds ]
[build] Summary: 4 of 50 packages succeeded.                                   
[build] Ignored: 1 packages were skipped or are blacklisted.                   
[build] Warnings: 1 packages succeeded with warnings.                          
[build] Abandoned: 45 packages were abandoned.                                 
[build] Failed: 1 packages failed.                                             
[build] Runtime: 12 minutes and 34.4 seconds total.                            
[build] Note: Workspace packages have changed, please re-source setup files to use them.
The command '/bin/bash -c catkin build' returned a non-zero code: 1
ZacRavichandran commented 4 years ago

It looks like one of the cloned ROS packages introduced a breaking change. I'll dig into this and let you when I have a fix.

In the meantime, it sounds like everything else is up and running so hopefully this isn't blocking!

ZacRavichandran commented 4 years ago

I should push a fix shortly. For now, a workaround is to build the required packages before kimera_vio. Try adding this line:

catkin build gtsam opengv_catkin opencv3_catkin kimera_rpgo dbow2_catkin

before the final catkin build call in the goseek-kimera Dockerfile here

akshay-sharma1995 commented 4 years ago

Hey @ZacRavichandran, this solution did work, and the docker image is getting built now.

I should push a fix shortly. For now, a workaround is to build the required packages before kimera_vio. Try adding this line:

catkin build gtsam opengv_catkin opencv3_catkin kimera_rpgo dbow2_catkin

before the final catkin build call in the goseek-kimera Dockerfile here

Thanks a lot for the help.

ZacRavichandran commented 4 years ago

np, good to hear!