search

AndrejOrsula / drl_grasping

Deep Reinforcement Learning for Robotic Grasping from Octrees
https://arxiv.org/pdf/2208.00818
BSD 3-Clause "New" or "Revised" License
404 stars 54 forks source link

Error running ex_train.bash in a headless way #96

Closed dsriaditya999 closed 2 years ago

dsriaditya999 commented 2 years ago

Hi,

I was trying to train my agent with the kinova robotic arm and sac algorithm using the ex_train.bash script. But I get the following error. Can you please look into this @AndrejOrsula ?( I actually wanted to train on a GPU Server (Nvidia V100 GPU Card) without graphical elements )

The error I am getting is as follows :

Inserting camera
[Err] [Ogre2RenderEngine.cc:376] Unable to open display: 
terminate called after throwing an instance of 'Ogre::RenderingAPIException'
  what():  OGRE EXCEPTION(3:RenderingAPIException): Couldn`t open X display  in GLXGLSupport::getGLDisplay at /home/jenkins/workspace/ogre-2.2-debbuilder/repo/RenderSystems/GL3Plus/src/windowing/GLX/OgreGLXGLSupport.cpp (line 796)
Received interrupt signal SIGABRT
Found signal callback
[Err] [GazeboSimulator.cpp:237] The simulator was not initialized
Traceback (most recent call last):
  File "/root/drl_grasping/drl_grasping/install/lib/drl_grasping/train.py", line 219, in <module>
    main(args=args)
  File "/root/drl_grasping/drl_grasping/install/lib/drl_grasping/train.py", line 97, in main
    exp_manager.learn(model)
  File "/root/drl_grasping/drl_grasping/install/lib/python3.8/site-packages/drl_grasping/utils/training/exp_manager.py", line 209, in learn
    model.learn(self.n_timesteps, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/sb3_contrib/tqc/tqc.py", line 289, in learn
    return super(TQC, self).learn(
  File "/usr/local/lib/python3.8/dist-packages/stable_baselines3/common/off_policy_algorithm.py", line 338, in learn
    total_timesteps, callback = self._setup_learn(
  File "/usr/local/lib/python3.8/dist-packages/stable_baselines3/common/off_policy_algorithm.py", line 314, in _setup_learn
    return super()._setup_learn(
  File "/usr/local/lib/python3.8/dist-packages/stable_baselines3/common/base_class.py", line 421, in _setup_learn
    self._last_obs = self.env.reset()  # pytype: disable=annotation-type-mismatch
  File "/usr/local/lib/python3.8/dist-packages/stable_baselines3/common/vec_env/dummy_vec_env.py", line 61, in reset
    obs = self.envs[env_idx].reset()
  File "/usr/local/lib/python3.8/dist-packages/stable_baselines3/common/monitor.py", line 79, in reset
    return self.env.reset(**kwargs)
  File "/usr/local/lib/python3.8/dist-packages/gym_ignition/randomizers/gazebo_env_randomizer.py", line 98, in reset
    self.randomize_task(task=self.env.task, gazebo=self.env.gazebo, **kwargs)
  File "/root/drl_grasping/drl_grasping/install/lib/python3.8/site-packages/drl_grasping/envs/randomizers/manipulation.py", line 127, in randomize_task
    self.init_models(task=task,
  File "/root/drl_grasping/drl_grasping/install/lib/python3.8/site-packages/drl_grasping/envs/randomizers/manipulation.py", line 218, in init_models
    self.add_invisible_world_bottom_collision_plane(task=task,
  File "/root/drl_grasping/drl_grasping/install/lib/python3.8/site-packages/drl_grasping/envs/randomizers/manipulation.py", line 397, in add_invisible_world_bottom_collision_plane
    raise RuntimeError("Failed to execute a paused Gazebo run")
RuntimeError: Failed to execute a paused Gazebo run
terminate called recursively
1643635721.148012 [69] parameter_: selected interface "lo" is not multicast-capable: disabling multicast
INFO: Caught signal, killing all subprocesses...
INFO: Caught signal, killing all subprocesses...
[ERROR] [parameter_bridge-4]: process has died [pid 107, exit code -15, cmd '/root/drl_grasping/ros_ign/install/lib/ros_ign_bridge/parameter_bridge /joint_trajectory@trajectory_msgs/msg/JointTrajectory]ignition.msgs.JointTrajectory --ros-args --log-level fatal --ros-args -r __node:=parameter_bridge_joint_trajectory --params-file /tmp/launch_params_2rxhlxgm'].
[ERROR] [parameter_bridge-5]: process has died [pid 109, exit code -15, cmd '/root/drl_grasping/ros_ign/install/lib/ros_ign_bridge/parameter_bridge /joint_trajectory_progress@std_msgs/msg/Float32[ignition.msgs.Float --ros-args --log-level fatal --ros-args -r __node:=parameter_bridge_joint_trajectory_progress --params-file /tmp/launch_params_0f2n2n14'].
[ERROR] [parameter_bridge-2]: process has died [pid 103, exit code -15, cmd '/root/drl_grasping/ros_ign/install/lib/ros_ign_bridge/parameter_bridge /clock@rosgraph_msgs/msg/Clock[ignition.msgs.Clock --ros-args --log-level fatal --ros-args -r __node:=parameter_bridge_block --params-file /tmp/launch_params_cmdun4_2'].
[ERROR] [parameter_bridge-6]: process has died [pid 111, exit code -15, cmd '/root/drl_grasping/ros_ign/install/lib/ros_ign_bridge/parameter_bridge /gripper_trajectory@trajectory_msgs/msg/JointTrajectory]ignition.msgs.JointTrajectory --ros-args --log-level fatal --ros-args -r __node:=parameter_bridge_gripper_trajectory --params-file /tmp/launch_params_v5tzo06_'].
[ERROR] [parameter_bridge-3]: process has died [pid 105, exit code -15, cmd '/root/drl_grasping/ros_ign/install/lib/ros_ign_bridge/parameter_bridge /world/default/model/j2s7s300/joint_state@sensor_msgs/msg/JointState[ignition.msgs.Model --ros-args --log-level fatal --ros-args -r __node:=parameter_bridge_joint_states --params-file /tmp/launch_params_rxm44l6w -r /world/default/model/j2s7s300/joint_state:=/joint_states'].
[ERROR] [move_group-1]: process has died [pid 101, exit code -15, cmd '/opt/ros/rolling/lib/moveit_ros_move_group/move_group --ros-args --log-level fatal --ros-args -r __node:=move_group --params-file /tmp/launch_params_b9bffj9c --params-file /tmp/launch_params_ukwqts0r --params-file /tmp/launch_params__n5pqs6b --params-file /tmp/launch_params_ybfsybp8 --params-file /tmp/launch_params_d35t94mw --params-file /tmp/launch_params_9bnr6z13 --params-file /tmp/launch_params_eirebeoc --params-file /tmp/launch_params_lleymg4o --params-file /tmp/launch_params_r2wt_332'].
[move_group-1]
AndrejOrsula commented 2 years ago

Hello,

The following error is indeed a result of ign-sensors (camera sensor) requiring X display.

[Err] [Ogre2RenderEngine.cc:376] Unable to open display: terminate called after throwing an instance of 'Ogre::RenderingAPIException' what():  OGRE EXCEPTION(3:RenderingAPIException): Couldn`t open X display  in GLXGLSupport::getGLDisplay at /home/jenkins/workspace/ogre-2.2-debbuilder/repo/RenderSystems/GL3Plus/src/windowing/GLX/OgreGLXGLSupport.cpp (line 796) Received interrupt signal SIGABRT

I have not yet used Ignition Gazebo in a fully headless more, so I unfortunately cannot give you an exact solution. However, there should be support for headless simulation using EGL in Ignition fortress. Therefore, it should be possible and "straightforward" to enable a fully headless training. Please, consider taking a look at the documentation to see if you find a tutorial on how to enable it. Otherwise, there might be some relevant information in Gazebo community and/or answers, e.g. https://community.gazebosim.org/t/running-ignition-gazebo-headless-in-docker-container/1093, https://answers.gazebosim.org/question/27676/running-ignition-fortress-headless-in-docker-environment-for-continous-integration-using-rostest.

In order to resolve selected interface "lo" is not multicast-capable: disabling multicast, you need to enable multicast on your loopback (lo) network interface. For that, you could use ifconfig lo multicast.

On a side-note:

Using docker is currently recommended as I have not tested whether this project at version 1.1.0 still works with the latest releases of Ignition fortress. ROS rolling (and rolling packages) are most likely not compatible anymore, so I would recommend to use foxy/galactic if you ever want to install 1.1.0 from source.

Although training on Kinova j2s7s300 should be functional and provide some decent success rate, the robot is not yet fully configured and tested as of 1.1.0.

dsriaditya999 commented 2 years ago

Thankyou so much for responding quickly. I will try to look into your suggestions and run it in a headless manner. I will post here in case I need little support.

Also as you have pointed out I tried a local installation using ROS rolling and I was not able to do it (facing errors). So I am currently using docker only. I will try using ROS galactic/foxy instead, as you suggested.

AndrejOrsula commented 2 years ago

Yeah, rolling will not work. However, there is a chance that foxy/galactic will have some issues as well. Apologies for the inconvenience.

I am hoping that I will have the opportunity to update/refactor this repository in a couple of months with a working 2.0.0 release.