ac-93 / tactile_gym

Suite of PyBullet reinforcement learning environments targeted towards using tactile data as the main form of observation.
GNU General Public License v3.0
117 stars 20 forks source link

How to use tactile_gym in updated pybullet? #10

Open CarlDegio opened 1 year ago

CarlDegio commented 1 year ago

For code reason,I want to use tactile_gym in pybullet 3.2.5. So I change the requirements.txt of tactile_gym version to >=3.1.0. But some example tests(not all, demo_roll_env could be run) occured problems such as :

python examples/demo_example_arm_env.py
...
Traceback (most recent call last):
  File "examples/demo_example_arm_env.py", line 85, in <module>
    main()
  File "examples/demo_example_arm_env.py", line 45, in main
    env = ExampleArmEnv(
  File "/home/demons/anaconda3/envs/aerm/lib/python3.8/site-packages/tactile_gym-0.0.1-py3.8.egg/tactile_gym/rl_envs/example_envs/example_arm_env/example_arm_env.py", line 43, in __init__
    self.arm_type = env_modes["arm_type"]
KeyError: 'arm_type'
...
python examples/demo_edge_auto_env.py
...
MotionThreadFunc thread started
Traceback (most recent call last):
  File "examples/demo_edge_auto_env.py", line 122, in <module>
    main()
  File "examples/demo_edge_auto_env.py", line 47, in main
    env = EdgeFollowAutoEnv(
  File "/home/demons/anaconda3/envs/aerm/lib/python3.8/site-packages/tactile_gym-0.0.1-py3.8.egg/tactile_gym/rl_envs/exploration/edge_follow/edge_follow_auto_env.py", line 53, in __init__
    self.stimulus = env_modes["stimulus"]
KeyError: 'stimulus'
...
python examples/demo_push_env.py
...
b3Printf: /home/demons/anaconda3/envs/aerm/lib/python3.8/site-packages/tactile_gym-0.0.1-py3.8.egg/tactile_gym/assets/robot_assets/mg400/tactip/mg400_with_mini_right_angle_tactip.urdf:63: cannot find 'mg400/meshes/base_link.STL' in any directory in urdf path

Warning: b3Error[examples/Importers/ImportURDFDemo/BulletUrdfImporter.cpp,121]:

Warning: Could not parse visual element for Link:

...

It seems that different version of pybullets are not compatible well. Do you have some advice about version compatibility about bullet? Or how do I change the tactile_gym to make it work in upgraded pybullet.

CarlDegio commented 1 year ago

Some of these question are because I didn't set the arm_type... Consider to set these parameter in single file?

CarlDegio commented 1 year ago

problems left: 1.self.stimulus = env_modes["stimulus"]in tactile_gym/rl_envs/exploration/edge_follow/edge_follow_auto_env.py still cause error. Because there is no relevant setting in demo_edge_auto_env.py 2.demo_push_env.py error with: pybullet.error: Cannot load URDF file. I think the missing of meshes files when install leads to this problem.

  1. demo_surf_auto_env error with: AttributeError: 'OpenSimplex' object has no attribute 'noise2d' (0.4.3 OpenSimplex). This method have been changed as https://github.com/lmas/opensimplex/blob/master/opensimplex/api.py
ac-93 commented 1 year ago

Apologies, we broke some of the demos on the last update. Just did some cleanup so hopefully the demos work now. There's still some things that need fixing that I hope to do soon.

The reason for using pybullet==3.1.0 was that later pybullet updates broke rendering of heightfields. I think this is related to this pr but haven't checked it out yet. It seems that installing pybullet==3.1.0 is broken on ubuntu so I've changed to >3.1.0 but for now the heighfield rendering is broken.

The edge_follow_auto_env was actually being used for a separate project and is no longer needed so I have removed. If you did have plans to use that let me know and I'll add it back in.

OpenSimplex methods have been updated, thanks for pointing these issues out!

CarlDegio commented 1 year ago

The meshes of mg400 seems still missing. I run all examples successfully except demo_surf_vert_env.py, and only this example use mg400 as manipulator.

b3Printf: b3Warning[examples/Importers/ImportURDFDemo/BulletUrdfImporter.cpp,126]:

b3Printf: Duplicate material
b3Printf: b3Warning[examples/Importers/ImportURDFDemo/UrdfFindMeshFile.h,102]:

b3Printf: /home/demons/anaconda3/envs/tactile_gym/lib/python3.8/site-packages/tactile_gym-0.0.1-py3.8.egg/tactile_gym/assets/robot_assets/mg400/tactip/mg400_with_forward_tactip.urdf:63: cannot find 'mg400/meshes/base_link.STL' in any directory in urdf path

Warning: b3Error[examples/Importers/ImportURDFDemo/BulletUrdfImporter.cpp,121]:

Warning: Could not parse visual element for Link:
Warning: b3Error[examples/Importers/ImportURDFDemo/BulletUrdfImporter.cpp,121]:

Warning: base_link
Warning: b3Error[examples/Importers/ImportURDFDemo/BulletUrdfImporter.cpp,121]:

Warning: failed to parse link
Traceback (most recent call last):
  File "examples/demo_surf_vert_env.py", line 98, in <module>
    main()
  File "examples/demo_surf_vert_env.py", line 50, in main
    env = SurfaceFollowVertEnv(
  File "/home/demons/anaconda3/envs/tactile_gym/lib/python3.8/site-packages/tactile_gym-0.0.1-py3.8.egg/tactile_gym/rl_envs/exploration/surface_follow/surface_follow_vert/surface_follow_vert_env.py", line 25, in __init__
    super(SurfaceFollowVertEnv, self).__init__(
  File "/home/demons/anaconda3/envs/tactile_gym/lib/python3.8/site-packages/tactile_gym-0.0.1-py3.8.egg/tactile_gym/rl_envs/exploration/surface_follow/base_surface_env.py", line 131, in __init__
    self.robot = Robot(
  File "/home/demons/anaconda3/envs/tactile_gym/lib/python3.8/site-packages/tactile_gym-0.0.1-py3.8.egg/tactile_gym/robots/arms/robot.py", line 43, in __init__
    self.robot_id = self.load_robot()
  File "/home/demons/anaconda3/envs/tactile_gym/lib/python3.8/site-packages/tactile_gym-0.0.1-py3.8.egg/tactile_gym/robots/arms/robot.py", line 108, in load_robot
    robot_id = self._pb.loadURDF(
pybullet.error: Cannot load URDF file.

Others use ur5 didn't meet this problems. Besides, I test another manipulator, both the franka_panda and kuka_iiwa have some error from tactile_gym/rl_envs/example_envs/example_arm_env/rest_poses.py. Franka_panda is under ur5, seems unreasonable. kuka_iiwa has no attribute of tactip or digitac or digit which lead error :rest_poses = rest_poses_dict[self.arm_type][self.t_s_name][self.t_s_type]

ac-93 commented 1 year ago

These issues should now be fixed.

The MG400 issues came from capitalization of the STL in the mesh files, which meant that the appropriate meshes were not installed with python setup.py install.

The issues with other manipulators were separate, there was some update to naming conventions when we added alternate sensors but we didn't apply these for the kuka or franka robots. I've now updated the example_arm_env such that each arm (ur5, kuka_iiwa, franka_panda, mg400) can be combined with each sensor (tactip, digit, digitac). I haven't updated these for environment other than the example env yet, if this is required then let me know. The example_arm_env should serve as a demo on how to do this though.

yuxwen commented 6 months ago

Compile failed: 0:1(10): error: GLSL 1.50 is not supported. Supported versions are: 1.10, 1.20, 1.30, 1.40, 1.00 ES, and 3.00 ES

Warning/Error in GLSL shader: error: vertex shader lacks `main'

Warning/Error in GLSL shader: error: vertex shader lacks `main'

Warning/Error in GLSL shader: error: vertex shader lacks `main'

Warning/Error in GLSL shader: error: vertex shader lacks `main'

Warning/Error in GLSL shader: error: vertex shader lacks `main'

Warning/Error in GLSL shader: error: vertex shader lacks `main'

GL error 0x501 detected in glGenBuffers ven = Mesa/X.org ven = Mesa/X.org Traceback (most recent call last): File "/home/ros2/miniconda3/tactile_gym_push/tactile_gym/sb3_helpers/train_agent.py", line 184, in train_agent( File "/home/ros2/miniconda3/tactile_gym_push/tactile_gym/sb3_helpers/train_agent.py", line 52, in train_agent env = make_training_envs( File "/home/ros2/miniconda3/tactile_gym_push/tactile_gym/sb3_helpers/rl_utils.py", line 16, in make_training_envs env = make_vec_env(env_id, File "/home/ros2/miniconda3/envs/python39/lib/python3.9/site-packages/stable_baselines3/common/env_util.py", line 105, in make_vec_env return vec_env_cls([make_env(i + start_index) for i in range(n_envs)], **vec_env_kwargs) File "/home/ros2/miniconda3/envs/python39/lib/python3.9/site-packages/stable_baselines3/common/vec_env/subproc_vec_env.py", line 111, in init observation_space, action_space = self.remotes[0].recv() File "/home/ros2/miniconda3/envs/python39/lib/python3.9/multiprocessing/connection.py", line 250, in recv buf = self._recv_bytes() File "/home/ros2/miniconda3/envs/python39/lib/python3.9/multiprocessing/connection.py", line 414, in _recv_bytes buf = self._recv(4) File "/home/ros2/miniconda3/envs/python39/lib/python3.9/multiprocessing/connection.py", line 379, in _recv chunk = read(handle, remaining) ConnectionResetError: [Errno 104] Connection reset by peer ven = Mesa/X.org

I am running the code on Ubuntu installed through a virtual machine. I wonder if this has any impact.