search

aidudezzz / deepbots

A wrapper framework for Reinforcement Learning in the Webots robot simulator using Python 3.
https://deepbots.readthedocs.io/
GNU General Public License v3.0
236 stars 50 forks source link

GoalEnv for Robot-Supervisor scheme #103

Closed KelvinYang0320 closed 2 years ago

KelvinYang0320 commented 2 years ago

I integrate GoalEnv from gym/core.py with deepbots for Robot-Supervisor scheme. GoalEnv also inherite Env, but it imposes a required structure on the observation_space.

  1. reset(self): self.observation_space must be a Goal-compatible observation space. It is required to contain at least three elements, namely observation, desired_goal, and achieved_goal. Refer to robot_env.py in gym.

  2. compute_reward(self, achieved_goal, desired_goal, info): Since in our panda example, we need to get the observation to compute achieved_goal, and then we can compute the reward by using achieved_goal and desired_goal. Our original get_reward(action) is actually not that suitable for robotics control task. So I think we can let users implement their own step(), so I add raise NotImplementedError in step() function of RobotGoalSupervisor.

I think my coding style or implementation is not good enough. Any suggestions would be appreciated! :smile:

tsampazk commented 2 years ago

Thanks @KelvinYang0320 ! I'll review this PR in the following days.

KelvinYang0320 commented 2 years ago

@tsampazk Thank you! Since it is mainly used for robotic control task in gym, I think I can make an deepworlds example with panda for GoalEnv demo.

tsampazk commented 2 years ago

@tsampazk Thank you! Since it is mainly used for robotic control task in gym, I think I can make an deepworlds example with panda for GoalEnv demo.

That sounds nice so we can actually have a usage example to make it more clear.

KelvinYang0320 commented 2 years ago

As a general comment, i think we should also look into how the new GoalEnv classes interact with the emitter/receiver classes. Right now emitter/receiver classes seem to not have access to the GoalEnv stuff.

Yes, I am still trying to integrate GoalEnv with emitter/receiver. However, we might need to decide how to design the highest level class to support both gym.GoalEnv and gym.Env first, and then we can think about how to design emitter-receiver and robot-supervisor.

Thank you for reviewing the code!

KelvinYang0320 commented 2 years ago

@tsampazk I have made an example in deepworlds to demo GoalEnv with Panda. Check this PR. Also, I have updated this GoalEnv PR for those we have discussed about. Thank you!