search

uuvsimulator / uuv_simulator

Gazebo/ROS packages for underwater robotics simulation
https://uuvsimulator.github.io/
Other
699 stars 361 forks source link

Reinforcement Learning with openai gym #433

Closed francescofraternali closed 2 years ago

francescofraternali commented 3 years ago

Hi all,

First of all, I want to thank you very much for your work as it is very powerful and exciting. Therefore, I would like to use your simulator with the openai gym and ray rllib tools to exploit state-of-the-art reinforcement learning techniques. Do you have already a guide or something on how to use your simulator with any reinforcement learning tool? Any advice would be greatly appreciated. Thank you very much in advance for your help.

MihirSPatil commented 3 years ago

Hi @francescofraternali , I've recently worked on using the uuv_simulator with openai-gym and stable-baselines to train and test state-of-the-art deep reinforcement learning algorithms.

The basic steps would be:

  1. Clone the uuv_simulator into your ROS workspace and run a basic simulation with the Gazebo simulator. The instructions for this are provided in the uuv_simulator github docs.

  2. Create your own openAI-Gym environment as currently there are no pre-defined environments for an underwater simulation. You will need to write publishers and subscribers here to obtain and process the data from the rostopics specific to your underwater vehicle.

Note: You could also use the openai-ros interface to define your openAI-Gym environment since it provides a simple interface between the gazebo sim and the openAI-Gym interface.

I don't exactly know what you are looking for, but I'd be glad to help you out if you have more specific questions.

francescofraternali commented 3 years ago

Hi @MihirSPatil, thank you very much for your answer, I really appreciate it. Finally, I can come back to work on this project and really need to connect the UUVSimulator with OpenAI-Gym to test RL algorithms. In this regard, I noticed there is a uuv_simulator_wrapper that can be used to run multiple Gazebo simulations. This is very useful, but I am still trying to figure out how to connect it to openAI-Gym. As you mentioned, I would need to build a custom openAI-Gym environment class with functions such as init, step, reset, render, etc. And in the init function I should initialize the UUV simulator environment, probably using the wrapper, right? Also, inside of the step function, I should run ros publishers commands that move the UUV in the environment following the actions selected by the RL algorithms, right? If you can give me any more details on this or also a very simple basic code that runs OpenAI-Gym which connects to the Gazebo uuv_simulator and gives commands to a UUV robot using ros services it would amazing. Having a simple basic code that does this for any underwater environment or for any UUV platform would be just fine as I am really looking for a basic implementation of this. Thank you very much for your time and availability

MihirSPatil commented 2 years ago

Hi @francescofraternali, I wasn't aware of the uuv_simulator_wrapper and maybe I will look into it sometime in the future. However, what you could do is use the openai_ros package to which already connects the Gazebo simulation to any custom environment defined by you. Also, you are correct in your understanding of the openAI-Gym class and step function. I don't have a basic code for you but you can have a look at my implementation(using openai_ros) here :Connecting openAI-gym and Gazebo sim here "robot_envs" contains all the publishers and subscribers and the "task_env" contains stuff like the reward function, state-space and so on. Once you have configured this you can simply create ros-nodes where you can instantiate your openai-gym environment and use it to run any DRL algorithm.

Hope that answers all your questions. happy coding