search

reiniscimurs / DRL-robot-navigation

Deep Reinforcement Learning for mobile robot navigation in ROS Gazebo simulator. Using Twin Delayed Deep Deterministic Policy Gradient (TD3) neural network, a robot learns to navigate to a random goal point in a simulated environment while avoiding obstacles.
MIT License
571 stars 119 forks source link

About the tumbling problem of robot during training #75

Closed mars31030 closed 11 months ago

mars31030 commented 1 year ago

Hi , Dr.reiniscimurs： Excellent work. I highly appreciate your work. When I tried to replicate it, I substituted my own map. After that, I made some changes to the boundary values in the Python file. Initially, everything was going smoothly during training, but around the tenth epoch, the robot started rotating in place or even rolling over. I attempted to modify the seed value, but the results seemed mediocre. Could you provide some insights into this confusion of mine?

If it were a coordinate issue in the URDF file or a weight problem with the robot, it should have caused rolling over from the beginning rather than occurring after two hours of training. It's worth noting that I only replaced the map portion, and no other files were modified.

https://github.com/reiniscimurs/DRL-robot-navigation/assets/108964552/a19879d8-73a3-447c-92df-12df735aaaba

reiniscimurs commented 1 year ago

I can only guess but it looks like a wheel is not spawned in the episode before the robot starts moving. So it is moving with a missing wheel and falling over. What boundary values did you end up changing?

To avoid similar issues, I added set python to sleep so that gazebo could spawn everything: https://github.com/reiniscimurs/DRL-robot-navigation/blob/101ed8eaba77b9f447e3d57cc71518cfedb7f65c/TD3/train_velodyne_td3.py#L253

You could try adding something similar in the rest function in the environment file.

mars31030 commented 1 year ago

Hello, thank you for your response. I modified the boundary content in the velogyne_env.py file within the check_pos function to change the robot's starting position. I don't believe it's an issue with a missing wheel. If it were a problem with the robot model, it should have caused rolling over from the beginning of the training. However, in my case, this issue only arises after training for a few epochs.

reiniscimurs commented 1 year ago

I would still suggest trying the proposed method. It does make sense that certain things would happen later in the training as more resources are taken up over time and time step synchronization issues become bigger for ROS nodes and topics.

I wouldn't think it is anything to do with the weights or network training. Rather something on the ROS side. You should visually check if all model elements are still part of the robot when this starts happening.