search

qgallouedec / panda-gym

Set of robotic environments based on PyBullet physics engine and gymnasium.
MIT License
492 stars 106 forks source link

Weird movement of the end-effector #49

Closed fikricanozgur closed 8 months ago

fikricanozgur commented 1 year ago

Hi,

I am observing a weird movement of the end-effector. See the video below. How could this be happening when there is only 1 action corresponding to the variation of the gripper opening?

The task is slightly different than the original PickAndPlace task. I added rotation control of the end-effector around the z axis and the goal state also includes the desired rotation of the block. Could this somehow trigger the weird movement?

Thanks!

https://user-images.githubusercontent.com/54752334/210052838-56a9b693-6ccf-48f2-b2b4-2baac447eb5c.mp4

qgallouedec commented 1 year ago

Thanks for the feedback!

It is indeed a weird movement that I also observed in pick and place task. The gripper is not supposed to be able to move like that. I have no explanation at this stage. Maybe we should look at the forces within the gripper. If you have an explanation, please share it.

fikricanozgur commented 1 year ago

I investigated the problem a little more and found out a possible reason. I think what happens is as follows:

When the RL agent chooses its desired gripper opening (1 coordinate) this value is used to control both fingers (2 coordinates) in an equal manner, here in the code. At first glance, this implementation seems fair. However, I think it is also the reason why we sometimes observe gripper oscillations in tasks with gripper control. Let's say the agent tries to pick an object and both the gripper fingers are commanded to be at a certain distance from the center point such that the object will be in the center of the gripper opening when picked. Now, imagine one of the fingers has already arrived at its desired position before the other did. The second finger is still coming closer to its desired position by also pushing the object towards the center where it will grasped by both fingers. In this situation, when the second finger arrives at its desired position some force will be exerted on the first finger through the object in between them because the first finger has already settled in its position and is no longer applying any force. This will make the first finger deviate from its position then try to recover it but by doing so push the second finger away and enter this oscillatory behavior. Additionally, oscillations magnitude might be amplified if the grippers are commanded to have gap that is too small from the object to fit in. In this case, since the grippers are not able to reach their desired position simultaneously the oscillations will continue forever.

I improvised a workaround for this problem by fixing the position of one finger and only controlling the position of the second finger (1 coordinate). I also allowed the finger with fixed position to have a much higher maximum force allowance than the other finger by modifying the forces in here. It seems to work. Here is a video:

https://user-images.githubusercontent.com/54752334/211791672-4803ac5a-34b8-4d0b-9557-dfd06b788772.mp4

qgallouedec commented 1 year ago

Thank you very much for this investigation and your detailed explanation! Your explanation seems quite correct and the solution you propose is interesting. However, does it imply that there is only one finger applying the force, and that the other is therefore a kind of support? I would like to implement an update like that. I still need to think about how to do something like what you did, but keeping a symmetrical application of force, to better fit the real behavior of the robot.

fikricanozgur commented 1 year ago

Yes, in my solution, one of the fingers has a fixed position and is there only for support. The other finger is free to move.

I also quickly tried to clip the commanded gripper opening to be minimum the width of the object so that both fingers can attain their corresponding desired positions simultaneously. In this case, both fingers were free to move as in the original panda-gym library. However, there were still oscillations. Maybe you would need to play with the parameters of the physics engine (controllers for the gripper fingers) to get this approach working properly. I did not have time for that.

fikricanozgur commented 1 year ago

I also realized a peculiarity with the sliding task. The puck goes down into the table when pushed from the top, see the video below. I am guessing this has nothing to do with panda-gym and it is a problem of PyBullet, would you agree? slide_non-rigid_obj

qgallouedec commented 1 year ago

I also realized a peculiarity with the sliding task. The puck goes down into the table when pushed from the top, see the video below. I am guessing this has nothing to do with panda-gym and it is a problem of PyBullet, would you agree?

I've noticed it. I guess yes, but I would need to investigate to be sure. Indeed, this is not very satisfactory concerning the fidelity of the physics. As it does not hinder the learning process, I will not treat it as a priority. Is it annoying for your use?

fikricanozgur commented 1 year ago

Is it annoying for your use?

No, not really.

mechtach714 commented 1 year ago

Hi Fikricanozgur

Could you perhaps share the code changes you made to achieve pick and place with the rotation of the end effector? I'm trying something similar but I don't know where to amend the code. Still a noob and trying to understand how to define the tasks. I'm also struggling to define a fixed position for the target position of the object, mine still randomizes for each different run. Any help would be greatly appreciated.

qgallouedec commented 1 year ago

Still a noob and trying to understand how to define the tasks. I'm also struggling to define a fixed position for the target position of the object, mine still randomizes for each different run.

See https://panda-gym.readthedocs.io/en/latest/custom/custom_task.html If it doesn't answer your question, please open a dedicated issue.