[Bug] joint delta pos to joint pos control conversion has 0 success

[sean1295]

In replay_trajectory.py, when trying to convert environment in pd_joint_delta_pos to pd_joint_pos using the following commands,

python -m mani_skill.trajectory.replay_trajectory --traj-path demo/StackCubeEnv/20240716_102711.h5 --save-traj -o rgb --target-control-mode pd_ee_delta_pose --num-procs 4 --use-first-env-state --max-retry 5

it keeps giving me following messages: "Episode xxx is not replayed successfully. Skipping."

I also kept track of the joint positions of the robots across two environments using the converted actions and they do not follow exactly. Please see below.

Was looking at the code but do not see anything that stands out. aside from one error inside clip_and_scale_action function where clip method from numpy should be used instead of torch.

Am wondering if there are ways to fix this.

[StoneT2000]

The problem of 100% accurate action space conversion is highly non-trivial (if you solve it it's worth a conference publication). It is quite difficult to ensure the resulting environment state from using a different controller is the same as the original trajectory data, we can only do approximations and at best, try to get a success rate that matches the trajectory data

[sean1295]

Hi Stone, thanks for your prompt reply. I agree. When trying to convert from pd_joint_pos to pd_ee_delta_pose, I noticed that ~1% of them didn't convert successfully, which I am totally fine with since I can simply generate more trajectories from original env. But this seemly easy conversion from pd_joint_delta_pos to pd_joint_pos actually has 0 % conversion rate. I think this particular conversion has a lot of utility since many RL agents can be trained with pd_joint_delta_pos and we may want to convert that to other action spaces. In order to do so, we first need to convert that back to pd_joint_pos, and then the target action space.

[StoneT2000]

If it has 0 then that sounds like a bug. I'll keep the issue up and take a look soon.