Open KiriyamaGK opened 2 months ago
Sorry,I have to rectify my statement.Error disappeared after I converted the script in demo_mp4.py like this:
all_hand_loc = np.concatenate((hand_loc_1[:, :, -1:], hand_loc[:, :, -2:]), axis=2)
Sorry,I have to rectify my statement.Error disappeared after I converted the script in demo_mp4.py like this:
all_hand_loc = np.concatenate((hand_loc_1[:, :, -1:], hand_loc[:, :, -2:]), axis=2)
I'm glad to see this issue being addressed here. Recently, I've also been working on this topic and am at the stage of applying it to real robotic arms. I was wondering if you have successfully implemented this and could share any experience or code related to applying the model to real robotic arms?
Sorry,I have to rectify my statement.Error disappeared after I converted the script in demo_mp4.py like this:
all_hand_loc = np.concatenate((hand_loc_1[:, :, -1:], hand_loc[:, :, -2:]), axis=2)
I'm glad to see this issue being addressed here. Recently, I've also been working on this topic and am at the stage of applying it to real robotic arms. I was wondering if you have successfully implemented this and could share any experience or code related to applying the model to real robotic arms?
Hi,the problem is due to theaction dim
is set 30(10xyz) in simulation, but 40(10xy*2(dual imgs)) in real world code.What I have done then is to project the x-y coordinates of each camara into x-y-z coordinates in robot base frame,and after which error disappeared. Also,don't forget to add gradient clipping in case of gradient explosion,which worses the training process most.Finally, if you want to use human playdata(I haven't tried it though) ,there should be a KL-loss set in latent planner,which is mentioned in the paper but not implemented in simulation code.
Hi,what a brilliant work of mimicplay! In fact,when implementing training lowlevel controller with my real-world playdata,aftering training my highlevel human playdata, error occured like this:
However,error disappeared after I converted the script in demo_mp4.py like this:
all_hand_loc = np.concatenate((hand_loc_1[:, :, -2:], hand_loc[:, :, -2:]), axis=2)
,settingac_dim=30
in the highlevel_human.json and finally retraining highlevel planner and lowlevel controller.Also,it seems no use settingac_dim=40
in the lowlevel.json to train lowlevel controller successfully without changingall_hand_loc
in demo_mp4.py.