destroy314
commented
5 months ago Hello, my goals and progress are the same as yours. I've found that learning from human data requires some code modifications. After recording dual-view operation videos, the h5 file obtained by running demo_mp4.py still needs to extract the trajectory of the end-effector (here, the hand). Before that, modify the target key names in scripts/dataset_extract_traj_plans.py, and add a num_samples attribute,: (the lengths of 'done' and 'action' are one less than obs)
for i in range(0, DEMO_COUNT):
# Extract the hand location data
hand_loc = f[f'data/demo_{i}/obs/hand_loc'][...]
# Calculate the future trajectory for each data point
future_traj_data = np.array([get_future_points(hand_loc[j:]) for j in range(len(hand_loc))])
# Create the new dataset for future trajectory
f.create_dataset(f'data/demo_{i}/obs/hand_loc_future_traj', data=future_traj_data)
# Add the 'num_samples' attribute to the demo (specifically for human play data)
f[f'data/demo_{i}'].attrs['num_samples'] = len(hand_loc) - 1Correspondingly, the high-level JSON configuration file also needs to have the observation key names modified:
"observation": {
"modalities": {
"obs": {
"low_dim": [
"hand_loc",
"hand_loc_future_traj"
],
"rgb": [
"front_image_1",
"front_image_2"
],
"depth": [],
"scan": []
},
"goal": {
"low_dim": [],
"rgb": [
"front_image_1",
"front_image_2"
],
"depth": [],
"scan": []
}
},
}In addition, mimicplay/algo/mimicplay.py also needs the following corresponding lines modified to match the new observations (hand_loc_future_traj, front_image_1, and front_image_2):
del self.obs_shapes['robot0_eef_pos_future_traj']
batch["goal_obs"]["agentview_image"] = batch["goal_obs"]["agentview_image"][:, 0]
log_probs = dists.log_prob(batch["obs"]["hand_loc_future_traj"])The end of last line should be ["hand_loc_future_traj"].reshape(-1,40), and I think it's also possible to modify the shape of the data recorded in dataset_extract_traj_plans.py. Also modify the dimension number in the corresponding JSON file to match the dim:
"highlevel": {
"ac_dim": 40,
},Since human play data doesn't have a train-validation split, set it not to validate:
"experiment": {
"validate": false,
}
"train": {
"hdf5_filter_key": null,
"hdf5_validation_filter_key": null,
}This could also be addressed by modifying demo_mp4.py and make a split. Given the amount of code involved, I guess renaming keys in human data to match those in the code might be a better solution. In any case, the above modifications were sufficient for me to train a high-level model (but no way to verify the correctness), and I hope I haven't missed anything!
Compared to play data provided in simulation environment, the model trained with my collected data saw the Train/Policy_Grad_Norms parameter begin to rise from a decline around 20 epochs, far less than the ~400 epochs for the former, which also coincided with the best validation results for the former. It would be helpful if the authors could provide guidance on data collection or training.
kadambaribhujbal
j96w
Hello, this is really impressive work. I have been able to replicate the simulation part of training the high and low level planner using play data. I am currently focusing on training a high-level latent plan using human play data. To this end, I have successfully generated an HDF5 file containing all the relevant information from the human play data extracted from the provided videos. However, I have encountered a hurdle in adapting the existing code for training the high level planner to utilize my generated HDF5 file. The current training code appears to be tailored for simulation data, leading to errors when I attempt to integrate my dataset. Any guidance or direction you could offer would be immensely helpful. Thank you!