Error: Mismatch in Betas Dimensions when Training on MOYO Dataset

[IshaanVyasIMP]

I'm encountering an issue when trying to train the HuMoR model on the MOYO dataset. The model is intended to work with 16 betas, but the MOYO dataset has 300 betas, leading to a runtime error during the training process.

Script Details: I've created a custom script, process_moyo_data.py, which is modelled after humor/scripts/process_amass_data.py. You can find it here but it is pretty much the same as the process_amass_data.py. On line 48 of my script, I have defined NUM_BETAS as 16, matching the expected number of parameters for the model. Despite this, the training process fails when trying to reshape the betas, as the model receives 300 betas instead of the expected 16.

Here is the error I get:

Traceback (most recent call last):
  File "humor/train/train_humor.py", line 313, in <module>
    main(args, config_file)
  File "humor/train/train_humor.py", line 308, in main
    train(args, config_file)
  File "humor/train/train_humor.py", line 225, in train
    raise e
  File "humor/train/train_humor.py", line 198, in train
    loss, stats_dict = model_class.step(model, loss_func, data, train_dataset, device, epoch, mode='train', use_gt_p=sched_samp_gt_p)
  File "/home/ishaan/Google-Deepmind-UROP/humor/humor/train/../models/humor_model.py", line 98, in step
    betas_in = meta['betas'].reshape((B, T, 1, -1)).expand((B, T, S_out, 16)).to(device)
RuntimeError: The expanded size of the tensor (16) must match the existing size (300) at non-singleton dimension 3.  Target sizes: [200, 10, 1, 16].  Tensor sizes: [200, 10, 1, 300]

Potential Cause: I suspect that the issue may be related to the .npz file data in the MOYO dataset. I compared the outputs of processing the AMASS and MOYO datasets, and noticed that the MOYO dataset has 300 parameters in certain fields where AMASS has 16.

Steps to reproduce (this is not exactly how I have it set up, but it does produce the same error. I have laid it out his way to make it easy for people and not create any new files only modifications):

1. Have the HuMoR model correctly set up
2. Go to the download AMASS page as suggested by the README here, but do not download all the datasets. Instead scroll all the way down and download MOYO SMPL+H G (Female) and SMPL+H N (Neutral). Extract these and have these files follow this folder structure and naming:

Initially

amass_raw
├── MOYO_YOGI_2_latest_smplh_female
│   └── YOGI_2_latest_smplh_female
│       ├── train
│       └── val
└── MOYO_YOGI_2_latest_smplh_neutral
    └── YOGI_2_latest_smplh_neutral
        ├── train
        └── val

Rename and reorder the files like this:

amass_raw/
├── Female_T
│   └── female_train
│       └── [raw_data]
├── Female_V
│   └── female_val
│       └── [raw_data]
├── Neutral_T
│   └── neutral_train
│       └── [raw_data]
└── Neutral_V
    └── neutral_val
        └── [raw_data]

3. Now the raw data is ready and formatted. Now edit the process_amass_data.py like this:

Line 352 becomes this: fps = 30

Line 600 becomes this: input_seqs = glob.glob(os.path.join(data_dir, '*/*_stageii.npz'))

Lines 37 to 44 become this:

ALL_DATASETS = ['Neutral_T', 'Neutral_V', 'Female_T', 'Female_V']  # everything in MOYO
TRAIN_DATASETS = ['Neutral_T', 'Female_T']  # HuMoR-MOYO training dataset
TEST_DATASETS = ['Neutral_T', 'Female_T']  # HuMoR-MOYO test datasets
VAL_DATASETS = ['Neutral_V', 'Female_V']  # HuMoR-MOYO validation datasets

4. You can now run the command python humor/scripts/process_amass_data.py --amass-root ./data/amass_raw --out ./data/amass_processed --smplh-root ./body_models/smplh to get amass_processed with only the MOYO files.
5. Once the amass_processed folder is ready, now what is left is ensuring the training script can run. Hence replace lines humor/humor/datasets/amass_utils.py lines 5-8 with

TRAIN_DATASETS = ['Neutral_T', 'Female_T']
TEST_DATASETS = ['Neutral_T', 'Female_T', 'Female_V', 'Neutral_V']
VAL_DATASETS = ['Female_V', 'Neutral_V']

Now you should be able to run python humor/train/train_humor.py @./configs/train_humor.cfg and you should encounter the same error as me.