Replicating the results of the pre-trained models.

weihaosky commented 9 months ago

Thanks for releasing this amazing work! However, I cannot replicate the results of the pre-trained models using the provided code. The results after training with the provided code are

RVQ Reconstruction:

FID: 0.032, conf. 0.000                                                                                     
Diversity: 9.676, conf. 0.054                                                                       
TOP1: 0.507, conf. 0.003, TOP2. 0.697, conf. 0.002, TOP3. 0.794, conf. 0.002                                
Matching: 3.024, conf. 0.009                                                                                
MAE:0.042, conf.0.000

Text2motion Generation

FID: 0.148, conf. 0.006
Diversity: 9.576, conf. 0.079
TOP1: 0.504, conf. 0.003, TOP2. 0.699, conf. 0.003, TOP3. 0.795, conf. 0.002
Matching: 3.062, conf. 0.010
Multimodality:1.327, conf.0.044

May I ask where is the problem? Many thanks!

Murrol commented 9 months ago

Hi, thanks for your interest!

We are happy to figure out the problem together with you. Could you please provide the configures for training (opt file) and testing (scripts) for each stage?

weihaosky commented 9 months ago

I just use the provided code and commands without any change. vq training: opt.txt mtrans training: opt.txt rtrans training: opt.txt vq test: rvq_nq6.log mtrans test: evaluation.log

Murrol commented 9 months ago

I just use the provided code and commands without any change. vq training: opt.txt mtrans training: opt.txt rtrans training: opt.txt vq test: rvq_nq6.log mtrans test: evaluation.log

Hi wenhao, thanks for your info.

I've checked the configures and corrected the scripts in our README. We used --gamma 0.05 to train the rvq. I just re-trained the rvq and got the following results:

Hope you will find it useful.

weihaosky commented 9 months ago

I just use the provided code and commands without any change. vq training: opt.txt mtrans training: opt.txt rtrans training: opt.txt vq test: rvq_nq6.log mtrans test: evaluation.log

Hi wenhao, thanks for your info.

I've checked the configures and corrected the scripts in our README. We used --gamma 0.05 to train the rvq. I just re-trained the rvq and got the following results:

Hope you will find it useful.

Hi, after setting --gamma 0.05 I got a worse result:

net_best_fid.tar final result, epoch 36
        FID: 0.040, conf. 0.000
        Diversity: 9.598, conf. 0.097
        TOP1: 0.506, conf. 0.002, TOP2. 0.696, conf. 0.002, TOP3. 0.793, conf. 0.002
        Matching: 3.027, conf. 0.008
        MAE:0.039, conf.0.000

aszxnm commented 8 months ago

I meet the same problem and get the same results with @weihaosky . Have you solved the problem？ Thanks.

weihaosky commented 8 months ago

I meet the same problem and get the same results with @weihaosky . Have you solved the problem？ Thanks.

No. I still cannot replicate the results.

seoneun commented 7 months ago

Have you solved the problem now? @weihaosky @aszxnm Thanks.

JHang2020 commented 7 months ago

the same for me...

I meet the same problem and get the same results with @weihaosky . Have you solved the problem？ Thanks.

No. I still cannot replicate the results.

wang-zm18 commented 5 months ago

The same for me, and another confusing problem is the MPJPE of the reconstruction setting on the kit dataset: 1719153702064

The command I run is python eval_t2m_vq.py --gpu_id 0 --name rvq_nq6_dc512_nc512_noshare_qdp0.2_k --dataset_name kit --ext rvq_nq6, and I used the default pretrained checkpoint.

Similarly, when I test it on the Humanml dataset, the results is :

It is obvious that the MPJPE results on these two datasets differ by several orders of magnitude. Does anyone else have the same problem?

HitBadTrap commented 5 months ago

I just use the provided code and commands without any change. vq training: opt.txt mtrans training: opt.txt rtrans training: opt.txt vq test: rvq_nq6.log mtrans test: evaluation.log

Hi wenhao, thanks for your info. I've checked the configures and corrected the scripts in our README. We used --gamma 0.05 to train the rvq. I just re-trained the rvq and got the following results:

Hope you will find it useful.

Hi, after setting --gamma 0.05 I got a worse result:
net_best_fid.tar final result, epoch 36
        FID: 0.040, conf. 0.000
        Diversity: 9.598, conf. 0.097
        TOP1: 0.506, conf. 0.002, TOP2. 0.696, conf. 0.002, TOP3. 0.793, conf. 0.002
        Matching: 3.027, conf. 0.008
        MAE:0.039, conf.0.000

Have you solved the problem now? @weihaosky @aszxnm

Thanks.

aszxnm commented 5 months ago

I just use the provided code and commands without any change. vq training: opt.txt mtrans training: opt.txt rtrans training: opt.txt vq test: rvq_nq6.log mtrans test: evaluation.log

Hi wenhao, thanks for your info. I've checked the configures and corrected the scripts in our README. We used --gamma 0.05 to train the rvq. I just re-trained the rvq and got the following results:

Hope you will find it useful.

Hi, after setting --gamma 0.05 I got a worse result:
net_best_fid.tar final result, epoch 36
        FID: 0.040, conf. 0.000
        Diversity: 9.598, conf. 0.097
        TOP1: 0.506, conf. 0.002, TOP2. 0.696, conf. 0.002, TOP3. 0.793, conf. 0.002
        Matching: 3.027, conf. 0.008
        MAE:0.039, conf.0.000
Have you solved the problem now? @weihaosky @aszxnm

Thanks.

No. I still cannot replicate the results.

Murrol commented 5 months ago

Thank you all for your attempts to replicate the results. I just got some time to re-train the masked-transformer and res-transformer using our released code. Here are some results for your reference:

I used the rvq checkpoint I obtained here https://github.com/EricGuo5513/momask-codes/issues/27#issuecomment-1956264373. I used the following scripts to train the m-trans and r-trans:

python train_t2m_transformer.py --name mtrans_replicate --gpu_id 1 --dataset_name t2m --batch_size 64 --vq_name rvq_replicate

python train_res_transformer.py --name rtrans_replicate --gpu_id 2 --dataset_name t2m --batch_size 64 --vq_name rvq_replicate --cond_drop_prob 0.2 --share_weight

evaluation script:

python eval_t2m_trans_res.py --res_name rtrans_replicate --dataset_name t2m --name mtrans_replicate --gpu_id 1 --cond_scale 4 --time_steps 10 --ext evaluation_replicate

The above results are obtained from these scripts without any modification to this code base. The replicate experiments were done on a single RTX 2080 Ti GPU, torch==1.7.1. For the processed dataset cloned from the original HumanML3D project, please send inquiry to cguo2@ualberta.ca or ymu3@ualberta.ca

wang-zm18 commented 5 months ago

https://github.com/EricGuo5513/momask-codes/issues/27#issuecomment-2185009113 Does anybody knows the problem?

Murrol commented 5 months ago

The same for me, and another confusing problem is the MPJPE of the reconstruction setting on the kit dataset:

The command I run is python eval_t2m_vq.py --gpu_id 0 --name rvq_nq6_dc512_nc512_noshare_qdp0.2_k --dataset_name kit --ext rvq_nq6, and I used the default pretrained checkpoint.

Similarly, when I test it on the Humanml dataset, the results is :

It is obvious that the MPJPE results on these two datasets differ by several orders of magnitude. Does anyone else have the same problem?

The scale is different. Check the Mean.py files.

wang-zm18 commented 5 months ago

I re-download the KIT-ML dataset from Link directly. While it still remains this problem. Maybe it is not the mean problem？Thank you in advance! @Murrol

wang-zm18 commented 4 months ago

https://github.com/EricGuo5513/momask-codes/issues/27#issuecomment-2198380894

Any other suggestions? Thanks!

Murrol commented 4 months ago

I re-download the KIT-ML dataset from Link directly. While it still remains this problem. Maybe it is not the mean problem？Thank you in advance! @Murrol

Scale difference means that the scale of AMASS(HumanML3D) and kit data is different. Take an example, inch vs cm.

"the MPJPE results on these two datasets differ by several orders of magnitude" should not be an issue.

wang-zm18 commented 4 months ago

I re-download the KIT-ML dataset from Link directly. While it still remains this problem. Maybe it is not the mean problem？Thank you in advance! @Murrol

Scale difference means that the scale of AMASS(HumanML3D) and kit data is different. Take an example, inch vs cm.

"the MPJPE results on these two datasets differ by several orders of magnitude" should not be an issue.

Oh, thanks! I checked the range of means from these two datasets, the mean of KIT datasets is three orders of magnitude higher than that of HumanML3D on average. Thank you! @Murrol

Suzixin7894946 commented 4 months ago

Thank you all for your attempts to replicate the results. I just got some time to re-train the masked-transformer and res-transformer using our released code. Here are some results for your reference:

I used the rvq checkpoint I obtained here #27 (comment). I used the following scripts to train the m-trans and r-trans:
python train_t2m_transformer.py --name mtrans_replicate --gpu_id 1 --dataset_name t2m --batch_size 64 --vq_name rvq_replicate
python train_res_transformer.py --name rtrans_replicate --gpu_id 2 --dataset_name t2m --batch_size 64 --vq_name rvq_replicate --cond_drop_prob 0.2 --share_weight
evaluation script:
python eval_t2m_trans_res.py --res_name rtrans_replicate --dataset_name t2m --name mtrans_replicate --gpu_id 1 --cond_scale 4 --time_steps 10 --ext evaluation_replicate
The above results are obtained from these scripts without any modification to this code base. The replicate experiments were done on a single RTX 2080 Ti GPU, torch==1.7.1. For the processed dataset cloned from the original HumanML3D project, please send inquiry to cguo2@ualberta.ca or ymu3@ualberta.ca

Hello, I have some doubts regarding the reproduction of results. Currently, the minimum FID value I am getting while replicating the results on t2m is 0.059, and other metrics are normal. However, after adding certain modules, the FID value is 0.047, but in your paper, the FID value is 0.045. I would like to know if this is an improvement? I am a bit unclear whether my modifications are effective, or if, in fact, my reproduced results are not correct.

Tiooo111 commented 1 month ago

My rvq replicated FID is also 0.032, has it been solved?

cr8br0ze commented 1 month ago

My rvq replicated FID is also 0.032, has it been solved?

Same question, my rvq replicated FID is similar, (0.033). Using this rvq and given terminal command here to train mask transformer and residual transformer leads me to a generation fid only around (0.182). But if I use the given vq checkpoint to train both transformers, I can achieve a more similar result to here Hope we can have a solution soon since rvq quality matters a lot for generation here.

EricGuo5513 / momask-codes

Replicating the results of the pre-trained models. #27