search

anindita127 / Complextext2animation

Proceedings of IEEE CVF. ICCV 2021 "Synthesis of Compositional Animations from Textual Descriptions."
MIT License
32 stars 6 forks source link

Evaluation Results with ICCV'21 Paper #4

Open ludy3996 opened 2 years ago

ludy3996 commented 2 years ago

Hi, thanks for your great work on ICCV'21. I have some questions and issues with your code. I just clone your code (didn't modify official code) and evaluated the provided pre-trained model with these codes. But I got so different results with your ICCV paper, only got the evaluation result like this.

Test APE [4.190873301445142, 15.652097037362648, 14.476590518962245, 17.752781245065474, 14.795194316051749, 13.24986283658489, 14.723207273690173, 14.2541448259369, 13.721858107615398, 13.556832176482954, 13.431385539737901, 13.9537314120196, 13.902865999886918, 12.268731942753899, 13.682642174726016, 13.855254758015043, 14.981251266174741, 12.796628382808017, 14.758570286169096, 8.079425356527656]

Test AVE [10.178670862604642, 15.219314325004246, 12.879057965257015, 14.139660217758474, 13.689597073921833, 8.454485149774731, 9.164908408357755, 6.780733172433479, 9.70430640765224, 7.1300269325197965, 5.382036185673594, 20.322144246346795, 13.266659138706576, 8.909401371480373, 12.573738910250547, 20.25171573373726, 14.21424176749085, 10.286702136810037, 13.424342641988945, 5.88676996947937]

Test CEE 1.2373948167908533

Test SEE 0.46560313116859303

I wanna reproduce the performance of ICCV'21 with your official model and code. But I don't know provided pre-trained is the best model or not, and how to modify this code to get it. Do I need to re-train the model or do you have any idea about this? And another issue is when using python dataProcessing/meanVariance.py -mask '[0]' -feats_kind rifke -dataset KITMocap -path2data ../dataset/kit-mocap -f_new 8 to calculate the mean and variance for Z-Normalization, got some errors about the length of mask. Then I used this one python dataProcessing/meanVariance.py -feats_kind rifke -dataset KITMocap -path2data ../dataset/kit-mocap -f_new 8 (I saw code from dataUtils.py and there is some defaults with mask=[0], but I don't know if this is correct.

Thanks,

anindita127 commented 2 years ago

Hi,

Thanks for your interest in our work. Recently the repository went through some updates. Can you check in once more and see if your issues sill persist.

HappyPiepie commented 2 years ago

Hi , thanks for your work for this field. I hava some problems when i run your eval coda. I got APE_LOSS： [1.6152539077560253, 3.8035130007033446, 3.801077495838645, 4.009686965172017, 3.7679262980997006, 3.590032654908319, 4.529732276969545, 4.89012817142057, 3.574709865812325, 4.043609161419107, 4.578285699350569, 3.8500040182153494, 4.156827169357215, 3.902301512267607, 4.1243365680896416, 3.839924943096757, 4.410920671962248, 3.9593780655352617, 4.329834376329709, 3.3412870973680375] is close your paper's results.

And , AVE_loss:
[0.44463602033627375, 0.7030023537442466, 0.7308105546397969, 0.805257988857508, 0.7744137068526745, 0.40802066304414186, 0.6775546575277528, 0.6388468728953295, 0.5033926311021557, 0.38493773164501277, 0.3819525868883675, 0.6934178872048002, 0.5638059260649851, 0.3720031997450148, 0.5573624620957186, 0.674750450305267, 0.6029626064853397, 0.4764308816370605, 0.6281975160633145, 0.4648465112427174]

test_loss_cee: 0.13350790943356924 test_loss_see: 0.047305966971594324 is so different with your paper. Can you help me ?

anindita127 commented 2 years ago

Hi, thanks for your interest in this work. Have you checked if all the sequences in the test set is the same? I have received a feedback saying the dataloader of the code was loading a lesser number of test data samples than what was originally noted in the paper. Please check into that!

HappyPiepie commented 2 years ago

Hi,thank you for your reply. I 'm interested your reserach. But i check the number of test is 1272. However the result AVE is so diffurent from paper. Can you help me？ **ntest = 1272 AVE_loss: [0.11498997069991236, 0.36148732210997464, 0.331772028897207, 0.3569755042491488, 0.3645807766719102, 0.348606916867408, 0.4495298685360269, 0.36152220935468593, 0.36813386458562714, 0.31260547344663797, 0.27812345266831623, 0.4559474845800431, 0.4220093915709082, 0.39660299227905194, 0.43399406737483454, 0.4691431327129764, 0.39240389395484915, 0.48269123247404677, 0.4334432511608902, 0.44937287047752356] (text2motion) beibeigh@beibeigh-System-Product-Name:~/Research/Projects/LanguageText2Pose/Complextext2animation-main/src$