search

jackyjsy / CVPR21Chal-SLR

This repo contains the official code of our work SAM-SLR which won the CVPR 2021 Challenge on Large Scale Signer Independent Isolated Sign Language Recognition.
Creative Commons Zero v1.0 Universal
205 stars 50 forks source link

Accuracy of V2 paper #17

Closed LiangSiyv closed 2 years ago

LiangSiyv commented 2 years ago

I'm sorry to ask question below. The result of v1 cited in V2 paper is 97.51%. The result of V2 is 98.00%, which reaches 98.53% after adding extra data. However, the results mentioned in the v1 paper have already reached 98.53%. Then there are two possibilities. Either the improvements mentioned in V2 have no effect at all, or the data have already used the technologies mentioned in V2 when the V1 paper comes out. /(ㄒoㄒ)/~~ Could you please help me to find out the all_modilities_cc 98.53% in V2 paper?

jackyjsy commented 2 years ago

Hi LiangSiyv,

No worries. Please feel free to raise questions or concerns and I am more than happy to help. As you've noticed, we report the same number (AUTSL finetuned w/ validation data ) in V2 as V1. It is because that our newly proposed Global Ensemble Model (GEM) in V2 relies on the train-val-test dataset split for the AUTSL dataset. In our experiment setting, GEM is trained on the validation set to learn the multi-modal ensemble. If we add extra data (i.e., validation set) in finetuning the model of each modality, the validation accuracy will reach nearly 100% and the losses are extremely low. In this case, since the output scores for validation set will be very different than the scores for test set, GEM can no longer be applied to this setting to learn the multi-modal ensemble. Therefore, we didn't update the +extra data accuracy number in our V2 version.

Songyao

LiangSiyv commented 2 years ago

Thanks for your reply! Now I know why the acc number is the same between your paper v1 and v2. Hope your v2 paper is received successfully!