search

ku21fan / STR-Fewer-Labels

Scene Text Recognition (STR) methods trained with fewer real labels (CVPR 2021)
MIT License
171 stars 27 forks source link

question about the experimental results #6

Closed hyangyu closed 2 years ago

hyangyu commented 2 years ago

Dear author, when I use the pretrained model of CRNN, the accuracy of each evaluation dataset cannot reach the result reported in the paper. I wonder what is wrong with that? Thank you.

ku21fan commented 2 years ago

Hello,

First of all, it should be slightly different because the reported value is the average value of three different trials. (See the "evaluation metric" part of subsection 5.1 Implementation Detail in our paper.)

If there is a huge difference between the reported value and the accuracy of the pretrained model, it seems to be something wrong. In this case, can you upload your evaluation log (the results of the pretrained model)? To check something wrong or not, seeing how much difference between the results can be helpful.

Hope it helps.

Best

ku21fan commented 2 years ago

Hello,

"TPS-ResNet-BiLSTM-Attn.pth" is a different model.

"TPS-ResNet-BiLSTM-Attn.pth" https://www.dropbox.com/sh/j3xmli4di1zuv3s/AAArdcPgz7UFxIHUuKNOeKv_a?dl=0 is the pretrained model of the ICCV2019 paper "What Is Wrong With Scene Text Recognition Model Comparisons? Dataset and Model Analysis."

In our CVPR2021 paper "What If We Only Use Real Datasets for Scene Text Recognition? Toward Scene Text Recognition With Fewer Labels," we used the real data for training, and you can download pretrained models of the paper in here https://www.dropbox.com/sh/23adceu2i85c4x1/AACLmaiL43Jy8eYIVVUkZ344a?dl=0. Check "TRBA-Baseline-real.pth"

Hope it helps :)

Best

2022년 4월 24일 (일) 오전 12:19, hyangyu @.***>님이 작성:

Sorry to bother you again. I have evaluated the performance of TRBA on six benchmarks with the pre-trained model "TPS-ResNet-BiLSTM-Attn.pth" and the accuracy of each dataset is as follows:

accuracy: IIIT5k_3000: 87.367 SVT: 87.326 IC13_1015: 92.217 IC15_2077: 75.390 SVTP: 80.155 CUTE80: 74.216 total_accuracy: 83.834 averaged_infer_time: 1.379 # parameters: 49.555

but I cannot find the corresponding results in the published paper. Did I get anything wrong?

— Reply to this email directly, view it on GitHub https://github.com/ku21fan/STR-Fewer-Labels/issues/6#issuecomment-1107519767, or unsubscribe https://github.com/notifications/unsubscribe-auth/AB4V4M2C2UPZOWHXJAVBCATVGQIJNANCNFSM5UEKQPJA . You are receiving this because you commented.Message ID: @.***>