search

hlt-mt / TranscRater

An open-source tool for automatic speech recognition ASR quality estimation.
23 stars 9 forks source link

issue: calculating MAE, NDCG #1

Open tien-le-grenoble opened 8 years ago

tien-le-grenoble commented 8 years ago

Hi,

I would like to run some commands in "2.2.1. First run" - Manual.pdf, for example: time . fast_run_RR.sh

However, I got the following result:

calculate MAE: python /home/tien/TranscRater/bin/compute_MAE.py /home/tien/TranscRater/data/RR_train_LEX_LM_POS.data /home/tien/TranscRater/temp/RR/train.pwer bash: printf: 0.0634067073171: nombre non valable MAE: 0,000 bash: printf: 88.2451328384: nombre non valable NDCG: 0,000

$ head -3 /home/tien/TranscRater/data/RR_train_LEX_LM_POS.data /home/tien/TranscRater/temp/RR/train.pwer ==> /home/tien/TranscRater/data/RR_train_LEX_LM_POS.data <== 0.100 qid:f01_050c0101_ped_real 1:0.03090 2:0.07000 3:0.11060 4:0.11060 5:-56.93430 6:-44.11260 7:-38.03130 8:-38.03130 9:20.01620 10:10.19310 11:7.40120 12:7.40120 13:0.99690 14:0.99620 15:0.99630 16:19.00000 17:0.31580 18:0.21050 19:0.05260 20:0.10530 21:0.63160 22:0.31580 23:0.00000 24:0.78950 25:0.36840 26:0.42110 27:0.89470 28:1.47370 29:4.78950 0.100 qid:f01_050c0101_ped_real 1:0.04170 2:0.07920 3:0.11160 4:0.11160 5:-59.77510 6:-36.43120 7:-29.99140 8:-29.99140 9:19.86090 10:6.80350 11:4.84760 12:4.84760 13:0.98750 14:0.99640 15:0.99650 16:20.00000 17:0.30000 18:0.25000 19:0.10000 20:0.10000 21:0.60000 22:0.35000 23:0.00000 24:0.75000 25:0.30000 26:0.45000 27:0.75000 28:1.40000 29:5.20000 0.200 qid:f01_050c0101_ped_real 1:0.04140 2:0.07390 3:0.10220 4:0.10220 5:-57.52060 6:-42.44310 7:-40.79450 8:-40.79450 9:17.74370 10:8.34910 11:7.68850 12:7.68850 13:0.98760 14:0.99790 15:0.99800 16:20.00000 17:0.30000 18:0.25000 19:0.10000 20:0.10000 21:0.60000 22:0.35000 23:0.00000 24:0.75000 25:0.30000 26:0.40000 27:0.90000 28:1.55000 29:5.45000

==> /home/tien/TranscRater/temp/RR/train.pwer <== 0.196 0.187 0.215

Could you help me to find out above issue? Thanks for your support! Tien LE

shahabjld commented 8 years ago

Dear Tien,

I'm out of the office till the day after tomorrow.

I'll answer you by Thursday.

Bests Shahab On May 14, 2016 03:24, "Tien N. LE" notifications@github.com wrote:

Hi,

I would like to run some commands in "2.2.1. First run" - Manual.pdf, for example: _time . fast_runRR.sh

However, I got the following result:

calculate MAE: python /home/tien/TranscRater/bin/compute_MAE.py /home/tien/TranscRater/data/RR_train_LEX_LM_POS.data /home/tien/TranscRater/temp/RR/train.pwer bash: printf: 0.0634067073171: nombre non valable MAE: 0,000 bash: printf: 88.2451328384: nombre non valable

NDCG: 0,000

$ head -3 /home/tien/TranscRater/data/RR_train_LEX_LM_POS.data /home/tien/TranscRater/temp/RR/train.pwer ==> /home/tien/TranscRater/data/RR_train_LEX_LM_POS.data <== 0.100 qid:f01_050c0101_ped_real 1:0.03090 2:0.07000 3:0.11060 4:0.11060 5:-56.93430 6:-44.11260 7:-38.03130 8:-38.03130 9:20.01620 10:10.19310 11:7.40120 12:7.40120 13:0.99690 14:0.99620 15:0.99630 16:19.00000 17:0.31580 18:0.21050 19:0.05260 20:0.10530 21:0.63160 22:0.31580 23:0.00000 24:0.78950 25:0.36840 26:0.42110 27:0.89470 28:1.47370 29:4.78950 0.100 qid:f01_050c0101_ped_real 1:0.04170 2:0.07920 3:0.11160 4:0.11160 5:-59.77510 6:-36.43120 7:-29.99140 8:-29.99140 9:19.86090 10:6.80350 11:4.84760 12:4.84760 13:0.98750 14:0.99640 15:0.99650 16:20.00000 17:0.30000 18:0.25000 19:0.10000 20:0.10000 21:0.60000 22:0.35000 23:0.00000 24:0.75000 25:0.30000 26:0.45000 27:0.75000 28:1.40000 29:5.20000 0.200 qid:f01_050c0101_ped_real 1:0.04140 2:0.07390 3:0.10220 4:0.10220 5:-57.52060 6:-42.44310 7:-40.79450 8:-40.79450 9:17.74370 10:8.34910 11:7.68850 12:7.68850 13:0.98760 14:0.99790 15:0.99800 16:20.00000 17:0.30000 18:0.25000 19:0.10000 20:0.10000 21:0.60000 22:0.35000 23:0.00000 24:0.75000 25:0.30000 26:0.40000 27:0.90000 28:1.55000 29:5.45000

==> /home/tien/TranscRater/temp/RR/train.pwer <== 0.196 0.187

0.215

Could you help me to find out above issue? Thanks for your support! Tien LE

— You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub https://github.com/hlt-mt/TranscRater/issues/1

tien-le-grenoble commented 8 years ago

Dear Shahab, Thanks for your reply! Best regards, Tien

shahabjld commented 8 years ago

Dear Tien,

I believe that this issue is because of not having control on the number of digits after decimal point in the terminal. I modified the code for both compute_MAE and compute_NDCG in order to control this number internally. Please could you check if the problem is solved?

Thanks a lot Shahab

tien-le-grenoble commented 8 years ago

Hi, I updated 2 above files. Then, I re-run command: time . fast_run_RR.sh

And our current result:

bash: printf: 0.063: nombre non valable MAE: 0,000 bash: printf: 0.8820.8820.8820.8820.8820.8820.8820.8820.8820.8820.8820.8820.8820.8820.8820.8820.8820.8820.8820.8820.8820.8820.8820.8820.8820.8820.8820.8820.8820.8820.8820.8820.8820.8820.8820.8820.8820.8820.8820.8820.8820.8820.8820.8820.8820.8820.8820.8820.8820.8820.8820.8820.8820.8820.8820.8820.8820.8820.8820.8820.8820.8820.8820.8820.8820.8820.8820.8820.8820.8820.8820.8820.8820.8820.8820.8820.8820.8820.8820.8820.8820.8820.8820.8820.8820.8820.8820.8820.8820.8820.8820.8820.8820.8820.8820.8820.8820.8820.8820.882: nombre non valable NDCG: 0,000

Thanks for your support! Tien

shahabjld commented 8 years ago

Do you mind trying it again? If the problem still exists please send me "/home/tien/TranscRater/temp/RR/train.pwer" file. Then I'll try to resolve it.

tien-le-grenoble commented 8 years ago

Hi, I would like to send you the following files: train.pwer and train.pwer.rank

train.pwer.rank.zip train.pwer.zip

Cheers, Tien