search

232525 / PureT

Implementation of 'End-to-End Transformer Based Model for Image Captioning' [AAAI 2022]
63 stars 12 forks source link

question about score #10

Open li512871889 opened 1 year ago

li512871889 commented 1 year ago

Author, thank you for your work.i have train your code with xe loss, i train it for 20 epoch .but i only get 122.5 cider, is it normal？ i want to know how high score you get when you train with xe loss.3q

232525 commented 1 year ago

Hi, 122.5 CIDEr score after the XE training? Actually, it is normal, you can continue the SCST training for higher metrics. log.txt

[INFO: 2021-08-23 01:52:14,715] ######## Epoch (VAL)16 ########
[INFO: 2021-08-23 01:52:14,715] {'Bleu_1': 0.7785170494529935, 'Bleu_2': 0.6202688540370216, 'Bleu_3': 0.48288052763278233, 'Bleu_4': 0.37480802263214913, 'METEOR': 0.2893485293726186, 'ROUGE_L': 0.577614615064298, 'CIDEr': 1.2182801264260739, 'SPICE': 0.2170227590036378}
[INFO: 2021-08-23 01:58:48,847] ######## Epoch (TEST)16 ########
[INFO: 2021-08-23 01:58:48,848] {'Bleu_1': 0.7777327417014939, 'Bleu_2': 0.6167787156303476, 'Bleu_3': 0.4783774841959114, 'Bleu_4': 0.36858402548570396, 'METEOR': 0.2889732147622691, 'ROUGE_L': 0.576869742406256, 'CIDEr': 1.2240928659565975, 'SPICE': 0.21760507825932882}
li512871889 commented 1 year ago

Thank you for your reply! This means that the highest score when using XE training is about 122 points? Because the SCST training time is too long, I want to compare the model performance through the final score of cross entropy. Some models can get 123 CIDEr after the XE training , but the scst can only improve by about 10 points.

232525 commented 1 year ago

Yes, the original Noam lr scheduler can get about 122 CIDEr score after XE training. If adopting Cosine lr scheduler may get higher CIDEr.

Actually, I think the comparison of SCST training performance may be more representative. (e.g. If I remember correctly: M2 Transformer can only get about 113-114 CIDEr score after XE training, but its SCST performance can achieve 130+ CIDEr score.)

li512871889 commented 1 year ago

Thank you for your patience, I learned a lot！

Dabiaoge111 commented 1 year ago

您好，我使用 xe 损失进行训练，但是效果并不如您给出的结果，请问这是为什么呢？ [INFO: 2023-05-18 13:44:44,544] ######## Epoch (VAL)1 ######## [INFO: 2023-05-18 13:44:44,720] {'Bleu_1': 0.3818399999999924, 'Bleu_2': 0.20107862475492425, 'Bleu_3': 0.08539206403435688, 'Bleu_4': 0.04516999958937476, 'METEOR': 0.10479199870334399, 'ROUGE_L': 0.31692343791287475, 'CIDEr': 0.07149305971582554, 'SPICE': 0.018785750055848042} [INFO: 2023-05-18 13:55:41,167] ######## Epoch (TEST)1 ######## [INFO: 2023-05-18 13:55:41,167] {'Bleu_1': 0.3790199999999924, 'Bleu_2': 0.1952244007973006, 'Bleu_3': 0.0772061291242391, 'Bleu_4': 0.03889189773339084, 'METEOR': 0.10305717766670115, 'ROUGE_L': 0.3141987201400116, 'CIDEr': 0.06542164677863438, 'SPICE': 0.017417374342758977} [INFO: 2023-05-18 17:22:34,463] ######## Epoch (VAL)2 ######## [INFO: 2023-05-18 17:22:34,484] {'Bleu_1': 0.35301135717582277, 'Bleu_2': 0.14146733112221543, 'Bleu_3': 0.0629978474612141, 'Bleu_4': 0.03486052920816647, 'METEOR': 0.10060064931708552, 'ROUGE_L': 0.3006993176045085, 'CIDEr': 0.06894694113136134, 'SPICE': 0.030987662433544005} [INFO: 2023-05-18 17:33:25,290] ######## Epoch (TEST)2 ######## [INFO: 2023-05-18 17:33:25,291] {'Bleu_1': 0.35240442849162784, 'Bleu_2': 0.1408084526990802, 'Bleu_3': 0.06367582200991487, 'Bleu_4': 0.035705580257913545, 'METEOR': 0.10050690800357387, 'ROUGE_L': 0.299936530796512, 'CIDEr': 0.07133710406163653, 'SPICE': 0.029902078514486858} [INFO: 2023-05-18 20:58:42,425] ######## Epoch (VAL)3 ######## [INFO: 2023-05-18 20:58:42,425] {'Bleu_1': 0.42905999999999145, 'Bleu_2': 0.16759811454786364, 'Bleu_3': 0.09384653514039774, 'Bleu_4': 0.05581579649727142, 'METEOR': 0.10942277416558514, 'ROUGE_L': 0.3097270434066501, 'CIDEr': 0.09253214546345255, 'SPICE': 0.0346987747890859} [INFO: 2023-05-18 21:09:53,312] ######## Epoch (TEST)3 ######## [INFO: 2023-05-18 21:09:53,313] {'Bleu_1': 0.4269199999999915, 'Bleu_2': 0.16632623364941213, 'Bleu_3': 0.09278572520597064, 'Bleu_4': 0.055206786223033315, 'METEOR': 0.10934402543406793, 'ROUGE_L': 0.3086351736281151, 'CIDEr': 0.09352031795884845, 'SPICE': 0.035185417606991135} [INFO: 2023-05-19 00:34:53,738] ######## Epoch (VAL)4 ######## [INFO: 2023-05-19 00:34:53,738] {'Bleu_1': 0.4118933150844191, 'Bleu_2': 0.2100678827597871, 'Bleu_3': 0.07343065704291461, 'Bleu_4': 0.03596276644937378, 'METEOR': 0.0973138396662465, 'ROUGE_L': 0.32639738461091783, 'CIDEr': 0.048224453824196746, 'SPICE': 0.02421402370913991} [INFO: 2023-05-19 00:45:42,314] ######## Epoch (TEST)4 ######## [INFO: 2023-05-19 00:45:42,314] {'Bleu_1': 0.41135318548727, 'Bleu_2': 0.20846477786608988, 'Bleu_3': 0.07824502812038898, 'Bleu_4': 0.04057557236874318, 'METEOR': 0.09656139773766664, 'ROUGE_L': 0.3251622071542118, 'CIDEr': 0.05062505115721602, 'SPICE': 0.023319224743997488} [INFO: 2023-05-19 04:11:22,113] ######## Epoch (VAL)5 ######## [INFO: 2023-05-19 04:11:22,113] {'Bleu_1': 0.50549999999999, 'Bleu_2': 0.2535071991088168, 'Bleu_3': 0.10767475042370363, 'Bleu_4': 0.048567924399837856, 'METEOR': 0.11003105994064094, 'ROUGE_L': 0.3402018804754431, 'CIDEr': 0.06426883799167149, 'SPICE': 0.01269253785397476} [INFO: 2023-05-19 04:22:38,880] ######## Epoch (TEST)5 ######## [INFO: 2023-05-19 04:22:38,880] {'Bleu_1': 0.5066999999999899, 'Bleu_2': 0.2526740983955366, 'Bleu_3': 0.10996019124241675, 'Bleu_4': 0.04988345260803159, 'METEOR': 0.10976608910336653, 'ROUGE_L': 0.33886501085754334, 'CIDEr': 0.06377643175175123, 'SPICE': 0.012576024426484582} [INFO: 2023-05-19 07:48:06,988] ######## Epoch (VAL)6 ######## [INFO: 2023-05-19 07:48:06,989] {'Bleu_1': 0.39514545454544736, 'Bleu_2': 0.1725176965889659, 'Bleu_3': 0.07516170190588022, 'Bleu_4': 0.03675907445855056, 'METEOR': 0.10273311281970786, 'ROUGE_L': 0.32044549238804804, 'CIDEr': 0.04869592330831971, 'SPICE': 0.038753006337179374} [INFO: 2023-05-19 07:59:22,259] ######## Epoch (TEST)6 ######## [INFO: 2023-05-19 07:59:22,259] {'Bleu_1': 0.3942363636363565, 'Bleu_2': 0.17377717184528094, 'Bleu_3': 0.07717698988541392, 'Bleu_4': 0.038690962348050306, 'METEOR': 0.10303209447952676, 'ROUGE_L': 0.3189842076052249, 'CIDEr': 0.0522295916796513, 'SPICE': 0.03778005099765087} [INFO: 2023-05-19 11:25:26,398] ######## Epoch (VAL)7 ######## [INFO: 2023-05-19 11:25:26,417] {'Bleu_1': 0.4118199999999918, 'Bleu_2': 0.14661856635501214, 'Bleu_3': 0.08154607032542044, 'Bleu_4': 0.046662244352125286, 'METEOR': 0.10743532461666641, 'ROUGE_L': 0.3038953623494287, 'CIDEr': 0.07778979394366252, 'SPICE': 0.03465007578929749} [INFO: 2023-05-19 11:36:35,791] ######## Epoch (TEST)7 ######## [INFO: 2023-05-19 11:36:35,791] {'Bleu_1': 0.40973999999999183, 'Bleu_2': 0.14562390371546988, 'Bleu_3': 0.08025460977715249, 'Bleu_4': 0.04614451368532797, 'METEOR': 0.10713992229189541, 'ROUGE_L': 0.30285356996257823, 'CIDEr': 0.07778699665092374, 'SPICE': 0.03498550900886459} [INFO: 2023-05-19 15:02:10,530] ######## Epoch (VAL)8 ######## [INFO: 2023-05-19 15:02:10,546] {'Bleu_1': 0.5024444444444334, 'Bleu_2': 0.25599085270115335, 'Bleu_3': 0.1405215778884449, 'Bleu_4': 0.08565598148593018, 'METEOR': 0.11639622241003743, 'ROUGE_L': 0.3460591644983278, 'CIDEr': 0.08727025798432432, 'SPICE': 0.03581775971017731} [INFO: 2023-05-19 15:13:22,843] ######## Epoch (TEST)8 ######## [INFO: 2023-05-19 15:13:22,843] {'Bleu_1': 0.5031999999999889, 'Bleu_2': 0.257065633642455, 'Bleu_3': 0.14062879262879904, 'Bleu_4': 0.08566815337061158, 'METEOR': 0.11615636483516203, 'ROUGE_L': 0.3460731780322913, 'CIDEr': 0.08924357223273957, 'SPICE': 0.03504227186102224} [INFO: 2023-05-19 18:40:15,531] ######## Epoch (VAL)9 ######## [INFO: 2023-05-19 18:40:15,532] {'Bleu_1': 0.41586551065761806, 'Bleu_2': 0.1736713416036651, 'Bleu_3': 0.09907689070918817, 'Bleu_4': 0.059897061008356754, 'METEOR': 0.10441264690704787, 'ROUGE_L': 0.31904993638933765, 'CIDEr': 0.0876320691875331, 'SPICE': 0.03309479295713423} [INFO: 2023-05-19 18:51:01,650] ######## Epoch (TEST)9 ######## [INFO: 2023-05-19 18:51:01,650] {'Bleu_1': 0.41421045539329643, 'Bleu_2': 0.1729510459644286, 'Bleu_3': 0.09865593671356451, 'Bleu_4': 0.06011694869450499, 'METEOR': 0.10401029393371367, 'ROUGE_L': 0.3179092233200732, 'CIDEr': 0.0884116002544005, 'SPICE': 0.0334441955883992} [INFO: 2023-05-19 22:17:03,389] ######## Epoch (VAL)10 ######## [INFO: 2023-05-19 22:17:03,389] {'Bleu_1': 0.44174545454544656, 'Bleu_2': 0.205587463360263, 'Bleu_3': 0.10412661171334027, 'Bleu_4': 0.05730231205336981, 'METEOR': 0.11730532567160329, 'ROUGE_L': 0.3367507971502956, 'CIDEr': 0.09565455860613091, 'SPICE': 0.03947567319929087} [INFO: 2023-05-19 22:28:18,736] ######## Epoch (TEST)10 ######## [INFO: 2023-05-19 22:28:18,737] {'Bleu_1': 0.43872727272726475, 'Bleu_2': 0.20451957896939008, 'Bleu_3': 0.10277797018660702, 'Bleu_4': 0.05703956189693085, 'METEOR': 0.11681861171205933, 'ROUGE_L': 0.33475096701070006, 'CIDEr': 0.09452298088423848, 'SPICE': 0.04029333724448294} [INFO: 2023-05-20 01:53:37,092] ######## Epoch (VAL)11 ######## [INFO: 2023-05-20 01:53:37,093] {'Bleu_1': 0.3808399999999924, 'Bleu_2': 0.1922456923372294, 'Bleu_3': 0.08745754250910905, 'Bleu_4': 0.048182931385741985, 'METEOR': 0.11163825505662674, 'ROUGE_L': 0.28199432338966735, 'CIDEr': 0.08833940609359202, 'SPICE': 0.03551818255032678} [INFO: 2023-05-20 02:04:24,244] ######## Epoch (TEST)11 ######## [INFO: 2023-05-20 02:04:24,244] {'Bleu_1': 0.37919999999999243, 'Bleu_2': 0.19095074408513085, 'Bleu_3': 0.08863910435319856, 'Bleu_4': 0.04959300268854557, 'METEOR': 0.11153979829772896, 'ROUGE_L': 0.28189391035497435, 'CIDEr': 0.0892316774203666, 'SPICE': 0.03384980940617799} [INFO: 2023-05-20 05:29:35,299] ######## Epoch (VAL)12 ######## [INFO: 2023-05-20 05:29:35,299] {'Bleu_1': 0.4365599999999913, 'Bleu_2': 0.2046476386377283, 'Bleu_3': 0.09250355568118845, 'Bleu_4': 0.043129524149255784, 'METEOR': 0.10404069861164401, 'ROUGE_L': 0.31463952579325943, 'CIDEr': 0.06911734074019524, 'SPICE': 0.016181802012579963} [INFO: 2023-05-20 05:40:49,646] ######## Epoch (TEST)12 ######## [INFO: 2023-05-20 05:40:49,646] {'Bleu_1': 0.43533999999999135, 'Bleu_2': 0.20195673023914595, 'Bleu_3': 0.09363053755914182, 'Bleu_4': 0.04401250852260147, 'METEOR': 0.10295618693744817, 'ROUGE_L': 0.31381064469958075, 'CIDEr': 0.06546284591841045, 'SPICE': 0.015587761718153837} [INFO: 2023-05-20 09:06:14,624] ######## Epoch (VAL)13 ######## [INFO: 2023-05-20 09:06:14,624] {'Bleu_1': 0.39514545454544736, 'Bleu_2': 0.1725176965889659, 'Bleu_3': 0.07516170190588022, 'Bleu_4': 0.03675907445855056, 'METEOR': 0.10273311281970786, 'ROUGE_L': 0.32044549238804804, 'CIDEr': 0.04869592330831971, 'SPICE': 0.038753006337179374} [INFO: 2023-05-20 09:17:29,308] ######## Epoch (TEST)13 ######## [INFO: 2023-05-20 09:17:29,308] {'Bleu_1': 0.3942363636363565, 'Bleu_2': 0.17377717184528094, 'Bleu_3': 0.07717698988541392, 'Bleu_4': 0.038690962348050306, 'METEOR': 0.10303209447952676, 'ROUGE_L': 0.3189842076052249, 'CIDEr': 0.0522295916796513, 'SPICE': 0.03778005099765087} [INFO: 2023-05-20 12:43:16,853] ######## Epoch (VAL)14 ######## [INFO: 2023-05-20 12:43:16,863] {'Bleu_1': 0.42905999999999145, 'Bleu_2': 0.16759811454786364, 'Bleu_3': 0.09384653514039774, 'Bleu_4': 0.05581579649727142, 'METEOR': 0.10942277416558514, 'ROUGE_L': 0.3097270434066501, 'CIDEr': 0.09253214546345255, 'SPICE': 0.0346987747890859} [INFO: 2023-05-20 12:54:24,108] ######## Epoch (TEST)14 ######## [INFO: 2023-05-20 12:54:24,108] {'Bleu_1': 0.4269199999999915, 'Bleu_2': 0.16632623364941213, 'Bleu_3': 0.09278572520597064, 'Bleu_4': 0.055206786223033315, 'METEOR': 0.10934402543406793, 'ROUGE_L': 0.3086351736281151, 'CIDEr': 0.09352031795884845, 'SPICE': 0.035185417606991135} [INFO: 2023-05-20 16:20:38,966] ######## Epoch (VAL)15 ######## [INFO: 2023-05-20 16:20:38,966] {'Bleu_1': 0.39514545454544736, 'Bleu_2': 0.1725176965889659, 'Bleu_3': 0.07516170190588022, 'Bleu_4': 0.03675907445855056, 'METEOR': 0.10273311281970786, 'ROUGE_L': 0.32044549238804804, 'CIDEr': 0.04869592330831971, 'SPICE': 0.038753006337179374} [INFO: 2023-05-20 16:31:53,736] ######## Epoch (TEST)15 ######## [INFO: 2023-05-20 16:31:53,736] {'Bleu_1': 0.3942363636363565, 'Bleu_2': 0.17377717184528094, 'Bleu_3': 0.07717698988541392, 'Bleu_4': 0.038690962348050306, 'METEOR': 0.10303209447952676, 'ROUGE_L': 0.3189842076052249, 'CIDEr': 0.0522295916796513, 'SPICE': 0.03778005099765087}