Training Loss

[usmanxia]

Can you please share the training loss that you experienced at the end of your training? I am trying to train the model but not getting expected results. Thank you

[deepsemantic]

Sorry for late response, I found one of my experimental logs on my old laptop, but it's similar to the provided examples, hope it helps.

I0121 11:22:25.310305 6981 image_data_layer.cpp:150] Restarting data prefetching from start. I0121 11:22:25.736824 6967 solver.cpp:414] Test net output #0: accuracy = 0.00039446 I0121 11:22:25.736872 6967 solver.cpp:414] Test net output #1: cross_entropy_loss = 7.73027 ( 20 = 154.605 loss) I0121 11:22:25.942699 6967 solver.cpp:242] Iteration 0, loss = 151.539 I0121 11:22:25.942756 6967 solver.cpp:258] Train net output #0: accuracy = 0 I0121 11:22:25.942773 6967 solver.cpp:258] Train net output #1: cross_entropy_loss = 7.57697 ( 20 = 151.539 loss) I0121 11:22:25.942802 6967 solver.cpp:571] Iteration 0, lr = 0.01 I0121 11:22:40.078191 6967 solver.cpp:242] Iteration 50, loss = 130.968 I0121 11:22:40.078258 6967 solver.cpp:258] Train net output #0: accuracy = 0.104478 I0121 11:22:40.078274 6967 solver.cpp:258] Train net output #1: cross_entropy_loss = 5.46427 ( 20 = 109.285 loss) I0121 11:22:40.078290 6967 solver.cpp:571] Iteration 50, lr = 0.01 I0121 11:22:54.318545 6967 solver.cpp:242] Iteration 100, loss = 120.63 I0121 11:22:54.318614 6967 solver.cpp:258] Train net output #0: accuracy = 0.057971 I0121 11:22:54.318639 6967 solver.cpp:258] Train net output #1: cross_entropy_loss = 5.69797 ( 20 = 113.959 loss) I0121 11:22:54.318652 6967 solver.cpp:571] Iteration 100, lr = 0.01 I0121 11:23:08.571382 6967 solver.cpp:242] Iteration 150, loss = 107.777 I0121 11:23:08.571498 6967 solver.cpp:258] Train net output #0: accuracy = 0.160714 I0121 11:23:08.571521 6967 solver.cpp:258] Train net output #1: cross_entropy_loss = 5.1158 ( 20 = 102.316 loss) I0121 11:23:08.571535 6967 solver.cpp:571] Iteration 150, lr = 0.01 I0121 11:23:22.804710 6967 solver.cpp:242] Iteration 200, loss = 103.695 I0121 11:23:22.804764 6967 solver.cpp:258] Train net output #0: accuracy = 0.181818 I0121 11:23:22.804780 6967 solver.cpp:258] Train net output #1: cross_entropy_loss = 4.86012 ( 20 = 97.2023 loss) I0121 11:23:22.804795 6967 solver.cpp:571] Iteration 200, lr = 0.01 I0121 11:23:36.999411 6967 solver.cpp:242] Iteration 250, loss = 102.157 I0121 11:23:36.999465 6967 solver.cpp:258] Train net output #0: accuracy = 0.15625 I0121 11:23:36.999483 6967 solver.cpp:258] Train net output #1: cross_entropy_loss = 4.81321 ( 20 = 96.2642 loss) I0121 11:23:36.999497 6967 solver.cpp:571] Iteration 250, lr = 0.01 I0121 11:23:51.203235 6967 solver.cpp:242] Iteration 300, loss = 100.736 I0121 11:23:51.203356 6967 solver.cpp:258] Train net output #0: accuracy = 0.132075 I0121 11:23:51.203375 6967 solver.cpp:258] Train net output #1: cross_entropy_loss = 4.8058 ( 20 = 96.116 loss) I0121 11:23:51.203397 6967 solver.cpp:571] Iteration 300, lr = 0.01 I0121 11:24:05.397879 6967 solver.cpp:242] Iteration 350, loss = 99.7511 I0121 11:24:05.397958 6967 solver.cpp:258] Train net output #0: accuracy = 0.137931 I0121 11:24:05.397976 6967 solver.cpp:258] Train net output #1: cross_entropy_loss = 5.43398 ( 20 = 108.68 loss) I0121 11:24:05.397989 6967 solver.cpp:571] Iteration 350, lr = 0.01 I0121 11:24:19.592331 6967 solver.cpp:242] Iteration 400, loss = 99.3335 I0121 11:24:19.592398 6967 solver.cpp:258] Train net output #0: accuracy = 0.104478 I0121 11:24:19.592424 6967 solver.cpp:258] Train net output #1: cross_entropy_loss = 4.83965 ( 20 = 96.793 loss) I0121 11:24:19.592437 6967 solver.cpp:571] Iteration 400, lr = 0.01 I0121 11:24:33.784838 6967 solver.cpp:242] Iteration 450, loss = 98.8016 I0121 11:24:33.784994 6967 solver.cpp:258] Train net output #0: accuracy = 0.15942 I0121 11:24:33.785017 6967 solver.cpp:258] Train net output #1: cross_entropy_loss = 4.7255 ( 20 = 94.51 loss) I0121 11:24:33.785035 6967 solver.cpp:571] Iteration 450, lr = 0.01 I0121 11:24:47.991668 6967 solver.cpp:242] Iteration 500, loss = 98.3618 I0121 11:24:47.991726 6967 solver.cpp:258] Train net output #0: accuracy = 0.09375 I0121 11:24:47.991745 6967 solver.cpp:258] Train net output #1: cross_entropy_loss = 4.941 ( 20 = 98.8199 loss) I0121 11:24:47.991763 6967 solver.cpp:571] Iteration 500, lr = 0.01 I0121 11:25:02.184698 6967 solver.cpp:242] Iteration 550, loss = 98.6693 I0121 11:25:02.184752 6967 solver.cpp:258] Train net output #0: accuracy = 0.111111 I0121 11:25:02.184772 6967 solver.cpp:258] Train net output #1: cross_entropy_loss = 4.78535 ( 20 = 95.7069 loss) I0121 11:25:02.184784 6967 solver.cpp:571] Iteration 550, lr = 0.01 I0121 11:25:16.382648 6967 solver.cpp:242] Iteration 600, loss = 98.394 I0121 11:25:16.382751 6967 solver.cpp:258] Train net output #0: accuracy = 0.20339 I0121 11:25:16.382772 6967 solver.cpp:258] Train net output #1: cross_entropy_loss = 4.87894 ( 20 = 97.5788 loss) I0121 11:25:16.382788 6967 solver.cpp:571] Iteration 600, lr = 0.01 I0121 11:25:30.572062 6967 solver.cpp:242] Iteration 650, loss = 97.412 I0121 11:25:30.572118 6967 solver.cpp:258] Train net output #0: accuracy = 0.0983607 I0121 11:25:30.572135 6967 solver.cpp:258] Train net output #1: cross_entropy_loss = 4.96279 ( 20 = 99.2559 loss) I0121 11:25:30.572150 6967 solver.cpp:571] Iteration 650, lr = 0.01 I0121 11:25:44.765990 6967 solver.cpp:242] Iteration 700, loss = 97.6146 I0121 11:25:44.766044 6967 solver.cpp:258] Train net output #0: accuracy = 0.226415 I0121 11:25:44.766060 6967 solver.cpp:258] Train net output #1: cross_entropy_loss = 4.87698 ( 20 = 97.5396 loss) I0121 11:25:44.766072 6967 solver.cpp:571] Iteration 700, lr = 0.01 I0121 11:25:58.954264 6967 solver.cpp:242] Iteration 750, loss = 96.491

....... I0121 23:45:21.838233 6967 solver.cpp:258] Train net output #1: cross_entropy_loss = 2.1057 ( 20 = 42.1139 loss) I0121 23:45:21.838251 6967 solver.cpp:571] Iteration 109850, lr = 0.0025 I0121 23:45:35.991255 6967 solver.cpp:242] Iteration 109900, loss = 44.926 I0121 23:45:35.991304 6967 solver.cpp:258] Train net output #0: accuracy = 0.396552 I0121 23:45:35.991319 6967 solver.cpp:258] Train net output #1: cross_entropy_loss = 2.51174 ( 20 = 50.2349 loss) I0121 23:45:35.991331 6967 solver.cpp:571] Iteration 109900, lr = 0.0025 I0121 23:45:50.162571 6967 solver.cpp:242] Iteration 109950, loss = 45.8994 I0121 23:45:50.162673 6967 solver.cpp:258] Train net output #0: accuracy = 0.377049 I0121 23:45:50.162689 6967 solver.cpp:258] Train net output #1: cross_entropy_loss = 3.05881 ( 20 = 61.1762 loss) I0121 23:45:50.162700 6967 solver.cpp:571] Iteration 109950, lr = 0.0025 I0121 23:46:04.124423 6967 solver.cpp:449] Snapshotting to binary proto file ./examples/flickr8K/trained_models/lrcn20160121_iter_110000.caffemodel I0121 23:46:04.124464 6967 net.cpp:948] Serializing 34 layers I0121 23:47:45.034051 6967 solver.cpp:734] Snapshotting solver state to binary proto file./examples/flickr8K/trained_models/lrcn20160121_iter_110000.solverstate I0121 23:47:47.449722 6967 solver.cpp:326] Iteration 110000, loss = 36.1393 I0121 23:47:47.449769 6967 solver.cpp:346] Iteration 110000, Testing net (#0) I0121 23:47:47.449782 6967 net.cpp:781] Copying source layer data I0121 23:47:47.449792 6967 net.cpp:781] Copying source layer data I0121 23:47:47.449798 6967 net.cpp:781] Copying source layer cont_sentence_data_0_split I0121 23:47:47.449803 6967 net.cpp:781] Copying source layer target_sentence_data_2_split I0121 23:47:47.449811 6967 net.cpp:781] Copying source layer silence I0121 23:47:47.449816 6967 net.cpp:781] Copying source layer conv1 I0121 23:47:47.449825 6967 net.cpp:781] Copying source layer relu1 I0121 23:47:47.449829 6967 net.cpp:781] Copying source layer pool1 I0121 23:47:47.449834 6967 net.cpp:781] Copying source layer norm1 I0121 23:47:47.449839 6967 net.cpp:781] Copying source layer conv2 I0121 23:47:47.449847 6967 net.cpp:781] Copying source layer relu2 I0121 23:47:47.449852 6967 net.cpp:781] Copying source layer pool2 I0121 23:47:47.449857 6967 net.cpp:781] Copying source layer norm2 I0121 23:47:47.449862 6967 net.cpp:781] Copying source layer conv3 I0121 23:47:47.449868 6967 net.cpp:781] Copying source layer relu3 I0121 23:47:47.449873 6967 net.cpp:781] Copying source layer conv4 I0121 23:47:47.449880 6967 net.cpp:781] Copying source layer relu4 I0121 23:47:47.449887 6967 net.cpp:781] Copying source layer conv5 I0121 23:47:47.449892 6967 net.cpp:781] Copying source layer relu5 I0121 23:47:47.449898 6967 net.cpp:781] Copying source layer pool5 I0121 23:47:47.449903 6967 net.cpp:781] Copying source layer fc6 I0121 23:47:47.449908 6967 net.cpp:781] Copying source layer relu6 I0121 23:47:47.449915 6967 net.cpp:781] Copying source layer drop6 I0121 23:47:47.449920 6967 net.cpp:781] Copying source layer fc7 I0121 23:47:47.449926 6967 net.cpp:781] Copying source layer relu7 I0121 23:47:47.449933 6967 net.cpp:781] Copying source layer drop7 I0121 23:47:47.449937 6967 net.cpp:781] Copying source layer fc8 I0121 23:47:47.449944 6967 net.cpp:781] Copying source layer embedding I0121 23:47:47.449949 6967 net.cpp:781] Copying source layer lstm1 I0121 23:47:47.449957 6967 net.cpp:781] Copying source layer lstm2 I0121 23:47:47.449964 6967 net.cpp:781] Copying source layer predict I0121 23:47:47.449970 6967 net.cpp:781] Copying source layer predict_predict_0_split I0121 23:47:47.449975 6967 net.cpp:781] Copying source layer cross_entropy_loss I0121 23:47:47.449990 6967 net.cpp:781] Copying source layer accuracy I0121 23:49:40.741700 6967 solver.cpp:414] Test net output #0: accuracy = 0.524921 I0121 23:49:40.741817 6967 solver.cpp:414] Test net output #1: cross_entropy_loss = 1.99833 ( 20 = 39.9666 loss) I0121 23:49:40.741830 6967 solver.cpp:346] Iteration 110000, Testing net (#1) I0121 23:49:40.741847 6967 net.cpp:781] Copying source layer data I0121 23:49:40.741857 6967 net.cpp:781] Copying source layer data I0121 23:49:40.741864 6967 net.cpp:781] Copying source layer cont_sentence_data_0_split I0121 23:49:40.741871 6967 net.cpp:781] Copying source layer target_sentence_data_2_split I0121 23:49:40.741881 6967 net.cpp:781] Copying source layer silence I0121 23:49:40.741890 6967 net.cpp:781] Copying source layer conv1 I0121 23:49:40.741905 6967 net.cpp:781] Copying source layer relu1 I0121 23:49:40.741914 6967 net.cpp:781] Copying source layer pool1 I0121 23:49:40.741925 6967 net.cpp:781] Copying source layer norm1 I0121 23:49:40.741932 6967 net.cpp:781] Copying source layer conv2 I0121 23:49:40.741945 6967 net.cpp:781] Copying source layer relu2 I0121 23:49:40.741955 6967 net.cpp:781] Copying source layer pool2 I0121 23:49:40.741966 6967 net.cpp:781] Copying source layer norm2 I0121 23:49:40.741976 6967 net.cpp:781] Copying source layer conv3 I0121 23:49:40.741986 6967 net.cpp:781] Copying source layer relu3 I0121 23:49:40.741997 6967 net.cpp:781] Copying source layer conv4 I0121 23:49:40.742007 6967 net.cpp:781] Copying source layer relu4 I0121 23:49:40.742018 6967 net.cpp:781] Copying source layer conv5 I0121 23:49:40.742029 6967 net.cpp:781] Copying source layer relu5 I0121 23:49:40.742040 6967 net.cpp:781] Copying source layer pool5 I0121 23:49:40.742049 6967 net.cpp:781] Copying source layer fc6 I0121 23:49:40.742061 6967 net.cpp:781] Copying source layer relu6 I0121 23:49:40.742074 6967 net.cpp:781] Copying source layer drop6 I0121 23:49:40.742089 6967 net.cpp:781] Copying source layer fc7 I0121 23:49:40.742105 6967 net.cpp:781] Copying source layer relu7 I0121 23:49:40.742116 6967 net.cpp:781] Copying source layer drop7 I0121 23:49:40.742127 6967 net.cpp:781] Copying source layer fc8 I0121 23:49:40.742139 6967 net.cpp:781] Copying source layer embedding I0121 23:49:40.742152 6967 net.cpp:781] Copying source layer lstm1 I0121 23:49:40.742166 6967 net.cpp:781] Copying source layer lstm2 I0121 23:49:40.742179 6967 net.cpp:781] Copying source layer predict I0121 23:49:40.742192 6967 net.cpp:781] Copying source layer predict_predict_0_split I0121 23:49:40.742203 6967 net.cpp:781] Copying source layer cross_entropy_loss I0121 23:49:40.742214 6967 net.cpp:781] Copying source layer accuracy I0121 23:51:33.523701 6981 image_data_layer.cpp:150] Restarting data prefetching from start. I0121 23:51:33.947804 6967 solver.cpp:414] Test net output #0: accuracy = 0.369726 I0121 23:51:33.947850 6967 solver.cpp:414] Test net output #1: cross_entropy_loss = 3.00547 (* 20 = 60.1093 loss) I0121 23:51:33.947860 6967 solver.cpp:331] Optimization Done. I0121 23:51:33.947875 6967 caffe.cpp:214] Optimization Done.

[usmanxia]

Thank you sir! Really appreciate!

On Dec 18, 2017 2:41 AM, "Cheng Wang" notifications@github.com wrote:

Sorry for late response, I found one of my experimental logs on my old laptop, but it's similar to the provided examples, hope it hopes.

I0121 11:22:25.310305 6981 image_data_layer.cpp:150] Restarting data prefetching from start. I0121 11:22:25.736824 6967 solver.cpp:414] Test net output #0: accuracy = 0.00039446 I0121 11:22:25.736872 6967 solver.cpp:414] Test net output #1 https://github.com/deepsemantic/image_captioning/issues/1: cross_entropy_loss = 7.73027 ( 20 = 154.605 loss) I0121 11:22:25.942699 6967 solver.cpp:242] Iteration 0, loss = 151.539 I0121 11:22:25.942756 6967 solver.cpp:258] Train net output #0: accuracy = 0 I0121 11:22:25.942773 6967 solver.cpp:258] Train net output #1 https://github.com/deepsemantic/image_captioning/issues/1: cross_entropy_loss = 7.57697 ( 20 = 151.539 loss) I0121 11:22:25.942802 6967 solver.cpp:571] Iteration 0, lr = 0.01 I0121 11:22:40.078191 6967 solver.cpp:242] Iteration 50, loss = 130.968 I0121 11:22:40.078258 6967 solver.cpp:258] Train net output #0: accuracy = 0.104478 I0121 11:22:40.078274 6967 solver.cpp:258] Train net output #1 https://github.com/deepsemantic/image_captioning/issues/1: cross_entropy_loss = 5.46427 ( 20 = 109.285 loss) I0121 11:22:40.078290 6967 solver.cpp:571] Iteration 50, lr = 0.01 I0121 11:22:54.318545 6967 solver.cpp:242] Iteration 100, loss = 120.63 I0121 11:22:54.318614 6967 solver.cpp:258] Train net output #0: accuracy = 0.057971 I0121 11:22:54.318639 6967 solver.cpp:258] Train net output #1 https://github.com/deepsemantic/image_captioning/issues/1: cross_entropy_loss = 5.69797 ( 20 = 113.959 loss) I0121 11:22:54.318652 6967 solver.cpp:571] Iteration 100, lr = 0.01 I0121 11:23:08.571382 6967 solver.cpp:242] Iteration 150, loss = 107.777 I0121 11:23:08.571498 6967 solver.cpp:258] Train net output #0: accuracy = 0.160714 I0121 11:23:08.571521 6967 solver.cpp:258] Train net output #1 https://github.com/deepsemantic/image_captioning/issues/1: cross_entropy_loss = 5.1158 ( 20 = 102.316 loss) I0121 11:23:08.571535 6967 solver.cpp:571] Iteration 150, lr = 0.01 I0121 11:23:22.804710 6967 solver.cpp:242] Iteration 200, loss = 103.695 I0121 11:23:22.804764 6967 solver.cpp:258] Train net output #0: accuracy = 0.181818 I0121 11:23:22.804780 6967 solver.cpp:258] Train net output #1 https://github.com/deepsemantic/image_captioning/issues/1: cross_entropy_loss = 4.86012 ( 20 = 97.2023 loss) I0121 11:23:22.804795 6967 solver.cpp:571] Iteration 200, lr = 0.01 I0121 11:23:36.999411 6967 solver.cpp:242] Iteration 250, loss = 102.157 I0121 11:23:36.999465 6967 solver.cpp:258] Train net output #0: accuracy = 0.15625 I0121 11:23:36.999483 6967 solver.cpp:258] Train net output #1 https://github.com/deepsemantic/image_captioning/issues/1: cross_entropy_loss = 4.81321 ( 20 = 96.2642 loss) I0121 11:23:36.999497 6967 solver.cpp:571] Iteration 250, lr = 0.01 I0121 11:23:51.203235 6967 solver.cpp:242] Iteration 300, loss = 100.736 I0121 11:23:51.203356 6967 solver.cpp:258] Train net output #0: accuracy = 0.132075 I0121 11:23:51.203375 6967 solver.cpp:258] Train net output #1 https://github.com/deepsemantic/image_captioning/issues/1: cross_entropy_loss = 4.8058 ( 20 = 96.116 loss) I0121 11:23:51.203397 6967 solver.cpp:571] Iteration 300, lr = 0.01 I0121 11:24:05.397879 6967 solver.cpp:242] Iteration 350, loss = 99.7511 I0121 11:24:05.397958 6967 solver.cpp:258] Train net output #0: accuracy = 0.137931 I0121 11:24:05.397976 6967 solver.cpp:258] Train net output #1 https://github.com/deepsemantic/image_captioning/issues/1: cross_entropy_loss = 5.43398 ( 20 = 108.68 loss) I0121 11:24:05.397989 6967 solver.cpp:571] Iteration 350, lr = 0.01 I0121 11:24:19.592331 6967 solver.cpp:242] Iteration 400, loss = 99.3335 I0121 11:24:19.592398 6967 solver.cpp:258] Train net output #0: accuracy = 0.104478 I0121 11:24:19.592424 6967 solver.cpp:258] Train net output #1 https://github.com/deepsemantic/image_captioning/issues/1: cross_entropy_loss = 4.83965 ( 20 = 96.793 loss) I0121 11:24:19.592437 6967 solver.cpp:571] Iteration 400, lr = 0.01 I0121 11:24:33.784838 6967 solver.cpp:242] Iteration 450, loss = 98.8016 I0121 11:24:33.784994 6967 solver.cpp:258] Train net output #0: accuracy = 0.15942 I0121 11:24:33.785017 6967 solver.cpp:258] Train net output #1 https://github.com/deepsemantic/image_captioning/issues/1: cross_entropy_loss = 4.7255 ( 20 = 94.51 loss) I0121 11:24:33.785035 6967 solver.cpp:571] Iteration 450, lr = 0.01 I0121 11:24:47.991668 6967 solver.cpp:242] Iteration 500, loss = 98.3618 I0121 11:24:47.991726 6967 solver.cpp:258] Train net output #0: accuracy = 0.09375 I0121 11:24:47.991745 6967 solver.cpp:258] Train net output #1 https://github.com/deepsemantic/image_captioning/issues/1: cross_entropy_loss = 4.941 ( 20 = 98.8199 loss) I0121 11:24:47.991763 6967 solver.cpp:571] Iteration 500, lr = 0.01 I0121 11:25:02.184698 6967 solver.cpp:242] Iteration 550, loss = 98.6693 I0121 11:25:02.184752 6967 solver.cpp:258] Train net output #0: accuracy = 0.111111 I0121 11:25:02.184772 6967 solver.cpp:258] Train net output #1 https://github.com/deepsemantic/image_captioning/issues/1: cross_entropy_loss = 4.78535 ( 20 = 95.7069 loss) I0121 11:25:02.184784 6967 solver.cpp:571] Iteration 550, lr = 0.01 I0121 11:25:16.382648 6967 solver.cpp:242] Iteration 600, loss = 98.394 I0121 11:25:16.382751 6967 solver.cpp:258] Train net output #0: accuracy = 0.20339 I0121 11:25:16.382772 6967 solver.cpp:258] Train net output #1 https://github.com/deepsemantic/image_captioning/issues/1: cross_entropy_loss = 4.87894 ( 20 = 97.5788 loss) I0121 11:25:16.382788 6967 solver.cpp:571] Iteration 600, lr = 0.01 I0121 11:25:30.572062 6967 solver.cpp:242] Iteration 650, loss = 97.412 I0121 11:25:30.572118 6967 solver.cpp:258] Train net output #0: accuracy = 0.0983607 I0121 11:25:30.572135 6967 solver.cpp:258] Train net output #1 https://github.com/deepsemantic/image_captioning/issues/1: cross_entropy_loss = 4.96279 ( 20 = 99.2559 loss) I0121 11:25:30.572150 6967 solver.cpp:571] Iteration 650, lr = 0.01 I0121 11:25:44.765990 6967 solver.cpp:242] Iteration 700, loss = 97.6146 I0121 11:25:44.766044 6967 solver.cpp:258] Train net output #0: accuracy = 0.226415 I0121 11:25:44.766060 6967 solver.cpp:258] Train net output #1 https://github.com/deepsemantic/image_captioning/issues/1: cross_entropy_loss = 4.87698 ( 20 = 97.5396 loss) I0121 11:25:44.766072 6967 solver.cpp:571] Iteration 700, lr = 0.01 I0121 11:25:58.954264 6967 solver.cpp:242] Iteration 750, loss = 96.491

....... I0121 23:45:21.838233 6967 solver.cpp:258] Train net output #1 https://github.com/deepsemantic/image_captioning/issues/1: cross_entropy_loss = 2.1057 ( 20 = 42.1139 loss) I0121 23:45:21.838251 6967 solver.cpp:571] Iteration 109850, lr = 0.0025 I0121 23:45:35.991255 6967 solver.cpp:242] Iteration 109900, loss = 44.926 I0121 23:45:35.991304 6967 solver.cpp:258] Train net output #0: accuracy = 0.396552 I0121 23:45:35.991319 6967 solver.cpp:258] Train net output #1 https://github.com/deepsemantic/image_captioning/issues/1: cross_entropy_loss = 2.51174 ( 20 = 50.2349 loss) I0121 23:45:35.991331 6967 solver.cpp:571] Iteration 109900, lr = 0.0025 I0121 23:45:50.162571 6967 solver.cpp:242] Iteration 109950, loss = 45.8994 I0121 23:45:50.162673 6967 solver.cpp:258] Train net output #0: accuracy = 0.377049 I0121 23:45:50.162689 6967 solver.cpp:258] Train net output #1 https://github.com/deepsemantic/image_captioning/issues/1: cross_entropy_loss = 3.05881 ( 20 = 61.1762 loss) I0121 23:45:50.162700 6967 solver.cpp:571] Iteration 109950, lr = 0.0025 I0121 23:46:04.124423 6967 solver.cpp:449] Snapshotting to binary proto file ./examples/flickr8K/trained_models/lrcn20160121iter 110000.caffemodel I0121 23:46:04.124464 6967 net.cpp:948] Serializing 34 layers I0121 23:47:45.034051 6967 solver.cpp:734] Snapshotting solver state to binary proto file./examples/flickr8K/trainedmodels/lrcn20160121 iter_110000.solverstate I0121 23:47:47.449722 6967 solver.cpp:326] Iteration 110000, loss = 36.1393 I0121 23:47:47.449769 6967 solver.cpp:346] Iteration 110000, Testing net (#0) I0121 23:47:47.449782 6967 net.cpp:781] Copying source layer data I0121 23:47:47.449792 6967 net.cpp:781] Copying source layer data I0121 23:47:47.449798 6967 net.cpp:781] Copying source layer cont_sentence_data_0_split I0121 23:47:47.449803 6967 net.cpp:781] Copying source layer target_sentence_data_2_split I0121 23:47:47.449811 6967 net.cpp:781] Copying source layer silence I0121 23:47:47.449816 6967 net.cpp:781] Copying source layer conv1 I0121 23:47:47.449825 6967 net.cpp:781] Copying source layer relu1 I0121 23:47:47.449829 6967 net.cpp:781] Copying source layer pool1 I0121 23:47:47.449834 6967 net.cpp:781] Copying source layer norm1 I0121 23:47:47.449839 6967 net.cpp:781] Copying source layer conv2 I0121 23:47:47.449847 6967 net.cpp:781] Copying source layer relu2 I0121 23:47:47.449852 6967 net.cpp:781] Copying source layer pool2 I0121 23:47:47.449857 6967 net.cpp:781] Copying source layer norm2 I0121 23:47:47.449862 6967 net.cpp:781] Copying source layer conv3 I0121 23:47:47.449868 6967 net.cpp:781] Copying source layer relu3 I0121 23:47:47.449873 6967 net.cpp:781] Copying source layer conv4 I0121 23:47:47.449880 6967 net.cpp:781] Copying source layer relu4 I0121 23:47:47.449887 6967 net.cpp:781] Copying source layer conv5 I0121 23:47:47.449892 6967 net.cpp:781] Copying source layer relu5 I0121 23:47:47.449898 6967 net.cpp:781] Copying source layer pool5 I0121 23:47:47.449903 6967 net.cpp:781] Copying source layer fc6 I0121 23:47:47.449908 6967 net.cpp:781] Copying source layer relu6 I0121 23:47:47.449915 6967 net.cpp:781] Copying source layer drop6 I0121 23:47:47.449920 6967 net.cpp:781] Copying source layer fc7 I0121 23:47:47.449926 6967 net.cpp:781] Copying source layer relu7 I0121 23:47:47.449933 6967 net.cpp:781] Copying source layer drop7 I0121 23:47:47.449937 6967 net.cpp:781] Copying source layer fc8 I0121 23:47:47.449944 6967 net.cpp:781] Copying source layer embedding I0121 23:47:47.449949 6967 net.cpp:781] Copying source layer lstm1 I0121 23:47:47.449957 6967 net.cpp:781] Copying source layer lstm2 I0121 23:47:47.449964 6967 net.cpp:781] Copying source layer predict I0121 23:47:47.449970 6967 net.cpp:781] Copying source layer predict_predict_0_split I0121 23:47:47.449975 6967 net.cpp:781] Copying source layer cross_entropy_loss I0121 23:47:47.449990 6967 net.cpp:781] Copying source layer accuracy I0121 23:49:40.741700 6967 solver.cpp:414] Test net output #0: accuracy = 0.524921 I0121 23:49:40.741817 6967 solver.cpp:414] Test net output #1 https://github.com/deepsemantic/image_captioning/issues/1: cross_entropy_loss = 1.99833 ( 20 = 39.9666 loss) I0121 23:49:40.741830 6967 solver.cpp:346] Iteration 110000, Testing net (

1 https://github.com/deepsemantic/image_captioning/issues/1)

I0121 23:49:40.741847 6967 net.cpp:781] Copying source layer data I0121 23:49:40.741857 6967 net.cpp:781] Copying source layer data I0121 23:49:40.741864 6967 net.cpp:781] Copying source layer cont_sentence_data_0_split I0121 23:49:40.741871 6967 net.cpp:781] Copying source layer target_sentence_data_2_split I0121 23:49:40.741881 6967 net.cpp:781] Copying source layer silence I0121 23:49:40.741890 6967 net.cpp:781] Copying source layer conv1 I0121 23:49:40.741905 6967 net.cpp:781] Copying source layer relu1 I0121 23:49:40.741914 6967 net.cpp:781] Copying source layer pool1 I0121 23:49:40.741925 6967 net.cpp:781] Copying source layer norm1 I0121 23:49:40.741932 6967 net.cpp:781] Copying source layer conv2 I0121 23:49:40.741945 6967 net.cpp:781] Copying source layer relu2 I0121 23:49:40.741955 6967 net.cpp:781] Copying source layer pool2 I0121 23:49:40.741966 6967 net.cpp:781] Copying source layer norm2 I0121 23:49:40.741976 6967 net.cpp:781] Copying source layer conv3 I0121 23:49:40.741986 6967 net.cpp:781] Copying source layer relu3 I0121 23:49:40.741997 6967 net.cpp:781] Copying source layer conv4 I0121 23:49:40.742007 6967 net.cpp:781] Copying source layer relu4 I0121 23:49:40.742018 6967 net.cpp:781] Copying source layer conv5 I0121 23:49:40.742029 6967 net.cpp:781] Copying source layer relu5 I0121 23:49:40.742040 6967 net.cpp:781] Copying source layer pool5 I0121 23:49:40.742049 6967 net.cpp:781] Copying source layer fc6 I0121 23:49:40.742061 6967 net.cpp:781] Copying source layer relu6 I0121 23:49:40.742074 6967 net.cpp:781] Copying source layer drop6 I0121 23:49:40.742089 6967 net.cpp:781] Copying source layer fc7 I0121 23:49:40.742105 6967 net.cpp:781] Copying source layer relu7 I0121 23:49:40.742116 6967 net.cpp:781] Copying source layer drop7 I0121 23:49:40.742127 6967 net.cpp:781] Copying source layer fc8 I0121 23:49:40.742139 6967 net.cpp:781] Copying source layer embedding I0121 23:49:40.742152 6967 net.cpp:781] Copying source layer lstm1 I0121 23:49:40.742166 6967 net.cpp:781] Copying source layer lstm2 I0121 23:49:40.742179 6967 net.cpp:781] Copying source layer predict I0121 23:49:40.742192 6967 net.cpp:781] Copying source layer predict_predict_0_split I0121 23:49:40.742203 6967 net.cpp:781] Copying source layer cross_entropy_loss I0121 23:49:40.742214 6967 net.cpp:781] Copying source layer accuracy I0121 23:51:33.523701 6981 image_data_layer.cpp:150] Restarting data prefetching from start. I0121 23:51:33.947804 6967 solver.cpp:414] Test net output #0: accuracy = 0.369726 I0121 23:51:33.947850 6967 solver.cpp:414] Test net output #1 https://github.com/deepsemantic/image_captioning/issues/1: cross_entropy_loss = 3.00547 (* 20 = 60.1093 loss) I0121 23:51:33.947860 6967 solver.cpp:331] Optimization Done. I0121 23:51:33.947875 6967 caffe.cpp:214] Optimization Done.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/deepsemantic/image_captioning/issues/9#issuecomment-352287785, or mute the thread https://github.com/notifications/unsubscribe-auth/Ab4h-TZKpiQMjjVwPGZX8FRUX3o_M5XIks5tBYqLgaJpZM4Q_VSc .

[usmanxia]

@deepsemantic can you please help me out in training the model. I have trained the model with 100 batch size as my GPU gives out of memory exception if batch size is 150.

The model gets trained and after 25000 iteration accuracy is 0.35 and loss is 3.7 but when i test the model using "bi_generation_retrieval.py", the output is as follows :

................... 755 ./data/flickr8K/images/1056249424_ef2a2e041c.jpg Two dog play in the water. 756 ./data/flickr8K/images/3265864834_e0229020dd.jpg Two dog play in the water. 757 ./data/flickr8K/images/802594049_289e3c8420.jpg Two dog play in the water. 758 ./data/flickr8K/images/3430779304_43a2146f4b.jpg A man in a red shirt be stand on a sidewalk. 759 ./data/flickr8K/images/2455286250_fb6a66175a.jpg Two dog play in the water. 760 ./data/flickr8K/images/1425919702_ddb761aeec.jpg Two dog play in the water. 761 ./data/flickr8K/images/1088767354_2acee738cf.jpg A man in a red shirt be stand on a bench. 762 ./data/flickr8K/images/540338917_57069687be.jpg Two dog play in the water. 763 ./data/flickr8K/images/3459871361_92d1ecda36.jpg A man in a red shirt be stand on a bench. 764 ./data/flickr8K/images/576920249_df1bdc2068.jpg A man in a red shirt be stand on a sidewalk. 765 ./data/flickr8K/images/2872197070_4e97c3ccfa.jpg Two dog play in the water. 766 ./data/flickr8K/images/3495349745_1b29a63571.jpg Two dog play in the water. 767 ./data/flickr8K/images/2943334864_6bab479a3e.jpg Two dog play in the water. ....................................

Its just these two sentences repeated for all 1000 val images. Any clue what I can do to trained it right? and will it be possible for you to share a trained model? My email is usman.xia@gmail.com.

Thank you