search

thtrieu / darkflow

Translate darknet to tensorflow. Load trained weights, retrain/fine-tune using tensorflow, export constant graph def to mobile devices
GNU General Public License v3.0
6.14k stars 2.08k forks source link

Training log #470

Closed kilianyp closed 6 years ago

kilianyp commented 6 years ago

Hi,

thank you for your work.

Could you, or anyone else share an exemplary training log of an successful training? With some details about the used parameters.

Thanks

gangooteli commented 6 years ago

Training is successful. But results are worse. Since you asked for log I am providing that.

Most of the details are in log itself. If you need anything else, please let me know

2017-12-16 19:19:51.510554: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.1 instructions, but these are available on your machine and could speed up CPU computations. 2017-12-16 19:19:51.510601: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.2 instructions, but these are available on your machine and could speed up CPU computations. 2017-12-16 19:19:51.510609: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX instructions, but these are available on your machine and could speed up CPU computations. 2017-12-16 19:19:51.510613: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX2 instructions, but these are available on your machine and could speed up CPU computations. 2017-12-16 19:19:51.510634: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use FMA instructions, but these are available on your machine and could speed up CPU computations. 2017-12-16 19:20:01.474360: E tensorflow/core/util/events_writer.cc:162] The events file ./summary/train/events.out.tfevents.1513432201.mobiliya-server1 has disappeared. 2017-12-16 19:20:01.474419: E tensorflow/core/util/events_writer.cc:131] Failed to flush 1 events to ./summary/train/events.out.tfevents.1513432201.mobiliya-server1

Parsing /home/nikhil/Documents/GTSDB_dataset_in_VOC_format/yolo-gtsdb-1c.cfg Loading None ... Finished in 0.0001239776611328125s

Building net ... Source | Train? | Layer description | Output size -------+--------+----------------------------------+--------------- | | input | (?, 416, 416, 3) Init | Yep! | conv 3x3p1_1 +bnorm leaky | (?, 416, 416, 32) Load | Yep! | maxp 2x2p0_2 | (?, 208, 208, 32) Init | Yep! | conv 3x3p1_1 +bnorm leaky | (?, 208, 208, 64) Load | Yep! | maxp 2x2p0_2 | (?, 104, 104, 64) Init | Yep! | conv 3x3p1_1 +bnorm leaky | (?, 104, 104, 128) Init | Yep! | conv 1x1p0_1 +bnorm leaky | (?, 104, 104, 64) Init | Yep! | conv 3x3p1_1 +bnorm leaky | (?, 104, 104, 128) Load | Yep! | maxp 2x2p0_2 | (?, 52, 52, 128) Init | Yep! | conv 3x3p1_1 +bnorm leaky | (?, 52, 52, 256) Init | Yep! | conv 1x1p0_1 +bnorm leaky | (?, 52, 52, 128) Init | Yep! | conv 3x3p1_1 +bnorm leaky | (?, 52, 52, 256) Load | Yep! | maxp 2x2p0_2 | (?, 26, 26, 256) Init | Yep! | conv 3x3p1_1 +bnorm leaky | (?, 26, 26, 512) Init | Yep! | conv 1x1p0_1 +bnorm leaky | (?, 26, 26, 256) Init | Yep! | conv 3x3p1_1 +bnorm leaky | (?, 26, 26, 512) Init | Yep! | conv 1x1p0_1 +bnorm leaky | (?, 26, 26, 256) Init | Yep! | conv 3x3p1_1 +bnorm leaky | (?, 26, 26, 512) Load | Yep! | maxp 2x2p0_2 | (?, 13, 13, 512) Init | Yep! | conv 3x3p1_1 +bnorm leaky | (?, 13, 13, 1024) Init | Yep! | conv 1x1p0_1 +bnorm leaky | (?, 13, 13, 512) Init | Yep! | conv 3x3p1_1 +bnorm leaky | (?, 13, 13, 1024) Init | Yep! | conv 1x1p0_1 +bnorm leaky | (?, 13, 13, 512) Init | Yep! | conv 3x3p1_1 +bnorm leaky | (?, 13, 13, 1024) Init | Yep! | conv 3x3p1_1 +bnorm leaky | (?, 13, 13, 1024) Init | Yep! | conv 3x3p1_1 +bnorm leaky | (?, 13, 13, 1024) Load | Yep! | concat [16] | (?, 26, 26, 512) Init | Yep! | conv 1x1p0_1 +bnorm leaky | (?, 26, 26, 64) Load | Yep! | local flatten 2x2 | (?, 13, 13, 256) Load | Yep! | concat [27, 24] | (?, 13, 13, 1280) Init | Yep! | conv 3x3p1_1 +bnorm leaky | (?, 13, 13, 1024) Init | Yep! | conv 1x1p0_1 linear | (?, 13, 13, 30) -------+--------+----------------------------------+--------------- GPU mode with 0.8 usage /home/nikhil/Documents/GTSDB_dataset_in_VOC_format/yolo-gtsdb-1c.cfg loss hyper-parameters: H = 13 W = 13 box = 5 classes = 1 scales = [1.0, 5.0, 1.0, 1.0] Building /home/nikhil/Documents/GTSDB_dataset_in_VOC_format/yolo-gtsdb-1c.cfg loss Building /home/nikhil/Documents/GTSDB_dataset_in_VOC_format/yolo-gtsdb-1c.cfg train op Finished in 18.700514793395996s

Enter training ...

/home/nikhil/Documents/GTSDB_dataset_in_VOC_format/yolo-gtsdb-1c.cfg parsing /home/nikhil/Documents/GTSDB_dataset_in_VOC_format/labels/single_class_annotation/ Parsing for ['0'] [> ]0% 00435.xml [> ]0% 00831.xml [> ]0% 00609.xml [> ]1% 00667.xml [> ]1% 00518.xml [> ]1% 00357.xml [> ]1% 00544.xml [> ]1% 00128.xml [> ]1% 00428.xml [> ]1% 00766.xml [> ]1% 00466.xml [> ]2% 00660.xml [> ]2% 00403.xml [> ]2% 00616.xml [> ]2% 00551.xml [> ]2% 00147.xml [> ]2% 00680.xml [> ]2% 00788.xml [> ]3% 00070.xml [> ]3% 00738.xml [> ]3% 00802.xml [> ]3% 00556.xml [> ]3% 00610.xml [> ]3% 00388.xml [> ]3% 00387.xml [> ]4% 00678.xml [> ]4% 00026.xml [> ]4% 00772.xml [> ]4% 00416.xml [> ]4% 00497.xml [> ]4% 00189.xml [> ]4% 00420.xml [> ]4% 00124.xml [> ]5% 00752.xml [> ]5% 00715.xml [> ]5% 00412.xml [> ]5% 00836.xml [=> ]5% 00299.xml [=> ]5% 00200.xml [=> ]5% 00470.xml [=> ]6% 00855.xml [=> ]6% 00250.xml [=> ]6% 00710.xml [=> ]6% 00349.xml [=> ]6% 00224.xml [=> ]6% 00882.xml [=> ]6% 00397.xml [=> ]6% 00031.xml [=> ]7% 00097.xml [=> ]7% 00373.xml [=> ]7% 00353.xml [=> ]7% 00129.xml [=> ]7% 00842.xml [=> ]7% 00603.xml [=> ]7% 00311.xml [=> ]8% 00694.xml [=> ]8% 00192.xml [=> ]8% 00868.xml [=> ]8% 00439.xml [=> ]8% 00635.xml [=> ]8% 00411.xml [=> ]8% 00367.xml [=> ]9% 00641.xml [=> ]9% 00851.xml [=> ]9% 00829.xml [=> ]9% 00079.xml [=> ]9% 00515.xml [=> ]9% 00844.xml [=> ]9% 00608.xml [=> ]9% 00238.xml [=> ]10% 00813.xml [=> ]10% 00024.xml [=> ]10% 00160.xml [=> ]10% 00848.xml [==> ]10% 00898.xml [==> ]10% 00375.xml [==> ]10% 00570.xml [==> ]11% 00780.xml [==> ]11% 00168.xml [==> ]11% 00690.xml [==> ]11% 00649.xml [==> ]11% 00225.xml [==> ]11% 00102.xml [==> ]11% 00473.xml [==> ]11% 00703.xml [==> ]12% 00068.xml [==> ]12% 00319.xml [==> ]12% 00889.xml [==> ]12% 00893.xml [==> ]12% 00347.xml [==> ]12% 00298.xml [==> ]12% 00762.xml [==> ]13% 00207.xml [==> ]13% 00080.xml [==> ]13% 00363.xml [==> ]13% 00560.xml [==> ]13% 00165.xml [==> ]13% 00312.xml [==> ]13% 00256.xml [==> ]13% 00177.xml [==> ]14% 00138.xml [==> ]14% 00849.xml [==> ]14% 00030.xml [==> ]14% 00899.xml [==> ]14% 00087.xml [==> ]14% 00440.xml [==> ]14% 00645.xml [==> ]15% 00133.xml [==> ]15% 00023.xml [==> ]15% 00041.xml [==> ]15% 00674.xml [===> ]15% 00148.xml [===> ]15% 00096.xml [===> ]15% 00390.xml [===> ]16% 00279.xml [===> ]16% 00212.xml [===> ]16% 00809.xml [===> ]16% 00090.xml [===> ]16% 00143.xml [===> ]16% 00456.xml [===> ]16% 00253.xml [===> ]16% 00604.xml [===> ]17% 00215.xml [===> ]17% 00077.xml [===> ]17% 00410.xml [===> ]17% 00146.xml [===> ]17% 00732.xml [===> ]17% 00632.xml [===> ]17% 00306.xml [===> ]18% 00434.xml [===> ]18% 00450.xml [===> ]18% 00628.xml [===> ]18% 00425.xml [===> ]18% 00758.xml [===> ]18% 00058.xml [===> ]18% 00344.xml [===> ]18% 00383.xml [===> ]19% 00055.xml [===> ]19% 00178.xml [===> ]19% 00733.xml [===> ]19% 00545.xml [===> ]19% 00009.xml [===> ]19% 00088.xml [===> ]19% 00622.xml [===> ]20% 00827.xml [===> ]20% 00272.xml [===> ]20% 00123.xml [===> ]20% 00210.xml [====> ]20% 00351.xml [====> ]20% 00887.xml [====> ]20% 00370.xml [====> ]21% 00016.xml [====> ]21% 00228.xml [====> ]21% 00169.xml [====> ]21% 00860.xml [====> ]21% 00740.xml [====> ]21% 00650.xml [====> ]21% 00293.xml [====> ]21% 00825.xml [====> ]22% 00824.xml [====> ]22% 00155.xml [====> ]22% 00013.xml [====> ]22% 00337.xml [====> ]22% 00193.xml [====> ]22% 00527.xml [====> ]22% 00048.xml [====> ]23% 00697.xml [====> ]23% 00637.xml [====> ]23% 00433.xml [====> ]23% 00218.xml [====> ]23% 00242.xml [====> ]23% 00105.xml [====> ]23% 00067.xml [====> ]23% 00707.xml [====> ]24% 00176.xml [====> ]24% 00377.xml [====> ]24% 00274.xml [====> ]24% 00419.xml [====> ]24% 00808.xml [====> ]24% 00810.xml [====> ]24% 00520.xml [====> ]25% 00422.xml [====> ]25% 00805.xml [====> ]25% 00828.xml [====> ]25% 00002.xml [=====> ]25% 00214.xml [=====> ]25% 00480.xml [=====> ]25% 00307.xml [=====> ]26% 00494.xml [=====> ]26% 00565.xml [=====> ]26% 00010.xml [=====> ]26% 00051.xml [=====> ]26% 00057.xml [=====> ]26% 00267.xml [=====> ]26% 00485.xml [=====> ]26% 00285.xml [=====> ]27% 00141.xml [=====> ]27% 00801.xml [=====> ]27% 00005.xml [=====> ]27% 00467.xml [=====> ]27% 00154.xml [=====> ]27% 00262.xml [=====> ]27% 00094.xml [=====> ]28% 00406.xml [=====> ]28% 00239.xml [=====> ]28% 00885.xml [=====> ]28% 00059.xml [=====> ]28% 00244.xml [=====> ]28% 00626.xml [=====> ]28% 00392.xml [=====> ]28% 00661.xml [=====> ]29% 00380.xml [=====> ]29% 00512.xml [=====> ]29% 00611.xml [=====> ]29% 00246.xml [=====> ]29% 00675.xml [=====> ]29% 00531.xml [=====> ]29% 00746.xml [=====> ]30% 00881.xml [=====> ]30% 00060.xml [=====> ]30% 00166.xml [=====> ]30% 00063.xml [======> ]30% 00431.xml [======> ]30% 00423.xml [======> ]30% 00859.xml [======> ]30% 00747.xml [======> ]31% 00047.xml [======> ]31% 00863.xml [======> ]31% 00864.xml [======> ]31% 00203.xml [======> ]31% 00065.xml [======> ]31% 00072.xml [======> ]31% 00257.xml [======> ]32% 00191.xml [======> ]32% 00647.xml [======> ]32% 00046.xml [======> ]32% 00557.xml [======> ]32% 00722.xml [======> ]32% 00838.xml [======> ]32% 00118.xml [======> ]33% 00270.xml [======> ]33% 00639.xml [======> ]33% 00014.xml [======> ]33% 00395.xml [======> ]33% 00259.xml [======> ]33% 00098.xml [======> ]33% 00714.xml [======> ]33% 00642.xml [======> ]34% 00159.xml [======> ]34% 00528.xml [======> ]34% 00602.xml [======> ]34% 00134.xml [======> ]34% 00530.xml [======> ]34% 00110.xml [======> ]34% 00391.xml [======> ]35% 00869.xml [======> ]35% 00408.xml [======> ]35% 00320.xml [======> ]35% 00277.xml [=======> ]35% 00850.xml [=======> ]35% 00107.xml [=======> ]35% 00853.xml [=======> ]35% 00734.xml [=======> ]36% 00683.xml [=======> ]36% 00717.xml [=======> ]36% 00723.xml [=======> ]36% 00672.xml [=======> ]36% 00818.xml [=======> ]36% 00271.xml [=======> ]36% 00783.xml [=======> ]37% 00648.xml [=======> ]37% 00121.xml [=======> ]37% 00854.xml [=======> ]37% 00744.xml [=======> ]37% 00689.xml [=======> ]37% 00245.xml [=======> ]37% 00086.xml [=======> ]38% 00183.xml [=======> ]38% 00438.xml [=======> ]38% 00452.xml [=======> ]38% 00696.xml [=======> ]38% 00455.xml [=======> ]38% 00508.xml [=======> ]38% 00106.xml [=======> ]38% 00475.xml [=======> ]39% 00043.xml [=======> ]39% 00461.xml [=======> ]39% 00310.xml [=======> ]39% 00876.xml [=======> ]39% 00140.xml [=======> ]39% 00181.xml [=======> ]39% 00263.xml [=======> ]40% 00479.xml [=======> ]40% 00791.xml [=======> ]40% 00756.xml [=======> ]40% 00216.xml [========> ]40% 00761.xml [========> ]40% 00226.xml [========> ]40% 00100.xml [========> ]40% 00265.xml [========> ]41% 00682.xml [========> ]41% 00823.xml [========> ]41% 00516.xml [========> ]41% 00841.xml [========> ]41% 00290.xml [========> ]41% 00513.xml [========> ]41% 00499.xml [========> ]42% 00289.xml [========> ]42% 00185.xml [========> ]42% 00188.xml [========> ]42% 00891.xml [========> ]42% 00208.xml [========> ]42% 00778.xml [========> ]42% 00115.xml [========> ]43% 00045.xml [========> ]43% 00774.xml [========> ]43% 00025.xml [========> ]43% 00820.xml [========> ]43% 00872.xml [========> ]43% 00186.xml [========> ]43% 00725.xml [========> ]43% 00371.xml [========> ]44% 00150.xml [========> ]44% 00220.xml [========> ]44% 00136.xml [========> ]44% 00132.xml [========> ]44% 00548.xml [========> ]44% 00038.xml [========> ]44% 00374.xml [========> ]45% 00300.xml [========> ]45% 00247.xml [========> ]45% 00393.xml [========> ]45% 00137.xml [=========> ]45% 00119.xml [=========> ]45% 00706.xml [=========> ]45% 00280.xml [=========> ]45% 00777.xml [=========> ]46% 00493.xml [=========> ]46% 00681.xml [=========> ]46% 00227.xml [=========> ]46% 00463.xml [=========> ]46% 00217.xml [=========> ]46% 00254.xml [=========> ]46% 00486.xml [=========> ]47% 00042.xml [=========> ]47% 00117.xml [=========> ]47% 00054.xml [=========> ]47% 00426.xml [=========> ]47% 00339.xml [=========> ]47% 00343.xml [=========> ]47% 00011.xml [=========> ]48% 00402.xml [=========> ]48% 00144.xml [=========> ]48% 00662.xml [=========> ]48% 00749.xml [=========> ]48% 00142.xml [=========> ]48% 00484.xml [=========> ]48% 00237.xml [=========> ]48% 00488.xml [=========> ]49% 00116.xml [=========> ]49% 00630.xml [=========> ]49% 00776.xml [=========> ]49% 00028.xml [=========> ]49% 00685.xml [=========> ]49% 00033.xml [=========> ]49% 00837.xml [=========> ]50% 00716.xml [=========> ]50% 00658.xml [=========> ]50% 00061.xml [=========> ]50% 00296.xml [==========> ]50% 00564.xml [==========> ]50% 00149.xml [==========> ]50% 00302.xml [==========> ]50% 00018.xml [==========> ]51% 00012.xml [==========> ]51% 00787.xml [==========> ]51% 00396.xml [==========> ]51% 00173.xml [==========> ]51% 00001.xml [==========> ]51% 00281.xml [==========> ]51% 00867.xml [==========> ]52% 00092.xml [==========> ]52% 00223.xml [==========> ]52% 00127.xml [==========> ]52% 00304.xml [==========> ]52% 00260.xml [==========> ]52% 00283.xml [==========> ]52% 00161.xml [==========> ]52% 00075.xml [==========> ]53% 00170.xml [==========> ]53% 00101.xml [==========> ]53% 00194.xml [==========> ]53% 00378.xml [==========> ]53% 00773.xml [==========> ]53% 00199.xml [==========> ]53% 00295.xml [==========> ]54% 00477.xml [==========> ]54% 00323.xml [==========> ]54% 00797.xml [==========> ]54% 00255.xml [==========> ]54% 00833.xml [==========> ]54% 00021.xml [==========> ]54% 00700.xml [==========> ]55% 00736.xml [==========> ]55% 00180.xml [==========> ]55% 00315.xml [==========> ]55% 00536.xml [===========> ]55% 00496.xml [===========> ]55% 00196.xml [===========> ]55% 00760.xml [===========> ]55% 00821.xml [===========> ]56% 00064.xml [===========> ]56% 00114.xml [===========> ]56% 00379.xml [===========> ]56% 00852.xml [===========> ]56% 00679.xml [===========> ]56% 00376.xml [===========> ]56% 00400.xml [===========> ]57% 00654.xml [===========> ]57% 00333.xml [===========> ]57% 00120.xml [===========> ]57% 00879.xml [===========> ]57% 00644.xml [===========> ]57% 00779.xml [===========> ]57% 00664.xml [===========> ]57% 00292.xml [===========> ]58% 00248.xml [===========> ]58% 00444.xml [===========> ]58% 00501.xml [===========> ]58% 00634.xml [===========> ]58% 00158.xml [===========> ]58% 00413.xml [===========> ]58% 00668.xml [===========> ]59% 00359.xml [===========> ]59% 00184.xml [===========> ]59% 00314.xml [===========> ]59% 00091.xml [===========> ]59% 00502.xml [===========> ]59% 00174.xml [===========> ]59% 00241.xml [===========> ]60% 00275.xml [===========> ]60% 00498.xml [===========> ]60% 00204.xml [===========> ]60% 00712.xml [============> ]60% 00737.xml [============> ]60% 00624.xml [============> ]60% 00870.xml [============> ]60% 00691.xml [============> ]61% 00330.xml [============> ]61% 00807.xml [============> ]61% 00436.xml [============> ]61% 00487.xml [============> ]61% 00481.xml [============> ]61% 00135.xml [============> ]61% 00751.xml [============> ]62% 00103.xml [============> ]62% 00646.xml [============> ]62% 00093.xml [============> ]62% 00673.xml [============> ]62% 00000.xml [============> ]62% 00728.xml [============> ]62% 00884.xml [============> ]62% 00113.xml [============> ]63% 00040.xml [============> ]63% 00482.xml [============> ]63% 00360.xml [============> ]63% 00698.xml [============> ]63% 00172.xml [============> ]63% 00449.xml [============> ]63% 00468.xml [============> ]64% 00301.xml [============> ]64% 00266.xml [============> ]64% 00264.xml [============> ]64% 00341.xml [============> ]64% 00443.xml [============> ]64% 00688.xml [============> ]64% 00794.xml [============> ]65% 00164.xml [============> ]65% 00448.xml [============> ]65% 00483.xml [============> ]65% 00593.xml [=============> ]65% 00785.xml [=============> ]65% 00032.xml [=============> ]65% 00104.xml [=============> ]65% 00231.xml [=============> ]66% 00044.xml [=============> ]66% 00446.xml [=============> ]66% 00066.xml [=============> ]66% 00131.xml [=============> ]66% 00219.xml [=============> ]66% 00073.xml [=============> ]66% 00053.xml [=============> ]67% 00182.xml [=============> ]67% 00157.xml [=============> ]67% 00249.xml [=============> ]67% 00623.xml [=============> ]67% 00386.xml [=============> ]67% 00417.xml [=============> ]67% 00686.xml [=============> ]67% 00441.xml [=============> ]68% 00500.xml [=============> ]68% 00554.xml [=============> ]68% 00613.xml [=============> ]68% 00222.xml [=============> ]68% 00704.xml [=============> ]68% 00514.xml [=============> ]68% 00671.xml [=============> ]69% 00567.xml [=============> ]69% 00269.xml [=============> ]69% 00019.xml [=============> ]69% 00845.xml [=============> ]69% 00074.xml [=============> ]69% 00049.xml [=============> ]69% 00015.xml [=============> ]70% 00857.xml [=============> ]70% 00286.xml [=============> ]70% 00022.xml [=============> ]70% 00078.xml [==============> ]70% 00234.xml [==============> ]70% 00834.xml [==============> ]70% 00839.xml [==============> ]70% 00233.xml [==============> ]71% 00111.xml [==============> ]71% 00523.xml [==============> ]71% 00095.xml [==============> ]71% 00784.xml [==============> ]71% 00670.xml [==============> ]71% 00702.xml [==============> ]71% 00152.xml [==============> ]72% 00358.xml [==============> ]72% 00897.xml [==============> ]72% 00162.xml [==============> ]72% 00221.xml [==============> ]72% 00491.xml [==============> ]72% 00230.xml [==============> ]72% 00817.xml [==============> ]72% 00754.xml [==============> ]73% 00205.xml [==============> ]73% 00029.xml [==============> ]73% 00034.xml [==============> ]73% 00187.xml [==============> ]73% 00492.xml [==============> ]73% 00083.xml [==============> ]73% 00282.xml [==============> ]74% 00291.xml [==============> ]74% 00318.xml [==============> ]74% 00729.xml [==============> ]74% 00297.xml [==============> ]74% 00771.xml [==============> ]74% 00822.xml [==============> ]74% 00474.xml [==============> ]74% 00619.xml [==============> ]75% 00081.xml [==============> ]75% 00631.xml [==============> ]75% 00789.xml [===============> ]75% 00476.xml [===============> ]75% 00340.xml [===============> ]75% 00858.xml [===============> ]75% 00458.xml [===============> ]76% 00334.xml [===============> ]76% 00112.xml [===============> ]76% 00517.xml [===============> ]76% 00151.xml [===============> ]76% 00764.xml [===============> ]76% 00153.xml [===============> ]76% 00035.xml [===============> ]77% 00261.xml [===============> ]77% 00720.xml [===============> ]77% 00607.xml [===============> ]77% 00252.xml [===============> ]77% 00305.xml [===============> ]77% 00653.xml [===============> ]77% 00695.xml [===============> ]77% 00770.xml [===============> ]78% 00471.xml [===============> ]78% 00460.xml [===============> ]78% 00003.xml [===============> ]78% 00062.xml [===============> ]78% 00017.xml [===============> ]78% 00571.xml [===============> ]78% 00229.xml [===============> ]79% 00651.xml [===============> ]79% 00519.xml [===============> ]79% 00036.xml [===============> ]79% 00451.xml [===============> ]79% 00232.xml [===============> ]79% 00421.xml [===============> ]79% 00669.xml [===============> ]79% 00346.xml [===============> ]80% 00509.xml [===============> ]80% 00206.xml [===============> ]80% 00317.xml [================> ]80% 00338.xml [================> ]80% 00601.xml [================> ]80% 00202.xml [================> ]80% 00175.xml [================> ]81% 00659.xml [================> ]81% 00076.xml [================> ]81% 00542.xml [================> ]81% 00303.xml [================> ]81% 00640.xml [================> ]81% 00004.xml [================> ]81% 00278.xml [================> ]82% 00521.xml [================> ]82% 00896.xml [================> ]82% 00811.xml [================> ]82% 00472.xml [================> ]82% 00599.xml [================> ]82% 00179.xml [================> ]82% 00719.xml [================> ]82% 00459.xml [================> ]83% 00362.xml [================> ]83% 00782.xml [================> ]83% 00007.xml [================> ]83% 00806.xml [================> ]83% 00865.xml [================> ]83% 00037.xml [================> ]83% 00089.xml [================> ]84% 00432.xml [================> ]84% 00209.xml [================> ]84% 00398.xml [================> ]84% 00606.xml [================> ]84% 00465.xml [================> ]84% 00415.xml [================> ]84% 00524.xml [================> ]84% 00888.xml [================> ]85% 00552.xml [================> ]85% 00656.xml [================> ]85% 00350.xml [=================> ]85% 00775.xml [=================> ]85% 00699.xml [=================> ]85% 00615.xml [=================> ]85% 00620.xml [=================> ]86% 00692.xml [=================> ]86% 00288.xml [=================> ]86% 00268.xml [=================> ]86% 00197.xml [=================> ]86% 00742.xml [=================> ]86% 00163.xml [=================> ]86% 00504.xml [=================> ]87% 00633.xml [=================> ]87% 00457.xml [=================> ]87% 00284.xml [=================> ]87% 00322.xml [=================> ]87% 00126.xml [=================> ]87% 00069.xml [=================> ]87% 00369.xml [=================> ]87% 00663.xml [=================> ]88% 00612.xml [=================> ]88% 00866.xml [=================> ]88% 00414.xml [=================> ]88% 00382.xml [=================> ]88% 00384.xml [=================> ]88% 00243.xml [=================> ]88% 00657.xml [=================> ]89% 00816.xml [=================> ]89% 00020.xml [=================> ]89% 00846.xml [=================> ]89% 00071.xml [=================> ]89% 00236.xml [=================> ]89% 00895.xml [=================> ]89% 00099.xml [=================> ]89% 00453.xml [=================> ]90% 00008.xml [=================> ]90% 00445.xml [=================> ]90% 00372.xml [==================> ]90% 00507.xml [==================> ]90% 00313.xml [==================> ]90% 00505.xml [==================> ]90% 00730.xml [==================> ]91% 00122.xml [==================> ]91% 00039.xml [==================> ]91% 00705.xml [==================> ]91% 00056.xml [==================> ]91% 00211.xml [==================> ]91% 00287.xml [==================> ]91% 00566.xml [==================> ]91% 00294.xml [==================> ]92% 00125.xml [==================> ]92% 00503.xml [==================> ]92% 00085.xml [==================> ]92% 00655.xml [==================> ]92% 00886.xml [==================> ]92% 00665.xml [==================> ]92% 00721.xml [==================> ]93% 00240.xml [==================> ]93% 00201.xml [==================> ]93% 00490.xml [==================> ]93% 00755.xml [==================> ]93% 00171.xml [==================> ]93% 00701.xml [==================> ]93% 00316.xml [==================> ]94% 00356.xml [==================> ]94% 00862.xml [==================> ]94% 00874.xml [==================> ]94% 00404.xml [==================> ]94% 00084.xml [==================> ]94% 00871.xml [==================> ]94% 00368.xml [==================> ]94% 00405.xml [==================> ]95% 00273.xml [==================> ]95% 00575.xml [==================> ]95% 00401.xml [===================>]95% 00385.xml [===================>]95% 00763.xml [===================>]95% 00795.xml [===================>]95% 00309.xml [===================>]96% 00418.xml [===================>]96% 00050.xml [===================>]96% 00130.xml [===================>]96% 00276.xml [===================>]96% 00741.xml [===================>]96% 00803.xml [===================>]96% 00447.xml [===================>]96% 00082.xml [===================>]97% 00621.xml [===================>]97% 00251.xml [===================>]97% 00348.xml [===================>]97% 00109.xml [===================>]97% 00430.xml [===================>]97% 00190.xml [===================>]97% 00352.xml [===================>]98% 00366.xml [===================>]98% 00167.xml [===================>]98% 00894.xml [===================>]98% 00027.xml [===================>]98% 00605.xml [===================>]98% 00052.xml [===================>]98% 00835.xml [===================>]99% 00006.xml [===================>]99% 00195.xml [===================>]99% 00355.xml [===================>]99% 00198.xml [===================>]99% 00427.xml [===================>]99% 00693.xml [===================>]99% 00798.xml [===================>]99% 00156.xml [===================>]100% 00625.xml [===================>]100% 00638.xml [===================>]100% 00258.xml [====================>]100% 00478.xml Statistics: 0: 1213 Dataset size: 741 Dataset of 741 instance(s) Training statistics: Learning rate : 1e-05 Batch size : 4 Epoch number : 10 Backup every : 50 step 1 - loss 107.2027816772461 - moving ave loss 107.20278167724611 step 2 - loss 106.665771484375 - moving ave loss 107.14908065795899 step 3 - loss 107.01924133300781 - moving ave loss 107.13609672546387 step 4 - loss 106.98098754882812 - moving ave loss 107.1205858078003 step 5 - loss 106.67323303222656 - moving ave loss 107.07585053024293 step 6 - loss 106.92096710205078 - moving ave loss 107.06036218742372 step 7 - loss 107.03022766113281 - moving ave loss 107.05734873479463 step 8 - loss 106.94993591308594 - moving ave loss 107.04660745262376 step 9 - loss 107.08029174804688 - moving ave loss 107.04997588216608 step 10 - loss 106.68739318847656 - moving ave loss 107.01371761279714 step 11 - loss 106.68819427490234 - moving ave loss 106.98116527900766 step 12 - loss 106.63270568847656 - moving ave loss 106.94631931995454 Checkpoint at step 12 step 13 - loss 107.26649475097656 - moving ave loss 106.97833686305675 step 14 - loss 106.47819519042969 - moving ave loss 106.92832269579405 step 15 - loss 106.63316345214844 - moving ave loss 106.89880677142949 step 16 - loss 106.82279205322266 - moving ave loss 106.8912052996088 step 17 - loss 106.72016906738281 - moving ave loss 106.8741016763862 step 18 - loss 107.05854797363281 - moving ave loss 106.89254630611087 step 19 - loss 106.76737976074219 - moving ave loss 106.880029651574 step 20 - loss 106.5975341796875 - moving ave loss 106.85178010438537 step 21 - loss 106.77951049804688 - moving ave loss 106.84455314375153 step 22 - loss 106.5695571899414 - moving ave loss 106.81705354837051 step 23 - loss 106.61922454833984 - moving ave loss 106.79727064836744 step 24 - loss 106.55591583251953 - moving ave loss 106.77313516678265 Checkpoint at step 24 step 25 - loss 106.78688049316406 - moving ave loss 106.77450969942079 step 26 - loss 106.7171630859375 - moving ave loss 106.76877503807248 step 27 - loss 106.887451171875 - moving ave loss 106.78064265145272 step 28 - loss 106.83073425292969 - moving ave loss 106.78565181160043 step 29 - loss 106.6050033569336 - moving ave loss 106.76758696613375 step 30 - loss 106.84686279296875 - moving ave loss 106.77551454881726 step 31 - loss 106.67445373535156 - moving ave loss 106.76540846747069 step 32 - loss 106.82987976074219 - moving ave loss 106.77185559679783 step 33 - loss 106.65704345703125 - moving ave loss 106.76037438282117 step 34 - loss 106.75775909423828 - moving ave loss 106.7601128539629 step 35 - loss 106.42225646972656 - moving ave loss 106.72632721553927 step 36 - loss 106.4888916015625 - moving ave loss 106.70258365414159 Checkpoint at step 36 step 37 - loss 106.52091979980469 - moving ave loss 106.6844172687079 step 38 - loss 107.07786560058594 - moving ave loss 106.72376210189572 step 39 - loss 106.37808227539062 - moving ave loss 106.68919411924522 step 40 - loss 106.77946472167969 - moving ave loss 106.69822117948867 step 41 - loss 106.8153305053711 - moving ave loss 106.70993211207691 step 42 - loss 106.29920196533203 - moving ave loss 106.66885909740243 step 43 - loss 106.2989501953125 - moving ave loss 106.63186820719343 step 44 - loss 106.44171142578125 - moving ave loss 106.61285252905222 step 45 - loss 107.34013366699219 - moving ave loss 106.68558064284622 step 46 - loss 106.43294525146484 - moving ave loss 106.66031710370808 step 47 - loss 106.88168334960938 - moving ave loss 106.68245372829821 step 48 - loss 106.59717559814453 - moving ave loss 106.67392591528285 Checkpoint at step 48 step 49 - loss 106.57827758789062 - moving ave loss 106.66436108254364 step 50 - loss 106.60226440429688 - moving ave loss 106.65815141471896 step 51 - loss 106.75173950195312 - moving ave loss 106.66751022344238 step 52 - loss 106.87220001220703 - moving ave loss 106.68797920231886 step 53 - loss 106.36550903320312 - moving ave loss 106.65573218540729 step 54 - loss 106.41397857666016 - moving ave loss 106.63155682453258 step 55 - loss 106.69821166992188 - moving ave loss 106.63822230907152 step 56 - loss 107.24430084228516 - moving ave loss 106.6988301623929 step 57 - loss 106.49462127685547 - moving ave loss 106.67840927383916 step 58 - loss 106.51053619384766 - moving ave loss 106.66162196584001 step 59 - loss 106.302001953125 - moving ave loss 106.62565996456851 step 60 - loss 106.90705871582031 - moving ave loss 106.6537998396937 Checkpoint at step 60 step 61 - loss 106.73187255859375 - moving ave loss 106.6616071115837 step 62 - loss 106.87931823730469 - moving ave loss 106.6833782241558 step 63 - loss 106.4859848022461 - moving ave loss 106.66363888196483 step 64 - loss 106.42460632324219 - moving ave loss 106.63973562609256 step 65 - loss 106.24432373046875 - moving ave loss 106.60019443653017 step 66 - loss 106.60664367675781 - moving ave loss 106.60083936055294 step 67 - loss 106.42277526855469 - moving ave loss 106.58303295135312 step 68 - loss 106.66944885253906 - moving ave loss 106.59167454147172 step 69 - loss 106.04557800292969 - moving ave loss 106.53706488761752 step 70 - loss 106.65421295166016 - moving ave loss 106.54877969402177 step 71 - loss 106.21815490722656 - moving ave loss 106.51571721534225 step 72 - loss 107.25535583496094 - moving ave loss 106.58968107730412 Checkpoint at step 72 step 73 - loss 106.37124633789062 - moving ave loss 106.56783760336276 step 74 - loss 106.37212371826172 - moving ave loss 106.54826621485267 step 75 - loss 106.17753601074219 - moving ave loss 106.51119319444163 step 76 - loss 106.38319396972656 - moving ave loss 106.49839327197013 step 77 - loss 106.2623291015625 - moving ave loss 106.47478685492936 step 78 - loss 106.44959259033203 - moving ave loss 106.47226742846964 step 79 - loss 106.2547607421875 - moving ave loss 106.45051675984142 step 80 - loss 106.49771118164062 - moving ave loss 106.45523620202135 step 81 - loss 106.39009094238281 - moving ave loss 106.44872167605752 step 82 - loss 106.51192474365234 - moving ave loss 106.455041982817 step 83 - loss 106.11557006835938 - moving ave loss 106.42109479137123 step 84 - loss 106.21826171875 - moving ave loss 106.40081148410911 Checkpoint at step 84 step 85 - loss 106.21946716308594 - moving ave loss 106.3826770520068 step 86 - loss 106.3611068725586 - moving ave loss 106.38052003406199 step 87 - loss 106.548095703125 - moving ave loss 106.3972776009683 step 88 - loss 106.18984985351562 - moving ave loss 106.37653482622302 step 89 - loss 106.4819107055664 - moving ave loss 106.38707241415736 step 90 - loss 106.04778289794922 - moving ave loss 106.35314346253655 step 91 - loss 105.96730041503906 - moving ave loss 106.3145591577868 step 92 - loss 105.889404296875 - moving ave loss 106.27204367169563 step 93 - loss 106.30978393554688 - moving ave loss 106.27581769808076 step 94 - loss 105.84703063964844 - moving ave loss 106.23293899223754 step 95 - loss 106.11000061035156 - moving ave loss 106.22064515404894 step 96 - loss 105.93508911132812 - moving ave loss 106.19208954977687 Checkpoint at step 96 step 97 - loss 105.95610809326172 - moving ave loss 106.16849140412536 step 98 - loss 106.21467590332031 - moving ave loss 106.17310985404485 step 99 - loss 106.4625015258789 - moving ave loss 106.20204902122826 step 100 - loss 105.7701644897461 - moving ave loss 106.15886056808006 step 101 - loss 106.15484619140625 - moving ave loss 106.15845913041268 step 102 - loss 105.89578247070312 - moving ave loss 106.13219146444172 step 103 - loss 106.38439178466797 - moving ave loss 106.15741149646435 step 104 - loss 105.84929656982422 - moving ave loss 106.12660000380033 step 105 - loss 105.87896728515625 - moving ave loss 106.10183673193593 step 106 - loss 105.83324432373047 - moving ave loss 106.07497749111539 step 107 - loss 105.81340026855469 - moving ave loss 106.04881976885933 step 108 - loss 105.93949127197266 - moving ave loss 106.03788691917066 Checkpoint at step 108 step 109 - loss 106.15354919433594 - moving ave loss 106.0494531466872 step 110 - loss 105.97056579589844 - moving ave loss 106.04156441160833 step 111 - loss 106.21704864501953 - moving ave loss 106.05911283494946 step 112 - loss 105.7951889038086 - moving ave loss 106.03272044183538 step 113 - loss 106.2662582397461 - moving ave loss 106.05607422162645 step 114 - loss 105.7263412475586 - moving ave loss 106.02310092421966 step 115 - loss 105.92390441894531 - moving ave loss 106.01318127369224 step 116 - loss 106.1472396850586 - moving ave loss 106.02658711482887 step 117 - loss 106.3017807006836 - moving ave loss 106.05410647341435 step 118 - loss 105.88153076171875 - moving ave loss 106.0368489022448 step 119 - loss 105.90078735351562 - moving ave loss 106.0232427473719 step 120 - loss 105.85284423828125 - moving ave loss 106.00620289646284 Checkpoint at step 120 step 121 - loss 105.86114501953125 - moving ave loss 105.99169710876967 step 122 - loss 105.69696807861328 - moving ave loss 105.96222420575404 step 123 - loss 105.54986572265625 - moving ave loss 105.92098835744426 step 124 - loss 105.90497589111328 - moving ave loss 105.91938711081116 step 125 - loss 106.010009765625 - moving ave loss 105.92844937629255 step 126 - loss 106.08826446533203 - moving ave loss 105.9444308851965 step 127 - loss 105.91373443603516 - moving ave loss 105.94136124028036 step 128 - loss 105.90892028808594 - moving ave loss 105.93811714506091 step 129 - loss 105.609619140625 - moving ave loss 105.90526734461733 step 130 - loss 105.40428924560547 - moving ave loss 105.85516953471614 step 131 - loss 105.96538543701172 - moving ave loss 105.8661911249457 step 132 - loss 105.82537078857422 - moving ave loss 105.86210909130855 Checkpoint at step 132 step 133 - loss 105.82861328125 - moving ave loss 105.8587595103027 step 134 - loss 105.54536437988281 - moving ave loss 105.8274199972607 step 135 - loss 105.85879516601562 - moving ave loss 105.8305575141362 step 136 - loss 105.67129516601562 - moving ave loss 105.81463127932415 step 137 - loss 105.81139373779297 - moving ave loss 105.81430752517103 step 138 - loss 105.82711791992188 - moving ave loss 105.8155885646461 step 139 - loss 105.65750122070312 - moving ave loss 105.79977983025181 step 140 - loss 105.74409484863281 - moving ave loss 105.79421133208992 step 141 - loss 105.65298461914062 - moving ave loss 105.78008866079499 step 142 - loss 105.69862365722656 - moving ave loss 105.77194216043814 step 143 - loss 105.62739562988281 - moving ave loss 105.75748750738262 step 144 - loss 105.28263854980469 - moving ave loss 105.71000261162483 Checkpoint at step 144 step 145 - loss 106.28074645996094 - moving ave loss 105.76707699645844 step 146 - loss 105.74500274658203 - moving ave loss 105.76486957147081 step 147 - loss 105.2305908203125 - moving ave loss 105.71144169635498 step 148 - loss 105.38233947753906 - moving ave loss 105.67853147447339 step 149 - loss 105.51787567138672 - moving ave loss 105.66246589416473 step 150 - loss 106.06008911132812 - moving ave loss 105.70222821588108 step 151 - loss 105.1966552734375 - moving ave loss 105.65167092163672 step 152 - loss 105.34188079833984 - moving ave loss 105.62069190930703 step 153 - loss 105.58403778076172 - moving ave loss 105.6170264964525 step 154 - loss 106.27583312988281 - moving ave loss 105.68290715979553 step 155 - loss 105.26405334472656 - moving ave loss 105.64102177828863 step 156 - loss 105.09485626220703 - moving ave loss 105.58640522668048 Checkpoint at step 156 step 157 - loss 105.18196105957031 - moving ave loss 105.54596080996947 step 158 - loss 105.55261993408203 - moving ave loss 105.54662672238074 step 159 - loss 105.55184936523438 - moving ave loss 105.54714898666612 step 160 - loss 105.24760437011719 - moving ave loss 105.51719452501123 step 161 - loss 105.47241973876953 - moving ave loss 105.51271704638707 step 162 - loss 105.63601684570312 - moving ave loss 105.52504702631867 step 163 - loss 105.3216552734375 - moving ave loss 105.50470785103056 step 164 - loss 105.3648681640625 - moving ave loss 105.49072388233375 step 165 - loss 105.00672149658203 - moving ave loss 105.44232364375858 step 166 - loss 105.49979400634766 - moving ave loss 105.4480706800175 step 167 - loss 105.72257232666016 - moving ave loss 105.47552084468177 step 168 - loss 105.51000213623047 - moving ave loss 105.47896897383663 Checkpoint at step 168 step 169 - loss 105.27354431152344 - moving ave loss 105.45842650760531 step 170 - loss 105.40800476074219 - moving ave loss 105.453384332919 step 171 - loss 105.22383117675781 - moving ave loss 105.43042901730288 step 172 - loss 105.69647979736328 - moving ave loss 105.45703409530893 step 173 - loss 105.43733978271484 - moving ave loss 105.45506466404953 step 174 - loss 105.36625671386719 - moving ave loss 105.4461838690313 step 175 - loss 105.09407043457031 - moving ave loss 105.41097252558521 step 176 - loss 105.13438415527344 - moving ave loss 105.38331368855404 step 177 - loss 105.15988159179688 - moving ave loss 105.36097047887833 step 178 - loss 105.33067321777344 - moving ave loss 105.35794075276783 step 179 - loss 105.0648193359375 - moving ave loss 105.3286286110848 step 180 - loss 105.00301361083984 - moving ave loss 105.2960671110603 Checkpoint at step 180 step 181 - loss 105.1250228881836 - moving ave loss 105.27896268877262 step 182 - loss 105.12751770019531 - moving ave loss 105.2638181899149 step 183 - loss 105.28411865234375 - moving ave loss 105.2658482361578 step 184 - loss 105.11140441894531 - moving ave loss 105.25040385443654 step 185 - loss 105.25332641601562 - moving ave loss 105.25069611059446 Finish 1 epoch(es) step 186 - loss 105.28966522216797 - moving ave loss 105.25459302175182 step 187 - loss 104.93852233886719 - moving ave loss 105.22298595346336 step 188 - loss 105.1244125366211 - moving ave loss 105.21312861177915 step 189 - loss 105.20828247070312 - moving ave loss 105.21264399767155 step 190 - loss 105.0288314819336 - moving ave loss 105.19426274609776 step 191 - loss 105.12317657470703 - moving ave loss 105.1871541289587 step 192 - loss 104.93281555175781 - moving ave loss 105.16172027123861 Checkpoint at step 192 step 193 - loss 104.95182037353516 - moving ave loss 105.14073028146827 step 194 - loss 104.96200561523438 - moving ave loss 105.12285781484488 step 195 - loss 104.81951141357422 - moving ave loss 105.09252317471783 step 196 - loss 105.02140808105469 - moving ave loss 105.08541166535151 step 197 - loss 105.30072021484375 - moving ave loss 105.10694252030073 step 198 - loss 104.77879333496094 - moving ave loss 105.07412760176675 step 199 - loss 104.92472839355469 - moving ave loss 105.05918768094554 step 200 - loss 104.77729797363281 - moving ave loss 105.03099871021428 step 201 - loss 104.755126953125 - moving ave loss 105.00341153450536 step 202 - loss 105.08045959472656 - moving ave loss 105.01111634052748 step 203 - loss 105.36325073242188 - moving ave loss 105.04632977971693 step 204 - loss 104.76773071289062 - moving ave loss 105.01846987303429 Checkpoint at step 204 step 205 - loss 104.56466674804688 - moving ave loss 104.97308956053556 step 206 - loss 104.53721618652344 - moving ave loss 104.92950222313434 step 207 - loss 104.5159683227539 - moving ave loss 104.8881488330963 step 208 - loss 104.97764587402344 - moving ave loss 104.89709853718901 step 209 - loss 105.31646728515625 - moving ave loss 104.93903541198574 step 210 - loss 104.80670166015625 - moving ave loss 104.92580203680279 step 211 - loss 104.68711853027344 - moving ave loss 104.90193368614986 step 212 - loss 104.78535461425781 - moving ave loss 104.89027577896066 step 213 - loss 104.65190887451172 - moving ave loss 104.86643908851576 step 214 - loss 104.86824035644531 - moving ave loss 104.86661921530872 step 215 - loss 104.8510971069336 - moving ave loss 104.86506700447121 step 216 - loss 104.73817443847656 - moving ave loss 104.85237774787174 Checkpoint at step 216 step 217 - loss 104.91670989990234 - moving ave loss 104.85881096307482 step 218 - loss 104.74560546875 - moving ave loss 104.84749041364233 step 219 - loss 105.24284362792969 - moving ave loss 104.88702573507108 step 220 - loss 104.74700927734375 - moving ave loss 104.87302408929834 step 221 - loss 104.7821044921875 - moving ave loss 104.86393212958725 step 222 - loss 104.65582275390625 - moving ave loss 104.84312119201917 step 223 - loss 104.90790557861328 - moving ave loss 104.84959963067858 step 224 - loss 104.30520629882812 - moving ave loss 104.79516029749354 step 225 - loss 104.70978546142578 - moving ave loss 104.78662281388677 step 226 - loss 104.77098846435547 - moving ave loss 104.78505937893364 step 227 - loss 104.87971496582031 - moving ave loss 104.79452493762231 step 228 - loss 104.72293853759766 - moving ave loss 104.78736629761985 Checkpoint at step 228 step 229 - loss 104.47381591796875 - moving ave loss 104.75601125965474 step 230 - loss 104.56600952148438 - moving ave loss 104.73701108583771 step 231 - loss 104.41580200195312 - moving ave loss 104.70489017744926 step 232 - loss 104.45884704589844 - moving ave loss 104.68028586429418 step 233 - loss 104.4417724609375 - moving ave loss 104.65643452395852 step 234 - loss 104.70344543457031 - moving ave loss 104.6611356150197 step 235 - loss 104.47187805175781 - moving ave loss 104.64220985869352 step 236 - loss 104.6623306274414 - moving ave loss 104.64422193556831 step 237 - loss 104.88406372070312 - moving ave loss 104.6682061140818 step 238 - loss 104.63192749023438 - moving ave loss 104.66457825169705 step 239 - loss 104.96418762207031 - moving ave loss 104.69453918873438 step 240 - loss 104.99859619140625 - moving ave loss 104.72494488900156 Checkpoint at step 240 step 241 - loss 104.5069580078125 - moving ave loss 104.70314620088266 step 242 - loss 104.93717956542969 - moving ave loss 104.72654953733736 step 243 - loss 104.50074768066406 - moving ave loss 104.70396935167003 step 244 - loss 104.61541748046875 - moving ave loss 104.69511416454989 step 245 - loss 104.24674987792969 - moving ave loss 104.65027773588787 step 246 - loss 104.36079406738281 - moving ave loss 104.62132936903737 step 247 - loss 104.20982360839844 - moving ave loss 104.58017879297348 step 248 - loss 104.54753112792969 - moving ave loss 104.57691402646911 step 249 - loss 104.41433715820312 - moving ave loss 104.5606563396425 step 250 - loss 104.49662780761719 - moving ave loss 104.55425348643999 step 251 - loss 104.51158142089844 - moving ave loss 104.54998627988584 step 252 - loss 104.51834106445312 - moving ave loss 104.54682175834257 Checkpoint at step 252 step 253 - loss 104.10497283935547 - moving ave loss 104.50263686644388 step 254 - loss 104.77682495117188 - moving ave loss 104.53005567491668 step 255 - loss 104.3299789428711 - moving ave loss 104.51004800171212 step 256 - loss 104.10298156738281 - moving ave loss 104.4693413582792 step 257 - loss 104.41996002197266 - moving ave loss 104.46440322464855 step 258 - loss 104.08502960205078 - moving ave loss 104.42646586238878 step 259 - loss 104.11544799804688 - moving ave loss 104.39536407595459 step 260 - loss 104.833740234375 - moving ave loss 104.43920169179664 step 261 - loss 104.48413848876953 - moving ave loss 104.44369537149393 step 262 - loss 104.36682891845703 - moving ave loss 104.43600872619025 step 263 - loss 103.91785430908203 - moving ave loss 104.38419328447942 step 264 - loss 104.20504760742188 - moving ave loss 104.36627871677368 Checkpoint at step 264 step 265 - loss 104.49337768554688 - moving ave loss 104.378988613651 step 266 - loss 104.03030395507812 - moving ave loss 104.34412014779372 step 267 - loss 103.92532348632812 - moving ave loss 104.30224048164716 step 268 - loss 104.2557373046875 - moving ave loss 104.2975901639512 step 269 - loss 104.19316864013672 - moving ave loss 104.28714801156974 step 270 - loss 104.47480773925781 - moving ave loss 104.30591398433856 step 271 - loss 103.9455337524414 - moving ave loss 104.26987596114886 step 272 - loss 103.81890106201172 - moving ave loss 104.22477847123514 step 273 - loss 104.2153091430664 - moving ave loss 104.22383153841827 step 274 - loss 103.95429229736328 - moving ave loss 104.19687761431277 step 275 - loss 104.31777954101562 - moving ave loss 104.20896780698305 step 276 - loss 104.05583190917969 - moving ave loss 104.19365421720272 Checkpoint at step 276 step 277 - loss 103.90361022949219 - moving ave loss 104.16464981843167 step 278 - loss 103.8858642578125 - moving ave loss 104.13677126236976 step 279 - loss 104.51109313964844 - moving ave loss 104.17420345009764 step 280 - loss 103.8587646484375 - moving ave loss 104.14265956993162 step 281 - loss 104.41291046142578 - moving ave loss 104.16968465908104 step 282 - loss 104.34651184082031 - moving ave loss 104.18736737725497 step 283 - loss 103.91970825195312 - moving ave loss 104.16060146472479 step 284 - loss 103.98069763183594 - moving ave loss 104.14261108143592 step 285 - loss 104.28459167480469 - moving ave loss 104.1568091407728 step 286 - loss 103.76969909667969 - moving ave loss 104.1180981363635 step 287 - loss 103.72776794433594 - moving ave loss 104.07906511716074 step 288 - loss 104.53068542480469 - moving ave loss 104.12422714792513 Checkpoint at step 288 step 289 - loss 103.87319946289062 - moving ave loss 104.09912437942168 step 290 - loss 104.0941162109375 - moving ave loss 104.09862356257327 step 291 - loss 103.83905029296875 - moving ave loss 104.07266623561281 step 292 - loss 103.97318267822266 - moving ave loss 104.0627178798738 step 293 - loss 104.0120620727539 - moving ave loss 104.05765229916182 step 294 - loss 103.57733154296875 - moving ave loss 104.00962022354251 step 295 - loss 103.89476013183594 - moving ave loss 103.99813421437186 step 296 - loss 104.01719665527344 - moving ave loss 104.00004045846202 step 297 - loss 103.74533081054688 - moving ave loss 103.97456949367051 step 298 - loss 104.18145751953125 - moving ave loss 103.9952582962566 step 299 - loss 103.67619323730469 - moving ave loss 103.9633517903614 step 300 - loss 103.95738983154297 - moving ave loss 103.96275559447956 Checkpoint at step 300 step 301 - loss 103.83807373046875 - moving ave loss 103.95028740807848 step 302 - loss 103.6563720703125 - moving ave loss 103.92089587430189 step 303 - loss 104.27848815917969 - moving ave loss 103.95665510278967 step 304 - loss 103.89167785644531 - moving ave loss 103.95015737815524 step 305 - loss 103.61709594726562 - moving ave loss 103.91685123506628 step 306 - loss 104.2301025390625 - moving ave loss 103.94817636546591 step 307 - loss 103.62674713134766 - moving ave loss 103.91603344205409 step 308 - loss 103.72113037109375 - moving ave loss 103.89654313495805 step 309 - loss 103.68976593017578 - moving ave loss 103.87586541447983 step 310 - loss 103.5346908569336 - moving ave loss 103.84174795872521 step 311 - loss 103.71687316894531 - moving ave loss 103.82926047974722 step 312 - loss 103.69410705566406 - moving ave loss 103.81574513733891 Checkpoint at step 312 step 313 - loss 103.69331359863281 - moving ave loss 103.8035019834683 step 314 - loss 103.868896484375 - moving ave loss 103.81004143355896 step 315 - loss 103.48406982421875 - moving ave loss 103.77744427262495 step 316 - loss 103.49241638183594 - moving ave loss 103.74894148354605 step 317 - loss 103.62982177734375 - moving ave loss 103.73702951292583 step 318 - loss 103.58283996582031 - moving ave loss 103.72161055821527 step 319 - loss 103.55399322509766 - moving ave loss 103.70484882490352 step 320 - loss 103.44601440429688 - moving ave loss 103.67896538284286 step 321 - loss 103.44976806640625 - moving ave loss 103.65604565119921 step 322 - loss 103.56463623046875 - moving ave loss 103.64690470912616 step 323 - loss 103.57154083251953 - moving ave loss 103.6393683214655 step 324 - loss 103.44559478759766 - moving ave loss 103.61999096807871 Checkpoint at step 324 step 325 - loss 103.20926666259766 - moving ave loss 103.57891853753061 step 326 - loss 104.18978881835938 - moving ave loss 103.64000556561349 step 327 - loss 103.39461517333984 - moving ave loss 103.61546652638613 step 328 - loss 103.85212707519531 - moving ave loss 103.63913258126705 step 329 - loss 103.30818939208984 - moving ave loss 103.60603826234933 step 330 - loss 103.75489807128906 - moving ave loss 103.62092424324331 step 331 - loss 103.76776123046875 - moving ave loss 103.63560794196586 step 332 - loss 103.31646728515625 - moving ave loss 103.6036938762849 step 333 - loss 103.64054870605469 - moving ave loss 103.60737935926188 step 334 - loss 103.58140563964844 - moving ave loss 103.60478198730056 step 335 - loss 103.30278015136719 - moving ave loss 103.57458180370722 step 336 - loss 103.37613677978516 - moving ave loss 103.55473730131501 Checkpoint at step 336 step 337 - loss 103.42684173583984 - moving ave loss 103.54194774476751 step 338 - loss 103.21878051757812 - moving ave loss 103.50963102204858 step 339 - loss 103.18643951416016 - moving ave loss 103.47731187125974 step 340 - loss 103.10888671875 - moving ave loss 103.44046935600878 step 341 - loss 103.18132019042969 - moving ave loss 103.41455443945088 step 342 - loss 103.3350601196289 - moving ave loss 103.4066050074687 step 343 - loss 103.4364242553711 - moving ave loss 103.40958693225895 step 344 - loss 103.57050323486328 - moving ave loss 103.4256785625194 step 345 - loss 103.25736999511719 - moving ave loss 103.40884770577918 step 346 - loss 104.0627670288086 - moving ave loss 103.47423963808212 step 347 - loss 103.44207763671875 - moving ave loss 103.47102343794579 step 348 - loss 103.12491607666016 - moving ave loss 103.43641270181723 Checkpoint at step 348 step 349 - loss 103.20448303222656 - moving ave loss 103.41321973485817 step 350 - loss 103.56339263916016 - moving ave loss 103.42823702528837 step 351 - loss 103.03517150878906 - moving ave loss 103.38893047363844 step 352 - loss 103.55999755859375 - moving ave loss 103.40603718213397 step 353 - loss 104.07288360595703 - moving ave loss 103.47272182451627 step 354 - loss 103.2120361328125 - moving ave loss 103.44665325534591 step 355 - loss 103.1159439086914 - moving ave loss 103.41358232068046 step 356 - loss 103.39776611328125 - moving ave loss 103.41200069994053 step 357 - loss 102.9975814819336 - moving ave loss 103.37055877813985 step 358 - loss 102.83781433105469 - moving ave loss 103.31728433343135 step 359 - loss 103.15869140625 - moving ave loss 103.30142504071321 step 360 - loss 103.29166412353516 - moving ave loss 103.30044894899541 Checkpoint at step 360 step 361 - loss 102.94651794433594 - moving ave loss 103.26505584852946 step 362 - loss 102.93436431884766 - moving ave loss 103.23198669556129 step 363 - loss 103.81672668457031 - moving ave loss 103.2904606944622 step 364 - loss 103.19287109375 - moving ave loss 103.28070173439097 step 365 - loss 102.9163589477539 - moving ave loss 103.24426745572727 step 366 - loss 102.84817504882812 - moving ave loss 103.20465821503736 step 367 - loss 103.3686752319336 - moving ave loss 103.22105991672699 step 368 - loss 102.84048461914062 - moving ave loss 103.18300238696835 step 369 - loss 103.1836166381836 - moving ave loss 103.18306381208988 step 370 - loss 102.82292175292969 - moving ave loss 103.14704960617387 Finish 2 epoch(es) step 371 - loss 103.17256927490234 - moving ave loss 103.14960157304671 step 372 - loss 102.98533630371094 - moving ave loss 103.13317504611314 Checkpoint at step 372 step 373 - loss 102.67414855957031 - moving ave loss 103.08727239745886 step 374 - loss 102.71714782714844 - moving ave loss 103.05025994042782 step 375 - loss 103.52777862548828 - moving ave loss 103.09801180893388 step 376 - loss 102.90436553955078 - moving ave loss 103.07864718199556 step 377 - loss 102.74359893798828 - moving ave loss 103.04514235759484 step 378 - loss 102.79058837890625 - moving ave loss 103.01968695972599 step 379 - loss 102.94744873046875 - moving ave loss 103.01246313680028 step 380 - loss 102.69012451171875 - moving ave loss 102.98022927429213 step 381 - loss 103.40890502929688 - moving ave loss 103.02309684979261 step 382 - loss 102.54660034179688 - moving ave loss 102.97544719899304 step 383 - loss 103.4028549194336 - moving ave loss 103.0181879710371 step 384 - loss 103.33696746826172 - moving ave loss 103.05006592075956 Checkpoint at step 384 step 385 - loss 103.41693115234375 - moving ave loss 103.08675244391799 step 386 - loss 102.495361328125 - moving ave loss 103.02761333233869 step 387 - loss 102.83299255371094 - moving ave loss 103.00815125447592 step 388 - loss 102.66644287109375 - moving ave loss 102.9739804161377 step 389 - loss 102.81659698486328 - moving ave loss 102.95824207301027 step 390 - loss 103.03097534179688 - moving ave loss 102.96551539988893 step 391 - loss 102.90319061279297 - moving ave loss 102.95928292117932 step 392 - loss 102.89497375488281 - moving ave loss 102.95285200454967 step 393 - loss 103.1679458618164 - moving ave loss 102.97436139027634 step 394 - loss 102.45478820800781 - moving ave loss 102.92240407204949 step 395 - loss 102.7194595336914 - moving ave loss 102.90210961821369 step 396 - loss 102.79136657714844 - moving ave loss 102.89103531410716 Checkpoint at step 396 step 397 - loss 103.0849838256836 - moving ave loss 102.9104301652648 step 398 - loss 102.9300537109375 - moving ave loss 102.91239251983208 step 399 - loss 102.75387573242188 - moving ave loss 102.89654084109107 step 400 - loss 102.51445007324219 - moving ave loss 102.85833176430619 step 401 - loss 102.81187438964844 - moving ave loss 102.85368602684042 step 402 - loss 102.89093780517578 - moving ave loss 102.85741120467395 step 403 - loss 102.35792541503906 - moving ave loss 102.80746262571047 step 404 - loss 102.28028869628906 - moving ave loss 102.75474523276833 step 405 - loss 103.10385131835938 - moving ave loss 102.78965584132744 step 406 - loss 102.792236328125 - moving ave loss 102.7899138900072 step 407 - loss 102.45938873291016 - moving ave loss 102.7568613742975 step 408 - loss 102.48301696777344 - moving ave loss 102.7294769336451 Checkpoint at step 408 step 409 - loss 103.20729064941406 - moving ave loss 102.77725830522199 step 410 - loss 102.20719146728516 - moving ave loss 102.7202516214283 step 411 - loss 102.74840545654297 - moving ave loss 102.72306700493978 step 412 - loss 102.42406463623047 - moving ave loss 102.69316676806886 step 413 - loss 102.20199584960938 - moving ave loss 102.64404967622292 step 414 - loss 102.36647033691406 - moving ave loss 102.61629174229205 step 415 - loss 102.43277740478516 - moving ave loss 102.59794030854137 step 416 - loss 102.43538665771484 - moving ave loss 102.58168494345871 step 417 - loss 102.44139862060547 - moving ave loss 102.5676563111734 step 418 - loss 102.98485565185547 - moving ave loss 102.6093762452416 step 419 - loss 102.44288635253906 - moving ave loss 102.59272725597135 step 420 - loss 102.4940414428711 - moving ave loss 102.58285867466134 Checkpoint at step 420 step 421 - loss 102.53239440917969 - moving ave loss 102.57781224811318 step 422 - loss 102.56214904785156 - moving ave loss 102.57624592808702 step 423 - loss 102.04633331298828 - moving ave loss 102.52325466657715 step 424 - loss 102.07861328125 - moving ave loss 102.47879052804444 step 425 - loss 102.45454406738281 - moving ave loss 102.4763658819783 step 426 - loss 102.23725891113281 - moving ave loss 102.45245518489375 step 427 - loss 102.47296905517578 - moving ave loss 102.45450657192197 step 428 - loss 102.50344848632812 - moving ave loss 102.4594007633626 step 429 - loss 102.0251235961914 - moving ave loss 102.41597304664549 step 430 - loss 102.24072265625 - moving ave loss 102.39844800760594 step 431 - loss 102.28772735595703 - moving ave loss 102.38737594244105 step 432 - loss 102.62509155273438 - moving ave loss 102.41114750347039 Checkpoint at step 432 step 433 - loss 102.572509765625 - moving ave loss 102.42728372968585 step 434 - loss 102.102783203125 - moving ave loss 102.39483367702977 step 435 - loss 102.54381561279297 - moving ave loss 102.4097318706061 step 436 - loss 102.40133666992188 - moving ave loss 102.40889235053768 step 437 - loss 101.8404541015625 - moving ave loss 102.35204852564016 step 438 - loss 102.00965881347656 - moving ave loss 102.3178095544238 step 439 - loss 101.90684509277344 - moving ave loss 102.27671310825878 step 440 - loss 101.88079071044922 - moving ave loss 102.23712086847783 step 441 - loss 102.337646484375 - moving ave loss 102.24717343006755 step 442 - loss 102.13389587402344 - moving ave loss 102.23584567446314 step 443 - loss 102.69757080078125 - moving ave loss 102.28201818709495 step 444 - loss 102.4117202758789 - moving ave loss 102.29498839597335 Checkpoint at step 444 step 445 - loss 101.80157470703125 - moving ave loss 102.24564702707914 step 446 - loss 102.24903869628906 - moving ave loss 102.24598619400014 step 447 - loss 102.02125549316406 - moving ave loss 102.22351312391653 step 448 - loss 102.08235168457031 - moving ave loss 102.20939697998192 step 449 - loss 101.8672103881836 - moving ave loss 102.17517832080209 step 450 - loss 101.86654663085938 - moving ave loss 102.14431515180783 step 451 - loss 101.70076751708984 - moving ave loss 102.09996038833604 step 452 - loss 101.72020721435547 - moving ave loss 102.06198507093798 step 453 - loss 101.70863342285156 - moving ave loss 102.02664990612934 step 454 - loss 102.24037170410156 - moving ave loss 102.04802208592656 step 455 - loss 101.63630676269531 - moving ave loss 102.00685055360344 step 456 - loss 102.63341522216797 - moving ave loss 102.0695070204599 Checkpoint at step 456 step 457 - loss 101.85310363769531 - moving ave loss 102.04786668218344 step 458 - loss 101.77021026611328 - moving ave loss 102.02010104057642 step 459 - loss 102.06440734863281 - moving ave loss 102.02453167138206 step 460 - loss 101.70831298828125 - moving ave loss 101.992909803072 step 461 - loss 102.00639343261719 - moving ave loss 101.99425816602653 step 462 - loss 101.89752197265625 - moving ave loss 101.98458454668949 step 463 - loss 101.5675048828125 - moving ave loss 101.94287658030179 step 464 - loss 101.53043365478516 - moving ave loss 101.90163228775012 step 465 - loss 102.03297424316406 - moving ave loss 101.91476648329152 step 466 - loss 102.18058776855469 - moving ave loss 101.94134861181784 step 467 - loss 101.73947143554688 - moving ave loss 101.92116089419075 step 468 - loss 102.40164947509766 - moving ave loss 101.96920975228144 Checkpoint at step 468 step 469 - loss 102.01058959960938 - moving ave loss 101.97334773701425 step 470 - loss 101.68313598632812 - moving ave loss 101.94432656194564 step 471 - loss 101.808349609375 - moving ave loss 101.93072886668858 step 472 - loss 101.82966613769531 - moving ave loss 101.92062259378926 step 473 - loss 101.68016815185547 - moving ave loss 101.89657714959588 step 474 - loss 101.98973083496094 - moving ave loss 101.9058925181324 step 475 - loss 101.96538543701172 - moving ave loss 101.91184181002035 step 476 - loss 101.77770233154297 - moving ave loss 101.89842786217261 step 477 - loss 101.84194946289062 - moving ave loss 101.89278002224442 step 478 - loss 102.22953796386719 - moving ave loss 101.9264558164067 step 479 - loss 101.49409484863281 - moving ave loss 101.88321971962932 step 480 - loss 101.65127563476562 - moving ave loss 101.86002531114295 Checkpoint at step 480 step 481 - loss 101.72419738769531 - moving ave loss 101.84644251879818 step 482 - loss 101.59091186523438 - moving ave loss 101.82088945344181 step 483 - loss 101.94462585449219 - moving ave loss 101.83326309354685 step 484 - loss 101.66590881347656 - moving ave loss 101.81652766553982 step 485 - loss 101.451416015625 - moving ave loss 101.78001650054834 step 486 - loss 101.40422058105469 - moving ave loss 101.74243690859898 step 487 - loss 101.52958679199219 - moving ave loss 101.7211518969383 step 488 - loss 101.81599426269531 - moving ave loss 101.730636133514 step 489 - loss 102.10572814941406 - moving ave loss 101.76814533510401 step 490 - loss 101.54566955566406 - moving ave loss 101.74589775716002 step 491 - loss 101.6512680053711 - moving ave loss 101.73643478198113 step 492 - loss 101.56990051269531 - moving ave loss 101.71978135505255 Checkpoint at step 492 step 493 - loss 101.77738189697266 - moving ave loss 101.72554140924456 step 494 - loss 102.12828826904297 - moving ave loss 101.76581609522441 step 495 - loss 101.13362121582031 - moving ave loss 101.702596607284 step 496 - loss 101.4368667602539 - moving ave loss 101.676023622581 step 497 - loss 101.45264434814453 - moving ave loss 101.65368569513736 step 498 - loss 101.2276611328125 - moving ave loss 101.61108323890488 step 499 - loss 101.42961120605469 - moving ave loss 101.59293603561986 step 500 - loss 101.46292114257812 - moving ave loss 101.5799345463157 step 501 - loss 101.1935806274414 - moving ave loss 101.54129915442827 step 502 - loss 101.39798736572266 - moving ave loss 101.52696797555771 step 503 - loss 101.18278503417969 - moving ave loss 101.49254968141992 step 504 - loss 101.2518310546875 - moving ave loss 101.46847781874668 Checkpoint at step 504 step 505 - loss 101.32782745361328 - moving ave loss 101.45441278223333 step 506 - loss 101.64089965820312 - moving ave loss 101.47306146983031 step 507 - loss 101.33332824707031 - moving ave loss 101.45908814755433 step 508 - loss 101.26261138916016 - moving ave loss 101.43944047171492 step 509 - loss 101.06312561035156 - moving ave loss 101.40180898557858 step 510 - loss 100.92987060546875 - moving ave loss 101.3546151475676 step 511 - loss 101.34371948242188 - moving ave loss 101.35352558105303 step 512 - loss 101.13542175292969 - moving ave loss 101.33171519824072 step 513 - loss 101.08655548095703 - moving ave loss 101.30719922651235 step 514 - loss 101.39865112304688 - moving ave loss 101.3163444161658 step 515 - loss 101.1508560180664 - moving ave loss 101.29979557635586 step 516 - loss 101.17623901367188 - moving ave loss 101.28743992008748 Checkpoint at step 516 step 517 - loss 101.21800231933594 - moving ave loss 101.28049616001232 step 518 - loss 100.96382141113281 - moving ave loss 101.24882868512437 step 519 - loss 100.99093627929688 - moving ave loss 101.22303944454163 step 520 - loss 101.0887451171875 - moving ave loss 101.20961001180622 step 521 - loss 101.42404174804688 - moving ave loss 101.23105318543028 step 522 - loss 101.4776611328125 - moving ave loss 101.25571398016851 step 523 - loss 101.10475158691406 - moving ave loss 101.24061774084308 step 524 - loss 101.18306732177734 - moving ave loss 101.2348626989365 step 525 - loss 101.37631225585938 - moving ave loss 101.24900765462878 step 526 - loss 100.85514831542969 - moving ave loss 101.20962172070887 step 527 - loss 100.73774719238281 - moving ave loss 101.16243426787626 step 528 - loss 100.70880126953125 - moving ave loss 101.11707096804177 Checkpoint at step 528 step 529 - loss 101.07286071777344 - moving ave loss 101.11264994301493 step 530 - loss 101.30931854248047 - moving ave loss 101.13231680296148 step 531 - loss 100.7828140258789 - moving ave loss 101.09736652525322 step 532 - loss 101.51333618164062 - moving ave loss 101.13896349089197 step 533 - loss 100.76194763183594 - moving ave loss 101.10126190498637 step 534 - loss 101.07261657714844 - moving ave loss 101.09839737220258 step 535 - loss 100.82401275634766 - moving ave loss 101.0709589106171 step 536 - loss 100.96398162841797 - moving ave loss 101.06026118239718 step 537 - loss 100.87565612792969 - moving ave loss 101.04180067695044 step 538 - loss 101.06874084472656 - moving ave loss 101.04449469372807 step 539 - loss 101.33421325683594 - moving ave loss 101.07346655003886 step 540 - loss 100.7088851928711 - moving ave loss 101.03700841432209 Checkpoint at step 540 step 541 - loss 100.84661102294922 - moving ave loss 101.01796867518482 step 542 - loss 101.13143157958984 - moving ave loss 101.02931496562533 step 543 - loss 100.49496459960938 - moving ave loss 100.97587992902373 step 544 - loss 100.77552795410156 - moving ave loss 100.95584473153151 step 545 - loss 100.76602172851562 - moving ave loss 100.93686243122993 step 546 - loss 100.95637512207031 - moving ave loss 100.93881370031397 step 547 - loss 100.6895751953125 - moving ave loss 100.91388984981383 step 548 - loss 101.86735534667969 - moving ave loss 101.00923639950042 step 549 - loss 100.54591369628906 - moving ave loss 100.9629041291793 step 550 - loss 100.92683410644531 - moving ave loss 100.95929712690591 step 551 - loss 100.56153869628906 - moving ave loss 100.91952128384423 step 552 - loss 100.49615478515625 - moving ave loss 100.87718463397545 Checkpoint at step 552 step 553 - loss 101.4144287109375 - moving ave loss 100.93090904167167 step 554 - loss 100.86811828613281 - moving ave loss 100.92462996611778 step 555 - loss 100.59489440917969 - moving ave loss 100.89165641042398 Finish 3 epoch(es) step 556 - loss 100.86231994628906 - moving ave loss 100.88872276401048 step 557 - loss 100.74357604980469 - moving ave loss 100.8742080925899 step 558 - loss 100.7225570678711 - moving ave loss 100.85904299011801 step 559 - loss 100.6252212524414 - moving ave loss 100.83566081635036 step 560 - loss 100.73794555664062 - moving ave loss 100.8258892903794 step 561 - loss 100.9449462890625 - moving ave loss 100.8377949902477 step 562 - loss 100.62996673583984 - moving ave loss 100.81701216480691 step 563 - loss 100.58646392822266 - moving ave loss 100.79395734114848 step 564 - loss 100.17436218261719 - moving ave loss 100.73199782529537 Checkpoint at step 564 step 565 - loss 100.41336822509766 - moving ave loss 100.7001348652756 step 566 - loss 100.80118560791016 - moving ave loss 100.71023993953906 step 567 - loss 100.81300354003906 - moving ave loss 100.72051629958906 step 568 - loss 100.35270690917969 - moving ave loss 100.68373536054813 step 569 - loss 100.73031616210938 - moving ave loss 100.68839344070426 step 570 - loss 100.26790618896484 - moving ave loss 100.64634471553032 step 571 - loss 100.40825653076172 - moving ave loss 100.62253589705345 step 572 - loss 100.25603485107422 - moving ave loss 100.58588579245553 step 573 - loss 100.683837890625 - moving ave loss 100.59568100227247 step 574 - loss 100.23382568359375 - moving ave loss 100.55949547040461 step 575 - loss 100.44610595703125 - moving ave loss 100.54815651906728 step 576 - loss 100.96538543701172 - moving ave loss 100.58987941086173 Checkpoint at step 576 step 577 - loss 100.4448471069336 - moving ave loss 100.57537618046891 step 578 - loss 100.03187561035156 - moving ave loss 100.52102612345718 step 579 - loss 100.4361572265625 - moving ave loss 100.51253923376771 step 580 - loss 100.51570892333984 - moving ave loss 100.51285620272493 step 581 - loss 100.08382415771484 - moving ave loss 100.46995299822393 step 582 - loss 100.14384460449219 - moving ave loss 100.43734215885077 step 583 - loss 99.93075561523438 - moving ave loss 100.38668350448913 step 584 - loss 99.89497375488281 - moving ave loss 100.3375125295285 step 585 - loss 100.81980895996094 - moving ave loss 100.38574217257174 step 586 - loss 100.35016632080078 - moving ave loss 100.38218458739465 step 587 - loss 100.24128723144531 - moving ave loss 100.36809485179971 step 588 - loss 100.49564361572266 - moving ave loss 100.380849728192 Checkpoint at step 588 step 589 - loss 100.3218994140625 - moving ave loss 100.37495469677907 step 590 - loss 100.37342071533203 - moving ave loss 100.37480129863437 step 591 - loss 99.9828872680664 - moving ave loss 100.33560989557756 step 592 - loss 100.13313293457031 - moving ave loss 100.31536219947684 step 593 - loss 99.9471435546875 - moving ave loss 100.2785403349979 step 594 - loss 99.87525939941406 - moving ave loss 100.23821224143951 step 595 - loss 100.07142639160156 - moving ave loss 100.22153365645573 step 596 - loss 100.16184997558594 - moving ave loss 100.21556528836877 step 597 - loss 99.99787139892578 - moving ave loss 100.19379589942447 step 598 - loss 99.86978912353516 - moving ave loss 100.16139522183555 step 599 - loss 100.26709747314453 - moving ave loss 100.17196544696645 step 600 - loss 99.75336456298828 - moving ave loss 100.13010535856864 Checkpoint at step 600 step 601 - loss 100.11061096191406 - moving ave loss 100.12815591890319 step 602 - loss 99.63960266113281 - moving ave loss 100.07930059312616 step 603 - loss 99.73590850830078 - moving ave loss 100.04496138464363 step 604 - loss 99.94727325439453 - moving ave loss 100.03519257161872 step 605 - loss 100.11735534667969 - moving ave loss 100.04340884912483 step 606 - loss 99.88877868652344 - moving ave loss 100.0279458328647 step 607 - loss 99.91419219970703 - moving ave loss 100.01657046954894 step 608 - loss 99.67215728759766 - moving ave loss 99.98212915135382 step 609 - loss 99.70704650878906 - moving ave loss 99.95462088709735 step 610 - loss 100.21638488769531 - moving ave loss 99.98079728715715 step 611 - loss 99.81257629394531 - moving ave loss 99.96397518783597 step 612 - loss 99.77729797363281 - moving ave loss 99.94530746641566 Checkpoint at step 612 step 613 - loss 99.87248229980469 - moving ave loss 99.93802494975456 step 614 - loss 99.77140808105469 - moving ave loss 99.92136326288458 step 615 - loss 100.51649475097656 - moving ave loss 99.98087641169379 step 616 - loss 100.18644714355469 - moving ave loss 100.00143348487988 step 617 - loss 99.4327392578125 - moving ave loss 99.94456406217316 step 618 - loss 99.8970718383789 - moving ave loss 99.93981483979374 step 619 - loss 100.12821960449219 - moving ave loss 99.95865531626359 step 620 - loss 100.1914291381836 - moving ave loss 99.9819326984556 step 621 - loss 99.67138671875 - moving ave loss 99.95087810048504 step 622 - loss 99.83612060546875 - moving ave loss 99.93940235098341 step 623 - loss 99.69879913330078 - moving ave loss 99.91534202921515 step 624 - loss 99.98624420166016 - moving ave loss 99.92243224645966 Checkpoint at step 624 step 625 - loss 99.70809173583984 - moving ave loss 99.90099819539769 step 626 - loss 99.47330474853516 - moving ave loss 99.85822885071144 step 627 - loss 99.94129943847656 - moving ave loss 99.86653590948795 step 628 - loss 99.38045501708984 - moving ave loss 99.81792782024814 step 629 - loss 99.24378204345703 - moving ave loss 99.76051324256903 step 630 - loss 100.04739379882812 - moving ave loss 99.78920129819494 step 631 - loss 99.91765594482422 - moving ave loss 99.80204676285788 step 632 - loss 99.67060089111328 - moving ave loss 99.78890217568343 step 633 - loss 99.45164489746094 - moving ave loss 99.75517644786119 step 634 - loss 99.3423843383789 - moving ave loss 99.71389723691296 step 635 - loss 99.4244384765625 - moving ave loss 99.68495136087792 step 636 - loss 99.41522216796875 - moving ave loss 99.657978441587 Checkpoint at step 636 step 637 - loss 99.43540954589844 - moving ave loss 99.63572155201815 step 638 - loss 99.27088928222656 - moving ave loss 99.599238325039 step 639 - loss 99.22703552246094 - moving ave loss 99.56201804478118 step 640 - loss 99.04257202148438 - moving ave loss 99.51007344245151 step 641 - loss 99.52053833007812 - moving ave loss 99.51111993121417 step 642 - loss 99.18272399902344 - moving ave loss 99.4782803379951 step 643 - loss 99.50608825683594 - moving ave loss 99.48106112987918 step 644 - loss 99.01782989501953 - moving ave loss 99.43473800639322 step 645 - loss 99.8088150024414 - moving ave loss 99.47214570599803 step 646 - loss 99.1268310546875 - moving ave loss 99.43761424086699 step 647 - loss 99.41380310058594 - moving ave loss 99.43523312683888 step 648 - loss 99.048095703125 - moving ave loss 99.3965193844675 Checkpoint at step 648 step 649 - loss 99.39262390136719 - moving ave loss 99.39612983615747 step 650 - loss 99.4306411743164 - moving ave loss 99.39958096997337 step 651 - loss 98.90851593017578 - moving ave loss 99.35047446599361 step 652 - loss 99.47102355957031 - moving ave loss 99.36252937535129 step 653 - loss 99.49632263183594 - moving ave loss 99.37590870099976 step 654 - loss 99.38029479980469 - moving ave loss 99.37634731088026 step 655 - loss 98.84370422363281 - moving ave loss 99.32308300215553 step 656 - loss 99.10231018066406 - moving ave loss 99.30100572000639 step 657 - loss 99.20854949951172 - moving ave loss 99.29176009795692 step 658 - loss 98.88656616210938 - moving ave loss 99.25124070437217 step 659 - loss 99.65023040771484 - moving ave loss 99.29113967470643 step 660 - loss 99.25833129882812 - moving ave loss 99.2878588371186 Checkpoint at step 660 step 661 - loss 98.91203308105469 - moving ave loss 99.25027626151223 step 662 - loss 98.87315368652344 - moving ave loss 99.21256400401336 step 663 - loss 99.21206665039062 - moving ave loss 99.21251426865109 step 664 - loss 99.19779205322266 - moving ave loss 99.21104204710825 step 665 - loss 99.43789672851562 - moving ave loss 99.23372751524899 step 666 - loss 99.0240478515625 - moving ave loss 99.21275954888034 step 667 - loss 99.18883514404297 - moving ave loss 99.21036710839661 step 668 - loss 98.92215728759766 - moving ave loss 99.18154612631672 step 669 - loss 98.96086120605469 - moving ave loss 99.15947763429052 step 670 - loss 98.93672180175781 - moving ave loss 99.13720205103726 step 671 - loss 98.5760498046875 - moving ave loss 99.08108682640228 step 672 - loss 99.29522705078125 - moving ave loss 99.10250084884018 Checkpoint at step 672 step 673 - loss 98.9147720336914 - moving ave loss 99.0837279673253 step 674 - loss 98.72525024414062 - moving ave loss 99.04788019500684 step 675 - loss 98.73982238769531 - moving ave loss 99.01707441427568 step 676 - loss 99.51905822753906 - moving ave loss 99.06727279560204 step 677 - loss 98.7012939453125 - moving ave loss 99.03067491057308 step 678 - loss 99.06916809082031 - moving ave loss 99.03452422859782 step 679 - loss 98.43592834472656 - moving ave loss 98.97466464021069 step 680 - loss 98.4169692993164 - moving ave loss 98.91889510612127 step 681 - loss 98.7540512084961 - moving ave loss 98.90241071635874 step 682 - loss 99.37725067138672 - moving ave loss 98.94989471186155 step 683 - loss 98.79226684570312 - moving ave loss 98.93413192524571 step 684 - loss 98.70681762695312 - moving ave loss 98.91140049541646 Checkpoint at step 684 step 685 - loss 98.38838195800781 - moving ave loss 98.85909864167559 step 686 - loss 98.57234191894531 - moving ave loss 98.83042296940256 step 687 - loss 99.1333999633789 - moving ave loss 98.8607206688002 step 688 - loss 98.50811767578125 - moving ave loss 98.8254603694983 step 689 - loss 98.28440856933594 - moving ave loss 98.77135518948207 step 690 - loss 98.46421813964844 - moving ave loss 98.7406414844987 step 691 - loss 98.87701416015625 - moving ave loss 98.75427875206447 step 692 - loss 98.51841735839844 - moving ave loss 98.73069261269787 step 693 - loss 98.2801284790039 - moving ave loss 98.68563619932847 step 694 - loss 98.8901138305664 - moving ave loss 98.70608396245227 step 695 - loss 98.34734344482422 - moving ave loss 98.67020991068947 step 696 - loss 98.43840026855469 - moving ave loss 98.647028946476 Checkpoint at step 696 step 697 - loss 98.7221908569336 - moving ave loss 98.65454513752177 step 698 - loss 98.501953125 - moving ave loss 98.6392859362696 step 699 - loss 98.72572326660156 - moving ave loss 98.64792966930281 step 700 - loss 98.64563751220703 - moving ave loss 98.64770045359323 step 701 - loss 98.22782897949219 - moving ave loss 98.60571330618313 step 702 - loss 98.45574188232422 - moving ave loss 98.59071616379724 step 703 - loss 98.11607360839844 - moving ave loss 98.54325190825736 step 704 - loss 98.41271209716797 - moving ave loss 98.53019792714842 step 705 - loss 98.75 - moving ave loss 98.55217813443357 step 706 - loss 98.5248031616211 - moving ave loss 98.54944063715233 step 707 - loss 98.24121856689453 - moving ave loss 98.51861843012655 step 708 - loss 98.31201171875 - moving ave loss 98.49795775898889 Checkpoint at step 708 step 709 - loss 98.20487976074219 - moving ave loss 98.46864995916422 step 710 - loss 98.71675109863281 - moving ave loss 98.4934600731111 step 711 - loss 98.27800750732422 - moving ave loss 98.4719148165324 step 712 - loss 98.08924102783203 - moving ave loss 98.43364743766237 step 713 - loss 98.12196350097656 - moving ave loss 98.4024790439938 step 714 - loss 98.65815734863281 - moving ave loss 98.42804687445769 step 715 - loss 98.12490844726562 - moving ave loss 98.3977330317385 step 716 - loss 97.87161254882812 - moving ave loss 98.34512098344747 step 717 - loss 98.02801513671875 - moving ave loss 98.31341039877461 step 718 - loss 98.05052947998047 - moving ave loss 98.2871223068952 step 719 - loss 98.4621810913086 - moving ave loss 98.30462818533654 step 720 - loss 98.13069915771484 - moving ave loss 98.28723528257439 Checkpoint at step 720 step 721 - loss 97.95252990722656 - moving ave loss 98.2537647450396 step 722 - loss 98.27409362792969 - moving ave loss 98.25579763332863 step 723 - loss 98.23210906982422 - moving ave loss 98.2534287769782 step 724 - loss 98.32742309570312 - moving ave loss 98.26082820885068 step 725 - loss 98.343017578125 - moving ave loss 98.26904714577812 step 726 - loss 98.31271362304688 - moving ave loss 98.273413793505 step 727 - loss 98.00389099121094 - moving ave loss 98.24646151327559 step 728 - loss 98.18623352050781 - moving ave loss 98.24043871399881 step 729 - loss 97.66941833496094 - moving ave loss 98.18333667609502 step 730 - loss 97.84967041015625 - moving ave loss 98.14997004950115 step 731 - loss 97.75682067871094 - moving ave loss 98.11065511242214 step 732 - loss 98.08383178710938 - moving ave loss 98.10797277989087 Checkpoint at step 732 step 733 - loss 97.75321197509766 - moving ave loss 98.07249669941156 step 734 - loss 98.20081329345703 - moving ave loss 98.0853283588161 step 735 - loss 97.705810546875 - moving ave loss 98.04737657762199 step 736 - loss 98.25492095947266 - moving ave loss 98.06813101580705 step 737 - loss 97.57060241699219 - moving ave loss 98.01837815592557 step 738 - loss 99.4597396850586 - moving ave loss 98.1625143088389 step 739 - loss 97.6942367553711 - moving ave loss 98.11568655349213 step 740 - loss 97.49353790283203 - moving ave loss 98.05347168842613 Finish 4 epoch(es) step 741 - loss 97.85920715332031 - moving ave loss 98.03404523491554 step 742 - loss 97.51214599609375 - moving ave loss 97.98185531103337 step 743 - loss 97.72158813476562 - moving ave loss 97.95582859340661 step 744 - loss 97.56500244140625 - moving ave loss 97.91674597820658 Checkpoint at step 744 step 745 - loss 98.00090026855469 - moving ave loss 97.92516140724139 step 746 - loss 98.4468994140625 - moving ave loss 97.97733520792352 step 747 - loss 97.86369323730469 - moving ave loss 97.96597101086164 step 748 - loss 97.67015075683594 - moving ave loss 97.93638898545908 step 749 - loss 97.65940856933594 - moving ave loss 97.90869094384676 step 750 - loss 97.46271514892578 - moving ave loss 97.86409336435467 step 751 - loss 97.86878967285156 - moving ave loss 97.86456299520437 step 752 - loss 97.79771423339844 - moving ave loss 97.85787811902378 step 753 - loss 97.33712005615234 - moving ave loss 97.80580231273663 step 754 - loss 97.61483764648438 - moving ave loss 97.7867058461114 step 755 - loss 97.5680923461914 - moving ave loss 97.76484449611941 step 756 - loss 97.69470977783203 - moving ave loss 97.75783102429067 Checkpoint at step 756 step 757 - loss 98.19099426269531 - moving ave loss 97.80114734813114 step 758 - loss 97.36216735839844 - moving ave loss 97.75724934915787 step 759 - loss 97.29405212402344 - moving ave loss 97.71092962664443 step 760 - loss 97.68832397460938 - moving ave loss 97.70866906144093 step 761 - loss 97.3112564086914 - moving ave loss 97.66892779616597 step 762 - loss 97.71925354003906 - moving ave loss 97.67396037055329 step 763 - loss 97.6954345703125 - moving ave loss 97.67610779052922 step 764 - loss 97.41626739501953 - moving ave loss 97.65012375097825 step 765 - loss 97.5772933959961 - moving ave loss 97.64284071548003 step 766 - loss 97.02583312988281 - moving ave loss 97.58113995692031 step 767 - loss 97.18830871582031 - moving ave loss 97.54185683281031 step 768 - loss 97.35594940185547 - moving ave loss 97.52326608971484 Checkpoint at step 768 step 769 - loss 97.6065673828125 - moving ave loss 97.5315962190246 step 770 - loss 97.16179656982422 - moving ave loss 97.49461625410457 step 771 - loss 97.32350158691406 - moving ave loss 97.47750478738551 step 772 - loss 97.51594543457031 - moving ave loss 97.48134885210399 step 773 - loss 97.27733612060547 - moving ave loss 97.46094757895415 step 774 - loss 96.96188354492188 - moving ave loss 97.41104117555093 step 775 - loss 97.03378295898438 - moving ave loss 97.37331535389427 step 776 - loss 97.04315948486328 - moving ave loss 97.34029976699117 step 777 - loss 97.19369506835938 - moving ave loss 97.32563929712799 step 778 - loss 96.95393371582031 - moving ave loss 97.28846873899722 step 779 - loss 96.90006256103516 - moving ave loss 97.24962812120101 step 780 - loss 96.97557830810547 - moving ave loss 97.22222313989147 Checkpoint at step 780 step 781 - loss 97.29303741455078 - moving ave loss 97.2293045673574 step 782 - loss 97.28700256347656 - moving ave loss 97.23507436696931 step 783 - loss 96.83472442626953 - moving ave loss 97.19503937289933 step 784 - loss 97.46402740478516 - moving ave loss 97.22193817608792 step 785 - loss 96.92575073242188 - moving ave loss 97.19231943172133 step 786 - loss 96.88027954101562 - moving ave loss 97.16111544265075 step 787 - loss 97.8154296875 - moving ave loss 97.22654686713568 step 788 - loss 96.91390991210938 - moving ave loss 97.19528317163305 step 789 - loss 96.86219024658203 - moving ave loss 97.16197387912794 step 790 - loss 97.02189636230469 - moving ave loss 97.14796612744563 step 791 - loss 96.61529541015625 - moving ave loss 97.09469905571669 step 792 - loss 97.46064758300781 - moving ave loss 97.1312939084458 Checkpoint at step 792 step 793 - loss 97.19835662841797 - moving ave loss 97.13800018044303 step 794 - loss 96.6986083984375 - moving ave loss 97.09406100224248 step 795 - loss 97.0897216796875 - moving ave loss 97.093627069987 step 796 - loss 97.07149505615234 - moving ave loss 97.09141386860352 step 797 - loss 96.68177032470703 - moving ave loss 97.05044951421388 step 798 - loss 96.8458251953125 - moving ave loss 97.02998708232374 step 799 - loss 96.78086853027344 - moving ave loss 97.00507522711871 step 800 - loss 96.88424682617188 - moving ave loss 96.99299238702403 step 801 - loss 96.79756164550781 - moving ave loss 96.97344931287242 step 802 - loss 96.63531494140625 - moving ave loss 96.93963587572581 step 803 - loss 96.809326171875 - moving ave loss 96.92660490534072 step 804 - loss 96.57354736328125 - moving ave loss 96.89129915113479 Checkpoint at step 804 step 805 - loss 96.556396484375 - moving ave loss 96.8578088844588 step 806 - loss 96.5311508178711 - moving ave loss 96.82514307780004 step 807 - loss 97.06744384765625 - moving ave loss 96.84937315478567 step 808 - loss 96.56065368652344 - moving ave loss 96.82050120795945 step 809 - loss 96.99063110351562 - moving ave loss 96.83751419751506 step 810 - loss 96.481201171875 - moving ave loss 96.80188289495106 step 811 - loss 97.76676177978516 - moving ave loss 96.89837078343447 step 812 - loss 96.52645111083984 - moving ave loss 96.86117881617501 step 813 - loss 96.46797943115234 - moving ave loss 96.82185887767274 step 814 - loss 96.54023742675781 - moving ave loss 96.79369673258125 step 815 - loss 96.24197387695312 - moving ave loss 96.73852444701845 step 816 - loss 96.90400695800781 - moving ave loss 96.7550726981174 Checkpoint at step 816 step 817 - loss 97.03924560546875 - moving ave loss 96.78348998885255 step 818 - loss 96.53269958496094 - moving ave loss 96.75841094846339 step 819 - loss 97.13753509521484 - moving ave loss 96.79632336313854 step 820 - loss 96.61054992675781 - moving ave loss 96.77774601950047 step 821 - loss 96.75309753417969 - moving ave loss 96.77528117096838 step 822 - loss 96.2802734375 - moving ave loss 96.72578039762155 step 823 - loss 96.08563232421875 - moving ave loss 96.66176559028126 step 824 - loss 96.38375854492188 - moving ave loss 96.63396488574531 step 825 - loss 96.71815490722656 - moving ave loss 96.64238388789343 step 826 - loss 96.538818359375 - moving ave loss 96.63202733504158 step 827 - loss 96.56245422363281 - moving ave loss 96.62507002390072 step 828 - loss 96.03823852539062 - moving ave loss 96.56638687404971 Checkpoint at step 828 step 829 - loss 96.24383544921875 - moving ave loss 96.53413173156662 step 830 - loss 96.2824478149414 - moving ave loss 96.5089633399041 step 831 - loss 96.59716796875 - moving ave loss 96.5177838027887 step 832 - loss 95.95376586914062 - moving ave loss 96.46138200942389 step 833 - loss 96.400146484375 - moving ave loss 96.455258456919 step 834 - loss 96.13717651367188 - moving ave loss 96.42345026259429 step 835 - loss 96.02793884277344 - moving ave loss 96.38389912061221 step 836 - loss 96.17898559570312 - moving ave loss 96.3634077681213 step 837 - loss 96.39122772216797 - moving ave loss 96.36618976352597 step 838 - loss 96.31350708007812 - moving ave loss 96.36092149518119 step 839 - loss 96.12324523925781 - moving ave loss 96.33715386958885 step 840 - loss 95.76480102539062 - moving ave loss 96.27991858516903 Checkpoint at step 840 step 841 - loss 96.76138305664062 - moving ave loss 96.32806503231619 step 842 - loss 96.07990264892578 - moving ave loss 96.30324879397715 step 843 - loss 96.206298828125 - moving ave loss 96.29355379739192 step 844 - loss 96.03643035888672 - moving ave loss 96.2678414535414 step 845 - loss 96.1342544555664 - moving ave loss 96.2544827537439 step 846 - loss 96.14952087402344 - moving ave loss 96.24398656577186 step 847 - loss 95.83812713623047 - moving ave loss 96.20340062281772 step 848 - loss 96.39134216308594 - moving ave loss 96.22219477684455 step 849 - loss 95.92973327636719 - moving ave loss 96.1929486267968 step 850 - loss 96.32671356201172 - moving ave loss 96.2063251203183 step 851 - loss 95.78807067871094 - moving ave loss 96.16449967615756 step 852 - loss 95.83096313476562 - moving ave loss 96.13114602201837 Checkpoint at step 852 step 853 - loss 95.91488647460938 - moving ave loss 96.10952006727747 step 854 - loss 95.97846984863281 - moving ave loss 96.09641504541301 step 855 - loss 95.5108642578125 - moving ave loss 96.03785996665296 step 856 - loss 95.61979675292969 - moving ave loss 95.99605364528064 step 857 - loss 95.51982116699219 - moving ave loss 95.9484303974518 step 858 - loss 95.74437713623047 - moving ave loss 95.92802507132967 step 859 - loss 95.80888366699219 - moving ave loss 95.91611093089593 step 860 - loss 95.85688781738281 - moving ave loss 95.91018861954461 step 861 - loss 96.46585845947266 - moving ave loss 95.96575560353743 step 862 - loss 95.72708129882812 - moving ave loss 95.9418881730665 step 863 - loss 95.85771942138672 - moving ave loss 95.93347129789854 step 864 - loss 95.6707763671875 - moving ave loss 95.90720180482744 Checkpoint at step 864 step 865 - loss 95.47053527832031 - moving ave loss 95.86353515217672 step 866 - loss 95.70527648925781 - moving ave loss 95.84770928588483 step 867 - loss 95.76325225830078 - moving ave loss 95.83926358312642 step 868 - loss 95.64236450195312 - moving ave loss 95.8195736750091 step 869 - loss 96.24992370605469 - moving ave loss 95.86260867811366 step 870 - loss 95.68290710449219 - moving ave loss 95.84463852075152 step 871 - loss 95.60404968261719 - moving ave loss 95.82057963693809 step 872 - loss 95.59100341796875 - moving ave loss 95.79762201504116 step 873 - loss 95.31437683105469 - moving ave loss 95.74929749664251 step 874 - loss 95.64302062988281 - moving ave loss 95.73866980996654 step 875 - loss 95.56098937988281 - moving ave loss 95.72090176695816 step 876 - loss 95.15754699707031 - moving ave loss 95.66456628996939 Checkpoint at step 876 step 877 - loss 95.30908966064453 - moving ave loss 95.62901862703691 step 878 - loss 95.45591735839844 - moving ave loss 95.61170850017308 step 879 - loss 95.79045104980469 - moving ave loss 95.62958275513624 step 880 - loss 95.89009094238281 - moving ave loss 95.6556335738609 step 881 - loss 95.35926818847656 - moving ave loss 95.62599703532247 step 882 - loss 95.23062133789062 - moving ave loss 95.5864594655793 step 883 - loss 95.40821838378906 - moving ave loss 95.56863535740027 step 884 - loss 95.2645492553711 - moving ave loss 95.53822674719736 step 885 - loss 95.16749572753906 - moving ave loss 95.50115364523153 step 886 - loss 95.7538833618164 - moving ave loss 95.52642661689002 step 887 - loss 95.1358642578125 - moving ave loss 95.48737038098227 step 888 - loss 95.29315948486328 - moving ave loss 95.46794929137037 Checkpoint at step 888 step 889 - loss 95.82250213623047 - moving ave loss 95.50340457585638 step 890 - loss 95.35749816894531 - moving ave loss 95.48881393516528 step 891 - loss 95.58409118652344 - moving ave loss 95.49834166030111 step 892 - loss 95.29489135742188 - moving ave loss 95.47799663001318 step 893 - loss 95.32527160644531 - moving ave loss 95.4627241276564 step 894 - loss 94.98503112792969 - moving ave loss 95.41495482768374 step 895 - loss 95.30068969726562 - moving ave loss 95.40352831464193 step 896 - loss 95.22959899902344 - moving ave loss 95.38613538308009 step 897 - loss 94.77071380615234 - moving ave loss 95.3245932253873 step 898 - loss 95.08354187011719 - moving ave loss 95.3004880898603 step 899 - loss 94.99976348876953 - moving ave loss 95.27041562975123 step 900 - loss 95.03213500976562 - moving ave loss 95.24658756775267 Checkpoint at step 900 step 901 - loss 95.13632202148438 - moving ave loss 95.23556101312585 step 902 - loss 95.30525207519531 - moving ave loss 95.2425301193328 step 903 - loss 94.9366455078125 - moving ave loss 95.21194165818076 step 904 - loss 94.99543762207031 - moving ave loss 95.19029125456971 step 905 - loss 94.80744934082031 - moving ave loss 95.15200706319477 step 906 - loss 94.55006408691406 - moving ave loss 95.0918127655667 step 907 - loss 94.7640380859375 - moving ave loss 95.05903529760377 step 908 - loss 94.56874084472656 - moving ave loss 95.01000585231606 step 909 - loss 95.17927551269531 - moving ave loss 95.02693281835398 step 910 - loss 95.31455993652344 - moving ave loss 95.05569553017094 step 911 - loss 95.61927032470703 - moving ave loss 95.11205300962455 step 912 - loss 94.42559814453125 - moving ave loss 95.0434075231152 Checkpoint at step 912 step 913 - loss 95.0815658569336 - moving ave loss 95.04722335649704 step 914 - loss 94.8795166015625 - moving ave loss 95.0304526810036 step 915 - loss 94.80012512207031 - moving ave loss 95.00741992511027 step 916 - loss 94.59280395507812 - moving ave loss 94.96595832810706 step 917 - loss 94.58140563964844 - moving ave loss 94.92750305926121 step 918 - loss 95.02748107910156 - moving ave loss 94.93750086124524 step 919 - loss 94.89144897460938 - moving ave loss 94.93289567258165 step 920 - loss 94.87206268310547 - moving ave loss 94.92681237363404 step 921 - loss 94.64794158935547 - moving ave loss 94.89892529520618 step 922 - loss 94.99141693115234 - moving ave loss 94.9081744588008 step 923 - loss 94.79712677001953 - moving ave loss 94.89706968992267 step 924 - loss 94.75942993164062 - moving ave loss 94.88330571409448 Checkpoint at step 924 step 925 - loss 94.81045532226562 - moving ave loss 94.87602067491159 Finish 5 epoch(es) step 926 - loss 94.20089721679688 - moving ave loss 94.80850832910012 step 927 - loss 94.84024810791016 - moving ave loss 94.81168230698113 step 928 - loss 94.53398132324219 - moving ave loss 94.78391220860723 step 929 - loss 94.34831237792969 - moving ave loss 94.74035222553948 step 930 - loss 94.31977844238281 - moving ave loss 94.6982948472238 step 931 - loss 94.4566650390625 - moving ave loss 94.67413186640768 step 932 - loss 94.4063720703125 - moving ave loss 94.64735588679815 step 933 - loss 94.35446166992188 - moving ave loss 94.61806646511053 step 934 - loss 94.27156066894531 - moving ave loss 94.583415885494 step 935 - loss 94.02887725830078 - moving ave loss 94.52796202277469 step 936 - loss 94.55777740478516 - moving ave loss 94.53094356097573 Checkpoint at step 936 step 937 - loss 94.71919250488281 - moving ave loss 94.54976845536645 step 938 - loss 94.15992736816406 - moving ave loss 94.5107843466462 step 939 - loss 94.69580078125 - moving ave loss 94.52928599010659 step 940 - loss 94.31137084960938 - moving ave loss 94.50749447605686 step 941 - loss 94.14430236816406 - moving ave loss 94.47117526526758 step 942 - loss 94.47044372558594 - moving ave loss 94.47110211129942 step 943 - loss 94.52020263671875 - moving ave loss 94.47601216384135 step 944 - loss 94.86554718017578 - moving ave loss 94.51496566547479 step 945 - loss 93.88654327392578 - moving ave loss 94.45212342631989 step 946 - loss 94.31497192382812 - moving ave loss 94.4384082760707 step 947 - loss 94.29442596435547 - moving ave loss 94.42401004489919 step 948 - loss 94.1113052368164 - moving ave loss 94.3927395640909 Checkpoint at step 948 step 949 - loss 94.29401397705078 - moving ave loss 94.3828670053869 step 950 - loss 93.77594757080078 - moving ave loss 94.32217506192829 step 951 - loss 95.0360107421875 - moving ave loss 94.39355862995421 step 952 - loss 94.04991149902344 - moving ave loss 94.35919391686114 step 953 - loss 94.27180480957031 - moving ave loss 94.35045500613207 step 954 - loss 93.99847412109375 - moving ave loss 94.31525691762825 step 955 - loss 94.05056762695312 - moving ave loss 94.28878798856073 step 956 - loss 93.89315795898438 - moving ave loss 94.24922498560309 step 957 - loss 94.08071899414062 - moving ave loss 94.23237438645685 step 958 - loss 94.59088134765625 - moving ave loss 94.26822508257679 step 959 - loss 94.17988586425781 - moving ave loss 94.2593911607449 step 960 - loss 93.9228515625 - moving ave loss 94.2257372009204 Checkpoint at step 960 step 961 - loss 94.15800476074219 - moving ave loss 94.2189639569026 step 962 - loss 93.8941879272461 - moving ave loss 94.18648635393696 step 963 - loss 93.50569152832031 - moving ave loss 94.1184068713753 step 964 - loss 93.66209411621094 - moving ave loss 94.07277559585887 step 965 - loss 93.94291687011719 - moving ave loss 94.05978972328471 step 966 - loss 93.6502456665039 - moving ave loss 94.01883531760663 step 967 - loss 93.85910034179688 - moving ave loss 94.00286182002566 step 968 - loss 93.55012512207031 - moving ave loss 93.95758815023012 step 969 - loss 93.53909301757812 - moving ave loss 93.91573863696492 step 970 - loss 93.57169342041016 - moving ave loss 93.88133411530944 step 971 - loss 93.68390655517578 - moving ave loss 93.86159135929607 step 972 - loss 93.53265380859375 - moving ave loss 93.82869760422584 Checkpoint at step 972 step 973 - loss 93.48847961425781 - moving ave loss 93.79467580522903 step 974 - loss 93.36942291259766 - moving ave loss 93.7521505159659 step 975 - loss 93.89364624023438 - moving ave loss 93.76630008839275 step 976 - loss 93.69464874267578 - moving ave loss 93.75913495382106 step 977 - loss 93.5825424194336 - moving ave loss 93.74147570038231 step 978 - loss 93.57331085205078 - moving ave loss 93.72465921554915 step 979 - loss 93.54460144042969 - moving ave loss 93.70665343803721 step 980 - loss 94.07560729980469 - moving ave loss 93.74354882421396 step 981 - loss 93.43339538574219 - moving ave loss 93.71253348036677 step 982 - loss 93.47200012207031 - moving ave loss 93.68848014453712 step 983 - loss 93.81546020507812 - moving ave loss 93.70117815059123 step 984 - loss 93.284423828125 - moving ave loss 93.65950271834461 Checkpoint at step 984 step 985 - loss 93.43441772460938 - moving ave loss 93.63699421897108 step 986 - loss 93.21426391601562 - moving ave loss 93.59472118867554 step 987 - loss 93.39126586914062 - moving ave loss 93.57437565672204 step 988 - loss 93.28202819824219 - moving ave loss 93.54514091087405 step 989 - loss 93.46980285644531 - moving ave loss 93.53760710543118 step 990 - loss 93.37578582763672 - moving ave loss 93.52142497765173 step 991 - loss 93.17350769042969 - moving ave loss 93.48663324892954 step 992 - loss 93.381103515625 - moving ave loss 93.47608027559909 step 993 - loss 93.27067565917969 - moving ave loss 93.45553981395716 step 994 - loss 93.28781127929688 - moving ave loss 93.43876696049114 step 995 - loss 93.17230987548828 - moving ave loss 93.41212125199085 step 996 - loss 93.16654968261719 - moving ave loss 93.38756409505349 Checkpoint at step 996 step 997 - loss 93.17820739746094 - moving ave loss 93.36662842529424 step 998 - loss 92.95014190673828 - moving ave loss 93.32497977343864 step 999 - loss 93.36820983886719 - moving ave loss 93.3293027799815 step 1000 - loss 93.25186920166016 - moving ave loss 93.32155942214938 step 1001 - loss 92.72566223144531 - moving ave loss 93.26196970307898 step 1002 - loss 93.26416015625 - moving ave loss 93.26218874839608 step 1003 - loss 93.2538833618164 - moving ave loss 93.26135820973812 step 1004 - loss 93.37399291992188 - moving ave loss 93.2726216807565 step 1005 - loss 92.98945617675781 - moving ave loss 93.24430513035664 step 1006 - loss 92.861328125 - moving ave loss 93.20600742982097 step 1007 - loss 93.07984161376953 - moving ave loss 93.19339084821583 step 1008 - loss 92.7561264038086 - moving ave loss 93.1496644037751 Checkpoint at step 1008 step 1009 - loss 93.06582641601562 - moving ave loss 93.14128060499917 step 1010 - loss 92.75902557373047 - moving ave loss 93.1030551018723 step 1011 - loss 92.57877349853516 - moving ave loss 93.05062694153858 step 1012 - loss 92.96794128417969 - moving ave loss 93.04235837580269 step 1013 - loss 92.69842529296875 - moving ave loss 93.00796506751931 step 1014 - loss 93.0374755859375 - moving ave loss 93.01091611936114 step 1015 - loss 92.7921142578125 - moving ave loss 92.98903593320628 step 1016 - loss 92.85511779785156 - moving ave loss 92.9756441196708 step 1017 - loss 92.80550384521484 - moving ave loss 92.95863009222522 step 1018 - loss 92.593505859375 - moving ave loss 92.9221176689402 step 1019 - loss 93.15127563476562 - moving ave loss 92.94503346552276 step 1020 - loss 92.72789001464844 - moving ave loss 92.92331912043532 Checkpoint at step 1020 step 1021 - loss 92.92632293701172 - moving ave loss 92.92361950209296 step 1022 - loss 92.5133285522461 - moving ave loss 92.88259040710827 step 1023 - loss 92.52908325195312 - moving ave loss 92.84723969159275 step 1024 - loss 92.36566925048828 - moving ave loss 92.79908264748231 step 1025 - loss 92.73883056640625 - moving ave loss 92.7930574393747 step 1026 - loss 92.76410675048828 - moving ave loss 92.79016237048606 step 1027 - loss 93.00997161865234 - moving ave loss 92.81214329530269 step 1028 - loss 92.38902282714844 - moving ave loss 92.76983124848726 step 1029 - loss 92.86404418945312 - moving ave loss 92.77925254258385 step 1030 - loss 92.14192962646484 - moving ave loss 92.71552025097195 step 1031 - loss 92.90005493164062 - moving ave loss 92.73397371903881 step 1032 - loss 92.56806182861328 - moving ave loss 92.71738252999626 Checkpoint at step 1032 step 1033 - loss 92.49803161621094 - moving ave loss 92.69544743861772 step 1034 - loss 92.58417510986328 - moving ave loss 92.68432020574228 step 1035 - loss 92.40507507324219 - moving ave loss 92.65639569249228 step 1036 - loss 92.90092468261719 - moving ave loss 92.68084859150477 step 1037 - loss 92.54395294189453 - moving ave loss 92.66715902654374 step 1038 - loss 92.1239013671875 - moving ave loss 92.61283326060813 step 1039 - loss 92.41705322265625 - moving ave loss 92.59325525681295 step 1040 - loss 93.17533111572266 - moving ave loss 92.65146284270392 step 1041 - loss 92.47361755371094 - moving ave loss 92.63367831380462 step 1042 - loss 92.24397277832031 - moving ave loss 92.59470776025618 step 1043 - loss 91.95441436767578 - moving ave loss 92.53067842099814 step 1044 - loss 92.71356964111328 - moving ave loss 92.54896754300967 Checkpoint at step 1044 step 1045 - loss 92.73091888427734 - moving ave loss 92.56716267713644 step 1046 - loss 91.87244415283203 - moving ave loss 92.497690824706 step 1047 - loss 92.54214477539062 - moving ave loss 92.50213621977447 step 1048 - loss 92.026123046875 - moving ave loss 92.45453490248453 step 1049 - loss 92.20140075683594 - moving ave loss 92.42922148791966 step 1050 - loss 92.46621704101562 - moving ave loss 92.43292104322926 step 1051 - loss 91.97019958496094 - moving ave loss 92.38664889740244 step 1052 - loss 92.07858276367188 - moving ave loss 92.35584228402939 step 1053 - loss 92.12581634521484 - moving ave loss 92.33283969014794 step 1054 - loss 92.54534912109375 - moving ave loss 92.35409063324252 step 1055 - loss 91.90147399902344 - moving ave loss 92.30882896982061 step 1056 - loss 91.78805541992188 - moving ave loss 92.25675161483075 Checkpoint at step 1056 step 1057 - loss 92.16859436035156 - moving ave loss 92.24793588938283 step 1058 - loss 91.60922241210938 - moving ave loss 92.18406454165549 step 1059 - loss 92.30010223388672 - moving ave loss 92.19566831087862 step 1060 - loss 92.12519836425781 - moving ave loss 92.18862131621654 step 1061 - loss 91.88774108886719 - moving ave loss 92.15853329348161 step 1062 - loss 91.75812530517578 - moving ave loss 92.11849249465102 step 1063 - loss 91.98097229003906 - moving ave loss 92.10474047418982 step 1064 - loss 91.79148864746094 - moving ave loss 92.07341529151694 step 1065 - loss 91.97451782226562 - moving ave loss 92.06352554459181 step 1066 - loss 91.79911041259766 - moving ave loss 92.03708403139241 step 1067 - loss 91.76695251464844 - moving ave loss 92.01007087971801 step 1068 - loss 92.40848541259766 - moving ave loss 92.04991233300598 Checkpoint at step 1068 step 1069 - loss 91.63561248779297 - moving ave loss 92.00848234848469 step 1070 - loss 91.36279296875 - moving ave loss 91.94391341051123 step 1071 - loss 91.902587890625 - moving ave loss 91.93978085852261 step 1072 - loss 92.20499420166016 - moving ave loss 91.96630219283637 step 1073 - loss 92.00617218017578 - moving ave loss 91.97028919157032 step 1074 - loss 91.66143798828125 - moving ave loss 91.93940407124141 step 1075 - loss 91.57974243164062 - moving ave loss 91.90343790728133 step 1076 - loss 91.80140686035156 - moving ave loss 91.89323480258835 step 1077 - loss 91.72313690185547 - moving ave loss 91.87622501251506 step 1078 - loss 91.69560241699219 - moving ave loss 91.85816275296277 step 1079 - loss 92.00450134277344 - moving ave loss 91.87279661194384 step 1080 - loss 91.71935272216797 - moving ave loss 91.85745222296626 Checkpoint at step 1080 step 1081 - loss 91.1512222290039 - moving ave loss 91.78682922357004 step 1082 - loss 91.48330688476562 - moving ave loss 91.7564769896896 step 1083 - loss 91.59811401367188 - moving ave loss 91.74064069208784 step 1084 - loss 91.38194274902344 - moving ave loss 91.7047708977814 step 1085 - loss 91.46427917480469 - moving ave loss 91.68072172548374 step 1086 - loss 91.20747375488281 - moving ave loss 91.63339692842364 step 1087 - loss 91.35298156738281 - moving ave loss 91.60535539231957 step 1088 - loss 91.48158264160156 - moving ave loss 91.59297811724777 step 1089 - loss 91.19770050048828 - moving ave loss 91.55345035557183 step 1090 - loss 91.66868591308594 - moving ave loss 91.56497391132325 step 1091 - loss 91.3761978149414 - moving ave loss 91.54609630168507 step 1092 - loss 91.255859375 - moving ave loss 91.51707260901657 Checkpoint at step 1092 step 1093 - loss 91.38209533691406 - moving ave loss 91.50357488180632 step 1094 - loss 91.44772338867188 - moving ave loss 91.49798973249287 step 1095 - loss 91.05802917480469 - moving ave loss 91.45399367672405 step 1096 - loss 92.26063537597656 - moving ave loss 91.53465784664931 step 1097 - loss 91.20677185058594 - moving ave loss 91.50186924704298 step 1098 - loss 91.03050231933594 - moving ave loss 91.45473255427227 step 1099 - loss 90.7560806274414 - moving ave loss 91.38486736158919 step 1100 - loss 91.0795669555664 - moving ave loss 91.35433732098691 step 1101 - loss 91.28726959228516 - moving ave loss 91.34763054811674 step 1102 - loss 90.92813873291016 - moving ave loss 91.30568136659609 step 1103 - loss 90.83450317382812 - moving ave loss 91.2585635473193 step 1104 - loss 90.79222106933594 - moving ave loss 91.21192929952097 Checkpoint at step 1104 step 1105 - loss 91.41825866699219 - moving ave loss 91.2325622362681 step 1106 - loss 90.96400451660156 - moving ave loss 91.20570646430144 step 1107 - loss 90.77082824707031 - moving ave loss 91.16221864257834 step 1108 - loss 91.40318298339844 - moving ave loss 91.18631507666035 step 1109 - loss 90.88761901855469 - moving ave loss 91.15644547084977 step 1110 - loss 90.5576171875 - moving ave loss 91.09656264251481 Finish 6 epoch(es) step 1111 - loss 90.67420959472656 - moving ave loss 91.05432733773598 step 1112 - loss 90.8208236694336 - moving ave loss 91.03097697090574 step 1113 - loss 90.43783569335938 - moving ave loss 90.97166284315111 step 1114 - loss 90.44802856445312 - moving ave loss 90.91929941528132 step 1115 - loss 90.82698822021484 - moving ave loss 90.91006829577466 step 1116 - loss 91.57147216796875 - moving ave loss 90.97620868299407 Checkpoint at step 1116 step 1117 - loss 90.95335388183594 - moving ave loss 90.97392320287827 step 1118 - loss 90.32772827148438 - moving ave loss 90.90930370973888 step 1119 - loss 90.49748229980469 - moving ave loss 90.86812156874547 step 1120 - loss 91.03546142578125 - moving ave loss 90.88485555444905 step 1121 - loss 90.97879028320312 - moving ave loss 90.89424902732445 step 1122 - loss 90.70451354980469 - moving ave loss 90.87527547957247 step 1123 - loss 91.09339141845703 - moving ave loss 90.89708707346092 step 1124 - loss 91.74371337890625 - moving ave loss 90.98174970400545 step 1125 - loss 90.96185302734375 - moving ave loss 90.97976003633927 step 1126 - loss 90.81097412109375 - moving ave loss 90.96288144481471 step 1127 - loss 90.71602630615234 - moving ave loss 90.93819593094848 step 1128 - loss 90.62640380859375 - moving ave loss 90.90701671871301 Checkpoint at step 1128 step 1129 - loss 90.74955749511719 - moving ave loss 90.89127079635342 step 1130 - loss 91.14777374267578 - moving ave loss 90.91692109098565 step 1131 - loss 90.2425308227539 - moving ave loss 90.84948206416249 step 1132 - loss 90.47554779052734 - moving ave loss 90.81208863679898 step 1133 - loss 90.43797302246094 - moving ave loss 90.77467707536518 step 1134 - loss 90.00601196289062 - moving ave loss 90.69781056411773 step 1135 - loss 90.31720733642578 - moving ave loss 90.65975024134853 step 1136 - loss 90.32584381103516 - moving ave loss 90.6263595983172 step 1137 - loss 90.33250427246094 - moving ave loss 90.59697406573156 step 1138 - loss 90.70851135253906 - moving ave loss 90.60812779441231 step 1139 - loss 90.4273681640625 - moving ave loss 90.59005183137732 step 1140 - loss 90.04179382324219 - moving ave loss 90.53522603056382 Checkpoint at step 1140 step 1141 - loss 90.2113037109375 - moving ave loss 90.50283379860119 step 1142 - loss 90.15064239501953 - moving ave loss 90.46761465824302 step 1143 - loss 90.23760986328125 - moving ave loss 90.44461417874685 step 1144 - loss 89.94621276855469 - moving ave loss 90.39477403772763 step 1145 - loss 90.22291564941406 - moving ave loss 90.37758819889628 step 1146 - loss 90.69978332519531 - moving ave loss 90.40980771152617 step 1147 - loss 90.61672973632812 - moving ave loss 90.43049991400636 step 1148 - loss 89.9332275390625 - moving ave loss 90.38077267651198 step 1149 - loss 90.2481689453125 - moving ave loss 90.36751230339205 step 1150 - loss 90.10842895507812 - moving ave loss 90.34160396856065 step 1151 - loss 89.94164276123047 - moving ave loss 90.30160784782763 step 1152 - loss 90.12724304199219 - moving ave loss 90.28417136724408 Checkpoint at step 1152 step 1153 - loss 90.11099243164062 - moving ave loss 90.26685347368375 step 1154 - loss 89.97347259521484 - moving ave loss 90.23751538583687 step 1155 - loss 89.9576416015625 - moving ave loss 90.20952800740945 step 1156 - loss 89.91944885253906 - moving ave loss 90.1805200919224 step 1157 - loss 89.85200500488281 - moving ave loss 90.14766858321845 step 1158 - loss 90.09403228759766 - moving ave loss 90.14230495365636 step 1159 - loss 89.64736938476562 - moving ave loss 90.09281139676729 step 1160 - loss 89.5901107788086 - moving ave loss 90.04254133497142 step 1161 - loss 90.14007568359375 - moving ave loss 90.05229476983365 step 1162 - loss 89.60516357421875 - moving ave loss 90.00758165027217 step 1163 - loss 89.77407836914062 - moving ave loss 89.98423132215902 step 1164 - loss 89.7714614868164 - moving ave loss 89.96295433862475 Checkpoint at step 1164 step 1165 - loss 90.07984924316406 - moving ave loss 89.97464382907869 step 1166 - loss 89.51097106933594 - moving ave loss 89.92827655310441 step 1167 - loss 89.24290466308594 - moving ave loss 89.85973936410257 step 1168 - loss 89.65543365478516 - moving ave loss 89.83930879317082 step 1169 - loss 89.80029296875 - moving ave loss 89.83540721072873 step 1170 - loss 89.61153411865234 - moving ave loss 89.8130199015211 step 1171 - loss 89.26499938964844 - moving ave loss 89.75821785033384 step 1172 - loss 90.28495025634766 - moving ave loss 89.81089109093523 step 1173 - loss 89.54093933105469 - moving ave loss 89.78389591494718 step 1174 - loss 89.65056610107422 - moving ave loss 89.77056293355989 step 1175 - loss 89.34852600097656 - moving ave loss 89.72835924030156 step 1176 - loss 89.67688751220703 - moving ave loss 89.72321206749211 Checkpoint at step 1176 step 1177 - loss 90.20597839355469 - moving ave loss 89.77148870009837 step 1178 - loss 89.14788818359375 - moving ave loss 89.7091286484479 step 1179 - loss 89.15028381347656 - moving ave loss 89.65324416495076 step 1180 - loss 89.054443359375 - moving ave loss 89.59336408439319 step 1181 - loss 89.37503051757812 - moving ave loss 89.57153072771169 step 1182 - loss 89.17860412597656 - moving ave loss 89.53223806753819 step 1183 - loss 89.58421325683594 - moving ave loss 89.53743558646796 step 1184 - loss 89.10932922363281 - moving ave loss 89.49462495018446 step 1185 - loss 89.10395812988281 - moving ave loss 89.4555582681543 step 1186 - loss 89.28587341308594 - moving ave loss 89.43858978264748 step 1187 - loss 89.86569213867188 - moving ave loss 89.48130001824993 step 1188 - loss 89.65243530273438 - moving ave loss 89.49841354669837 Checkpoint at step 1188 step 1189 - loss 89.03990936279297 - moving ave loss 89.45256312830784 step 1190 - loss 89.24909973144531 - moving ave loss 89.4322167886216 step 1191 - loss 89.35176849365234 - moving ave loss 89.42417195912468 step 1192 - loss 89.47735595703125 - moving ave loss 89.42949035891533 step 1193 - loss 88.90316772460938 - moving ave loss 89.37685809548474 step 1194 - loss 88.69558715820312 - moving ave loss 89.30873100175658 step 1195 - loss 88.90029907226562 - moving ave loss 89.26788780880749 step 1196 - loss 89.05307006835938 - moving ave loss 89.24640603476269 step 1197 - loss 89.23934936523438 - moving ave loss 89.24570036780986 step 1198 - loss 89.14131164550781 - moving ave loss 89.23526149557966 step 1199 - loss 89.02961730957031 - moving ave loss 89.21469707697874 step 1200 - loss 88.78528594970703 - moving ave loss 89.17175596425157 Checkpoint at step 1200 step 1201 - loss 88.81487274169922 - moving ave loss 89.13606764199635 step 1202 - loss 89.01521301269531 - moving ave loss 89.12398217906625 step 1203 - loss 89.29890441894531 - moving ave loss 89.14147440305415 step 1204 - loss 88.46052551269531 - moving ave loss 89.07337951401827 step 1205 - loss 88.66641235351562 - moving ave loss 89.03268279796801 step 1206 - loss 88.9725112915039 - moving ave loss 89.0266656473216 step 1207 - loss 90.02884674072266 - moving ave loss 89.1268837566617 step 1208 - loss 88.80692291259766 - moving ave loss 89.0948876722553 step 1209 - loss 89.74415588378906 - moving ave loss 89.15981449340867 step 1210 - loss 88.51669311523438 - moving ave loss 89.09550235559125 step 1211 - loss 88.5518798828125 - moving ave loss 89.04114010831337 step 1212 - loss 88.33607482910156 - moving ave loss 88.97063358039219 Checkpoint at step 1212 step 1213 - loss 88.77019500732422 - moving ave loss 88.95058972308539 step 1214 - loss 88.48321533203125 - moving ave loss 88.90385228397997 step 1215 - loss 88.70687866210938 - moving ave loss 88.88415492179293 step 1216 - loss 88.85089111328125 - moving ave loss 88.88082854094176 step 1217 - loss 88.81214141845703 - moving ave loss 88.87395982869329 step 1218 - loss 88.614501953125 - moving ave loss 88.84801404113647 step 1219 - loss 88.72322082519531 - moving ave loss 88.83553471954235 step 1220 - loss 88.51510620117188 - moving ave loss 88.8034918677053 step 1221 - loss 88.10272979736328 - moving ave loss 88.7334156606711 step 1222 - loss 88.4188232421875 - moving ave loss 88.70195641882273 step 1223 - loss 88.22821807861328 - moving ave loss 88.65458258480179 step 1224 - loss 88.76910400390625 - moving ave loss 88.66603472671224 Checkpoint at step 1224 step 1225 - loss 88.60161590576172 - moving ave loss 88.6595928446172 step 1226 - loss 88.52706909179688 - moving ave loss 88.64634046933516 step 1227 - loss 88.15963745117188 - moving ave loss 88.59767016751883 step 1228 - loss 88.29158020019531 - moving ave loss 88.56706117078649 step 1229 - loss 88.2578125 - moving ave loss 88.53613630370785 step 1230 - loss 88.78265380859375 - moving ave loss 88.56078805419644 step 1231 - loss 88.673583984375 - moving ave loss 88.5720676472143 step 1232 - loss 88.30731201171875 - moving ave loss 88.54559208366474 step 1233 - loss 88.265380859375 - moving ave loss 88.51757096123578 step 1234 - loss 88.42120361328125 - moving ave loss 88.50793422644033 step 1235 - loss 88.23126983642578 - moving ave loss 88.48026778743888 step 1236 - loss 88.00355529785156 - moving ave loss 88.43259653848014 Checkpoint at step 1236 step 1237 - loss 88.52226257324219 - moving ave loss 88.44156314195635 step 1238 - loss 87.77104187011719 - moving ave loss 88.37451101477244 step 1239 - loss 88.07405090332031 - moving ave loss 88.34446500362722 step 1240 - loss 87.71720886230469 - moving ave loss 88.28173938949497 step 1241 - loss 88.05902099609375 - moving ave loss 88.25946755015485 step 1242 - loss 88.27738952636719 - moving ave loss 88.2612597477761 step 1243 - loss 87.86532592773438 - moving ave loss 88.22166636577192 step 1244 - loss 87.57357788085938 - moving ave loss 88.15685751728067 step 1245 - loss 87.88591003417969 - moving ave loss 88.12976276897056 step 1246 - loss 87.88063049316406 - moving ave loss 88.10484954138991 step 1247 - loss 88.0622329711914 - moving ave loss 88.10058788437007 step 1248 - loss 88.31632995605469 - moving ave loss 88.12216209153854 Checkpoint at step 1248 step 1249 - loss 87.72622680664062 - moving ave loss 88.08256856304875 step 1250 - loss 87.7002944946289 - moving ave loss 88.04434115620677 step 1251 - loss 88.31065368652344 - moving ave loss 88.07097240923844 step 1252 - loss 87.92422485351562 - moving ave loss 88.05629765366616 step 1253 - loss 87.70287322998047 - moving ave loss 88.02095521129759 step 1254 - loss 87.39349365234375 - moving ave loss 87.95820905540221 step 1255 - loss 87.95689392089844 - moving ave loss 87.95807754195182 step 1256 - loss 88.10302734375 - moving ave loss 87.97257252213164 step 1257 - loss 87.72055053710938 - moving ave loss 87.94737032362943 step 1258 - loss 87.50345611572266 - moving ave loss 87.90297890283875 step 1259 - loss 87.46217346191406 - moving ave loss 87.85889835874629 step 1260 - loss 87.27212524414062 - moving ave loss 87.80022104728572 Checkpoint at step 1260 step 1261 - loss 88.21926879882812 - moving ave loss 87.84212582243995 step 1262 - loss 87.9927978515625 - moving ave loss 87.8571930253522 step 1263 - loss 87.8322525024414 - moving ave loss 87.85469897306113 step 1264 - loss 87.76710510253906 - moving ave loss 87.84593958600892 step 1265 - loss 87.91079711914062 - moving ave loss 87.8524253393221 step 1266 - loss 87.62356567382812 - moving ave loss 87.8295393727727 step 1267 - loss 87.84285736083984 - moving ave loss 87.83087117157942 step 1268 - loss 87.45770263671875 - moving ave loss 87.79355431809336 step 1269 - loss 87.02589416503906 - moving ave loss 87.71678830278793 step 1270 - loss 87.60772705078125 - moving ave loss 87.70588217758727 step 1271 - loss 87.58058166503906 - moving ave loss 87.69335212633244 step 1272 - loss 86.93083190917969 - moving ave loss 87.61710010461718 Checkpoint at step 1272 step 1273 - loss 87.10025024414062 - moving ave loss 87.56541511856953 step 1274 - loss 87.91372680664062 - moving ave loss 87.60024628737665 step 1275 - loss 87.51252746582031 - moving ave loss 87.591474405221 step 1276 - loss 87.59915161132812 - moving ave loss 87.59224212583172 step 1277 - loss 87.18782806396484 - moving ave loss 87.55180071964504 step 1278 - loss 87.31626892089844 - moving ave loss 87.52824753977039 step 1279 - loss 87.90939331054688 - moving ave loss 87.56636211684804 step 1280 - loss 87.15483856201172 - moving ave loss 87.52520976136441 step 1281 - loss 87.46614074707031 - moving ave loss 87.519302859935 step 1282 - loss 87.05760955810547 - moving ave loss 87.47313352975205 step 1283 - loss 86.8827133178711 - moving ave loss 87.41409150856396 step 1284 - loss 86.88822174072266 - moving ave loss 87.36150453177983 Checkpoint at step 1284 step 1285 - loss 86.94078826904297 - moving ave loss 87.31943290550615 step 1286 - loss 86.92015838623047 - moving ave loss 87.27950545357858 step 1287 - loss 86.87796020507812 - moving ave loss 87.23935092872854 step 1288 - loss 86.75131225585938 - moving ave loss 87.19054706144163 step 1289 - loss 86.95697021484375 - moving ave loss 87.16718937678185 step 1290 - loss 86.72959899902344 - moving ave loss 87.123430339006 step 1291 - loss 87.28709411621094 - moving ave loss 87.13979671672651 step 1292 - loss 86.84895324707031 - moving ave loss 87.11071236976089 step 1293 - loss 86.84219360351562 - moving ave loss 87.08386049313637 step 1294 - loss 86.52535247802734 - moving ave loss 87.02800969162547 step 1295 - loss 86.553955078125 - moving ave loss 86.98060423027543 Finish 7 epoch(es) step 1296 - loss 86.7017822265625 - moving ave loss 86.95272202990414 Checkpoint at step 1296 step 1297 - loss 86.59548950195312 - moving ave loss 86.91699877710904 step 1298 - loss 86.52545166015625 - moving ave loss 86.87784406541377 step 1299 - loss 86.66693115234375 - moving ave loss 86.85675277410678 step 1300 - loss 86.83029174804688 - moving ave loss 86.85410667150079 step 1301 - loss 86.67741394042969 - moving ave loss 86.83643739839367 step 1302 - loss 86.65089416503906 - moving ave loss 86.8178830750582 step 1303 - loss 86.23271942138672 - moving ave loss 86.75936670969107 step 1304 - loss 86.26722717285156 - moving ave loss 86.71015275600712 step 1305 - loss 86.42176818847656 - moving ave loss 86.68131429925407 step 1306 - loss 86.18158721923828 - moving ave loss 86.6313415912525 step 1307 - loss 86.8764419555664 - moving ave loss 86.65585162768389 step 1308 - loss 86.52801513671875 - moving ave loss 86.64306797858738 Checkpoint at step 1308 step 1309 - loss 87.25623321533203 - moving ave loss 86.70438450226185 step 1310 - loss 86.64189910888672 - moving ave loss 86.69813596292433 step 1311 - loss 87.1084976196289 - moving ave loss 86.73917212859479 step 1312 - loss 86.28804016113281 - moving ave loss 86.69405893184859 step 1313 - loss 86.17066955566406 - moving ave loss 86.64171999423013 step 1314 - loss 86.56901550292969 - moving ave loss 86.6344495451001 step 1315 - loss 86.16941833496094 - moving ave loss 86.58794642408618 step 1316 - loss 86.59969329833984 - moving ave loss 86.58912111151155 step 1317 - loss 86.634765625 - moving ave loss 86.59368556286039 step 1318 - loss 86.68730926513672 - moving ave loss 86.60304793308802 step 1319 - loss 85.90831756591797 - moving ave loss 86.53357489637102 step 1320 - loss 86.61326599121094 - moving ave loss 86.54154400585502 Checkpoint at step 1320 step 1321 - loss 86.36906433105469 - moving ave loss 86.52429603837498 step 1322 - loss 85.94795989990234 - moving ave loss 86.46666242452773 step 1323 - loss 86.11223602294922 - moving ave loss 86.43121978436989 step 1324 - loss 85.9669189453125 - moving ave loss 86.38478970046415 step 1325 - loss 86.27890014648438 - moving ave loss 86.37420074506619 step 1326 - loss 86.10423278808594 - moving ave loss 86.34720394936816 step 1327 - loss 86.24527740478516 - moving ave loss 86.33701129490986 step 1328 - loss 86.24966430664062 - moving ave loss 86.32827659608293 step 1329 - loss 86.49786376953125 - moving ave loss 86.34523531342776 step 1330 - loss 85.82630920410156 - moving ave loss 86.29334270249514 step 1331 - loss 86.36163330078125 - moving ave loss 86.30017176232374 step 1332 - loss 86.52003479003906 - moving ave loss 86.32215806509528 Checkpoint at step 1332 step 1333 - loss 86.23649597167969 - moving ave loss 86.31359185575373 step 1334 - loss 86.09542083740234 - moving ave loss 86.29177475391859 step 1335 - loss 85.69895935058594 - moving ave loss 86.23249321358533 step 1336 - loss 85.69471740722656 - moving ave loss 86.17871563294946 step 1337 - loss 85.8555679321289 - moving ave loss 86.14640086286741 step 1338 - loss 85.62100219726562 - moving ave loss 86.09386099630723 step 1339 - loss 85.41731262207031 - moving ave loss 86.02620615888354 step 1340 - loss 85.46723937988281 - moving ave loss 85.97030948098346 step 1341 - loss 85.51507568359375 - moving ave loss 85.92478610124448 step 1342 - loss 85.8519287109375 - moving ave loss 85.91750036221379 step 1343 - loss 85.84613800048828 - moving ave loss 85.91036412604124 step 1344 - loss 85.510009765625 - moving ave loss 85.87032868999962 Checkpoint at step 1344 step 1345 - loss 85.83528137207031 - moving ave loss 85.8668239582067 step 1346 - loss 86.19709777832031 - moving ave loss 85.89985134021805 step 1347 - loss 85.55984497070312 - moving ave loss 85.86585070326656 step 1348 - loss 85.44359588623047 - moving ave loss 85.82362522156296 step 1349 - loss 85.76400756835938 - moving ave loss 85.81766345624261 step 1350 - loss 85.21766662597656 - moving ave loss 85.75766377321601 step 1351 - loss 85.72299194335938 - moving ave loss 85.75419659023035 step 1352 - loss 85.52420043945312 - moving ave loss 85.73119697515264 step 1353 - loss 85.34637451171875 - moving ave loss 85.69271472880925 step 1354 - loss 85.42318725585938 - moving ave loss 85.66576198151427 step 1355 - loss 85.31536865234375 - moving ave loss 85.63072264859721 step 1356 - loss 85.18120574951172 - moving ave loss 85.58577095868867 Checkpoint at step 1356 step 1357 - loss 85.9896240234375 - moving ave loss 85.62615626516354 step 1358 - loss 85.97046661376953 - moving ave loss 85.66058730002415 step 1359 - loss 85.58012390136719 - moving ave loss 85.65254096015846 step 1360 - loss 85.0762939453125 - moving ave loss 85.59491625867386 step 1361 - loss 85.83879089355469 - moving ave loss 85.61930372216194 step 1362 - loss 85.86415100097656 - moving ave loss 85.6437884500434 step 1363 - loss 85.17964935302734 - moving ave loss 85.59737454034179 step 1364 - loss 85.39010620117188 - moving ave loss 85.5766477064248 step 1365 - loss 85.35330963134766 - moving ave loss 85.55431389891709 step 1366 - loss 85.00494384765625 - moving ave loss 85.49937689379101 step 1367 - loss 85.07939910888672 - moving ave loss 85.45737911530058 step 1368 - loss 85.04341125488281 - moving ave loss 85.41598232925881 Checkpoint at step 1368 step 1369 - loss 85.27862548828125 - moving ave loss 85.40224664516106 step 1370 - loss 85.14007568359375 - moving ave loss 85.37602954900433 step 1371 - loss 85.06249237060547 - moving ave loss 85.34467583116444 step 1372 - loss 85.20064544677734 - moving ave loss 85.33027279272574 step 1373 - loss 84.62452697753906 - moving ave loss 85.25969821120707 step 1374 - loss 85.04344177246094 - moving ave loss 85.23807256733245 step 1375 - loss 85.70850372314453 - moving ave loss 85.28511568291366 step 1376 - loss 84.71057891845703 - moving ave loss 85.227662006468 step 1377 - loss 85.06367492675781 - moving ave loss 85.21126329849697 step 1378 - loss 85.06138610839844 - moving ave loss 85.19627557948712 step 1379 - loss 84.66355895996094 - moving ave loss 85.1430039175345 step 1380 - loss 85.659423828125 - moving ave loss 85.19464590859354 Checkpoint at step 1380 step 1381 - loss 84.79096221923828 - moving ave loss 85.15427753965801 step 1382 - loss 84.40455627441406 - moving ave loss 85.07930541313362 step 1383 - loss 84.74859619140625 - moving ave loss 85.0462344909609 step 1384 - loss 84.9581069946289 - moving ave loss 85.0374217413277 step 1385 - loss 84.90383911132812 - moving ave loss 85.02406347832775 step 1386 - loss 84.89867401123047 - moving ave loss 85.01152453161802 step 1387 - loss 84.27775573730469 - moving ave loss 84.9381476521867 step 1388 - loss 84.43115997314453 - moving ave loss 84.88744888428248 step 1389 - loss 84.47798156738281 - moving ave loss 84.84650215259252 step 1390 - loss 85.03007507324219 - moving ave loss 84.86485944465748 step 1391 - loss 84.74281311035156 - moving ave loss 84.8526548112269 step 1392 - loss 84.34596252441406 - moving ave loss 84.80198558254563 Checkpoint at step 1392 step 1393 - loss 84.7900161743164 - moving ave loss 84.8007886417227 step 1394 - loss 85.16752624511719 - moving ave loss 84.83746240206216 step 1395 - loss 84.44451904296875 - moving ave loss 84.79816806615281 step 1396 - loss 84.5698013305664 - moving ave loss 84.77533139259418 step 1397 - loss 84.62017822265625 - moving ave loss 84.75981607560038 step 1398 - loss 84.83953857421875 - moving ave loss 84.76778832546223 step 1399 - loss 84.17255401611328 - moving ave loss 84.70826489452735 step 1400 - loss 83.99678039550781 - moving ave loss 84.63711644462539 step 1401 - loss 84.12123107910156 - moving ave loss 84.58552790807302 step 1402 - loss 84.14144897460938 - moving ave loss 84.54112001472666 step 1403 - loss 84.28666687011719 - moving ave loss 84.5156747002657 step 1404 - loss 84.38247680664062 - moving ave loss 84.5023549109032 Checkpoint at step 1404 step 1405 - loss 84.55460357666016 - moving ave loss 84.5075797774789 step 1406 - loss 84.44667053222656 - moving ave loss 84.50148885295367 step 1407 - loss 84.29200744628906 - moving ave loss 84.48054071228721 step 1408 - loss 84.34728240966797 - moving ave loss 84.46721488202527 step 1409 - loss 83.97703552246094 - moving ave loss 84.41819694606885 step 1410 - loss 84.27860260009766 - moving ave loss 84.40423751147173 step 1411 - loss 83.9224624633789 - moving ave loss 84.35606000666246 step 1412 - loss 84.06240844726562 - moving ave loss 84.32669485072279 step 1413 - loss 84.28919982910156 - moving ave loss 84.32294534856067 step 1414 - loss 83.78707122802734 - moving ave loss 84.26935793650733 step 1415 - loss 84.33819580078125 - moving ave loss 84.27624172293473 step 1416 - loss 84.02589416503906 - moving ave loss 84.25120696714515 Checkpoint at step 1416 step 1417 - loss 83.73171997070312 - moving ave loss 84.19925826750095 step 1418 - loss 84.60650634765625 - moving ave loss 84.23998307551649 step 1419 - loss 84.11988830566406 - moving ave loss 84.22797359853125 step 1420 - loss 84.04695129394531 - moving ave loss 84.20987136807265 step 1421 - loss 83.62205505371094 - moving ave loss 84.15108973663648 step 1422 - loss 83.61067199707031 - moving ave loss 84.09704796267987 step 1423 - loss 83.60359191894531 - moving ave loss 84.04770235830641 step 1424 - loss 83.77714538574219 - moving ave loss 84.02064666105 step 1425 - loss 83.66130828857422 - moving ave loss 83.98471282380243 step 1426 - loss 84.16777038574219 - moving ave loss 84.00301857999641 step 1427 - loss 83.55537414550781 - moving ave loss 83.95825413654755 step 1428 - loss 83.8568115234375 - moving ave loss 83.94810987523655 Checkpoint at step 1428 step 1429 - loss 83.8017578125 - moving ave loss 83.9334746689629 step 1430 - loss 83.64459228515625 - moving ave loss 83.90458643058224 step 1431 - loss 83.89278411865234 - moving ave loss 83.90340619938925 step 1432 - loss 83.39016723632812 - moving ave loss 83.85208230308314 step 1433 - loss 83.38211059570312 - moving ave loss 83.80508513234514 step 1434 - loss 83.13265991210938 - moving ave loss 83.73784261032156 step 1435 - loss 83.60191345214844 - moving ave loss 83.72424969450425 step 1436 - loss 83.46446990966797 - moving ave loss 83.69827171602063 step 1437 - loss 83.24822998046875 - moving ave loss 83.65326754246544 step 1438 - loss 83.0374526977539 - moving ave loss 83.5916860579943 step 1439 - loss 83.27963256835938 - moving ave loss 83.56048070903081 step 1440 - loss 83.18726348876953 - moving ave loss 83.52315898700468 Checkpoint at step 1440 step 1441 - loss 83.47901916503906 - moving ave loss 83.51874500480811 step 1442 - loss 83.50685119628906 - moving ave loss 83.51755562395621 step 1443 - loss 82.93740844726562 - moving ave loss 83.45954090628715 step 1444 - loss 83.74050903320312 - moving ave loss 83.48763771897875 step 1445 - loss 83.16858673095703 - moving ave loss 83.45573262017658 step 1446 - loss 83.95149230957031 - moving ave loss 83.50530858911596 step 1447 - loss 83.18335723876953 - moving ave loss 83.47311345408131 step 1448 - loss 83.42611694335938 - moving ave loss 83.46841380300913 step 1449 - loss 82.81398010253906 - moving ave loss 83.40297043296212 step 1450 - loss 82.88523864746094 - moving ave loss 83.35119725441201 step 1451 - loss 83.125 - moving ave loss 83.32857752897081 step 1452 - loss 82.94098663330078 - moving ave loss 83.2898184394038 Checkpoint at step 1452 step 1453 - loss 82.81217956542969 - moving ave loss 83.2420545520064 step 1454 - loss 83.03109741210938 - moving ave loss 83.22095883801668 step 1455 - loss 82.6096420288086 - moving ave loss 83.15982715709588 step 1456 - loss 83.21383666992188 - moving ave loss 83.16522810837849 step 1457 - loss 82.72268676757812 - moving ave loss 83.12097397429845 step 1458 - loss 82.75125122070312 - moving ave loss 83.08400169893892 step 1459 - loss 83.13565063476562 - moving ave loss 83.0891665925216 step 1460 - loss 83.4488525390625 - moving ave loss 83.12513518717569 step 1461 - loss 82.64263153076172 - moving ave loss 83.07688482153429 step 1462 - loss 82.79149627685547 - moving ave loss 83.04834596706641 step 1463 - loss 82.60289001464844 - moving ave loss 83.00380037182461 step 1464 - loss 82.77427673339844 - moving ave loss 82.98084800798199 Checkpoint at step 1464 step 1465 - loss 83.3488540649414 - moving ave loss 83.01764861367793 step 1466 - loss 83.08076477050781 - moving ave loss 83.02396022936092 step 1467 - loss 82.32990264892578 - moving ave loss 82.95455447131741 step 1468 - loss 82.251220703125 - moving ave loss 82.88422109449816 step 1469 - loss 82.7270736694336 - moving ave loss 82.86850635199171 step 1470 - loss 82.66621398925781 - moving ave loss 82.84827711571832 step 1471 - loss 82.57748413085938 - moving ave loss 82.82119781723242 step 1472 - loss 83.16679382324219 - moving ave loss 82.8557574178334 step 1473 - loss 82.32714080810547 - moving ave loss 82.80289575686062 step 1474 - loss 82.49169921875 - moving ave loss 82.77177610304956 step 1475 - loss 82.33711242675781 - moving ave loss 82.72830973542038 step 1476 - loss 82.30193328857422 - moving ave loss 82.68567209073576 Checkpoint at step 1476 step 1477 - loss 82.56144714355469 - moving ave loss 82.67324959601765 step 1478 - loss 82.71273040771484 - moving ave loss 82.67719767718738 step 1479 - loss 82.47235107421875 - moving ave loss 82.65671301689052 step 1480 - loss 82.99486541748047 - moving ave loss 82.69052825694952 Finish 8 epoch(es) step 1481 - loss 82.11557006835938 - moving ave loss 82.6330324380905 step 1482 - loss 82.94673156738281 - moving ave loss 82.66440235101973 step 1483 - loss 82.31915283203125 - moving ave loss 82.62987739912089 step 1484 - loss 82.02796173095703 - moving ave loss 82.5696858323045 step 1485 - loss 82.02860260009766 - moving ave loss 82.51557750908383 step 1486 - loss 82.04203033447266 - moving ave loss 82.46822279162272 step 1487 - loss 82.22088623046875 - moving ave loss 82.44348913550732 step 1488 - loss 82.36980438232422 - moving ave loss 82.43612066018902 Checkpoint at step 1488 step 1489 - loss 82.72872924804688 - moving ave loss 82.46538151897481 step 1490 - loss 81.92813110351562 - moving ave loss 82.41165647742889 step 1491 - loss 82.27763366699219 - moving ave loss 82.39825419638522 step 1492 - loss 81.87043762207031 - moving ave loss 82.34547253895373 step 1493 - loss 82.2326431274414 - moving ave loss 82.33418959780249 step 1494 - loss 82.06522369384766 - moving ave loss 82.30729300740701 step 1495 - loss 82.91949462890625 - moving ave loss 82.36851316955693 step 1496 - loss 82.14954376220703 - moving ave loss 82.34661622882194 step 1497 - loss 82.15113830566406 - moving ave loss 82.32706843650617 step 1498 - loss 82.28374481201172 - moving ave loss 82.32273607405672 step 1499 - loss 81.89946746826172 - moving ave loss 82.28040921347721 step 1500 - loss 82.08247375488281 - moving ave loss 82.26061566761778 Checkpoint at step 1500 step 1501 - loss 82.02799987792969 - moving ave loss 82.23735408864898 step 1502 - loss 81.78380584716797 - moving ave loss 82.19199926450088 step 1503 - loss 81.57257843017578 - moving ave loss 82.13005718106837 step 1504 - loss 82.19601440429688 - moving ave loss 82.13665290339122 step 1505 - loss 81.7334213256836 - moving ave loss 82.09632974562047 step 1506 - loss 81.48931884765625 - moving ave loss 82.03562865582406 step 1507 - loss 81.70947265625 - moving ave loss 82.00301305586666 step 1508 - loss 81.65531921386719 - moving ave loss 81.96824367166671 step 1509 - loss 81.88548278808594 - moving ave loss 81.95996758330864 step 1510 - loss 81.5658950805664 - moving ave loss 81.92056033303442 step 1511 - loss 82.19294738769531 - moving ave loss 81.9477990385005 step 1512 - loss 81.96876525878906 - moving ave loss 81.94989566052936 Checkpoint at step 1512 step 1513 - loss 82.34011840820312 - moving ave loss 81.98891793529674 step 1514 - loss 81.81712341308594 - moving ave loss 81.97173848307567 step 1515 - loss 81.44900512695312 - moving ave loss 81.91946514746341 step 1516 - loss 81.86309814453125 - moving ave loss 81.9138284471702 step 1517 - loss 81.1446533203125 - moving ave loss 81.83691093448444 step 1518 - loss 82.42054748535156 - moving ave loss 81.89527458957114 step 1519 - loss 81.36170959472656 - moving ave loss 81.84191809008668 step 1520 - loss 80.89987182617188 - moving ave loss 81.7477134636952 step 1521 - loss 81.56388854980469 - moving ave loss 81.72933097230614 step 1522 - loss 81.10812377929688 - moving ave loss 81.66721025300522 step 1523 - loss 81.47175598144531 - moving ave loss 81.64766482584923 step 1524 - loss 81.58212280273438 - moving ave loss 81.64111062353774 Checkpoint at step 1524 step 1525 - loss 82.18560028076172 - moving ave loss 81.69555958926014 step 1526 - loss 80.74386596679688 - moving ave loss 81.60039022701382 step 1527 - loss 80.81913757324219 - moving ave loss 81.52226496163665 step 1528 - loss 81.01704406738281 - moving ave loss 81.47174287221127 step 1529 - loss 80.95394897460938 - moving ave loss 81.41996348245108 step 1530 - loss 80.990234375 - moving ave loss 81.37699057170597 step 1531 - loss 80.91796875 - moving ave loss 81.33108838953538 step 1532 - loss 81.35405731201172 - moving ave loss 81.33338528178302 step 1533 - loss 81.03174591064453 - moving ave loss 81.30322134466918 step 1534 - loss 80.99920654296875 - moving ave loss 81.27281986449914 step 1535 - loss 81.13904571533203 - moving ave loss 81.25944244958242 step 1536 - loss 80.58348083496094 - moving ave loss 81.19184628812029 Checkpoint at step 1536 step 1537 - loss 81.41168212890625 - moving ave loss 81.21382987219889 step 1538 - loss 80.6843490600586 - moving ave loss 81.16088179098486 step 1539 - loss 81.0176773071289 - moving ave loss 81.14656134259927 step 1540 - loss 80.65121459960938 - moving ave loss 81.09702666830027 step 1541 - loss 81.05618286132812 - moving ave loss 81.09294228760305 step 1542 - loss 80.60709381103516 - moving ave loss 81.04435743994627 step 1543 - loss 80.92095947265625 - moving ave loss 81.03201764321726 step 1544 - loss 80.53865814208984 - moving ave loss 80.98268169310452 step 1545 - loss 80.54705810546875 - moving ave loss 80.93911933434096 step 1546 - loss 80.26313018798828 - moving ave loss 80.87152041970569 step 1547 - loss 80.5727767944336 - moving ave loss 80.84164605717848 step 1548 - loss 80.40923309326172 - moving ave loss 80.79840476078681 Checkpoint at step 1548 step 1549 - loss 80.35853576660156 - moving ave loss 80.7544178613683 step 1550 - loss 80.56929016113281 - moving ave loss 80.73590509134476 step 1551 - loss 80.40313720703125 - moving ave loss 80.70262830291341 step 1552 - loss 80.2999267578125 - moving ave loss 80.66235814840331 step 1553 - loss 81.10337829589844 - moving ave loss 80.70646016315283 step 1554 - loss 80.47711181640625 - moving ave loss 80.68352532847817 step 1555 - loss 80.23484802246094 - moving ave loss 80.63865759787645 step 1556 - loss 80.42886352539062 - moving ave loss 80.61767819062786 step 1557 - loss 79.97197723388672 - moving ave loss 80.55310809495374 step 1558 - loss 79.9583969116211 - moving ave loss 80.49363697662048 step 1559 - loss 80.15098571777344 - moving ave loss 80.45937185073578 step 1560 - loss 81.13595581054688 - moving ave loss 80.52703024671689 Checkpoint at step 1560 step 1561 - loss 80.13232421875 - moving ave loss 80.4875596439202 step 1562 - loss 80.44447326660156 - moving ave loss 80.48325100618835 step 1563 - loss 79.9787368774414 - moving ave loss 80.43279959331366 step 1564 - loss 80.43941497802734 - moving ave loss 80.43346113178504 step 1565 - loss 80.13356018066406 - moving ave loss 80.40347103667294 step 1566 - loss 80.13270568847656 - moving ave loss 80.37639450185331 step 1567 - loss 80.46795654296875 - moving ave loss 80.38555070596486 step 1568 - loss 79.85018920898438 - moving ave loss 80.33201455626681 step 1569 - loss 80.2446517944336 - moving ave loss 80.32327828008349 step 1570 - loss 80.13164520263672 - moving ave loss 80.30411497233881 step 1571 - loss 79.61774444580078 - moving ave loss 80.235477919685 step 1572 - loss 80.1358413696289 - moving ave loss 80.22551426467939 Checkpoint at step 1572 step 1573 - loss 79.70690155029297 - moving ave loss 80.17365299324075 step 1574 - loss 80.47114562988281 - moving ave loss 80.20340225690497 step 1575 - loss 80.31353759765625 - moving ave loss 80.2144157909801 step 1576 - loss 79.82546997070312 - moving ave loss 80.1755212089524 step 1577 - loss 79.80436706542969 - moving ave loss 80.13840579460015 step 1578 - loss 79.88444519042969 - moving ave loss 80.11300973418311 step 1579 - loss 79.82400512695312 - moving ave loss 80.0841092734601 step 1580 - loss 79.69331359863281 - moving ave loss 80.04502970597737 step 1581 - loss 79.90491485595703 - moving ave loss 80.03101822097533 step 1582 - loss 79.9302978515625 - moving ave loss 80.02094618403405 step 1583 - loss 79.96324157714844 - moving ave loss 80.01517572334548 step 1584 - loss 80.12924194335938 - moving ave loss 80.02658234534687 Checkpoint at step 1584 step 1585 - loss 79.74886322021484 - moving ave loss 79.99881043283368 step 1586 - loss 79.43714904785156 - moving ave loss 79.94264429433547 step 1587 - loss 79.671875 - moving ave loss 79.91556736490192 step 1588 - loss 79.35319519042969 - moving ave loss 79.85933014745468 step 1589 - loss 79.14108276367188 - moving ave loss 79.7875054090764 step 1590 - loss 79.81867980957031 - moving ave loss 79.7906228491258 step 1591 - loss 79.85736846923828 - moving ave loss 79.79729741113705 step 1592 - loss 79.23606872558594 - moving ave loss 79.74117454258194 step 1593 - loss 79.2403564453125 - moving ave loss 79.691092732855 step 1594 - loss 79.43304443359375 - moving ave loss 79.66528790292888 step 1595 - loss 79.43800354003906 - moving ave loss 79.64255946663991 step 1596 - loss 79.49774169921875 - moving ave loss 79.62807768989781 Checkpoint at step 1596 step 1597 - loss 79.45313262939453 - moving ave loss 79.61058318384748 step 1598 - loss 79.28302764892578 - moving ave loss 79.57782763035532 step 1599 - loss 79.26544189453125 - moving ave loss 79.54658905677292 step 1600 - loss 79.00069427490234 - moving ave loss 79.49199957858586 step 1601 - loss 79.46351623535156 - moving ave loss 79.48915124426243 step 1602 - loss 78.74311065673828 - moving ave loss 79.41454718551002 step 1603 - loss 79.31788635253906 - moving ave loss 79.40488110221293 step 1604 - loss 79.38880920410156 - moving ave loss 79.40327391240179 step 1605 - loss 78.93798828125 - moving ave loss 79.35674534928661 step 1606 - loss 79.2636489868164 - moving ave loss 79.34743571303959 step 1607 - loss 78.86247253417969 - moving ave loss 79.2989393951536 step 1608 - loss 79.05992126464844 - moving ave loss 79.27503758210308 Checkpoint at step 1608 step 1609 - loss 78.78880310058594 - moving ave loss 79.22641413395137 step 1610 - loss 79.43989562988281 - moving ave loss 79.2477622835445 step 1611 - loss 79.55904388427734 - moving ave loss 79.2788904436178 step 1612 - loss 78.73402404785156 - moving ave loss 79.22440380404117 step 1613 - loss 78.9271469116211 - moving ave loss 79.19467811479917 step 1614 - loss 78.72406768798828 - moving ave loss 79.14761707211808 step 1615 - loss 78.62752532958984 - moving ave loss 79.09560789786525 step 1616 - loss 78.873291015625 - moving ave loss 79.07337620964122 step 1617 - loss 79.63758850097656 - moving ave loss 79.12979743877477 step 1618 - loss 79.05773162841797 - moving ave loss 79.1225908577391 step 1619 - loss 78.81298065185547 - moving ave loss 79.09162983715073 step 1620 - loss 78.80612182617188 - moving ave loss 79.06307903605284 Checkpoint at step 1620 step 1621 - loss 78.63288879394531 - moving ave loss 79.02006001184209 step 1622 - loss 78.62317657470703 - moving ave loss 78.98037166812858 step 1623 - loss 78.92474365234375 - moving ave loss 78.9748088665501 step 1624 - loss 78.50418853759766 - moving ave loss 78.92774683365487 step 1625 - loss 79.41062927246094 - moving ave loss 78.97603507753547 step 1626 - loss 79.32951354980469 - moving ave loss 79.0113829247624 step 1627 - loss 78.5863265991211 - moving ave loss 78.96887729219827 step 1628 - loss 78.69389343261719 - moving ave loss 78.94137890624016 step 1629 - loss 78.07667541503906 - moving ave loss 78.85490855712005 step 1630 - loss 78.46073913574219 - moving ave loss 78.81549161498226 step 1631 - loss 79.02664184570312 - moving ave loss 78.83660663805435 step 1632 - loss 78.45897674560547 - moving ave loss 78.79884364880947 Checkpoint at step 1632 step 1633 - loss 78.19766998291016 - moving ave loss 78.73872628221953 step 1634 - loss 78.34223175048828 - moving ave loss 78.69907682904642 step 1635 - loss 78.30236053466797 - moving ave loss 78.65940519960857 step 1636 - loss 78.15931701660156 - moving ave loss 78.60939638130787 step 1637 - loss 78.1116943359375 - moving ave loss 78.55962617677083 step 1638 - loss 77.86189270019531 - moving ave loss 78.48985282911329 step 1639 - loss 78.21527862548828 - moving ave loss 78.4623954087508 step 1640 - loss 78.43164825439453 - moving ave loss 78.45932069331518 step 1641 - loss 77.94058227539062 - moving ave loss 78.40744685152274 step 1642 - loss 77.71946716308594 - moving ave loss 78.33864888267905 step 1643 - loss 77.67970275878906 - moving ave loss 78.27275427029006 step 1644 - loss 78.10951232910156 - moving ave loss 78.25643007617121 Checkpoint at step 1644 step 1645 - loss 77.87161254882812 - moving ave loss 78.21794832343691 step 1646 - loss 77.79042053222656 - moving ave loss 78.17519554431588 step 1647 - loss 78.41156005859375 - moving ave loss 78.19883199574367 step 1648 - loss 80.6184310913086 - moving ave loss 78.44079190530017 step 1649 - loss 77.71392059326172 - moving ave loss 78.36810477409634 step 1650 - loss 77.47605895996094 - moving ave loss 78.2789001926828 step 1651 - loss 77.49937438964844 - moving ave loss 78.20094761237937 step 1652 - loss 77.7137680053711 - moving ave loss 78.15222965167854 step 1653 - loss 77.64540100097656 - moving ave loss 78.10154678660835 step 1654 - loss 77.78929138183594 - moving ave loss 78.07032124613112 step 1655 - loss 77.5848388671875 - moving ave loss 78.02177300823675 step 1656 - loss 77.36271667480469 - moving ave loss 77.95586737489354 Checkpoint at step 1656 step 1657 - loss 77.97238159179688 - moving ave loss 77.95751879658387 step 1658 - loss 77.66802978515625 - moving ave loss 77.92856989544111 step 1659 - loss 77.45591735839844 - moving ave loss 77.88130464173685 step 1660 - loss 77.41122436523438 - moving ave loss 77.83429661408661 step 1661 - loss 77.5269775390625 - moving ave loss 77.8035647065842 step 1662 - loss 77.54647827148438 - moving ave loss 77.77785606307421 step 1663 - loss 77.62275695800781 - moving ave loss 77.76234615256757 step 1664 - loss 77.9901123046875 - moving ave loss 77.78512276777957 step 1665 - loss 77.32134246826172 - moving ave loss 77.73874473782779 Finish 9 epoch(es) step 1666 - loss 77.42930603027344 - moving ave loss 77.70780086707235 step 1667 - loss 77.6712417602539 - moving ave loss 77.70414495639051 step 1668 - loss 77.24789428710938 - moving ave loss 77.6585198894624 Checkpoint at step 1668 step 1669 - loss 78.76517486572266 - moving ave loss 77.76918538708843 step 1670 - loss 77.38816833496094 - moving ave loss 77.73108368187567 step 1671 - loss 77.35566711425781 - moving ave loss 77.69354202511389 step 1672 - loss 76.9260025024414 - moving ave loss 77.61678807284665 step 1673 - loss 77.53118896484375 - moving ave loss 77.60822816204636 step 1674 - loss 77.06787109375 - moving ave loss 77.55419245521672 step 1675 - loss 77.3197021484375 - moving ave loss 77.53074342453881 step 1676 - loss 77.23822784423828 - moving ave loss 77.50149186650876 step 1677 - loss 77.19337463378906 - moving ave loss 77.47068014323679 step 1678 - loss 77.5582275390625 - moving ave loss 77.47943488281936 step 1679 - loss 77.40300750732422 - moving ave loss 77.47179214526984 step 1680 - loss 76.94639587402344 - moving ave loss 77.41925251814521 Checkpoint at step 1680 step 1681 - loss 76.91111755371094 - moving ave loss 77.36843902170179 step 1682 - loss 77.26092529296875 - moving ave loss 77.35768764882849 step 1683 - loss 76.60166931152344 - moving ave loss 77.28208581509799 step 1684 - loss 77.06082153320312 - moving ave loss 77.2599593869085 step 1685 - loss 77.15339660644531 - moving ave loss 77.2493031088622 step 1686 - loss 76.74488067626953 - moving ave loss 77.19886086560292 step 1687 - loss 76.7050552368164 - moving ave loss 77.14948030272427 step 1688 - loss 76.69086456298828 - moving ave loss 77.10361872875067 step 1689 - loss 76.63394165039062 - moving ave loss 77.05665102091467 step 1690 - loss 76.88439178466797 - moving ave loss 77.03942509729 step 1691 - loss 77.27232360839844 - moving ave loss 77.06271494840085 step 1692 - loss 76.54493713378906 - moving ave loss 77.01093716693968 Checkpoint at step 1692 step 1693 - loss 77.11567687988281 - moving ave loss 77.021411138234 step 1694 - loss 76.75822448730469 - moving ave loss 76.99509247314107 step 1695 - loss 76.49050903320312 - moving ave loss 76.94463412914727 step 1696 - loss 76.48310852050781 - moving ave loss 76.89848156828333 step 1697 - loss 76.48383331298828 - moving ave loss 76.85701674275383 step 1698 - loss 77.17623901367188 - moving ave loss 76.88893896984564 step 1699 - loss 76.78241729736328 - moving ave loss 76.8782868025974 step 1700 - loss 76.57807159423828 - moving ave loss 76.84826528176148 step 1701 - loss 76.104248046875 - moving ave loss 76.77386355827284 step 1702 - loss 77.12959289550781 - moving ave loss 76.80943649199634 step 1703 - loss 76.49752807617188 - moving ave loss 76.7782456504139 step 1704 - loss 76.47624969482422 - moving ave loss 76.74804605485492 Checkpoint at step 1704 step 1705 - loss 76.28923034667969 - moving ave loss 76.7021644840374 step 1706 - loss 77.41645812988281 - moving ave loss 76.77359384862194 step 1707 - loss 75.995361328125 - moving ave loss 76.69577059657226 step 1708 - loss 76.75375366210938 - moving ave loss 76.70156890312597 step 1709 - loss 76.14178466796875 - moving ave loss 76.64559047961025 step 1710 - loss 75.88565063476562 - moving ave loss 76.5695964951258 step 1711 - loss 76.4671401977539 - moving ave loss 76.55935086538861 step 1712 - loss 76.22156524658203 - moving ave loss 76.52557230350796 step 1713 - loss 76.19093322753906 - moving ave loss 76.49210839591107 step 1714 - loss 76.79853820800781 - moving ave loss 76.52275137712076 step 1715 - loss 76.51075744628906 - moving ave loss 76.5215519840376 step 1716 - loss 76.12977600097656 - moving ave loss 76.48237438573149 Checkpoint at step 1716 step 1717 - loss 76.758056640625 - moving ave loss 76.50994261122084 step 1718 - loss 76.14253997802734 - moving ave loss 76.47320234790149 step 1719 - loss 75.8846206665039 - moving ave loss 76.41434417976174 step 1720 - loss 76.48326110839844 - moving ave loss 76.42123587262542 step 1721 - loss 75.86480712890625 - moving ave loss 76.36559299825349 step 1722 - loss 75.79051971435547 - moving ave loss 76.30808566986369 step 1723 - loss 76.23442077636719 - moving ave loss 76.30071918051405 step 1724 - loss 76.37120056152344 - moving ave loss 76.30776731861499 step 1725 - loss 75.47055053710938 - moving ave loss 76.22404564046442 step 1726 - loss 75.65554809570312 - moving ave loss 76.1671958859883 step 1727 - loss 75.64501953125 - moving ave loss 76.11497825051448 step 1728 - loss 76.28106689453125 - moving ave loss 76.13158711491616 Checkpoint at step 1728 step 1729 - loss 75.62126159667969 - moving ave loss 76.08055456309252 step 1730 - loss 75.5517349243164 - moving ave loss 76.02767259921491 step 1731 - loss 76.28638458251953 - moving ave loss 76.05354379754537 step 1732 - loss 75.78872680664062 - moving ave loss 76.0270620984549 step 1733 - loss 75.51983642578125 - moving ave loss 75.97633953118753 step 1734 - loss 75.70242309570312 - moving ave loss 75.94894788763911 step 1735 - loss 75.41046905517578 - moving ave loss 75.89510000439277 step 1736 - loss 76.24480438232422 - moving ave loss 75.93007044218592 step 1737 - loss 75.58493041992188 - moving ave loss 75.89555643995952 step 1738 - loss 76.14442443847656 - moving ave loss 75.92044323981122 step 1739 - loss 76.1319580078125 - moving ave loss 75.94159471661136 step 1740 - loss 75.49679565429688 - moving ave loss 75.89711481037992 Checkpoint at step 1740 step 1741 - loss 75.31509399414062 - moving ave loss 75.83891272875599 step 1742 - loss 75.50641632080078 - moving ave loss 75.80566308796047 step 1743 - loss 75.18031311035156 - moving ave loss 75.74312809019958 step 1744 - loss 75.49774169921875 - moving ave loss 75.7185894511015 step 1745 - loss 75.58910369873047 - moving ave loss 75.70564087586439 step 1746 - loss 75.13729858398438 - moving ave loss 75.64880664667639 step 1747 - loss 75.27853393554688 - moving ave loss 75.61177937556344 step 1748 - loss 74.90939331054688 - moving ave loss 75.5415407690618 step 1749 - loss 75.75018310546875 - moving ave loss 75.5624050027025 step 1750 - loss 75.73741912841797 - moving ave loss 75.57990641527405 step 1751 - loss 75.00714111328125 - moving ave loss 75.52262988507478 step 1752 - loss 75.24519348144531 - moving ave loss 75.49488624471184 Checkpoint at step 1752 step 1753 - loss 76.0329360961914 - moving ave loss 75.5486912298598 step 1754 - loss 75.3665771484375 - moving ave loss 75.53047982171758 step 1755 - loss 75.09748840332031 - moving ave loss 75.48718067987785 step 1756 - loss 74.91004943847656 - moving ave loss 75.42946755573773 step 1757 - loss 74.82925415039062 - moving ave loss 75.36944621520303 step 1758 - loss 74.75823974609375 - moving ave loss 75.30832556829209 step 1759 - loss 75.01040649414062 - moving ave loss 75.27853366087695 step 1760 - loss 74.95039367675781 - moving ave loss 75.24571966246504 step 1761 - loss 74.7092514038086 - moving ave loss 75.1920728365994 step 1762 - loss 74.64935302734375 - moving ave loss 75.13780085567383 step 1763 - loss 74.87852478027344 - moving ave loss 75.1118732481338 step 1764 - loss 75.2609634399414 - moving ave loss 75.12678226731457 Checkpoint at step 1764 step 1765 - loss 74.82005310058594 - moving ave loss 75.09610935064171 step 1766 - loss 75.06562805175781 - moving ave loss 75.09306122075331 step 1767 - loss 74.42910766601562 - moving ave loss 75.02666586527954 step 1768 - loss 74.3704833984375 - moving ave loss 74.96104761859534 step 1769 - loss 74.69662475585938 - moving ave loss 74.93460533232175 step 1770 - loss 74.88568115234375 - moving ave loss 74.92971291432396 step 1771 - loss 74.9225082397461 - moving ave loss 74.92899244686618 step 1772 - loss 74.86721801757812 - moving ave loss 74.92281500393737 step 1773 - loss 74.36610412597656 - moving ave loss 74.8671439161413 step 1774 - loss 74.31832122802734 - moving ave loss 74.8122616473299 step 1775 - loss 74.55059051513672 - moving ave loss 74.78609453411059 step 1776 - loss 74.07542419433594 - moving ave loss 74.71502750013313 Checkpoint at step 1776 step 1777 - loss 74.24766540527344 - moving ave loss 74.66829129064716 step 1778 - loss 74.91511535644531 - moving ave loss 74.69297369722698 step 1779 - loss 75.23258972167969 - moving ave loss 74.74693529967224 step 1780 - loss 74.63880920410156 - moving ave loss 74.73612269011518 step 1781 - loss 74.45452117919922 - moving ave loss 74.7079625390236 step 1782 - loss 74.56866455078125 - moving ave loss 74.69403274019936 step 1783 - loss 73.8749008178711 - moving ave loss 74.61211954796653 step 1784 - loss 74.77444458007812 - moving ave loss 74.62835205117769 step 1785 - loss 74.32937622070312 - moving ave loss 74.59845446813023 step 1786 - loss 74.04112243652344 - moving ave loss 74.54272126496956 step 1787 - loss 73.89151000976562 - moving ave loss 74.47760013944917 step 1788 - loss 74.05137634277344 - moving ave loss 74.43497775978159 Checkpoint at step 1788 step 1789 - loss 73.76985168457031 - moving ave loss 74.36846515226046 step 1790 - loss 73.88674926757812 - moving ave loss 74.32029356379223 step 1791 - loss 73.71884155273438 - moving ave loss 74.26014836268645 step 1792 - loss 74.52913665771484 - moving ave loss 74.28704719218929 step 1793 - loss 73.83325958251953 - moving ave loss 74.24166843122232 step 1794 - loss 73.73706817626953 - moving ave loss 74.19120840572704 step 1795 - loss 74.04966735839844 - moving ave loss 74.17705430099419 step 1796 - loss 74.20782470703125 - moving ave loss 74.1801313415979 step 1797 - loss 74.19235229492188 - moving ave loss 74.1813534369303 step 1798 - loss 73.8838882446289 - moving ave loss 74.15160691770016 step 1799 - loss 73.46316528320312 - moving ave loss 74.08276275425047 step 1800 - loss 73.84053039550781 - moving ave loss 74.05853951837621 Checkpoint at step 1800 step 1801 - loss 73.87773132324219 - moving ave loss 74.0404586988628 step 1802 - loss 73.537841796875 - moving ave loss 73.99019700866403 step 1803 - loss 74.62770080566406 - moving ave loss 74.05394738836402 step 1804 - loss 74.14259338378906 - moving ave loss 74.06281198790653 step 1805 - loss 73.90630340576172 - moving ave loss 74.04716112969204 step 1806 - loss 73.69194030761719 - moving ave loss 74.01163904748456 step 1807 - loss 74.287841796875 - moving ave loss 74.0392593224236 step 1808 - loss 73.65873718261719 - moving ave loss 74.00120710844297 step 1809 - loss 73.84561157226562 - moving ave loss 73.98564755482523 step 1810 - loss 74.65031433105469 - moving ave loss 74.05211423244818 step 1811 - loss 73.65147399902344 - moving ave loss 74.0120502091057 step 1812 - loss 73.497314453125 - moving ave loss 73.96057663350763 Checkpoint at step 1812 step 1813 - loss 73.30215454101562 - moving ave loss 73.89473442425843 step 1814 - loss 73.55902099609375 - moving ave loss 73.86116308144197 step 1815 - loss 73.58641052246094 - moving ave loss 73.83368782554388 step 1816 - loss 74.30105590820312 - moving ave loss 73.88042463380981 step 1817 - loss 73.64169311523438 - moving ave loss 73.85655148195227 step 1818 - loss 73.35405731201172 - moving ave loss 73.80630206495822 step 1819 - loss 73.37177276611328 - moving ave loss 73.76284913507372 step 1820 - loss 73.09721374511719 - moving ave loss 73.69628559607807 step 1821 - loss 73.96670532226562 - moving ave loss 73.72332756869683 step 1822 - loss 73.4549560546875 - moving ave loss 73.6964904172959 step 1823 - loss 73.24995422363281 - moving ave loss 73.6518367979296 step 1824 - loss 73.24771118164062 - moving ave loss 73.6114242363007 Checkpoint at step 1824 step 1825 - loss 72.97657775878906 - moving ave loss 73.54793958854954 step 1826 - loss 73.21296691894531 - moving ave loss 73.51444232158912 step 1827 - loss 73.555419921875 - moving ave loss 73.5185400816177 step 1828 - loss 73.4560546875 - moving ave loss 73.51229154220593 step 1829 - loss 73.5794677734375 - moving ave loss 73.5190091653291 step 1830 - loss 73.14073181152344 - moving ave loss 73.48118142994852 step 1831 - loss 73.25155639648438 - moving ave loss 73.45821892660211 step 1832 - loss 72.82179260253906 - moving ave loss 73.39457629419582 step 1833 - loss 72.58578491210938 - moving ave loss 73.31369715598717 step 1834 - loss 72.95674896240234 - moving ave loss 73.27800233662869 step 1835 - loss 72.66548919677734 - moving ave loss 73.21675102264355 step 1836 - loss 72.44499969482422 - moving ave loss 73.13957588986162 Checkpoint at step 1836 step 1837 - loss 73.169921875 - moving ave loss 73.14261048837547 step 1838 - loss 73.26011657714844 - moving ave loss 73.15436109725276 step 1839 - loss 72.77566528320312 - moving ave loss 73.1164915158478 step 1840 - loss 72.63217163085938 - moving ave loss 73.06805952734896 step 1841 - loss 73.21997833251953 - moving ave loss 73.08325140786603 step 1842 - loss 72.98775482177734 - moving ave loss 73.07370174925715 step 1843 - loss 72.23185729980469 - moving ave loss 72.98951730431192 step 1844 - loss 72.8983154296875 - moving ave loss 72.98039711684949 step 1845 - loss 72.43527221679688 - moving ave loss 72.92588462684422 step 1846 - loss 72.59123992919922 - moving ave loss 72.89242015707973 step 1847 - loss 72.16282653808594 - moving ave loss 72.81946079518035 step 1848 - loss 72.73353576660156 - moving ave loss 72.81086829232247 Checkpoint at step 1848 step 1849 - loss 72.48533630371094 - moving ave loss 72.77831509346132 step 1850 - loss 73.43997192382812 - moving ave loss 72.844480776498 Finish 10 epoch(es) Checkpoint at step 1850 Training finished, exit.

kilianyp commented 6 years ago

@gangooteli thanks I should have been clearer, but I wanted to compare my training log against one that achieved close to paper accuracy. But I guess thats good enough, as it looks so far pretty much as mine and in some other issues people mentioned how much they trained it.

aswin-raghavan commented 5 years ago

FYI: I used ADAM and was able to reach loss close to 2.0, but the detections were not great. Happy to provide log if someone is still interested.