Closed dongzhuoyao closed 6 years ago
fix 1: 2/3=0 error.
fix 2: beta default value should not be zero.
before fix log:
Begin to run SVRG-BB: Info: set m=n by default Epoch.: 0, Step size: 1.00e-03, Grad. norm: 4.31e-01, Func. value: 6.931472e-01 Epoch.: 1, Step size: 3.40e-03, Grad. norm: 3.21e-01, Func. value: 5.531818e-01 Epoch.: 2, Step size: 4.88e-03, Grad. norm: 1.68e-01, Func. value: 3.759994e-01 Epoch.: 3, Step size: 8.70e-03, Grad. norm: 9.85e-02, Func. value: 2.968144e-01 Epoch.: 4, Step size: 1.39e-02, Grad. norm: 5.55e-02, Func. value: 2.502696e-01 Epoch.: 5, Step size: 1.99e-02, Grad. norm: 2.97e-02, Func. value: 2.278909e-01 Epoch.: 6, Step size: 2.62e-02, Grad. norm: 1.55e-02, Func. value: 2.193573e-01 Epoch.: 7, Step size: 2.78e-02, Grad. norm: 7.71e-03, Func. value: 2.167048e-01 Epoch.: 8, Step size: 2.53e-02, Grad. norm: 3.91e-03, Func. value: 2.161033e-01 Epoch.: 9, Step size: 2.39e-02, Grad. norm: 2.15e-03, Func. value: 2.159635e-01 Epoch.: 10, Step size: 2.34e-02, Grad. norm: 1.10e-03, Func. value: 2.159267e-01 Epoch.: 11, Step size: 2.52e-02, Grad. norm: 5.44e-04, Func. value: 2.159164e-01 Epoch.: 12, Step size: 2.61e-02, Grad. norm: 2.94e-04, Func. value: 2.159134e-01 Epoch.: 13, Step size: 2.62e-02, Grad. norm: 1.60e-04, Func. value: 2.159126e-01 Epoch.: 14, Step size: 2.52e-02, Grad. norm: 8.38e-05, Func. value: 2.159124e-01 Epoch.: 15, Step size: 2.32e-02, Grad. norm: 4.48e-05, Func. value: 2.159123e-01 Epoch.: 16, Step size: 2.32e-02, Grad. norm: 2.31e-05, Func. value: 2.159123e-01 Epoch.: 17, Step size: 2.39e-02, Grad. norm: 1.23e-05, Func. value: 2.159123e-01 Epoch.: 18, Step size: 2.63e-02, Grad. norm: 6.06e-06, Func. value: 2.159123e-01 Epoch.: 19, Step size: 2.75e-02, Grad. norm: 3.25e-06, Func. value: 2.159123e-01 Epoch.: 20, Step size: 2.68e-02, Grad. norm: 1.67e-06, Func. value: 2.159123e-01 Epoch.: 21, Step size: 2.45e-02, Grad. norm: 9.16e-07, Func. value: 2.159123e-01 Epoch.: 22, Step size: 2.28e-02, Grad. norm: 4.68e-07, Func. value: 2.159123e-01 Epoch.: 23, Step size: 2.30e-02, Grad. norm: 2.26e-07, Func. value: 2.159123e-01 Epoch.: 24, Step size: 2.60e-02, Grad. norm: 1.15e-07, Func. value: 2.159123e-01 Epoch.: 25, Step size: 3.05e-02, Grad. norm: 6.31e-08, Func. value: 2.159123e-01 Epoch.: 26, Step size: 2.61e-02, Grad. norm: 3.53e-08, Func. value: 2.159123e-01 Epoch.: 27, Step size: 1.98e-02, Grad. norm: 1.66e-08, Func. value: 2.159123e-01 Epoch.: 28, Step size: 2.25e-02, Grad. norm: 8.76e-09, Func. value: 2.159123e-01 Epoch.: 29, Step size: 2.59e-02, Grad. norm: 4.59e-09, Func. value: 2.159123e-01 Epoch.: 30, Step size: 2.88e-02, Grad. norm: 2.53e-09, Func. value: 2.159123e-01 Epoch.: 31, Step size: 2.71e-02, Grad. norm: 1.26e-09, Func. value: 2.159123e-01 Epoch.: 32, Step size: 2.23e-02, Grad. norm: 6.29e-10, Func. value: 2.159123e-01 Epoch.: 33, Step size: 2.26e-02, Grad. norm: 3.40e-10, Func. value: 2.159123e-01 Epoch.: 34, Step size: 2.46e-02, Grad. norm: 1.66e-10, Func. value: 2.159123e-01 Epoch.: 35, Step size: 2.79e-02, Grad. norm: 8.54e-11, Func. value: 2.159123e-01 Epoch.: 36, Step size: 2.87e-02, Grad. norm: 4.36e-11, Func. value: 2.159123e-01 Epoch.: 37, Step size: 2.81e-02, Grad. norm: 2.42e-11, Func. value: 2.159123e-01 Epoch.: 38, Step size: 2.35e-02, Grad. norm: 1.22e-11, Func. value: 2.159123e-01 Epoch.: 39, Step size: 2.17e-02, Grad. norm: 5.89e-12, Func. value: 2.159123e-01 Epoch.: 40, Step size: 2.53e-02, Grad. norm: 3.15e-12, Func. value: 2.159123e-01 Epoch.: 41, Step size: 2.46e-02, Grad. norm: 1.64e-12, Func. value: 2.159123e-01 Epoch.: 42, Step size: 2.63e-02, Grad. norm: 8.22e-13, Func. value: 2.159123e-01 Epoch.: 43, Step size: 2.74e-02, Grad. norm: 4.36e-13, Func. value: 2.159123e-01 Epoch.: 44, Step size: 2.41e-02, Grad. norm: 2.38e-13, Func. value: 2.159123e-01 Epoch.: 45, Step size: 2.26e-02, Grad. norm: 1.19e-13, Func. value: 2.159123e-01 Epoch.: 46, Step size: 2.52e-02, Grad. norm: 5.95e-14, Func. value: 2.159123e-01 Epoch.: 47, Step size: 2.83e-02, Grad. norm: 3.38e-14, Func. value: 2.159123e-01 Epoch.: 48, Step size: 2.63e-02, Grad. norm: 1.81e-14, Func. value: 2.159123e-01 Epoch.: 49, Step size: 2.04e-02, Grad. norm: 9.91e-15, Func. value: 2.159123e-01 Test accuracy: 0.000000
Begin to run SGD-BB: Info: set m=n by default Info: set beta=10/m by default Epoch.: 0, Step size: 1.00e-03, Grad. norm: 4.31e-01, Func. value: 6.931472e-01 Epoch.: 1, Step size: 1.00e-03, Grad. norm: 3.21e-01, Func. value: 5.526788e-01 Epoch.: 2, Step size: inf, Grad. norm: 2.53e-01, Func. value: 4.723846e-01 Epoch.: 3, Step size: 3.33e-01, Grad. norm: nan, Func. value: nan Epoch.: 4, Step size: 2.50e-01, Grad. norm: nan, Func. value: nan Epoch.: 5, Step size: 2.00e-01, Grad. norm: nan, Func. value: nan Epoch.: 6, Step size: 1.67e-01, Grad. norm: nan, Func. value: nan Epoch.: 7, Step size: 1.43e-01, Grad. norm: nan, Func. value: nan Epoch.: 8, Step size: 1.25e-01, Grad. norm: nan, Func. value: nan Epoch.: 9, Step size: 1.11e-01, Grad. norm: nan, Func. value: nan Epoch.: 10, Step size: 1.00e-01, Grad. norm: nan, Func. value: nan Epoch.: 11, Step size: 9.09e-02, Grad. norm: nan, Func. value: nan Epoch.: 12, Step size: 8.33e-02, Grad. norm: nan, Func. value: nan Epoch.: 13, Step size: 7.69e-02, Grad. norm: nan, Func. value: nan Epoch.: 14, Step size: 7.14e-02, Grad. norm: nan, Func. value: nan Epoch.: 15, Step size: 6.67e-02, Grad. norm: nan, Func. value: nan Epoch.: 16, Step size: 6.25e-02, Grad. norm: nan, Func. value: nan Epoch.: 17, Step size: 5.88e-02, Grad. norm: nan, Func. value: nan Epoch.: 18, Step size: 5.56e-02, Grad. norm: nan, Func. value: nan Epoch.: 19, Step size: 5.26e-02, Grad. norm: nan, Func. value: nan Epoch.: 20, Step size: 5.00e-02, Grad. norm: nan, Func. value: nan Epoch.: 21, Step size: 4.76e-02, Grad. norm: nan, Func. value: nan Epoch.: 22, Step size: 4.55e-02, Grad. norm: nan, Func. value: nan Epoch.: 23, Step size: 4.35e-02, Grad. norm: nan, Func. value: nan Epoch.: 24, Step size: 4.17e-02, Grad. norm: nan, Func. value: nan Epoch.: 25, Step size: 4.00e-02, Grad. norm: nan, Func. value: nan Epoch.: 26, Step size: 3.85e-02, Grad. norm: nan, Func. value: nan Epoch.: 27, Step size: 3.70e-02, Grad. norm: nan, Func. value: nan Epoch.: 28, Step size: 3.57e-02, Grad. norm: nan, Func. value: nan Epoch.: 29, Step size: 3.45e-02, Grad. norm: nan, Func. value: nan Epoch.: 30, Step size: 3.33e-02, Grad. norm: nan, Func. value: nan Epoch.: 31, Step size: 3.23e-02, Grad. norm: nan, Func. value: nan Epoch.: 32, Step size: 3.12e-02, Grad. norm: nan, Func. value: nan Epoch.: 33, Step size: 3.03e-02, Grad. norm: nan, Func. value: nan Epoch.: 34, Step size: 2.94e-02, Grad. norm: nan, Func. value: nan Epoch.: 35, Step size: 2.86e-02, Grad. norm: nan, Func. value: nan Epoch.: 36, Step size: 2.78e-02, Grad. norm: nan, Func. value: nan Epoch.: 37, Step size: 2.70e-02, Grad. norm: nan, Func. value: nan Epoch.: 38, Step size: 2.63e-02, Grad. norm: nan, Func. value: nan Epoch.: 39, Step size: 2.56e-02, Grad. norm: nan, Func. value: nan Epoch.: 40, Step size: 2.50e-02, Grad. norm: nan, Func. value: nan Epoch.: 41, Step size: 2.44e-02, Grad. norm: nan, Func. value: nan Epoch.: 42, Step size: 2.38e-02, Grad. norm: nan, Func. value: nan Epoch.: 43, Step size: 2.33e-02, Grad. norm: nan, Func. value: nan Epoch.: 44, Step size: 2.27e-02, Grad. norm: nan, Func. value: nan Epoch.: 45, Step size: 2.22e-02, Grad. norm: nan, Func. value: nan Epoch.: 46, Step size: 2.17e-02, Grad. norm: nan, Func. value: nan Epoch.: 47, Step size: 2.13e-02, Grad. norm: nan, Func. value: nan Epoch.: 48, Step size: 2.08e-02, Grad. norm: nan, Func. value: nan Epoch.: 49, Step size: 2.04e-02, Grad. norm: nan, Func. value: nan Test accuracy: 0.000000
Process finished with exit code 0
after fix log:
Begin to run SVRG-BB: Info: set m=n by default Epoch.: 0, Step size: 1.00e-03, Grad. norm: 4.34e-01, Func. value: 6.931472e-01 Epoch.: 1, Step size: 3.40e-03, Grad. norm: 3.24e-01, Func. value: 5.511448e-01 Epoch.: 2, Step size: 4.86e-03, Grad. norm: 1.64e-01, Func. value: 3.681168e-01 Epoch.: 3, Step size: 8.89e-03, Grad. norm: 9.70e-02, Func. value: 2.913540e-01 Epoch.: 4, Step size: 1.44e-02, Grad. norm: 5.43e-02, Func. value: 2.453456e-01 Epoch.: 5, Step size: 2.05e-02, Grad. norm: 2.98e-02, Func. value: 2.234416e-01 Epoch.: 6, Step size: 2.52e-02, Grad. norm: 1.56e-02, Func. value: 2.150486e-01 Epoch.: 7, Step size: 2.73e-02, Grad. norm: 8.21e-03, Func. value: 2.125679e-01 Epoch.: 8, Step size: 2.56e-02, Grad. norm: 4.32e-03, Func. value: 2.119171e-01 Epoch.: 9, Step size: 2.45e-02, Grad. norm: 2.21e-03, Func. value: 2.117595e-01 Epoch.: 10, Step size: 2.33e-02, Grad. norm: 1.18e-03, Func. value: 2.117190e-01 Epoch.: 11, Step size: 2.25e-02, Grad. norm: 5.72e-04, Func. value: 2.117072e-01 Epoch.: 12, Step size: 2.49e-02, Grad. norm: 2.91e-04, Func. value: 2.117041e-01 Epoch.: 13, Step size: 2.96e-02, Grad. norm: 1.51e-04, Func. value: 2.117032e-01 Epoch.: 14, Step size: 3.06e-02, Grad. norm: 7.84e-05, Func. value: 2.117029e-01 Epoch.: 15, Step size: 2.81e-02, Grad. norm: 4.34e-05, Func. value: 2.117028e-01 Epoch.: 16, Step size: 2.17e-02, Grad. norm: 2.07e-05, Func. value: 2.117028e-01 Epoch.: 17, Step size: 1.88e-02, Grad. norm: 9.15e-06, Func. value: 2.117028e-01 Epoch.: 18, Step size: 2.58e-02, Grad. norm: 4.94e-06, Func. value: 2.117028e-01 Epoch.: 19, Step size: 3.18e-02, Grad. norm: 2.67e-06, Func. value: 2.117028e-01 Epoch.: 20, Step size: 2.91e-02, Grad. norm: 1.52e-06, Func. value: 2.117028e-01 Epoch.: 21, Step size: 2.33e-02, Grad. norm: 7.34e-07, Func. value: 2.117028e-01 Epoch.: 22, Step size: 1.96e-02, Grad. norm: 3.60e-07, Func. value: 2.117028e-01 Epoch.: 23, Step size: 2.22e-02, Grad. norm: 1.70e-07, Func. value: 2.117028e-01 Epoch.: 24, Step size: 2.96e-02, Grad. norm: 9.09e-08, Func. value: 2.117028e-01 Epoch.: 25, Step size: 3.16e-02, Grad. norm: 5.27e-08, Func. value: 2.117028e-01 Epoch.: 26, Step size: 2.44e-02, Grad. norm: 2.97e-08, Func. value: 2.117028e-01 Epoch.: 27, Step size: 1.86e-02, Grad. norm: 1.35e-08, Func. value: 2.117028e-01 Epoch.: 28, Step size: 1.96e-02, Grad. norm: 6.46e-09, Func. value: 2.117028e-01 Epoch.: 29, Step size: 2.97e-02, Grad. norm: 3.66e-09, Func. value: 2.117028e-01 Epoch.: 30, Step size: 3.14e-02, Grad. norm: 1.86e-09, Func. value: 2.117028e-01 Epoch.: 31, Step size: 2.78e-02, Grad. norm: 1.05e-09, Func. value: 2.117028e-01 Epoch.: 32, Step size: 2.28e-02, Grad. norm: 5.28e-10, Func. value: 2.117028e-01 Epoch.: 33, Step size: 1.95e-02, Grad. norm: 2.42e-10, Func. value: 2.117028e-01 Epoch.: 34, Step size: 2.25e-02, Grad. norm: 1.30e-10, Func. value: 2.117028e-01 Epoch.: 35, Step size: 2.69e-02, Grad. norm: 6.74e-11, Func. value: 2.117028e-01 Epoch.: 36, Step size: 2.88e-02, Grad. norm: 3.46e-11, Func. value: 2.117028e-01 Epoch.: 37, Step size: 2.83e-02, Grad. norm: 1.88e-11, Func. value: 2.117028e-01 Epoch.: 38, Step size: 2.41e-02, Grad. norm: 1.08e-11, Func. value: 2.117028e-01 Epoch.: 39, Step size: 2.00e-02, Grad. norm: 5.33e-12, Func. value: 2.117028e-01 Epoch.: 40, Step size: 1.89e-02, Grad. norm: 2.61e-12, Func. value: 2.117028e-01 Epoch.: 41, Step size: 2.29e-02, Grad. norm: 1.35e-12, Func. value: 2.117028e-01 Epoch.: 42, Step size: 2.95e-02, Grad. norm: 7.62e-13, Func. value: 2.117028e-01 Epoch.: 43, Step size: 3.07e-02, Grad. norm: 4.23e-13, Func. value: 2.117028e-01 Epoch.: 44, Step size: 2.74e-02, Grad. norm: 2.41e-13, Func. value: 2.117028e-01 Epoch.: 45, Step size: 1.88e-02, Grad. norm: 1.25e-13, Func. value: 2.117028e-01 Epoch.: 46, Step size: 1.70e-02, Grad. norm: 5.69e-14, Func. value: 2.117028e-01 Epoch.: 47, Step size: 2.48e-02, Grad. norm: 3.21e-14, Func. value: 2.117028e-01 Epoch.: 48, Step size: 3.11e-02, Grad. norm: 1.76e-14, Func. value: 2.117028e-01 Epoch.: 49, Step size: 3.01e-02, Grad. norm: 8.87e-15, Func. value: 2.117028e-01 Test accuracy: 0.944000
Begin to run SGD-BB: Info: set m=n by default Epoch.: 0, Step size: 1.00e-03, Grad. norm: 4.34e-01, Func. value: 6.931472e-01 Epoch.: 1, Step size: 1.00e-03, Grad. norm: 3.26e-01, Func. value: 5.535531e-01 Epoch.: 2, Step size: 2.15e-03, Grad. norm: 2.57e-01, Func. value: 4.727686e-01 Epoch.: 3, Step size: 3.33e-01, Grad. norm: 1.77e-01, Func. value: 3.795690e-01 Epoch.: 4, Step size: 2.50e-01, Grad. norm: 3.82e-01, Func. value: 2.273602e+00 Epoch.: 5, Step size: 2.00e-01, Grad. norm: 3.29e-01, Func. value: 1.532060e+00 Epoch.: 6, Step size: 1.67e-01, Grad. norm: 2.95e-01, Func. value: 1.135588e+00 Epoch.: 7, Step size: 1.43e-01, Grad. norm: 2.82e-01, Func. value: 9.321407e-01 Epoch.: 8, Step size: 1.25e-01, Grad. norm: 2.18e-01, Func. value: 6.674450e-01 Epoch.: 9, Step size: 1.11e-01, Grad. norm: 2.35e-01, Func. value: 6.177261e-01 Epoch.: 10, Step size: 1.00e-01, Grad. norm: 2.07e-01, Func. value: 5.256017e-01 Epoch.: 11, Step size: 9.09e-02, Grad. norm: 1.96e-01, Func. value: 4.756571e-01 Epoch.: 12, Step size: 8.33e-02, Grad. norm: 1.67e-01, Func. value: 4.041156e-01 Epoch.: 13, Step size: 7.69e-02, Grad. norm: 1.79e-01, Func. value: 3.986054e-01 Epoch.: 14, Step size: 7.14e-02, Grad. norm: 1.53e-01, Func. value: 3.580356e-01 Epoch.: 15, Step size: 6.67e-02, Grad. norm: 1.50e-01, Func. value: 3.388369e-01 Epoch.: 16, Step size: 6.25e-02, Grad. norm: 1.59e-01, Func. value: 3.490977e-01 Epoch.: 17, Step size: 5.88e-02, Grad. norm: 1.29e-01, Func. value: 3.056023e-01 Epoch.: 18, Step size: 5.56e-02, Grad. norm: 1.53e-01, Func. value: 3.339887e-01 Epoch.: 19, Step size: 5.26e-02, Grad. norm: 1.21e-01, Func. value: 2.931325e-01 Epoch.: 20, Step size: 5.00e-02, Grad. norm: 1.10e-01, Func. value: 2.834734e-01 Epoch.: 21, Step size: 4.76e-02, Grad. norm: 1.19e-01, Func. value: 2.850853e-01 Epoch.: 22, Step size: 4.55e-02, Grad. norm: 9.65e-02, Func. value: 2.665883e-01 Epoch.: 23, Step size: 4.35e-02, Grad. norm: 1.28e-01, Func. value: 2.950536e-01 Epoch.: 24, Step size: 4.17e-02, Grad. norm: 1.08e-01, Func. value: 2.705244e-01 Epoch.: 25, Step size: 4.00e-02, Grad. norm: 8.00e-02, Func. value: 2.459204e-01 Epoch.: 26, Step size: 3.85e-02, Grad. norm: 1.03e-01, Func. value: 2.622630e-01 Epoch.: 27, Step size: 3.70e-02, Grad. norm: 1.16e-01, Func. value: 2.701666e-01 Epoch.: 28, Step size: 3.57e-02, Grad. norm: 1.15e-01, Func. value: 2.682768e-01 Epoch.: 29, Step size: 3.45e-02, Grad. norm: 8.22e-02, Func. value: 2.466660e-01 Epoch.: 30, Step size: 3.33e-02, Grad. norm: 1.05e-01, Func. value: 2.621299e-01 Epoch.: 31, Step size: 3.23e-02, Grad. norm: 8.21e-02, Func. value: 2.468232e-01 Epoch.: 32, Step size: 3.12e-02, Grad. norm: 9.51e-02, Func. value: 2.526804e-01 Epoch.: 33, Step size: 3.03e-02, Grad. norm: 8.48e-02, Func. value: 2.498060e-01 Epoch.: 34, Step size: 2.94e-02, Grad. norm: 9.00e-02, Func. value: 2.496647e-01 Epoch.: 35, Step size: 2.86e-02, Grad. norm: 7.60e-02, Func. value: 2.392340e-01 Epoch.: 36, Step size: 2.78e-02, Grad. norm: 7.93e-02, Func. value: 2.411619e-01 Epoch.: 37, Step size: 2.70e-02, Grad. norm: 9.16e-02, Func. value: 2.485319e-01 Epoch.: 38, Step size: 2.63e-02, Grad. norm: 7.61e-02, Func. value: 2.395331e-01 Epoch.: 39, Step size: 2.56e-02, Grad. norm: 7.99e-02, Func. value: 2.392843e-01 Epoch.: 40, Step size: 2.50e-02, Grad. norm: 8.43e-02, Func. value: 2.442082e-01 Epoch.: 41, Step size: 2.44e-02, Grad. norm: 7.65e-02, Func. value: 2.381724e-01 Epoch.: 42, Step size: 2.38e-02, Grad. norm: 8.04e-02, Func. value: 2.393173e-01 Epoch.: 43, Step size: 2.33e-02, Grad. norm: 6.94e-02, Func. value: 2.347828e-01 Epoch.: 44, Step size: 2.27e-02, Grad. norm: 5.97e-02, Func. value: 2.294437e-01 Epoch.: 45, Step size: 2.22e-02, Grad. norm: 6.52e-02, Func. value: 2.313249e-01 Epoch.: 46, Step size: 2.17e-02, Grad. norm: 7.09e-02, Func. value: 2.339068e-01 Epoch.: 47, Step size: 2.13e-02, Grad. norm: 6.53e-02, Func. value: 2.323820e-01 Epoch.: 48, Step size: 2.08e-02, Grad. norm: 6.44e-02, Func. value: 2.311835e-01 Epoch.: 49, Step size: 2.04e-02, Grad. norm: 6.81e-02, Func. value: 2.324369e-01 Test accuracy: 0.933000
fix 1: 2/3=0 error.
fix 2: beta default value should not be zero.
before fix log:
Begin to run SVRG-BB: Info: set m=n by default Epoch.: 0, Step size: 1.00e-03, Grad. norm: 4.31e-01, Func. value: 6.931472e-01 Epoch.: 1, Step size: 3.40e-03, Grad. norm: 3.21e-01, Func. value: 5.531818e-01 Epoch.: 2, Step size: 4.88e-03, Grad. norm: 1.68e-01, Func. value: 3.759994e-01 Epoch.: 3, Step size: 8.70e-03, Grad. norm: 9.85e-02, Func. value: 2.968144e-01 Epoch.: 4, Step size: 1.39e-02, Grad. norm: 5.55e-02, Func. value: 2.502696e-01 Epoch.: 5, Step size: 1.99e-02, Grad. norm: 2.97e-02, Func. value: 2.278909e-01 Epoch.: 6, Step size: 2.62e-02, Grad. norm: 1.55e-02, Func. value: 2.193573e-01 Epoch.: 7, Step size: 2.78e-02, Grad. norm: 7.71e-03, Func. value: 2.167048e-01 Epoch.: 8, Step size: 2.53e-02, Grad. norm: 3.91e-03, Func. value: 2.161033e-01 Epoch.: 9, Step size: 2.39e-02, Grad. norm: 2.15e-03, Func. value: 2.159635e-01 Epoch.: 10, Step size: 2.34e-02, Grad. norm: 1.10e-03, Func. value: 2.159267e-01 Epoch.: 11, Step size: 2.52e-02, Grad. norm: 5.44e-04, Func. value: 2.159164e-01 Epoch.: 12, Step size: 2.61e-02, Grad. norm: 2.94e-04, Func. value: 2.159134e-01 Epoch.: 13, Step size: 2.62e-02, Grad. norm: 1.60e-04, Func. value: 2.159126e-01 Epoch.: 14, Step size: 2.52e-02, Grad. norm: 8.38e-05, Func. value: 2.159124e-01 Epoch.: 15, Step size: 2.32e-02, Grad. norm: 4.48e-05, Func. value: 2.159123e-01 Epoch.: 16, Step size: 2.32e-02, Grad. norm: 2.31e-05, Func. value: 2.159123e-01 Epoch.: 17, Step size: 2.39e-02, Grad. norm: 1.23e-05, Func. value: 2.159123e-01 Epoch.: 18, Step size: 2.63e-02, Grad. norm: 6.06e-06, Func. value: 2.159123e-01 Epoch.: 19, Step size: 2.75e-02, Grad. norm: 3.25e-06, Func. value: 2.159123e-01 Epoch.: 20, Step size: 2.68e-02, Grad. norm: 1.67e-06, Func. value: 2.159123e-01 Epoch.: 21, Step size: 2.45e-02, Grad. norm: 9.16e-07, Func. value: 2.159123e-01 Epoch.: 22, Step size: 2.28e-02, Grad. norm: 4.68e-07, Func. value: 2.159123e-01 Epoch.: 23, Step size: 2.30e-02, Grad. norm: 2.26e-07, Func. value: 2.159123e-01 Epoch.: 24, Step size: 2.60e-02, Grad. norm: 1.15e-07, Func. value: 2.159123e-01 Epoch.: 25, Step size: 3.05e-02, Grad. norm: 6.31e-08, Func. value: 2.159123e-01 Epoch.: 26, Step size: 2.61e-02, Grad. norm: 3.53e-08, Func. value: 2.159123e-01 Epoch.: 27, Step size: 1.98e-02, Grad. norm: 1.66e-08, Func. value: 2.159123e-01 Epoch.: 28, Step size: 2.25e-02, Grad. norm: 8.76e-09, Func. value: 2.159123e-01 Epoch.: 29, Step size: 2.59e-02, Grad. norm: 4.59e-09, Func. value: 2.159123e-01 Epoch.: 30, Step size: 2.88e-02, Grad. norm: 2.53e-09, Func. value: 2.159123e-01 Epoch.: 31, Step size: 2.71e-02, Grad. norm: 1.26e-09, Func. value: 2.159123e-01 Epoch.: 32, Step size: 2.23e-02, Grad. norm: 6.29e-10, Func. value: 2.159123e-01 Epoch.: 33, Step size: 2.26e-02, Grad. norm: 3.40e-10, Func. value: 2.159123e-01 Epoch.: 34, Step size: 2.46e-02, Grad. norm: 1.66e-10, Func. value: 2.159123e-01 Epoch.: 35, Step size: 2.79e-02, Grad. norm: 8.54e-11, Func. value: 2.159123e-01 Epoch.: 36, Step size: 2.87e-02, Grad. norm: 4.36e-11, Func. value: 2.159123e-01 Epoch.: 37, Step size: 2.81e-02, Grad. norm: 2.42e-11, Func. value: 2.159123e-01 Epoch.: 38, Step size: 2.35e-02, Grad. norm: 1.22e-11, Func. value: 2.159123e-01 Epoch.: 39, Step size: 2.17e-02, Grad. norm: 5.89e-12, Func. value: 2.159123e-01 Epoch.: 40, Step size: 2.53e-02, Grad. norm: 3.15e-12, Func. value: 2.159123e-01 Epoch.: 41, Step size: 2.46e-02, Grad. norm: 1.64e-12, Func. value: 2.159123e-01 Epoch.: 42, Step size: 2.63e-02, Grad. norm: 8.22e-13, Func. value: 2.159123e-01 Epoch.: 43, Step size: 2.74e-02, Grad. norm: 4.36e-13, Func. value: 2.159123e-01 Epoch.: 44, Step size: 2.41e-02, Grad. norm: 2.38e-13, Func. value: 2.159123e-01 Epoch.: 45, Step size: 2.26e-02, Grad. norm: 1.19e-13, Func. value: 2.159123e-01 Epoch.: 46, Step size: 2.52e-02, Grad. norm: 5.95e-14, Func. value: 2.159123e-01 Epoch.: 47, Step size: 2.83e-02, Grad. norm: 3.38e-14, Func. value: 2.159123e-01 Epoch.: 48, Step size: 2.63e-02, Grad. norm: 1.81e-14, Func. value: 2.159123e-01 Epoch.: 49, Step size: 2.04e-02, Grad. norm: 9.91e-15, Func. value: 2.159123e-01 Test accuracy: 0.000000
Begin to run SGD-BB: Info: set m=n by default Info: set beta=10/m by default Epoch.: 0, Step size: 1.00e-03, Grad. norm: 4.31e-01, Func. value: 6.931472e-01 Epoch.: 1, Step size: 1.00e-03, Grad. norm: 3.21e-01, Func. value: 5.526788e-01 Epoch.: 2, Step size: inf, Grad. norm: 2.53e-01, Func. value: 4.723846e-01 Epoch.: 3, Step size: 3.33e-01, Grad. norm: nan, Func. value: nan Epoch.: 4, Step size: 2.50e-01, Grad. norm: nan, Func. value: nan Epoch.: 5, Step size: 2.00e-01, Grad. norm: nan, Func. value: nan Epoch.: 6, Step size: 1.67e-01, Grad. norm: nan, Func. value: nan Epoch.: 7, Step size: 1.43e-01, Grad. norm: nan, Func. value: nan Epoch.: 8, Step size: 1.25e-01, Grad. norm: nan, Func. value: nan Epoch.: 9, Step size: 1.11e-01, Grad. norm: nan, Func. value: nan Epoch.: 10, Step size: 1.00e-01, Grad. norm: nan, Func. value: nan Epoch.: 11, Step size: 9.09e-02, Grad. norm: nan, Func. value: nan Epoch.: 12, Step size: 8.33e-02, Grad. norm: nan, Func. value: nan Epoch.: 13, Step size: 7.69e-02, Grad. norm: nan, Func. value: nan Epoch.: 14, Step size: 7.14e-02, Grad. norm: nan, Func. value: nan Epoch.: 15, Step size: 6.67e-02, Grad. norm: nan, Func. value: nan Epoch.: 16, Step size: 6.25e-02, Grad. norm: nan, Func. value: nan Epoch.: 17, Step size: 5.88e-02, Grad. norm: nan, Func. value: nan Epoch.: 18, Step size: 5.56e-02, Grad. norm: nan, Func. value: nan Epoch.: 19, Step size: 5.26e-02, Grad. norm: nan, Func. value: nan Epoch.: 20, Step size: 5.00e-02, Grad. norm: nan, Func. value: nan Epoch.: 21, Step size: 4.76e-02, Grad. norm: nan, Func. value: nan Epoch.: 22, Step size: 4.55e-02, Grad. norm: nan, Func. value: nan Epoch.: 23, Step size: 4.35e-02, Grad. norm: nan, Func. value: nan Epoch.: 24, Step size: 4.17e-02, Grad. norm: nan, Func. value: nan Epoch.: 25, Step size: 4.00e-02, Grad. norm: nan, Func. value: nan Epoch.: 26, Step size: 3.85e-02, Grad. norm: nan, Func. value: nan Epoch.: 27, Step size: 3.70e-02, Grad. norm: nan, Func. value: nan Epoch.: 28, Step size: 3.57e-02, Grad. norm: nan, Func. value: nan Epoch.: 29, Step size: 3.45e-02, Grad. norm: nan, Func. value: nan Epoch.: 30, Step size: 3.33e-02, Grad. norm: nan, Func. value: nan Epoch.: 31, Step size: 3.23e-02, Grad. norm: nan, Func. value: nan Epoch.: 32, Step size: 3.12e-02, Grad. norm: nan, Func. value: nan Epoch.: 33, Step size: 3.03e-02, Grad. norm: nan, Func. value: nan Epoch.: 34, Step size: 2.94e-02, Grad. norm: nan, Func. value: nan Epoch.: 35, Step size: 2.86e-02, Grad. norm: nan, Func. value: nan Epoch.: 36, Step size: 2.78e-02, Grad. norm: nan, Func. value: nan Epoch.: 37, Step size: 2.70e-02, Grad. norm: nan, Func. value: nan Epoch.: 38, Step size: 2.63e-02, Grad. norm: nan, Func. value: nan Epoch.: 39, Step size: 2.56e-02, Grad. norm: nan, Func. value: nan Epoch.: 40, Step size: 2.50e-02, Grad. norm: nan, Func. value: nan Epoch.: 41, Step size: 2.44e-02, Grad. norm: nan, Func. value: nan Epoch.: 42, Step size: 2.38e-02, Grad. norm: nan, Func. value: nan Epoch.: 43, Step size: 2.33e-02, Grad. norm: nan, Func. value: nan Epoch.: 44, Step size: 2.27e-02, Grad. norm: nan, Func. value: nan Epoch.: 45, Step size: 2.22e-02, Grad. norm: nan, Func. value: nan Epoch.: 46, Step size: 2.17e-02, Grad. norm: nan, Func. value: nan Epoch.: 47, Step size: 2.13e-02, Grad. norm: nan, Func. value: nan Epoch.: 48, Step size: 2.08e-02, Grad. norm: nan, Func. value: nan Epoch.: 49, Step size: 2.04e-02, Grad. norm: nan, Func. value: nan Test accuracy: 0.000000
Process finished with exit code 0
after fix log:
Begin to run SVRG-BB: Info: set m=n by default Epoch.: 0, Step size: 1.00e-03, Grad. norm: 4.34e-01, Func. value: 6.931472e-01 Epoch.: 1, Step size: 3.40e-03, Grad. norm: 3.24e-01, Func. value: 5.511448e-01 Epoch.: 2, Step size: 4.86e-03, Grad. norm: 1.64e-01, Func. value: 3.681168e-01 Epoch.: 3, Step size: 8.89e-03, Grad. norm: 9.70e-02, Func. value: 2.913540e-01 Epoch.: 4, Step size: 1.44e-02, Grad. norm: 5.43e-02, Func. value: 2.453456e-01 Epoch.: 5, Step size: 2.05e-02, Grad. norm: 2.98e-02, Func. value: 2.234416e-01 Epoch.: 6, Step size: 2.52e-02, Grad. norm: 1.56e-02, Func. value: 2.150486e-01 Epoch.: 7, Step size: 2.73e-02, Grad. norm: 8.21e-03, Func. value: 2.125679e-01 Epoch.: 8, Step size: 2.56e-02, Grad. norm: 4.32e-03, Func. value: 2.119171e-01 Epoch.: 9, Step size: 2.45e-02, Grad. norm: 2.21e-03, Func. value: 2.117595e-01 Epoch.: 10, Step size: 2.33e-02, Grad. norm: 1.18e-03, Func. value: 2.117190e-01 Epoch.: 11, Step size: 2.25e-02, Grad. norm: 5.72e-04, Func. value: 2.117072e-01 Epoch.: 12, Step size: 2.49e-02, Grad. norm: 2.91e-04, Func. value: 2.117041e-01 Epoch.: 13, Step size: 2.96e-02, Grad. norm: 1.51e-04, Func. value: 2.117032e-01 Epoch.: 14, Step size: 3.06e-02, Grad. norm: 7.84e-05, Func. value: 2.117029e-01 Epoch.: 15, Step size: 2.81e-02, Grad. norm: 4.34e-05, Func. value: 2.117028e-01 Epoch.: 16, Step size: 2.17e-02, Grad. norm: 2.07e-05, Func. value: 2.117028e-01 Epoch.: 17, Step size: 1.88e-02, Grad. norm: 9.15e-06, Func. value: 2.117028e-01 Epoch.: 18, Step size: 2.58e-02, Grad. norm: 4.94e-06, Func. value: 2.117028e-01 Epoch.: 19, Step size: 3.18e-02, Grad. norm: 2.67e-06, Func. value: 2.117028e-01 Epoch.: 20, Step size: 2.91e-02, Grad. norm: 1.52e-06, Func. value: 2.117028e-01 Epoch.: 21, Step size: 2.33e-02, Grad. norm: 7.34e-07, Func. value: 2.117028e-01 Epoch.: 22, Step size: 1.96e-02, Grad. norm: 3.60e-07, Func. value: 2.117028e-01 Epoch.: 23, Step size: 2.22e-02, Grad. norm: 1.70e-07, Func. value: 2.117028e-01 Epoch.: 24, Step size: 2.96e-02, Grad. norm: 9.09e-08, Func. value: 2.117028e-01 Epoch.: 25, Step size: 3.16e-02, Grad. norm: 5.27e-08, Func. value: 2.117028e-01 Epoch.: 26, Step size: 2.44e-02, Grad. norm: 2.97e-08, Func. value: 2.117028e-01 Epoch.: 27, Step size: 1.86e-02, Grad. norm: 1.35e-08, Func. value: 2.117028e-01 Epoch.: 28, Step size: 1.96e-02, Grad. norm: 6.46e-09, Func. value: 2.117028e-01 Epoch.: 29, Step size: 2.97e-02, Grad. norm: 3.66e-09, Func. value: 2.117028e-01 Epoch.: 30, Step size: 3.14e-02, Grad. norm: 1.86e-09, Func. value: 2.117028e-01 Epoch.: 31, Step size: 2.78e-02, Grad. norm: 1.05e-09, Func. value: 2.117028e-01 Epoch.: 32, Step size: 2.28e-02, Grad. norm: 5.28e-10, Func. value: 2.117028e-01 Epoch.: 33, Step size: 1.95e-02, Grad. norm: 2.42e-10, Func. value: 2.117028e-01 Epoch.: 34, Step size: 2.25e-02, Grad. norm: 1.30e-10, Func. value: 2.117028e-01 Epoch.: 35, Step size: 2.69e-02, Grad. norm: 6.74e-11, Func. value: 2.117028e-01 Epoch.: 36, Step size: 2.88e-02, Grad. norm: 3.46e-11, Func. value: 2.117028e-01 Epoch.: 37, Step size: 2.83e-02, Grad. norm: 1.88e-11, Func. value: 2.117028e-01 Epoch.: 38, Step size: 2.41e-02, Grad. norm: 1.08e-11, Func. value: 2.117028e-01 Epoch.: 39, Step size: 2.00e-02, Grad. norm: 5.33e-12, Func. value: 2.117028e-01 Epoch.: 40, Step size: 1.89e-02, Grad. norm: 2.61e-12, Func. value: 2.117028e-01 Epoch.: 41, Step size: 2.29e-02, Grad. norm: 1.35e-12, Func. value: 2.117028e-01 Epoch.: 42, Step size: 2.95e-02, Grad. norm: 7.62e-13, Func. value: 2.117028e-01 Epoch.: 43, Step size: 3.07e-02, Grad. norm: 4.23e-13, Func. value: 2.117028e-01 Epoch.: 44, Step size: 2.74e-02, Grad. norm: 2.41e-13, Func. value: 2.117028e-01 Epoch.: 45, Step size: 1.88e-02, Grad. norm: 1.25e-13, Func. value: 2.117028e-01 Epoch.: 46, Step size: 1.70e-02, Grad. norm: 5.69e-14, Func. value: 2.117028e-01 Epoch.: 47, Step size: 2.48e-02, Grad. norm: 3.21e-14, Func. value: 2.117028e-01 Epoch.: 48, Step size: 3.11e-02, Grad. norm: 1.76e-14, Func. value: 2.117028e-01 Epoch.: 49, Step size: 3.01e-02, Grad. norm: 8.87e-15, Func. value: 2.117028e-01 Test accuracy: 0.944000
Begin to run SGD-BB: Info: set m=n by default Epoch.: 0, Step size: 1.00e-03, Grad. norm: 4.34e-01, Func. value: 6.931472e-01 Epoch.: 1, Step size: 1.00e-03, Grad. norm: 3.26e-01, Func. value: 5.535531e-01 Epoch.: 2, Step size: 2.15e-03, Grad. norm: 2.57e-01, Func. value: 4.727686e-01 Epoch.: 3, Step size: 3.33e-01, Grad. norm: 1.77e-01, Func. value: 3.795690e-01 Epoch.: 4, Step size: 2.50e-01, Grad. norm: 3.82e-01, Func. value: 2.273602e+00 Epoch.: 5, Step size: 2.00e-01, Grad. norm: 3.29e-01, Func. value: 1.532060e+00 Epoch.: 6, Step size: 1.67e-01, Grad. norm: 2.95e-01, Func. value: 1.135588e+00 Epoch.: 7, Step size: 1.43e-01, Grad. norm: 2.82e-01, Func. value: 9.321407e-01 Epoch.: 8, Step size: 1.25e-01, Grad. norm: 2.18e-01, Func. value: 6.674450e-01 Epoch.: 9, Step size: 1.11e-01, Grad. norm: 2.35e-01, Func. value: 6.177261e-01 Epoch.: 10, Step size: 1.00e-01, Grad. norm: 2.07e-01, Func. value: 5.256017e-01 Epoch.: 11, Step size: 9.09e-02, Grad. norm: 1.96e-01, Func. value: 4.756571e-01 Epoch.: 12, Step size: 8.33e-02, Grad. norm: 1.67e-01, Func. value: 4.041156e-01 Epoch.: 13, Step size: 7.69e-02, Grad. norm: 1.79e-01, Func. value: 3.986054e-01 Epoch.: 14, Step size: 7.14e-02, Grad. norm: 1.53e-01, Func. value: 3.580356e-01 Epoch.: 15, Step size: 6.67e-02, Grad. norm: 1.50e-01, Func. value: 3.388369e-01 Epoch.: 16, Step size: 6.25e-02, Grad. norm: 1.59e-01, Func. value: 3.490977e-01 Epoch.: 17, Step size: 5.88e-02, Grad. norm: 1.29e-01, Func. value: 3.056023e-01 Epoch.: 18, Step size: 5.56e-02, Grad. norm: 1.53e-01, Func. value: 3.339887e-01 Epoch.: 19, Step size: 5.26e-02, Grad. norm: 1.21e-01, Func. value: 2.931325e-01 Epoch.: 20, Step size: 5.00e-02, Grad. norm: 1.10e-01, Func. value: 2.834734e-01 Epoch.: 21, Step size: 4.76e-02, Grad. norm: 1.19e-01, Func. value: 2.850853e-01 Epoch.: 22, Step size: 4.55e-02, Grad. norm: 9.65e-02, Func. value: 2.665883e-01 Epoch.: 23, Step size: 4.35e-02, Grad. norm: 1.28e-01, Func. value: 2.950536e-01 Epoch.: 24, Step size: 4.17e-02, Grad. norm: 1.08e-01, Func. value: 2.705244e-01 Epoch.: 25, Step size: 4.00e-02, Grad. norm: 8.00e-02, Func. value: 2.459204e-01 Epoch.: 26, Step size: 3.85e-02, Grad. norm: 1.03e-01, Func. value: 2.622630e-01 Epoch.: 27, Step size: 3.70e-02, Grad. norm: 1.16e-01, Func. value: 2.701666e-01 Epoch.: 28, Step size: 3.57e-02, Grad. norm: 1.15e-01, Func. value: 2.682768e-01 Epoch.: 29, Step size: 3.45e-02, Grad. norm: 8.22e-02, Func. value: 2.466660e-01 Epoch.: 30, Step size: 3.33e-02, Grad. norm: 1.05e-01, Func. value: 2.621299e-01 Epoch.: 31, Step size: 3.23e-02, Grad. norm: 8.21e-02, Func. value: 2.468232e-01 Epoch.: 32, Step size: 3.12e-02, Grad. norm: 9.51e-02, Func. value: 2.526804e-01 Epoch.: 33, Step size: 3.03e-02, Grad. norm: 8.48e-02, Func. value: 2.498060e-01 Epoch.: 34, Step size: 2.94e-02, Grad. norm: 9.00e-02, Func. value: 2.496647e-01 Epoch.: 35, Step size: 2.86e-02, Grad. norm: 7.60e-02, Func. value: 2.392340e-01 Epoch.: 36, Step size: 2.78e-02, Grad. norm: 7.93e-02, Func. value: 2.411619e-01 Epoch.: 37, Step size: 2.70e-02, Grad. norm: 9.16e-02, Func. value: 2.485319e-01 Epoch.: 38, Step size: 2.63e-02, Grad. norm: 7.61e-02, Func. value: 2.395331e-01 Epoch.: 39, Step size: 2.56e-02, Grad. norm: 7.99e-02, Func. value: 2.392843e-01 Epoch.: 40, Step size: 2.50e-02, Grad. norm: 8.43e-02, Func. value: 2.442082e-01 Epoch.: 41, Step size: 2.44e-02, Grad. norm: 7.65e-02, Func. value: 2.381724e-01 Epoch.: 42, Step size: 2.38e-02, Grad. norm: 8.04e-02, Func. value: 2.393173e-01 Epoch.: 43, Step size: 2.33e-02, Grad. norm: 6.94e-02, Func. value: 2.347828e-01 Epoch.: 44, Step size: 2.27e-02, Grad. norm: 5.97e-02, Func. value: 2.294437e-01 Epoch.: 45, Step size: 2.22e-02, Grad. norm: 6.52e-02, Func. value: 2.313249e-01 Epoch.: 46, Step size: 2.17e-02, Grad. norm: 7.09e-02, Func. value: 2.339068e-01 Epoch.: 47, Step size: 2.13e-02, Grad. norm: 6.53e-02, Func. value: 2.323820e-01 Epoch.: 48, Step size: 2.08e-02, Grad. norm: 6.44e-02, Func. value: 2.311835e-01 Epoch.: 49, Step size: 2.04e-02, Grad. norm: 6.81e-02, Func. value: 2.324369e-01 Test accuracy: 0.933000
Process finished with exit code 0