Tests seem to randomly fail sometimes...?

[jgranek]

Some of the derivative tests seem to fail on occasion - I'm guessing due to the random initializations?

test d2Y: Test Failed
  Expression: chkDer
Stacktrace:
 [1] macro expansion at /home/travis/.julia/v0.6/Meganet/src/utils/testLossFunction.jl:64 [inlined]
 [2] macro expansion at ./test.jl:860 [inlined]
 [3] #testLossFunction#83(::Int64, ::Int64, ::Int64, ::Bool, ::Function, ::Meganet.SoftmaxLoss{Float64}) at /home/travis/.julia/v0.6/Meganet/src/utils/testLossFunction.jl:53
SoftmaxLoss(shift=0.0,theta=0.001,addBias=true)
Test Summary:                     | Pass  Fail  Total
Meganet                           |  511     1    512
  integrators                     |   74           74
  kernels                         |   18           18
  layer                           |  396          396
  optim                           |    6            6
  loss                            |   15     1     16
    loss                          |   15     1     16
      softmax (with bias) Float64 |    4            4
      softmax (no bias) Float64   |    3     1      4
        test dW                   |    1            1
        test d2W                  |    1            1
        test dY                   |    1            1
        test d2Y                  |          1      1
      softmax (with bias) Float32 |    4            4
      softmax (no bias) Float32   |    4            4
  regularizer                     |    2            2
ERROR: LoadError: Some tests did not pass: 511 passed, 1 failed, 0 errored, 0 broken.

[klensink]

I've seen this with test d2W

[jgranek]

Ya me too - it only ever seems to be one that fails at a time though - I'm guessing that's just random luck. Assuming it has to do with initialization and should be an easy fix, but since I was able to get the OSX build of TravisCI to pass I want to implement the status checks on merges - need the tests to reliably pass in order to do that though. Looking into it now..

[klensink]

Also worth checking how the comparisons are being evaluated in the tests. If I remember correctly a == b isn't reliable for floats, and you can use isapprox(a, b) to check if they are within machine precision.

On Jan 29, 2018 2:05 PM, "Justin" notifications@github.com wrote:

Ya me too - it only ever seems to be one that fails at a time though - I'm guessing that's just random luck. Assuming it has to do with initialization and should be an easy fix, but since I was able to get the OSX build of TravisCI to pass I want to implement the status checks on merges - need the tests to reliably pass in order to do that though. Looking into it now..

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/XtractOpen/Meganet.jl/issues/29#issuecomment-361402909, or mute the thread https://github.com/notifications/unsubscribe-auth/ATplBUGfjQb9Lm_LSxcYydQHJV1uL4yNks5tPkChgaJpZM4RxbE5 .

[lruthotto]

The problem is that we verify the accuracy of derivatives by comparing the order of convergence of a zero-order Taylor (no derivative) to a first-order Taylor (with derivative) sequence. By default, a derivative is considered correct if the latter shows 2nd order convergence for at least 4 consecutive elements (see https://github.com/XtractOpen/Meganet.jl/blob/836b5d2c72eb6dedc9ceaf4047f355349ae75102/src/utils/checkDerivative.jl#L29) Maybe changing this to 3 already helps. Since this is a keyword argument we should probably only do this when testing the Hessians, e.g., here https://github.com/XtractOpen/Meganet.jl/blob/836b5d2c72eb6dedc9ceaf4047f355349ae75102/src/utils/testLossFunction.jl#L63.

Thoguhts? @eldadHaber ?

[eldadHaber]

I think that the test is too tight for single precision. I’d like to see the table before we decide what are the parameters of success when thinking single precision.

E

On Jan 29, 2018, at 6:13 PM, Lars Ruthotto notifications@github.com wrote:

The problem is that we verify the accuracy of derivatives by comparing the order of convergence of a zero-order Taylor (no derivative) to a first-order Taylor (with derivative) sequence. By default, a derivative is considered correct if the latter shows 2nd order convergence for at least 4 consecutive elements (see https://github.com/XtractOpen/Meganet.jl/blob/836b5d2c72eb6dedc9ceaf4047f355349ae75102/src/utils/checkDerivative.jl#L29) Maybe changing this to 3 already helps. Since this is a keyword argument we should probably only do this when testing the Hessians, e.g., here https://github.com/XtractOpen/Meganet.jl/blob/836b5d2c72eb6dedc9ceaf4047f355349ae75102/src/utils/testLossFunction.jl#L63.

Thoguhts? @eldadHaber ?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[lruthotto]

Note that the tests fail for Float64.

One change I always wanted to make is to print the history of checkDerivative when the test fails. Then we can just look at it in the log. May do this later...

[jgranek]

@lruthotto I agree - I looked at the output from some of the tests, and you're getting -Inf and NaN..didn't have a chance yet to figure out where they come from or if they're a serious problem.

##################################################
Meganet.testdYFun

        h              E0              E1              O1              O2         OK?
5.000e-01       6.842e-03       5.739e-03       0.000e+00       0.000e+00           0
2.500e-01       4.860e-03       1.430e-03       4.935e-01       2.004e+00           1
1.250e-01       2.788e-03       3.568e-04       8.015e-01       2.003e+00           1
6.250e-02       1.484e-03       8.899e-05       9.103e-01       2.003e+00           1
3.125e-02       7.641e-04       2.214e-05       9.572e-01       2.007e+00           1
1.563e-02       3.877e-04       5.474e-06       9.789e-01       2.016e+00           1
7.813e-03       1.952e-04       1.353e-06       9.900e-01       2.016e+00           1
3.906e-03       9.809e-05       1.845e-07       9.927e-01       2.874e+00           1
1.953e-03       4.908e-05       6.150e-08       9.991e-01       1.585e+00           1
9.766e-04       2.448e-05       1.230e-07       1.004e+00       -1.000e+00          1
4.883e-04       1.218e-05       1.230e-07       1.007e+00       0.000e+00           0
2.441e-04       6.089e-06       6.150e-08       1.000e+00       1.000e+00           0
1.221e-04       3.137e-06       6.150e-08       9.569e-01       0.000e+00           0
6.104e-05       1.538e-06       0.000e+00       1.029e+00       Inf         1
3.052e-05       7.380e-07       0.000e+00       1.059e+00       NaN         1
1.526e-05       3.690e-07       0.000e+00       1.000e+00       NaN         1
7.629e-06       2.460e-07       6.150e-08       5.850e-01       -Inf        0
3.815e-06       2.460e-07       1.230e-07       0.000e+00       -1.000e+00          0
1.907e-06       6.150e-08       0.000e+00       2.000e+00       Inf         1
9.537e-07       6.150e-08       6.150e-08       0.000e+00       -Inf        0

[lruthotto]

The Inf and NaN's are not a problem :) Look at the last column. The test passes if the sum of that column is larger than 4.