Tests fail on v0.5

[ranjanan]

 Pkg.add("BlackBoxOptim"); Pkg.checkout("BlackBoxOptim"); Pkg.test("BlackBoxOptim")

gives

ERROR: LoadError: FactCheck finished with 2 non-successful tests.
 in exitstatus() at /Users/ranjan/.julia/v0.5/FactCheck/src/FactCheck.jl:568
 in include_from_node1(::String) at ./loading.jl:426
 in process_options(::Base.JLOptions) at ./client.jl:262
 in _start() at ./client.jl:318
while loading /Users/ranjan/.julia/v0.5/BlackBoxOptim/test/runtests.jl, in expression starting on line 44

[robertfeldt]

Thanks, yes, this is a known problem since there have been a somewhat changing API for some new parts of the 0.5 release, right, @alyst ?

[alyst]

@robertfeldt Yes, parallelization in particular. I've not tried BBO with 0.5 yet, I plan to do that after the release is out.

[robertfeldt]

I actually built the latest 0.5 Version 0.5.0-rc0+53 (2016-07-29 04:07 UTC) and then

Pkg.add("BlackBoxOptim"); Pkg.checkout("BlackBoxOptim"); Pkg.test("BlackBoxOptim")

and got:

...
INFO: BlackBoxOptim tests passed
459.801207 seconds (605 allocations: 31.203 KB)

[robertfeldt]

@ranjanan which Julia 0.5 version did you try on?

[ranjanan]

@robertfeldt This is my Julia version:

Julia Version 0.5.0-rc0+30
Commit e717ded* (2016-07-28 00:39 UTC)
Platform Info:
  System: Darwin (x86_64-apple-darwin15.4.0)
  CPU: Intel(R) Core(TM) i5-5257U CPU @ 2.70GHz
  WORD_SIZE: 64
  BLAS: libopenblas (USE64BITINT DYNAMIC_ARCH NO_AFFINITY Haswell)
  LAPACK: libopenblas64_
  LIBM: libopenlibm
  LLVM: libLLVM-3.7.1 (ORCJIT, broadwell)

and these are the specific errors I got:

bboptimize() multi-objective methods smoketest
  > borg_moea
    Failure :: (line:272) :: borg_moea :: fact was false
      Expression: f[1] --> roughly(2.0,atol=0.08)
        Expected: 2.9814853844725633 ≅ 2.0
    Failure :: (line:272) :: borg_moea :: fact was false
      Expression: f[2] --> roughly(2.0,atol=0.08)
        Expected: 1.2141471519399443 ≅ 2.0
Out of 4 total facts:
  Verified: 2
  Failed:   2

[robertfeldt]

Ah ok, this might be because of the stochastic nature of optimization and that sometimes any non-convex optimization algorithm might get stuck in some local optima. Then again it should not be very likely to happen twice in a row so this might be something we need to investigate or what do you think @alyst ? This does not seem to be related to 0.5. @ranjanan if you run the test a few times do you get the same failures each time?

[ranjanan]

@robertfeldt include("test_smoketest_bboptimize.jl") consistently fails for me, and the same facts fail. (Not running the entire test suite because it takes too long. Probably slowed down by dep warns, which reminds me, I need to send a PR for that.)

[robertfeldt]

Ok, thanks, @ranjanan we need to investigate this then. If it is consistent something might be wrong when running Borg MOEA on your machine/setup. Not sure we can address this before we have a stable 0.5 though, sorry, but lets see if @alyst has any ideas.

[ranjanan]

Hey, I just managed to reproduce this on another machine with this configuration:

Julia Version 0.5.0-dev+5567                                                                                                                                        
Commit b82d923 (2016-07-21 05:04 UTC)                                                                                                                               
Platform Info:                                                                                                                                                      
  System: Linux (x86_64-unknown-linux-gnu)                                                                                                                          
  CPU: Intel(R) Xeon(R) CPU @ 2.60GHz                                                                                                                               
  WORD_SIZE: 64                                                                                                                                                     
  BLAS: libopenblas (USE64BITINT DYNAMIC_ARCH NO_AFFINITY Sandybridge)                                                                                              
  LAPACK: libopenblas64_                                                                                                                                            
  LIBM: libopenlibm                                                                                                                                                 
  LLVM: libLLVM-3.7.1 (ORCJIT, sandybridge)     

[robertfeldt]

Thanks @ranjanan , if you can update to latest 0.5 branch and try again on one of these machines that would be great since I do not see the same problems on any of my machines on latest 0.5. Would help a lot in pinpointing this.

[robertfeldt]

Actually, I ran a few times and now had the same failure as you did on latest 0.5. Ok, might be related to parallelization on 0.5 then since this never seems to happen on 0.4 for me.

[tkelman]

@robertfeldt these tests are consistently flaky on PackageEvaluator: http://pkg.julialang.org/detail/BlackBoxOptim.html

[robertfeldt]

Ok, thanks, @tkelman and @ranjanan , the problem seems to be on our side then and not solely related to 0.5. We will investigate but I'll need your help here @alyst , most likely.

[robertfeldt]

Ok, I added a Borg MOEA result stability test and ran it for both Julia 0.4 and 0.5 for different optimization times and it seems there really is a problem with stability on 0.5:

# Results below from running on Robert's MacBook Pro 13" from 2015, on 2016-07-29:

# julia04 -L src/BlackBoxOptim.jl examples/stability_test.jl 0.3 50
# Fitness 1: 1.999 +/- 0.007 (1.974-2.016)
# Fitness 2: 2.001 +/- 0.007 (1.985-2.026)
# Time: 15.028

# julia05 -L src/BlackBoxOptim.jl examples/stability_test.jl 0.3 50
# Fitness 1: 2.031 +/- 0.234 (1.655-2.85)
# Fitness 2: 2.059 +/- 0.423 (1.48-4.653)
# Time: 15.112

# julia04 -L src/BlackBoxOptim.jl examples/stability_test.jl 3.0 50
# Fitness 1: 2.0 +/- 0.005 (1.995-2.005)
# Fitness 2: 2.0 +/- 0.005 (1.995-2.005)
# Time: 147.778

# julia05 -L src/BlackBoxOptim.jl examples/stability_test.jl 3.0 50
# Fitness 1: 2.004 +/- 0.019 (1.955-2.048)
# Fitness 2: 1.996 +/- 0.019 (1.952-2.046)
# Time: 151.186

The variation is consistently higher on 0.5 than on 0.4 so something is different.

[robertfeldt]

I updated the smoketest so that it will hopefully give more stable results also on 0.5. This is a temp fix though, we should investigate this on 0.5 more and on more complex test functions to ensure stable results longer-term. @ranjan please check latest version and if it is more stable on your end, thanks.

[ranjanan]

@robertfeldt Thanks for updating the tests. They seem to be failing a lot less often. Tests failed for me only twice in around 10 tries, as opposed to consistently failing earlier.

[robertfeldt]

Ok, yes, we will revisit this once 0.5 is out to make sure test results are consistent. It is genuinely hard to test stochastic optimization algorithms though since there is always a small chance they will not converge... ;)

[robertfeldt]

@ranjanan if you care to test latest/HEAD on 0.5 on your side I'd appreciate it. With @alyst 's latest fixes I'm getting 10 out of 10 test runs on 0.5 without any failure so seems even more stable now and might be a candidate for a 0.2.2 tag.

The problem seems to have been that compilation time was sometimes messing up the MaxTime optimization limit => the "fails" meant that the Borg optimizer did not get much time to actually optimize... ;) So nothing wrong with the actual algorithm/optimizer itself.