Non-population optimizer test fails intermittently

[tkelman]

I've been running PackageEvaluator a bunch over the past few weeks and this package has been intermittently failing. Something about the tolerances or random inputs, or an actual bug?

Top-level interface
  > run a simple optimization
    > using bboptimize() with mostly defaults
    > using bbsetup()/bboptimize() with mostly defaults
    > using non-population optimizer
      Failure :: (line:-1) :: using non-population optimizer :: fact was false
        Expression: best_fitness(res) --> less_than(1.0)
          Expected: 3.5080192708609457 < 1.0
    > using population optimizer
  > continue running an optimization after it finished
  > continue running an optimization after serializing to disc
Out of 46 total facts:
  Verified: 45
  Failed:   1

[robertfeldt]

Should only be too tight tolerances so I upped the iteration count and hopefully this should be very unlikely. Get back to me if not.

[tkelman]

Appreciate it! I'll have to check in a non-pkgeval run to see if I can reproduce the issue and if checkout to master makes it go away. PkgEval runs on the most recently tagged version.

[robertfeldt]

Ok, I run a number of times here without reproducing it now so should be very unlikely. Since most of our testing is "random" (i.e. starts from random seeds) it will sometimes happen though; this is the nature of testing non-deterministic opt algs imho since fixing seeds would not provide as diverse testing. So I'd rather tune the tolerances...

[tkelman]

I got another one happening at a different place, probably also intermittent?

bboptimize smoketest
  > testing resampling_memetic_search method to ensure it works
  > testing de_rand_2_bin_radiuslimited method to ensure it works
  > testing de_rand_2_bin method to ensure it works
  > testing de_rand_1_bin_radiuslimited method to ensure it works
  > testing adaptive_de_rand_1_bin_radiuslimited method to ensure it works
  > testing adaptive_de_rand_1_bin method to ensure it works
  > testing separable_nes method to ensure it works
  > testing dxnes method to ensure it works
ERROR: LoadError: LoadError: AssertionError: target[2]=NaN is out of [-2.0, 2.0]
 in apply! at /home/vagrant/.julia/v0.4/BlackBoxOptim/src/genetic_operators/embedding/random_bound.jl:27
 in update_parameters! at /home/vagrant/.julia/v0.4/BlackBoxOptim/src/natural_evolution_strategies.jl:164
 in tell! at /home/vagrant/.julia/v0.4/BlackBoxOptim/src/dx_nes.jl:104
 in run! at /home/vagrant/.julia/v0.4/BlackBoxOptim/src/opt_controller.jl:173
 in run! at /home/vagrant/.julia/v0.4/BlackBoxOptim/src/opt_controller.jl:282
 in bboptimize at /home/vagrant/.julia/v0.4/BlackBoxOptim/src/bboptimize.jl:44
 in anonymous at /home/vagrant/.julia/v0.4/BlackBoxOptim/test/test_smoketest_bboptimize.jl:12
 in context at /home/vagrant/.julia/v0.4/FactCheck/src/FactCheck.jl:474
 in anonymous at /home/vagrant/.julia/v0.4/BlackBoxOptim/test/test_smoketest_bboptimize.jl:7
 in facts at /home/vagrant/.julia/v0.4/FactCheck/src/FactCheck.jl:448
 in include at ./boot.jl:261
 in include_from_node1 at ./loading.jl:304
 [inlined code] from /home/vagrant/.julia/v0.4/BlackBoxOptim/test/runtests.jl:32
 in anonymous at no file:0
 in include at ./boot.jl:261
 in include_from_node1 at ./loading.jl:304
 in process_options at ./client.jl:308
 in _start at ./client.jl:411
while loading /home/vagrant/.julia/v0.4/BlackBoxOptim/test/test_smoketest_bboptimize.jl, in expression starting on line 5
while loading /home/vagrant/.julia/v0.4/BlackBoxOptim/test/runtests.jl, in expression starting on line 31
============================[ ERROR: BlackBoxOptim ]============================

failed process: Process(`/home/vagrant/julia/bin/julia --check-bounds=yes --code-coverage=none --color=no /home/vagrant/.julia/v0.4/BlackBoxOptim/test/runtests.jl`, ProcessExited(1)) [1]

[robertfeldt]

@alyst you may be better equipped to check this one since it seems to be dxnes related?

[alyst]

@robertfeldt Sorry for long the delay. From the trace it looks like DX-NES has diverged (sigma step size overflow). I've just made a PR #37 that might improve the situation.

[robertfeldt]

@alyst no worries, I've had lots to do myself. Thanks, I'll check PR #37 in a day or so ad then merge all your recent changes if things look good.

[robertfeldt]

Ok, I've merged so lets see if @tkelman keeps having intermittent failures...

[tkelman]

Can you publish a tag to metadata that has this fix? PackageEvaluator is still running the 0.0.1 tag.

[robertfeldt]

Ok, I tagged and published 0.1.2.

[tkelman]

Thanks, hopefully that'll take you off my intermittent failures list. I'll be sure to let you know if I see otherwise when I'm working on the next backport release.