Pre-compiling BlackBoxOptim causes sysimg error with multi-core start

[multidis]

This started as a general Julia issue on building pre-compiled images, but narrowed down to specifically BlackBoxOptim.jl causing it: https://github.com/JuliaLang/julia/issues/11063

Thought it made sense to cross-reference here. The Parameter-type may be involved.

PS: Very helpful package, thanks for making it public!

[robertfeldt]

Thanks for using BlackBoxOptim and for letting me know you find it useful. I haven't heard much feedback about it and github does not give download statistics so I have not been super-motivated to keep up the work on it unless my research needs it. But there are large updates coming up, especially when it comes to multi-objective optimization.

Anyway, back to the issue. Have you understood any more what might cause it? I have had problems with this type before and I'm sure there is a bug with it but I'm not sure at all where it is. Would be great with a small test case or script that shows it happening. Only happening with build sysimg and parallel threads is a bit too involved for my taste and hard to ensure for a test case. Have you seen this in any other situation?

The Parameters type seems to be a bit of a hack though so I have wanted to fix the whole parameter parsing issue in a better way anyway. Any input appreciated.

[multidis]

The issue is indeed fairly involved and comes up upon pre-compilation only. I had limited progress with debugging and hope to hear from someone working with Julia parallelization on that filed bug report. The core code also changes quite a bit between Julia releases so it is difficult to keep up with my debugging. It could be something as simple as a name clash but that is yet to be explored.

Otherwise, I have not noticed any problems running the package functions, in particular :adaptive_de_rand_1_bin_radiuslimited. I was actually in the process of implementing Tsallis (generalized) simulated annealing (SA) in Julia when I discovered BlackBoxOptim and this particular implementation of evolutionary methods outperformed SA for a few problems of interest. Stopping criteria are a bit unclear (currently using your heuristic 20000 steps) but convergence is a challenging question for genetic algorithms. In case you are familiar with any recent developments in the literature the implementation of some stopping logic would help quite a few use cases.

[multidis]

This is getting more broad (back to the Parameters-type issue): now a related error while using Lexicon.jl docstring processing.

Basically, adding using BlackBoxOptim to any module produces errors not only with pre-compilation but also with an (unrelated) docstring generation.

For example, in TradingLogic.jl package, when adding BlackBoxOptim (NOT in the current master-branch) and generating documentation with

using Lexicon, TradingLogic
save("api.md", TradingLogic)

this error comes up:

ERROR: start has no method matching start(::Parameters) in Config at /home/.../.julia/v0.3/Lexicon/src/render.jl:34 in save at /home/.../.julia/v0.3/Lexicon/src/render.jl:64

Once using BlackBoxOptim is removed, the error is gone. The same Parameters-type is mentioned in the error message with pre-compilation above. Perhaps Lexicon code could make it easier to sort out what is happening as compared to Julia base code.

[robertfeldt]

Ok, thanks for the report. I'm traveling this week but will then investigate and find a better way than the Parameters type to do this.

[multidis]

To simplify testing, the shortest example: error just by loading 2 packages

using BlackBoxOptim
using DataFrames

ERROR: convert has no method matching convert(::Type{Dict{Symbol,Any}}, ::Parameters) in Metadata at /home/.../.julia/v0.3/Docile/src/Docile.jl:52 in include at ./boot.jl:245 in include_from_node1 at ./loading.jl:128 in reload_path at loading.jl:152 in _require at loading.jl:67 in require at loading.jl:54 in include at ./boot.jl:245 in include_from_node1 at ./loading.jl:128 in reload_path at loading.jl:152 in _require at loading.jl:67 in require at loading.jl:51 while loading /home/.../.julia/v0.3/Docile/src/Docile.jl, in expression starting on line 95 while loading /home/.../.julia/v0.3/DataFrames/src/DataFrames.jl, in expression starting on line 16

[MichaelHatherly]

Thanks for notifying me about this @multidis. From a quick look at this it seems that the merge call in Docile is dispatching to the wrong merge, one defined here. The signatures are:

merge(p1::Union(Dict{K,V},Parameters),p2::Union(Dict{K,V},Parameters))

and for Base:

merge(d::Associative{K,V},others::Associative{K,V}...)

so the one in BlackBoxOptim is being picked in preference to the usual one in Base when Docile calls it.

I'm not sure how best we should handle this. @robertfeldt can that signature be made more specific? (I've not had a chance to read through that area of the package in-depth yet.)

OT, @robertfeldt

github does not give download statistics

If you go to the graphs link on the right of the repo page and then click on the "traffic" tab you should be able to see "clones" count, which is basically a download stat.

[multidis]

This PR seems to resolve the issue, my fork now works fine with Lexicon.jl and DataFrames.jl https://github.com/robertfeldt/BlackBoxOptim.jl/pull/8