Search is not an exported method

[JonnyCBB]

Hi @phrb. So I've come stuck at the next hurdle. In the README it states

Now we are ready to use the search method, which could be called with just a Configuration and a function that accepts a Configuration as input.

The problem is that search doesn't seem to be an exported method so I end up with the following error:

ERROR: LoadError: MethodError: `search` has no method matching search(::Function, ::StochasticSearch.Configuration{StochasticSearch.NumberParameter{AbstractFloat}})
 in include at boot.jl:261
 in include_from_node1 at loading.jl:304
while loading C:\Users\jonathan\Documents\UniOfOxford\HelpingOthers\DavidBrown\FRAP_filter\filter_FRAP.jl, in expression starting on line 350

What method should I use if I want to run simulated annealing?
I assume something in the README or the code needs to be changed.

Thanks again in advance for helping me out

[phrb]

Hi @JonnyCBB, I'm really sorry for this. I'll have to rewrite the README because there are multiple undocumented changes. One of the changes is that search was renamed optimize. But this will not make the README example work.

Please, use the following as a substitute for the example in the README:

I'll use the rosenbrock example. The code for this example is available here.

@everywhere begin
    using StochasticSearch
    function rosenbrock(x::Configuration, parameters::Dict{Symbol, Any})
        return (1.0 - x["i0"].value)^2 + 100.0 * (x["i1"].value - x["i0"].value^2)^2
    end
end

This part just defines the cost function in every Julia process. Then, we define the configuration, the same way as before:

configuration = Configuration([FloatParameter(-2.0, 2.0, 0.0, "i0"),
                               FloatParameter(-2.0, 2.0, 0.0, "i1")],
                               "rosenbrock_config")

Now, instead of using a Dict, we build a tuning run configuration using the Run type. This type encapsulates all tuning run parameters. You can see its code here.

We use the methods array to define all search methods that will be used, and the number of instances of each one. This example uses one instance of every implemented method. Note that report_after and duration are in seconds, by default:

tuning_run = Run(cost               = rosenbrock,
                 starting_point     = configuration,
                 methods            = [[:simulated_annealing 1];
                                       [:iterative_first_improvement 1];
                                       [:randomized_first_improvement 1];
                                       [:iterative_greedy_construction 1];
                                       [:iterative_probabilistic_improvement 1];],
                 stopping_criterion = elapsed_time_criterion,
                 report_after       = 4,
                 duration           = 30,
                 measurement_method = sequential_measure_mean!)

Finally, we create a search task using the optimize method, and consume the task for the results:

search_task = @task optimize(tuning_run)
result = consume(search_task)
print(result)
while result.is_final == false
    result = consume(search_task)
    print(result)
end

I hope this works for you. Thanks for not giving up, and I'm really sorry for this terrible user experience. I'll update the wiki and the README. Please, keep posting if you find more hurdles, or need help of any kind.

[JonnyCBB]

No problem whatsoever. I trust you know what you're doing and this was the best simulated annealing package I could find in terms of flexibility and documentation so I'll continue to persevere with this package. Plus you've been really quick and responsive at getting back to me.

I'm busy for the next few hours but I'll try what you suggested later on and get back to you about it.

[phrb]

Thanks @JonnyCBB. I've updated the README example as well. If you can, please tell me what features you intend to use so I can document them first.

One more thing, make sure you are using the latest version from github:

julia> Pkg.clone("StochasticSearch")

[JonnyCBB]

Hi @phrb thank you very much for your comprehensive explanation of the problem and solution. I've managed to get it working. I appreciate the amount of work you put into this to make it work for me.

I'll close the issue because you have solved the original problem but I do have 2 more questions:

1) for the method field of the Run type, I'm not certain what it means to specify the method e.g. simulated annealing and then the integer that comes after it. Also does applying several methods to a run mean that the methods are applied consecutively during the search?

2) The output gives a Starting Configuration. This isn't necessarily the initial configuration that is specified by the user. What does the Starting Configuration represent?

Sorry for all of the questions.

[phrb]

Hi @JonnyCBB, that was no problem at all, I've been working on this project for some time and I really appreciate that someone took the time to use it. Please feel free to ask more questions, I'll be glad to answer them and help you to use the package.

1a The methods array has the format:

methods::Array{Any, 2} = [[:method1_name::Symbol instances::Integer];
                          [:method2_name::Symbol instances::Integer]; ...;]

This means that methods is a matrix with two "columns" and n "rows", where n is the number of methods specified. Every method Symbol specifies a search technique and is a Function name, such as :simulated_annealing. The integer that follows is the number of concurrent instances of that technique running on the same problem. Different instances of a technique may search different parts of the search space and use different starting configurations, but every one uses the same search algorithm.

1b) Applying several method instances creates separate Julia Tasks, that will then use remotecalls to distribute the calculations of cost measurements between different Julia workers or processes. This allows the measurements to be made distributed or in parallel if there are Julia workers active in different processor cores or machines. Using different techniques on the same problem provides different ways to explore the search space, and hopefully speeds up finding good solutions in complicated spaces.

2) When starting a tuning run, StochasticSearch perturbs the user-provided starting configuration generating a group of candidate starting points. The starting configuration listed in the output is the lowest-cost point between all those measured candidates, including the user provided point.

[JonnyCBB]

@phrb Once again, thank you so much for your thorough explanation. That all makes sense. I didn't know these types of distributed techniques could have such a simple API. This is so much better than I thought it already was. I was only going to run simulated annealing search technique but I may as well try others. You're an absolute legend.

As an aside, I stumbled across this package through quite a lot of Googling. In fact I didn't know these techniques as "Stochastic Search" algorithms. I was just looking for a package that had a nice API for simulated annealing. I think this package should be part of an organisation like JuliaOpt (assuming that it fits nicely in that umbrella). Have you got any plans to incorporate that there? It would've saved me so much time searching and I think more people would see it and use it. It's a brilliant package!

[phrb]

Wow thanks @JonnyCBB ! I took the ideia of using ensembles of search techniques from OpenTuner, an autotuning framework written in python.

I research program autotuning, and I wrote the package to make it easy to study parallel and concurrent measurement of cost functions. I thought of those cost functions as measurements of program execution time or energy consumed by an execution, for example, and the autotuner would use stochastic search to find good program optimizations according to these objectives. That's why the set of parameters is called a "configuration", as in a program configuration, and the optimization process is called a "tuning run". Of course the cost functions and sets of parameters can be anything, so the package ends up also working as a general-purpose tool.

I would really like to be part of some bigger Julia project or organisation, but I believe the package needs to mature a bit more before that!

Thanks for your feedback, I hope the package is useful and I'm looking forward for more questions.

[phrb]

Hey @JonnyCBB,

What did you end up doing with the package? Did you find any other hurdles?

[JonnyCBB]

Thanks for everything @phrb. It works really well and I haven't come across anymore hurdles. I really appreciate you taking the time to make the changes for me.

I was using it because I've been doing some modelling for a friend of mine. Basically we have a system of reaction diffusion equations for some FRAP data (Fluorescence Recovery After Photobleaching) and there are quite a few parameters - 9 in total I think. He's trying to write a paper to describe the dynamics that he thinks are going on and asked if I could see what the models say. So far so good but the real test comes over the next few days to see if the parameter values he expects are going to come out. So it's finished yet and I'm still using your package

[phrb]

@JonnyCBB No problems, it was actually fun, I got motivated to write a proper documentation page, that I'll be updating. I'm really glad it is working for you, If you have any questions or need any help, feel free to ask!

Did you use more than one Julia worker? I would like to see how the package's parallelism performs in your application. I can help you set it up if want to.

I'll release a new version, with a new search technique (Iterated Local Search) that could also be useful. The lastest version is already in GitHub.

[JonnyCBB]

@phrb Sorry for the late reply. I'm currently writing my thesis on top of the modelling and I just completely forgot to get back to you.

I've been using 2 workers (setting instances::Integer=2 if that's what you mean) but I haven't played around with this. Exploring this would actually be pretty useful because the reaction diffusion equation takes a fairly long time to run - 10,000 iterations takes about half a day or something like that. If you have some tips, I would very much welcome them.

I don't know the Iterated Local Search algorithm so that would be new to me.

[phrb]

@JonnyCBB No problem at all, good luck on your thesis!

Actually I meant actual Julia processes, or workers. Try setting instances=2 at the tuning run configuration, and add a call to addprocs(n) at the top of your script. You can make n=2 for two Julia processes, and bigger ns to test the performance. You'll have to adjust both the number of instances and Julia processes to improve your performance.

I'm available to help in any hurdles you may find!