Cooperation

[pkofod]

We're/I'm still interested in cooperating and getting this in Optim. I was reluctant at some point, but adding StaticArrays as a dependency to Optim would be worth it to have this functionality in Optim.

edit: let me say why I was reluctant. My main problem was that I prefer not to depend on the actual type packages (numbers, arrays, ...) as it discourages generic interfaces. However, StaticOptim is almost stdlib, and the traits interface I had been hoping for in v1.0 that describes mutability is probably not coming any time soon.

[aaowens]

One straightforward way would be to copy and paste this code as is into Optim and have optimize(f, x::StaticArray, BFGS()) call the static version and return a StaticOptimizationResult, but this type doesn't have most of the fields of the expected MultivariateOptimizationResults. The code here doesn't currently support any optimization options, but it shouldn't be hard to add them.

I think it should be possible to fill in the fields of MultivariateOptimizationResults with some modifications to the code here, but I'm not sure what that would do to performance. Also, MultivariateOptimizationResults is mutable, so I think constructing it would allocate.

Would you be happy with a new type struct StaticOptimizationResults <: OptimizationResults which has most (all?) of the fields of MultivariateOptimizationResults?

[pkofod]

For now, I have no problem prepending Static on whatever needs a Static to be performant. But I will have a look at MultivariateOptimizationResults and see if we actually use that mutability anywhere.

[pkofod]

I should add, that I'd be happy to participate in "experimenting" in this repo before we move stuff into Optim.

[aaowens]

Ok, I started this over here https://github.com/aaowens/StaticOptim.jl/pull/17 . I left out some fields from MultivariateOptimizationResults because I wasn't sure what they were.

[pkofod]

I'm wondering, how much did you actually modify from LineSearches?

edit: seems like it's just checking the order

[aaowens]

There should be two changes.

1. Modify operators to work on StaticArrays
2. Skip the make ϕ functions by copying the code into the loop body. I couldn't make it not allocate without that. Possibly this is no longer necessary on 1.0.

[pkofod]

Okay, I will try to see if we can't just call the linesearches functions directly, as that would certainly simplify the code.

[pkofod]

Hm, I basically took the code for the line search, and inserted it into a function, and I get

julia> @btime  soptimize(rosenbrock, sx, StaticOptim.Order2(), nothing; tol = 1e-8)
  30.671 μs (901 allocations: 28.59 KiB)
Results of Static Optimization Algorithm
 * Minimizer: [0.9999999999990606,0.9999999999980389]
 * Minimum: [1.5610191722141176e-24]
 * Hf(x): [801.6874976886638,-399.8345645795701,-399.83456457957504,199.9124176978296]
 * Number of iterations: [31]
 * Number of function calls: [69]
 * Number of gradient calls: [31]
 * Converged: [true]

instead of

julia> @btime  soptimize(rosenbrock, sx, StaticOptim.Order2(), nothing; tol = 1e-8)
  2.398 μs (4 allocations: 192 bytes)
Results of Static Optimization Algorithm
 * Minimizer: [0.9999999999990606,0.9999999999980389]
 * Minimum: [1.5610191722141176e-24]
 * Hf(x): [801.6874976886638,-399.8345645795701,-399.83456457957504,199.9124176978296]
 * Number of iterations: [31]
 * Number of function calls: [69]
 * Number of gradient calls: [31]
 * Converged: [true]

I wonder what optimization we're cheating ourselves for by putting it in a function.

[aaowens]

I've also tried to put the linesearch code in a function, but my results are similar to yours. Using @inline makes no difference.

[pkofod]

I never got around to figuring this out, but might have some time tonight

[aaowens]

I tagged an initial version for this package and updated the readme. I'm not sure if you still want the functionality in Optim since there might not be a way to reuse the existing linesearch code. This package does a few other things besides multivariate optimization too (root-finding, univariate optimization) so it's useful by itself.

[pkofod]

That's fine! I have another package that covers the same things you mention in the making, that I intend to be a back-end for Optim, but since you're using this already don't let me hold you back on anything :)