Add more optimisers!

[samuel-watson]

• [ ] NEWUOA: There's a c++ implementation in this library: https://github.com/elsid/newuoa-cpp but also look at other libraries. see comment below for better implementation
• [x] LBFGS: https://github.com/ZJU-FAST-Lab/LBFGS-Lite/tree/master

Optimisers can easily be added using the direct optim class, see how BOBYQA and DIRECT are added here. Basically, create a new specialisation of the optim<T(const std::vector<T>&),algo> class, copying the function binding methods and then just define the minimise function.

[zaikunzhang]

Hi @samuel-watson,

If you are interested in NEWUOA and other derivative-free solvers by the late Professor M.J.D. Powell, I suggest you have a look at PRIMA:

https://github.com/libprima/prima

The implementation you mentioned is based on the Fortran 77 version of NEWUOA, which is buggy, unmaintained, and should not be used any more.

Thanks.

Zaikun

[samuel-watson]

Hi @zaikunzhang Thanks so much for the pointer - I'll use the implementation you point to. Ideally I'm looking for derivative-free solvers that are competitive with BOBYQA for minimising convex functions - nothing seems to beat BOBYQA for the problems I have tried but NEWUOA looked promising.

[zaikunzhang]

PRIMA provides BOBYQA as well, in addition to NEWUOA, UOBYQA, LINCOA, and COBYLA.

The PRIMA implementation of BOBYQA performs much better than any version based on the original Fortran 77 implementation in terms of number of function evaluations.

[samuel-watson]

That's interesting @zaikunzhang . We are currently using this implementation, which looks like its based on the old Fortran version - so I'll update that as well.

[zaikunzhang]

That's interesting @zaikunzhang . We are currently using this implementation, which looks like its based on the old Fortran version - so I'll update that as well.

Yes, I am sure it is, considering the GOTO's. Indeed, PRIMA is the first (and only) modernized implementation of Powell's solvers, in the sense that it is totally GOTO-free and makes use of matrix-vector procedures.

[samuel-watson]

Hi @zaikunzhang this looks great. My knowledge of Fortran is basically nonexistent though so even with the C bindings I'm not sure how to incorporate it. The problem is this package is mainly written in C++ to then be used in a package for R. Unfortunately R's package repository CRAN is really strict about things to include in package, so you couldn't incorporate a dll or anything pre-built and it looks like you have to build the fortran program first and then link to the C program. I'll try to figure this out in the future. It'd be great if this could eventually be used with R's C++ interface as there are a huge number of statistical applications this would be great for.

[zaikunzhang]

Hi @samuel-watson ,

It would be great if PRIMA can be bonded to R. Although I am ignorant of R, I will be very glad to provide assistance from the Fortran side.

With the binding / implementation of PRIMA in C, Python, Julia, and MATLAB all being worked on, the only major scientific-computing-oriented language missing from the list is R.

For your information, see

https://github.com/lme4/lme4/issues/744

https://github.com/libprima/prima/issues/104

[samuel-watson]

Thanks @zaikunzhang - interesting to see that they are looking at it for lme4. For R there are two ways that are of interest: making the fortran functions available as calls in R, and making the fortran code callable from C++ for R (with Rcpp). I'm interested in the latter as a lot of useful code and R packages are written in C++, like this package. It is possible to use both C++ and Fortran in R, see here. However, that example has a simple fortran function rather than a larger library like prima. The C code is useful, but trying to get an R package working with what looks like a relatively complex build process with CMake is tricky with the (frustrating) CRAN standards. I'm sure there must be a way of extending the methods in that webpage and using the C code, the question is just how you could get it to build as an R package.

[samuel-watson]

@zaikunzhang Your comments have me thinking whether it would be worth developing a native C++ implementation of these algorithms using modern C++ and incorporating your improvements to the algorithms. Certainly a lot of work, but would be interesting to translate your project as a header only library. I'd be interested in giving this a go if that would be of interest. I can open an issue on the prima repo

[zaikunzhang]

Hi @samuel-watson ,

Certainly a lot of work, but would be interesting to translate your project as a header only library. I'd be interested in giving this a go if that would be of interest. I can open an issue on the prima repo

Certainly! This is highly interesting. Indeed, just this morning I was thinking about the native C or C++ implementation of PRIMA. However, my knowledge about C/C++ is limited and outdated, so I cannot do that myself. It would be great if you could do it.

Thank you very much!