How should I install the package (to be used as a fortran dependency)?

[beddalumia]

Hi, I saw this package some week ago in the Fortran-lang discourse. It rose a lot of interest in my group, as we have widespread needs for well written optimization packages. Now we are starting to explore a bit on a new project and came here to try adding PRIMA as a dependency. I know there are plans to make it buildable with fpm, but since we are not there yet, and I cannot see a CMake or other build systems around (except for the matlab-specific MEX setup), how can we deal with fortran source building? Looking at the actions for the testsuite has not helped so far, maybe we are missing something.

[zaikunzhang]

Hi, I saw this package some week ago in the Fortran-lang discourse. It rose a lot of interest in my group, as we have widespread needs for well written optimization packages. Now we are starting to explore a bit on a new project and came here to try adding PRIMA as a dependency. I know there are plans to make it buildable with fpm, but since we are not there yet, and I cannot see a CMake or other build systems around (except for the matlab-specific MEX setup), how can we deal with fortran source building? Looking at the actions for the testsuite has not helped so far, maybe we are missing something.

Hi @bellomia thank you for your interest. Would the Makefiles in fortran/examples help? You are right --- as mentioned on Fortran discourse, the Fortran building facility for this package is (totally) missing. My intention was first and foremost to make the solvers available under MATLAB and Python. Now I have very limited time, if any, to work on it.

Before you start exploring these solvers, I invite you to have a look at https://www.pdfo.net/docs.html#introduction (it is a webpage for PDFO, which provides the F77 version of the solvers). I want to make it crystal clear that these solvers are NOT intended to be used as general-purpose solvers. They are designed for a class of VERY SPECIAL problems, namely derivative-free / black-box optimization problems. For problems not in this category, MUCH BETTER solvers exist, and it is STRONGLY discouraged to use the solvers here. Below I quote the above mentioned webpage. See the first two sections of https://arxiv.org/pdf/2302.13246.pdf for more information.

"Professor Powell devised these solvers to tackle general nonlinear optimization problems of continuous variables with or without constraints using only function values but not derivatives of the objective function or nonlinear constraint functions. In practice, such functions are often black boxes defined by simulations. Consequently, the corresponding optimization problems are often categorized as black-box optimization or simulation-based optimization. Problem specified by explicit formulas can probably be handled by other methods more efficiently."

[jschueller]

I added cmake files for this purpose in #28 to install the fortran & C lib

[beddalumia]

@zaikunzhang Thanks a lot for the very detailed clarification. We are more or less into the proper domain of applicability, but to be honest we are indeed still investigating on which minimizer would best fit our needs.

More in general we maintain https://github.com/QcmPlab/SciFortran and may potentially be interested in embedding PRIMA into the library. Would you be comfortable with us doing that? License wise it should be feasible, if I'm not missing something obvious. Any comment or advice in this regard would be very welcome. Currently SciFor implements some of the methods in SciPy's minimize package, and Powell's methods have been for a long time in the TODO pipeline :)

@jschueller thanks a lot for the effort, much appreciated! I will look closely into the fate of that PR :)

[zaikunzhang]

Hi @bellomia ,

More in general we maintain https://github.com/QcmPlab/SciFortran and may potentially be interested in embedding PRIMA into the library. Would you be comfortable with us doing that?

Absolutely. I am very glad if you embed PRIMA in your library. This is why I work on PRIMA --- to make Powell's methods easily available to as many people as possible.

I am in academia, so citation and acknowledgment are all that matter to me. As long as you do the following, it is perfect for me.

1. Acknowledge in your code and documentation that your library includes code (derived) from PRIMA, and make a link to https://github.com/libprima/prima .

2. In any paper / technical report / manual about your library, cite PRIMA as follows:

   • Z. Zhang, PRIMA: Reference Implementation for Powell's Methods with Modernization and Amelioration, available at http://www.libprima.net, 2023

I will write a paper on PRIMA later. When the paper is ready to cite, I will inform you to update the citation.

3. Respect the license of PRIMA, which is 3-clause BSD. The same as you, I am flexible with the license. If it is not compatible with your need, let us discuss it.

We are more or less into the proper domain of applicability, but to be honest we are indeed still investigating on which minimizer would best fit our needs.

It will be super interesting to me if your optimization problem is a true derivative-free / black-box problem. If you are comfortable, I am interested in hearing more about this problem. I need interesting applications to support my research.

Thanks.

Best regards, Zaikun

[zaikunzhang]

Hi, @bellomia , I should give another comment about derivative-free optimization (DFO).

In DFO, we assume that the major cost comes from the evaluation of the objective / constraint functions, not from the numerical linear algebra of the solver. In practice, it is not uncommon that each function evaluation takes minutes, hours, or even days to finish. Therefore, the numerical linear algebra cost of the solver is negligible.

This justifies the following.

1. In the benchmarking of PRIMA against Powell's F77 implementation, I use the number of function evaluations to measure the cost / performance of the solvers. If you measure the computing time, the results will be totally different, because the functions of the test problems (see MatCUTEst ) are inexpensive to evaluate.

2. In PRIMA, I use matrix-vector procedures instead of loops, even if the former is less efficient in terms of computing time. When I need to choose between efficiency and simplicity&clarity of the code, I always choose simplicity&clarity. This is not a good choice if we were not dealing with DFO.

Thanks, Zaikun

[zaikunzhang]

Hi @beddalumia ,

FYI, PRIMA now has a CMake building system, thanks to @jschueller .

I hope this is useful to you.

Are you still interested in including PRIMA into SciFortran?

Best regards, Zaikun

[beddalumia]

Hi @zaikunzhang!

Sorry for the very late response, I've been extremely busy between writing a paper and going to conferences. Also, the much needed holiday break did not help. I sincerely apologize.

Coming to the questions, in inverse order:

• Are you still interested in including PRIMA into SciFortran?

  Yes, but this has a little went to the back in our priority list, right now. We are definitely interested, but for now the Fortran project has been temporarily halted while we work long term on a new solver (it makes for the heaviest part of the cost-function evaluation, in the method). For now we are running quick experiments in Julia, where we can play around with an existing library for designing said new solver. The main end goal remains to have a Fortran package to embed in our wide ecosystem, but we'll have to wait for the exploratory phase to reach its goals.

• Thanks for the detailed info about licensing and how to cite your work. We are perfectly fine with all the conditions. When we would eventually return to it I may contact you to discuss the finer details, but overall looks good and fair.

• It will be super interesting to me if your optimization problem is a true derivative-free / black-box problem. If you are comfortable, I am interested in hearing more about this problem. I need interesting applications to support my research.

  Thanks for the extensive explanation, truly appreciated. And by the way I like the guiding principles in the development of PRIMA. Regarding our application: it is a newly proposed scheme to solve strongly correlated electronic problems in condensed matter. The optimization step is essentially a multi-root problem, which can be as usual formulated in many ways (self-consistent loops, which is what has been explored the most in the emerging literature... or multivariable, nonlinear optimization). But this multi-root condition arises from equating the ~wavefunctions (more accurately they are a kind of "density matrices") evaluated in two distinct auxiliary problems, to which the original many-body problem is mapped into. In particular one of these two auxiliary problems is quite heavy to solve for the relevant density matrix and constitutes the genuine dominant bottleneck in the whole method. Hence, each evaluation of the cost function, for a given set of parameters (to be optimized) requires a full solution of such hard problem (you can imagine a few minutes to half an hour, depending on the physical system).

  The current, working, implementation we have is based on MINPACK and formulated as a genuine multi-root problem, for which we cannot provide a Jacobian. Hence we spend a significant amount of time at evaluating the cost function to numerically estimate the derivatives. In this sense a derivative-free optimizer, specifically tailored to minimize cost-function calls would probably fit the bill quite well. The current, work-in-progress, Julia port is still relying on MINPACK.jl (we wanted to avoid changing to many things at first, so to really compare if the new solver for the hard auxiliary problem is giving a significant advantage). But we will soon try to move to the Powell's methods available in the SciML ecosystem and explore if and how they improve our convergence behavior (and cost!). Then hopefully, we could return to the Fortran package (after backporting the solver, which is already an ongoing effort) and evaluate to adopt PRIMA instead of MINPACK, therein.

I hope this information is clear enough and of some value to you. I did not explicitly reference the method name and all actual details since for now we want to make progress without attracting too much attention, on a public venue.

Final note: more in general there is a whole class of methods, relevant for the theoretical study of strongly correlated materials, that amount to map the original, intractable, electronic problem to a simpler (set of) easier problems. The mapping holds and gives access to the physics of the original system under suitable self-consistent conditions, that may often be cast as multi-root problems (not what is usually done nowadays, for the common implementations of this idea). The auxiliary problem(s) to which the intractable model is mapped to are tractable but always significantly hard (no free lunch I'd say) so evaluating many times the cost-function brings these methods to potentially large timescales on HPC clusters. Currently the number of correlated orbitals one can address (and so the range of materials one can study) is quite limited in fact, for this exact reason (other than intrinsic limitations on how one can or cannot solve the auxiliary problem). So if we succeed with this specific instance (which is particularly nasty about the optimization step, since it involves two auxiliary problems and hence a lot of parameters and constraints) we may want to extend the gained know-how to other incarnations of the general idea. To end with a pinch of excitement: moving higher the bar with the number of correlated orbitals is relevant for high-temperature superconductors (among many things). :)

[beddalumia]

I close the issue, since the merged CMake installation effectively solves the underlying problem I had. Thanks again for the effort and reactive response!