Use automatic differentiation to calculate derivatives

[speth]

Abstract

Providing derivatives of quantities in Cantera using automatic differentiation would enable the use of many sophisticated capabilities, such as sparse iterative solvers and machine learning algorithms. It would also eliminate the need for many tedious and error prone manual derivatives currently calculated within Cantera.

Motivation

Most of the computationally expensive routines in Cantera involve linear algebra. Integrating stiff chemical kinetics requires factorization of Jacobian matrices, which grow in size with the kinetic model. Jacobian matrix sparsity, which reduces the cost of factorization, can be obtained by evaluating the matrices through automatic differentiation, rather than the current finite difference evaluation. For the comparatively simple case of ideal gas kinetics, sparse Jacobians can be derived analytically with some effort, for example Cantera/cantera#951. However, I don't think this scales well beyond that, for example to real gases.

Many calculations in Cantera inherently involve calculations of derivatives, and manual implementations of these calculations could be avoided if they could be provided by automatic differentiation instead. For example, while c_p for an ideal gas is calculated trivially by differentiating the NASA polynomials, this calculation is much more complex in non-ideal models, e.g. RedlichKwongMFTP::cp_mole which in turn invokes the pressureDerivatives, dpdVCalc, and da_dt methods. And this is all for a comparatively simple equation of state. For a real treat, see the 4000+ line implementation of the HMWSoln class, most of which is spent calculating various derivatives.

Most machine learning algorithms require calculations of gradients. If Cantera were able to provide this information without introducing new methods for calculating every possible derivative, it would greatly enhance Cantera's ability to be used in such algorithms.

Possible Solutions

There are several libraries for doing automatic differentiation of C++ codes. I think it is probably worth experimenting with a couple of them to see which ones can be used in the least complex/invasive manner in Cantera. Some possibilities:

• Adept 2 - seems to be mature, well documented, and actively maintained
• autodiff - A relatively new header-only library that uses C++17 features
• Enzyme - Automatic differentiation based on the LLVM intermediate representation with minimal or no changes to the existing library code. Bleeding edge.

I have no experience with any of these packages, so suggestions of packages you have used and like are welcome.

One complexity I forsee is figuring out how AD would work if there's any desire to cross language barriers, such as user-defined models in other languages as envisioned in #79, or with ML algorithms. I don't know whether it is sufficient to have numerical values for derivatives at the language interface, or whether a more complex data type needs to be transmitted (if that's even possible).

References

• Cantera/cantera#951 - Adding adaptive preconditioner functionality to Cantera

• 19 - A manual approach to calculating Jacobian elements analytically

P.S. This post is updated to reflect some of the comments below

P.P.S.: I labeled this as a "feature request" because while it is on my to-do list, I have several other projects that I'm prioritizing over this, so this could get done much sooner if someone else wants to tackle it.

[ischoegl]

@speth ... I am really happy to see this request (I'm also tagging @jiweiqi here, as this is to some extent a spillover from a discussion on the user group). At some point, I looked into writing a Jacobian in Python (which may now be more workable with numba), so I am following these discussions closely. If some of the model equations were re-written in Julia instead, I guess alternative automatic differentiation approaches from other data science research would apply (see also #79).

[ischoegl]

@jiweiqi ... my interest pertained to reaction rate expressions, see here (it's been a while). We ended up evaluating Jacobians numerically.

[ischoegl]

@jiweiqi ... let's please focus on a discussion on how to get AD capabilities into Cantera.

[speth]

@jiweiqi, Yes, I suppose automated symbolic differentiation is an alternative to AD in at least some places, although I'm not sure how well it scales or generalizes to code that contains branches. The paper you linked to is light on the details, to say the least.

From the documentation and paper on Adept, it's clear that efficient Jacobian calculations is very much something that they're targeting, and they claim competitive performance with hand-written Jacobians.

In any case, I think the main goal of doing something smarter than generating a dense finite difference Jacobian is to improve the "big-O" performance of the integrators. If there's a small multiplicative constant on the Jacobian evaluation, it's not something I'm going to worry about. Furthermore, I expect that the changes that will be needed to enable any AD approach (namely, making a lot of Cantera classes templated on the type of their input/output) will make it easier to try other approaches like symbolic differentiation as well.

[ischoegl]

Linking #47 here as it is related (via Cantera/cantera#951).

[anthony-walker]

Adaptive Preconditioning was moved to from #951 to #1010.

[corykinney]

I'm curious about this and its applications to the Jacobians and ML algorithms. Is there still interest in utilizing AD in Cantera? If so, is there a consensus on what approach might be most flexible?

There are several libraries for doing automatic differentiation of C++ codes. I think it is probably worth experimenting with a couple of them to see which ones can be used in the least complex/invasive manner in Cantera. Some possibilities:

* [Adept 2](https://github.com/rjhogan/Adept-2) - seems to be mature, well documented, and actively maintained

* [autodiff](https://github.com/autodiff/autodiff) - A relatively new header-only library that uses C++17 features

* [Enzyme](https://github.com/wsmoses/Enzyme) - Automatic differentiation based on the LLVM intermediate representation with minimal or no changes to the existing library code. Bleeding edge.

I have no experience with any of these packages, so suggestions of packages you have used and like are welcome.

I don't have a ton of experience in this area, so I wanted to check before potentially messing around with it as I don't want to waste time going the wrong direction.