Mismatch between Cantera Python and Cantera shared library

[dcmvdbekerom]

Problem description Good evening, first off congratulations with your recent 3.0.0 release 😄 !

I'm a developer over at Radis and Radis uses Cantera as a convenience tool to calculate equilibrium compositions.

As part of CI-tests we calculate some equilibrium composition to make sure Cantera is imported properly. Since the release of 3.0.0, the test is failing because there is a "mismatch between Cantera Python and shared library".

Steps to reproduce

For full steps please see here: https://app.travis-ci.com/github/radis/radis/jobs/608700261 Relevant lines are:

404  + cantera                           3.0.0  py38hb1af7a3_1          conda-forge/linux-64        2MB
477  + libcantera                        3.0.0  h6578b86_1              cantera/linux-64            3MB
920 Linking libcantera-3.0.0-h6578b86_1
953 Linking cantera-3.0.0-py38hb1af7a3_1

And finally lines 1531-1671:

__________________________ test_get_eq_mole_fraction ___________________________
initial_mixture = 'CO2:1', T_K = 3000, p_Pa = 100000.0
    def get_eq_mole_fraction(initial_mixture, T_K, p_Pa):
        """Calculates chemical equilibrium mole fraction at temperature T, using
        the CANTERA :py:meth:`~cantera.ThermoPhase.equilibrate` function.

        The calculation uses the default GRI3.0 mechanism, which was
        designed to model natural gas combustion, including NO formation
        and reburn chemistry. See `GRI 3.0 <combustion.berkeley.edu/gri-mech/version30/text30.html>`__.

        When using, cite the [CANTERA]_ package.

        Parameters
        ----------

        initial_mixture: str
            Gas composition. Example::

                 'N2:0.79, O2:0.21, CO2:363e-6'

            Or::

                 'CO2:1'

        T_K: float (K)
            temperature (Kelvin) to calculate equilibrium

        P_Pa: float (Pa)
            temperature (Pascal) to calculate equilibrium

        Examples
        --------

        Calculate equilibrium mixture of CO2 at 2000 K, 1 atm::

            get_eq_mole_fraction('CO2:1', 2000, 101325)

            >>> {'C': 1.7833953335281855e-19,
                'CO': 0.01495998583472384,
                'CO2': 0.9775311634424326,
                'O': 5.7715610124613225e-05,
                'O2': 0.007451135112719029}

        References
        ----------

        [CANTERA]_
        """

        try:
>           import cantera as ct
radis/tools/gascomp.py:58: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
../../../micromamba/envs/radis-env/lib/python3.8/site-packages/cantera/__init__.py:4: in <module>
    from ._cantera import *
build/python/cantera/_cantera.pyx:33: in init cantera._cantera
    ???
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
>   ???
E   **ImportError: Mismatch between Cantera Python module Git commit (be8cff9) and Cantera shared library git commit (806842dac)**
build/python/cantera/_utils.pyx:66: ImportError
The above exception was the direct cause of the following exception:
args = (), kwargs = {}, all_in = <function all_in at 0x7f55fa9164c0>
get_eq_mole_fraction = <function get_eq_mole_fraction at 0x7f55eb2b6b80>
    def test_get_eq_mole_fraction(*args, **kwargs):
        from radis.misc.basics import all_in
        from radis.tools.gascomp import get_eq_mole_fraction

>       gas = get_eq_mole_fraction("CO2:1", 3000, 1e5)
radis/test/tools/test_gascomp.py:15: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
initial_mixture = 'CO2:1', T_K = 3000, p_Pa = 100000.0
    def get_eq_mole_fraction(initial_mixture, T_K, p_Pa):
        """Calculates chemical equilibrium mole fraction at temperature T, using
        the CANTERA :py:meth:`~cantera.ThermoPhase.equilibrate` function.

        The calculation uses the default GRI3.0 mechanism, which was
        designed to model natural gas combustion, including NO formation
        and reburn chemistry. See `GRI 3.0 <combustion.berkeley.edu/gri-mech/version30/text30.html>`__.

        When using, cite the [CANTERA]_ package.

        Parameters
        ----------

        initial_mixture: str
            Gas composition. Example::

                 'N2:0.79, O2:0.21, CO2:363e-6'

            Or::

                 'CO2:1'

        T_K: float (K)
            temperature (Kelvin) to calculate equilibrium

        P_Pa: float (Pa)
            temperature (Pascal) to calculate equilibrium

        Examples
        --------

        Calculate equilibrium mixture of CO2 at 2000 K, 1 atm::

            get_eq_mole_fraction('CO2:1', 2000, 101325)

            >>> {'C': 1.7833953335281855e-19,
                'CO': 0.01495998583472384,
                'CO2': 0.9775311634424326,
                'O': 5.7715610124613225e-05,
                'O2': 0.007451135112719029}

        References
        ----------

        [CANTERA]_
        """

        try:
            import cantera as ct
        except ImportError as err:
>           raise ImportError(
                "Cantera is needed to calculate equilibrium mole fractions"
                + ". Install with  `pip install cantera` or (better) `conda install -c cantera cantera`",
            ) from err
E           ImportError: Cantera is needed to calculate equilibrium mole fractions. Install with  `pip install cantera` or (better) `conda install -c cantera cantera`
radis/tools/gascomp.py:60: ImportError

Behavior

This mismatch is new since version 3.0.0, and our solution for the time being is to cap the version to <3.0.0.

I'm not certain the issue lies with Cantera, but I thought at least this might be a good place to start - it is fully possible that this is just an issue with our CI server.

System information

• Cantera version: 3.0.0
• OS: Xenial (Unix)
• Python 3.8

Installed via: micromamba create -f ./environment.yml -y And in environment.yml we have:

channels: 
 - conda-forge 
 - astropy 
 - cantera 
 - plotly 
dependencies:
(...)
- cantera>=2.5.1

The full environment.yml is here

[bryanwweber]

Hi, how did you install Cantera? What OS is this? Can you please complete the template we provide in the Issue? If it's in the workflow file,can you please copy it here so we don't have to go digging? Thanks!

Edit to add: and since it seems like you install from conda, what channels you're using for Cantera and Python?

[dcmvdbekerom]

Sorry I missed the bottom part of the template, I have added the missing info on the top post

[bryanwweber]

Thanks for the update! Most likely the problem is that Python is coming from conda-forge and Cantera is coming from our channel (-c cantera). On our channel, we compile against the libpython in the defaults channel, not the one in the conda-forge channel. To fix this, you need to install all the C-module dependencies from the same channel (including NumPy, Python, and any other libraries that use compiled code). My suggestion is to just install everything from conda-forge, and also recommend your users do the same 😀

[speth]

I think the key thing is that cantera and libcantera need to be installed from the same source, though Bryan's suggestion to just install everything from conda-forge is probably the safest solution.

We do an explicit test to make sure these two packages are built from the same source, since they are so tightly coupled. However, I was a little surprised by the result here, since the packages in both channels are built from the 3.0.0 release. The commit hash mentioned for libcantera from the cantera channel, 806842dac, is in fact the commit for the v3.0.0 tag. However, the commit hash mentioned for the cantera Python package from conda-forge, be8cff9, is actually not a hash in the Cantera/cantera repo at all, but is the commit to the conda-forge/cantera-feedstock that was used to build these particular binaries.

I think this comes from the fact that the conda-forge build downloads a .tar.gz of Cantera, rather than using a git checkout. Normally, this would result in the commit hash just being "unknown", but the working directory for these builds is apparently the feedstock recipe's repo, and so that git commit gets picked up instead. This is definitely not what was intended to end up in this variable, but it might end up being worthwhile to prevent just this sort of mismatch between Cantera components compiled in different environments.

[ischoegl]

While I believe the issue itself is tracked down (does not appear to be a Cantera issue after all :tada:), I believe that the error message itself could be improved. I.e.

   ???
E   **ImportError: Mismatch between Cantera Python module Git commit (be8cff9) and Cantera shared library git commit (806842dac)**
build/python/cantera/_utils.pyx:66: ImportError

could point out likely scenarios (mismatch of conda channels or mix/match of pip/conda/etc if this even works)

[bryanwweber]

I think the key thing is that cantera and libcantera need to be installed from the same source

Thanks @speth I completely misread the error message! That's what I get for replying late at night on my phone 😓😩

[speth]

could point out likely scenarios (mismatch of conda channels or mix/match of pip/conda/etc if this even works)

Pip is not an issue -- it does not link to libcantera, but rather bundles everything into the Python extension module.

Rather than trying to guess possible causes of the error, I think I'd rather see if we can update the Conda recipes to be more specific about the version match required between cantera and libcantera. I think this is what pin_subpackage is for?

[dcmvdbekerom]

Hi all, thanks for the fast and helpful responses!

I have removed the cantera channel from our environment and it works now!

FYI, it does choose v2.6.0 for both cantera and libcantera, even though I removed the <3.0.0 from environment.yml.

Is that what you would expect?

[ischoegl]

FYI, it does choose v2.6.0 for both cantera and libcantera, even though I removed the <3.0.0 from environment.yml.

Is that what you would expect?

That’s definitely not the expected behavior: it should be 3.0.0. Fwiw, there was a similar issue on the conda cantera channel a while ago, see Cantera/conda-recipes#31 and fix in Cantera/conda-recipes#33. I wonder whether it is the same issue.

PS: what version does conda list show?

PPS: I cannot reproduce this behavior on macOS, with an environment created using either mamba or conda (everything looks correct: conda list shows cantera 3.0.0, and running

>>> import cantera as ct
>>> ct.__version__

likewise shows 3.0.0). There could be some library mismatch that forces a downgrade of Cantera in your environment?

[erwanp]

I tried on my side (Windows); I get a mismatch in between libcantera and pytables because of the HDF5 version

Could not solve for environment specs
The following packages are incompatible
├─ libcantera   is installable and it requires
│  └─ hdf5 >=1.14.2,<1.14.3.0a0 , which can be installed;
└─ pytables   is uninstallable because there are no viable options
   ├─ pytables 3.3.0 would require
   │  └─ numpy 1.10* , which does not exist (perhaps a missing channel);
   ├─ pytables [3.3.0|3.4.1|3.4.2] would require
   │  └─ hdf5 1.8.17|1.8.17.*  but there are no viable options
   │     ├─ hdf5 1.8.17 would require
   │     │  └─ vc 10.* , which does not exist (perhaps a missing channel);
   │     └─ hdf5 1.8.17 conflicts with any installable versions previously reported;
   ├─ pytables 3.3.0 would require
   │  └─ python 3.4*  but there are no viable options
   │     ├─ python 3.4.5 would require
   │     │  └─ vc 10.* , which does not exist (perhaps a missing channel);
   │     └─ python 3.4.5 would require
   │        └─ vs2010_runtime  , which does not exist (perhaps a missing channel);
   ├─ pytables [3.4.2|3.4.3|3.4.4] would require
   │  └─ hdf5 [1.8.18|1.8.18.* |>=1.8.18,<1.8.19.0a0 ] but there are no viable options
   │     ├─ hdf5 [1.8.18|1.8.19] would require
   │     │  └─ vc 10 , which does not exist (perhaps a missing channel);
   │     └─ hdf5 1.8.18 conflicts with any installable versions previously reported;
   ├─ pytables [3.4.2|3.4.3|3.4.4] would require
   │  └─ hdf5 [1.10.1 |1.10.1.* |>=1.10.1,<1.10.2.0a0 ], which conflicts with any installable versions previously reported            ;
   ├─ pytables [3.4.3|3.4.4] would require
   │  └─ hdf5 >=1.10.2,<1.10.3.0a0 , which conflicts with any installable versions previously reported;
   ├─ pytables 3.4.4 would require
   │  └─ hdf5 >=1.10.3,<1.10.4.0a0 , which conflicts with any installable versions previously reported;
   ├─ pytables [3.4.4|3.5.0|3.5.1|3.5.2|3.6.1] would require
   │  └─ hdf5 >=1.10.4,<1.10.5.0a0 , which conflicts with any installable versions previously reported;
   ├─ pytables [3.5.2|3.6.0|3.6.1] would require
   │  └─ hdf5 >=1.10.5,<1.10.6.0a0 , which conflicts with any installable versions previously reported;
   ├─ pytables [3.6.1|3.7.0|3.8.0] would require
   │  └─ hdf5 >=1.12.1,<1.12.2.0a0 , which conflicts with any installable versions previously reported;
   ├─ pytables [3.6.1|3.7.0|3.8.0] would require
   │  └─ hdf5 >=1.10.6,<1.10.7.0a0 , which conflicts with any installable versions previously reported;
   ├─ pytables 3.7.0 would require
   │  └─ hdf5 >=1.12.2,<1.12.3.0a0 , which conflicts with any installable versions previously reported;
   ├─ pytables [3.7.0|3.8.0] would require
   │  └─ hdf5 >=1.14.0,<1.14.1.0a0 , which conflicts with any installable versions previously reported;
   ├─ pytables 3.8.0 would require
   │  └─ hdf5 >=1.14.1,<1.14.2.0a0 , which conflicts with any installable versions previously reported;
   ├─ pytables [3.4.3|3.4.4] would require
   │  └─ hdf5 >=1.8.18,<1.9.0a0  but there are no viable options
   │     ├─ hdf5 [1.8.18|1.8.19], which cannot be installed (as previously explained);
   │     ├─ hdf5 1.8.18 conflicts with any installable versions previously reported;
   │     └─ hdf5 [1.8.19|1.8.20] conflicts with any installable versions previously reported;
   └─ pytables 3.7.0 would require
      └─ hdf5 >=1.12.0,<1.12.1.0a0 , which conflicts with any installable versions previously reported.

[ischoegl]

Thanks - that explains things. They go up to hdf5 1.14.1, whereas Cantera uses 1.14.2.

[speth]

Actually, the version requirements here are very narrow:

   ├─ pytables 3.8.0 would require
   │  └─ hdf5 >=1.14.1,<1.14.2.0a0 , which conflicts with any installable versions previously reported;

while Cantera requires

├─ libcantera   is installable and it requires
│  └─ hdf5 >=1.14.2,<1.14.3.0a0 , which can be installed;

I assume there's a reason that conda-forge requires matching the patch version with HDF5 instead of just matching 1.14.*.

[erwanp]

Is the -c cantera channel Cantera more flexible on the versions ? Do you think we should also reach out to the PyTables developer to understand why they also require to match the version ?

[erwanp]

Hello @speth do you have any update on the above ?

[speth]

Neither Cantera nor PyTables are responsible for the requirement of an exact match of the patch version. At least to my understanding, that is set by the feedstock for the hdf5 package. I don't know if it is a mistake in that package, or an actual requirement of the upstream library.

In any case, it looks like PyTables has already been rebuilt for HDF 1.14.2 -- see https://github.com/conda-forge/pytables-feedstock/pull/89 -- and I am able to simultaneously install both Cantera 3.0 and PyTables 3.8.0.

[ischoegl]

The discussion has drifted pretty far from the original issue. It appears that Radis dependencies include two packages that both build on libhdf. When the issue was first posted, Cantera used a newer libhdf version, where pytables didn’t offer a conda package yet. This appears to be resolved, so this issue should likely be closed?

if there is any other issue we can resolve on our side, please open a new issue.