`pypardiso` (PyPI): Error `libiomp5md.dll` already initialized

[adtzlr]

Hi @haasad!

For Win10, WinPython 3.11.5 and PyPardiso installed via PyPI I get a Fatal Python error: Aborted if I try to use PyPardiso with FElupe. However, I'm nearly 100% sure that this is not related to FElupe as it works with the sparse-solver from scipy.

A minimal code that crashes using FElupe (it works with the sparse-solver from scipy) (pip install felupe):

import felupe as fem
from pypardiso import spsolve

# from scipy.sparse.linalg import spsolve

mesh = fem.Cube(n=6)
region = fem.RegionHexahedron(mesh)
field = fem.FieldContainer([fem.Field(region, dim=3)])
boundaries, loadcase = fem.dof.uniaxial(field, clamped=True)
solid = fem.SolidBodyNearlyIncompressible(fem.NeoHooke(mu=1), field, bulk=5000)
step = fem.Step(items=[solid], boundaries=boundaries)
fem.Job(steps=[step]).evaluate(solver=spsolve)

I finally tracked this down to a lib-error. Do you have any advises for that?

MP: Error #15: Initializing libiomp5md.dll, but found libiomp5md.dll already initialized.
OMP: Hint This means that multiple copies of the OpenMP runtime have been linked into the program. That is dangerous, since it can degrade performance or cause incorrect results. The best thing to do is to ensure that only a single OpenMP runtime is linked into the process, e.g. by avoiding static linking of the OpenMP runtime in any library. As an unsafe, unsupported, undocumented workaround you can set the environment variable KMP_DUPLICATE_LIB_OK=TRUE to allow the program to continue to execute, but that may cause crashes or silently produce incorrect results. For more information, please see http://www.intel.com/software/products/support/.

This one works, but as described above this is not a very good idea.

```python import felupe as fem from pypardiso import spsolve import os os.environ["KMP_DUPLICATE_LIB_OK"] = "TRUE" mesh = fem.Cube(n=6) region = fem.RegionHexahedron(mesh) field = fem.FieldContainer([fem.Field(region, dim=3)]) boundaries, loadcase = fem.dof.uniaxial(field, clamped=True) solid = fem.SolidBodyNearlyIncompressible(fem.NeoHooke(mu=1), field, bulk=5000) step = fem.Step(items=[solid], boundaries=boundaries) fem.Job(steps=[step]).evaluate(solver=spsolve) ```

[haasad]

Hi @adtzlr,

I have a noticed the same today, but haven't found time to look into it yet.

I have a pipeline that runs every 24h and runs the tests and since this morning all windows pypi tests fail: https://github.com/haasad/PyPardisoProject/actions

I assume there's been a new release of one of the dependencies, which breaks pypi pypardiso. I'll get back to you once I know more.

[adtzlr]

Hi, perfect! I found some questions on stackoverflow related to that, like https://stackoverflow.com/questions/20554074/sklearn-omp-error-15-initializing-libiomp5md-dll-but-found-mk2iomp5md-dll-a

The WinPython distribution comes pre-bundled with a version of libiomp5md.dll (C:\...\WinPython\WPy64-31150\python-3.11.5.amd64\Library\bin\libiomp5md.dll), I removed it and pypardiso works as expected. However, this will probably break one of the other packages (which currently I'm not using).

[haasad]

There was indeed a new release of mkl 2024.0.0 and intel-openmp 2024.0.0 on 14th of November (for windows and linux).

Downgrading MKL should get rid of the error:

pip install "mkl<2024"

I'm still deciding how to proceed, I could in theory make a new pypardiso release which pins mkl to <2024. But I'm probably going to wait a few days and hope that this is fixed upstream.

[ma-sadeghi]

It's strange, IIRC a similar thing happened last year when Intel released mkl2023...

[ghsilva]

Hello @haasad,

I am the product manager for MKL at Intel and I've been looking for issues in GitHub since our latest release (2024.0.0) to be sure we can keep improving the quality of our releases. Do you know if the issue described here happens with WinPython only? Were you able to reproduce the issue in other configurations or Operating Systems? I wonder why pip could not prevent the installation of multiple instances of intel-openmp... anyway, I will try to reproduce it in my environment, if you have more details to share I would really appreciate. I hope I can help to get rid of the "<2024" from your current workaround ;-)

[ghsilva]

@haasad there are 2 workarounds to make the tests pass using mkl==2024.0.0 on Windows/Conda environment described in the link below: Intel(R) oneAPI Math Kernel Library Forum The long term solution will be a fix that is likely to take place in the intel-openmp package in pip and conda-forge repositories.

[ghsilva]

@haasad I am happy to inform that we have 2 good versions of intel-open for Windows that solve this issue in the correct way: 1) intel-openmp 2024.0.3 released on March 22nd 2) intel-openmp 2024.1 released on March 26th (few hours ago) --> https://pypi.org/project/mkl/2024.1.0/

I already tested pypardiso master branch for both versions, that means you can close this issue and revert the pin to mkl<2024. Let me know if you have any question.

[haasad]

Hi @ghsilva, thanks for the heads-up. I couldn't completely remove the pin, because the new releases aren't yet available on conda-forge. But it looks like the folks at https://github.com/conda-forge/intel_repack-feedstock are working on it.

I opted for excluding 2024.0 instead of removing the pin completely. Like this the latest releases are used where available.

I'm closing this issue. Thanks for your efforts :+1:

[haasad]

for future reference: https://github.com/haasad/PyPardiso/pull/71#issuecomment-2027984427

Hi ma-sadeghi, I also considered it but decided against it. I had a look a some of the previous issues and there isn't a clear pattern where major MKL versions cause issues. E.g #26 was caused by python 3.9 dropping support for a old dependency, #12 was caused by MKL 2021.2.0, #36 was caused by a new incompatible release of intel-openmp where mkl itself didn't properly pin it (here the release of MKL 2022.0 actually made the install work again). The same problem as #36 happened again with intel-openmp 2023.0, however MKL now sets the correct pins (see conda-forge/intel_repack-feedstock#33 and conda-forge/intel_repack-feedstock#50) and this particular problem shouldn't appear anymore.

As far as I can tell, mkl 2024.0 was the first major release, where the MKL release itself was the problem.