Incompatibilities and collisions between Matlab and numpy/scipy BLAS/LAPACK

[mducle]

Because we're using "in-process" loading (where Matlab is loaded into the Python process like another module), libraries needed by Matlab are also loaded into the same process. Matlab ships with its own version of the BLAS/LAPACK libraries (Intel MKL), and there appears to be a name collision with the BLAS/LAPACK libraries used by numpy / scipy. Specifically calls from numpy / scipy seem to reach the Matlab libraries rather than the libraries numpy / scipy was originally linked to on a particular system. This is because in order to get pace_python to work, we had to load Matlab first, so its libraries get loaded before others. This probably means the numpy / scipy BLAS/LAPACK libraries never get loaded.

This is fine if the numpy / scipy on a particular environment was linked to the MKL, but not if it was linked to another library. The specific errors seen (e.g. Intel MKL ERROR: Parameter 5 was incorrect on entry to ZHEEVD.) seems to indicate that the problem is due to Fortran vs C calling conventions for some parameters. numpy / scipy exposes the Fortran interface in _flapack which expects all parameters to be passed by reference; the Matlab MKL seems to expects scalar parameters (such as the leading dimension lda) to be passed by value.

So far systems with numpy / scipy packages which work are:

• conda using conda-forge packages.
• Ubuntu 18.04 provided packages.
• Windows pip packages.

System which doesn't

• PyPi / pip provided numpy / scipy (manylinux) packages.

[mducle]

A temporary fix is to import scipy.linalg with sys.setdlopenflags(os.RLTD_NOW | os.RLTD_DEEPBIND).

This has been added to the main pace-python import, so if the user imports us before they import scipy.linalg, then we're fine...