Shared library mkl_rt not found in azure web app

[kimc-fls]

I'm installing pypardiso in a Azure pipeline on a Ubuntu-latest system, but it's raising an ImportError: Shared library mkl_rt not found during initialization. Seems related to closed issue #54, but I have not been able to make it work.

A snippet from the .yml file that creates a virtual environment and install libraries

 - script: | python -m venv antenv
        source antenv/bin/activate
        python -m pip install --upgrade pip
        pip install -r requirements.txt
      workingDirectory: $(projectRoot)
      displayName: "Install requirements"

Requirements.txt:

numpy==1.22.4
scipy==1.10.0
pypardiso==0.4.2
pandas==1.4.2
flask==2.2.5
openpyxl==3.0.10
xlrd==2.0.1
scikit-learn==1.1.3
sphinx
sphinx-rtd-theme
sphinx-jsonschema

During import of pypardiso an error is raised: ImportError: Shared library mkl_rt not

The mkl_rt file is in the build artifact in folder antenv/lib and called libmkl_rt.so.2

I have tried to create a symbolic link ln -s antenv/lib/libmkl_rt.so.2 antenv/lib/libmkl_rt.so Also tried to install an older version of mkl, but so far no success

[haasad]

Could you maybe add the following line to your install script and check the output:

 - script: | python -m venv antenv
        source antenv/bin/activate
        python -m pip install --upgrade pip
        pip install -r requirements.txt
        python -c "import sys; print(sys.prefix, sys.executable)"  # <---

[kimc-fls]

Output is: /home/azureuser/myagent/_work/1/s/antenv /home/azureuser/myagent/_work/1/s/antenv/bin/python

[haasad]

and these two lines?

python -c "import glob; import sys; print(glob.glob(f'{sys.prefix}/[Ll]ib*/**/*mkl_rt*', recursive=True))"
python -c "import pypardiso"

[kimc-fls]

Output: ['/home/azureuser/myagent/_work/1/s/antenv/lib/libmkl_rt.so.2', '/home/azureuser/myagent/_work/1/s/antenv/lib64/libmkl_rt.so.2']

[kimc-fls]

It didn't raise an error in this case. After building, it is deployed to an Azure web app using Flask. But the web app fails to launch due to the ImportError,

[kimc-fls]

I have tried to read out the same paths from within the web app by this function:

@app.route('/', methods=['GET', 'POST'])
def calculate():   
    data = [sys.prefix, sys.executable, glob.glob(f'{sys.prefix}/[Ll]ib*/**/*mkl_rt*', recursive=True)]

    return data

This returns the following:

[
"/opt/python/3.10.12",
"/opt/python/3.10.12/bin/python3.10",
[ ]
]

[haasad]

Just to help me understand what's going on: You build your application in an azure pipeline and it then gets deployed as a flask azure web app similar to what's described here: https://learn.microsoft.com/en-us/azure/app-service/quickstart-python

Is that correct?

[kimc-fls]

Yes, that is correct. I have found this page which might describe my problem in section "Missing shared libraries": https://azureossd.github.io/2022/11/24/Python-on-Linux-App-Service-and-ModuleNotFound-Errors/

[haasad]

I don't think that this is the same problem that you're seeing. The mentioned section talks about missing shared libraries that should be available on OS level. However we have checked the libmkl_rt.so.2 is actually available in the venv and as far as I understand the whole venv is copied into the web app container.

I think the problem is that the libmkl_rt_so.2 shared library is in an unexpected location and that's why pypardiso can't find it. Can you figure out where the venv is located inside the container? And check if the antenv/lib folder is present?

[haasad]

I could add a feature to pypardiso to supply the full path to the mkl_rt library via environment variable.

[kimc-fls]

You are probably right. The mkl_rt is in folder home/site/wwwroot/antenv/lib

[haasad]

I've just released v0.4.3 which adds the option to configure the mkl_rt path with the PYPARDISO_MKL_RT variable. Can you try if it works when you set PYPARDISO_MKL_RT=/home/site/wwwroot/antenv/lib/libmkl_rt.so.2 as env variable with the new version?

[kimc-fls]

It working now :) But I'm also running the code local on Windows machines. I have worked around it by looking at the sys.platform:

if sys.platform == "linux" or sys.platform == "linux2":
    os.environ['PYPARDISO_MKL_RT'] = '/home/site/wwwroot/antenv/lib/libmkl_rt.so.2'

However, I'm not seeing the the same speed improvement when running in web app compared to local Windows, but that might be related to the Azure subscription. To be investigated.

[haasad]

One possible explanation could be that your azure web app is most likely single-core. One of the reasons pypardiso can give big speed improvements is because it's multi-threaded on can make good use of the available CPUs compared to the single-threaded solver in scipy. But it's also possible that your application runs into some container CPU limit and is being throttled.