Python configration fails to load numpy

[MarAlder]

Dear RCE-Team,

I have a question to the python configuration in RCE. I would like to define my own sampling via Python script which provides a matrix:

with:

from SALib.sample import saltelli

problem = {'num_vars': 2,
           'names' : ['var1','var2'],
           'bounds' : [[-1., 1.,],
                      [-1.5, 1.5]]}

samples = saltelli.sample(problem, 100)
RCE.write_output("Samples",samples.tolist())

I configured python (using a Conda environment) via:

    "thirdPartyIntegration": {
            "python":{
            "binaryPath": "D:/Programme/Miniconda3/envs/python3_cpacs/python.exe"
                    }
            }

Usually these scripts are running without problems. However, when I load a module which uses Numpy (such as SALib in the above example) the following error occurs:

2021-04-26-14:18:47,127 TOOL ERROR: D:\Programme\Miniconda3\envs\python3_cpacs\lib\site-packages\numpy\__init__.py:143: UserWarning: mkl-service package failed to import, therefore Intel(R) MKL initialization ensuring its correct out-of-the box operation under condition when Gnu OpenMP had already been loaded by Python process is not assured. Please install mkl-service package, see http://github.com/IntelPython/mkl-service
2021-04-26-14:18:47,127 TOOL ERROR:   from . import _distributor_init
2021-04-26-14:18:47,147 TOOL ERROR: Traceback (most recent call last):
2021-04-26-14:18:47,147 TOOL ERROR:   File "D:\Programme\Miniconda3\envs\python3_cpacs\lib\site-packages\numpy\core\__init__.py", line 22, in <module>
2021-04-26-14:18:47,147 TOOL ERROR:     from . import multiarray
2021-04-26-14:18:47,147 TOOL ERROR:   File "C:\Users\alde_ma\AppData\Local\Temp\rce-temp\default-1619437204945\424-pythonAgent/RCE_PythonClient.py", line 147, in executeUserscriptJSON
2021-04-26-14:18:47,147 TOOL ERROR:   File "D:\Programme\Miniconda3\envs\python3_cpacs\lib\site-packages\numpy\core\multiarray.py", line 12, in <module>
2021-04-26-14:18:47,147 TOOL ERROR:     exec(codeObject, CONTEXT)
2021-04-26-14:18:47,147 TOOL ERROR:     from . import overrides
2021-04-26-14:18:47,147 TOOL ERROR:   File "script", line 1, in <module>
2021-04-26-14:18:47,147 TOOL ERROR:   File "D:\Programme\Miniconda3\envs\python3_cpacs\lib\site-packages\numpy\core\overrides.py", line 7, in <module>
2021-04-26-14:18:47,147 TOOL ERROR:   File "D:\Programme\Miniconda3\envs\python3_cpacs\lib\site-packages\SALib\sample\saltelli.py", line 3, in <module>
2021-04-26-14:18:47,147 TOOL ERROR:     from numpy.core._multiarray_umath import (
2021-04-26-14:18:47,147 TOOL ERROR:     import numpy as np
2021-04-26-14:18:47,147 TOOL ERROR: ImportError: DLL load failed: Das angegebene Modul wurde nicht gefunden.
2021-04-26-14:18:47,147 TOOL ERROR:   File "D:\Programme\Miniconda3\envs\python3_cpacs\lib\site-packages\numpy\__init__.py", line 145, in <module>
2021-04-26-14:18:47,147 TOOL ERROR:     from . import core
2021-04-26-14:18:47,147 TOOL ERROR: During handling of the above exception, another exception occurred:
2021-04-26-14:18:47,147 TOOL ERROR:   File "D:\Programme\Miniconda3\envs\python3_cpacs\lib\site-packages\numpy\core\__init__.py", line 48, in <module>
2021-04-26-14:18:47,147 TOOL ERROR:     raise ImportError(msg)
2021-04-26-14:18:47,147 TOOL ERROR: Traceback (most recent call last):
2021-04-26-14:18:47,147 TOOL ERROR: ImportError: 
2021-04-26-14:18:47,147 TOOL ERROR: IMPORTANT: PLEASE READ THIS FOR ADVICE ON HOW TO SOLVE THIS ISSUE!
2021-04-26-14:18:47,147 TOOL ERROR: Importing the numpy C-extensions failed. This error can happen for
2021-04-26-14:18:47,147 TOOL ERROR: many reasons, often due to issues with your setup or how NumPy was
2021-04-26-14:18:47,147 TOOL ERROR: installed.
2021-04-26-14:18:47,147 TOOL ERROR: We have compiled some common reasons and troubleshooting tips at:
2021-04-26-14:18:47,147 TOOL ERROR:     https://numpy.org/devdocs/user/troubleshooting-importerror.html
2021-04-26-14:18:47,147 TOOL ERROR: Please note and check the following:
2021-04-26-14:18:47,147 TOOL ERROR:   * The Python version is: Python3.7 from "D:\Programme\Miniconda3\envs\python3_cpacs\python.exe"
2021-04-26-14:18:47,147 TOOL ERROR:   * The NumPy version is: "1.20.1"
2021-04-26-14:18:47,147 TOOL ERROR: and make sure that they are the versions you expect.
2021-04-26-14:18:47,147 TOOL ERROR: Please carefully study the documentation linked above for further help.
2021-04-26-14:18:47,147 TOOL ERROR: Original error was: DLL load failed: Das angegebene Modul wurde nicht gefunden.
2021-04-26-14:18:47,155 COMPONENT ERROR: Failed to execute script. (E#1619439527218)
[Log file format version: 1.0]

In my Conda environment itself, Numpy loads without problems and the example above runs without errors. I have observed a similar error when I have included my own python tool. Here I could work around the error by specifying the Python path via the execution command:

set PATH=D:\Programme\Miniconda3\envs\python3_cpacs\Scripts;D:\Programme\Miniconda3\envs\python3_cpacs;%PATH
D:\Programme\Miniconda3\condabin\activate.bat python3_cpacs
python run.py

In the configuration file under thirdPartyIntegration only the binaryPath can be specified, right?

Thanks for your help!

[ArneBachmannDLR]

Did you try running the script via activate python3_cpacs && python run.py? This is the approach I use for conda-env enabled tool integrations, but it might fail for script integration. You could try to replace python.exe with a batch script that sets all required paths prior to running python.exe with the given arguments.

The entire approach, however, is not very portable: it will run only on your specific setup.

You might alternatively activate the conda environment before starting rce.exe. This way python.exe will automatically refer to the correct environment's interpreter and you can forgo all other path setup. This would be somewhat more portable.

Or don't use external libraries in rce scripts at all...

[MarAlder]

Thanks @ArneBachmannDLR, the solution via batch script is working fine for me.