make test fails to find python

[100souci]

Hello again,

I'm trying to run make test and make mediumtest following a successful make install on an HPC cluster. However, it looks like the test files cannot detect the Python that was used earlier. I don't know why it picks a local version on the master node I'm running the tests on (/usr/bin/python3), whereas it should be the shared Python for all users (/softs/python/3.9.6-shared).

My Python environment variables are the following: PYTHONPATH=/softs/python/3.9.6-shared/lib/python3.9/site-packages PYTHON=/softs/python/3.9.6-shared/bin/python3.9

**Running simulation ../../bin/fluidity -v3 -l mphase_divergence_free_velocity.flml make[1]: Leaving directory `/softs/Fluidity/tests/mphase_divergence_free_velocity' mphase_divergence_free_velocity: Running mphase_divergence_free_velocity: Calling 'make input': mphase_divergence_free_velocity: make run

make[1]: Entering directory /softs/Fluidity/tests/mphase_dusty_gas_shock_tube' make[1]: warning: jobserver unavailable: using -j1. Add+' to parent make rule. rm -f .stat .steady_state rm -f .d. .vtu rm -f .msh rm -f .ele .edge .node .poly .bound rm -f matrixdump .log .err* **Creating 1D mesh ../../bin/interval --dx=0.00271002710027 -- -1.35501355014 1.35501355014 line Traceback (most recent call last): File "/softs/python/3.9.6-shared/lib/python3.9/site-packages/numpy/core/init.py", line 22, in from . import multiarray File "/softs/python/3.9.6-shared/lib/python3.9/site-packages/numpy/core/multiarray.py", line 12, in from . import overrides File "/softs/python/3.9.6-shared/lib/python3.9/site-packages/numpy/core/overrides.py", line 7, in from numpy.core._multiarray_umath import ( ModuleNotFoundError: No module named 'numpy.core._multiarray_umath'

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "../../bin/interval", line 31, in from numpy import * File "/softs/python/3.9.6-shared/lib/python3.9/site-packages/numpy/init.py", line 150, in from . import core File "/softs/python/3.9.6-shared/lib/python3.9/site-packages/numpy/core/init.py", line 48, in raise ImportError(msg) ImportError:

IMPORTANT: PLEASE READ THIS FOR ADVICE ON HOW TO SOLVE THIS ISSUE!

Importing the numpy C-extensions failed. This error can happen for many reasons, often due to issues with your setup or how NumPy was installed.

We have compiled some common reasons and troubleshooting tips at:

https://numpy.org/devdocs/user/troubleshooting-importerror.html

Please note and check the following:

• The Python version is: Python3.6 from "/usr/bin/python3"
• The NumPy version is: "1.21.0"

and make sure that they are the versions you expect. Please carefully study the documentation linked above for further help.

Original error was: No module named 'numpy.core._multiarray_umath'

make[1]: *** [preprocess] Error 1 make[1]: Leaving directory `/softs/Fluidity/tests/mphase_dusty_gas_shock_tube' mphase_dusty_gas_shock_tube: Running mphase_dusty_gas_shock_tube: Calling 'make input': Error: mphase_dusty_gas_shock_tube.xml raised an exception while running: Traceback (most recent call last):

File "../bin/testharness", line 437, in threadrun runtime=test.run(dir)

File "/softs/Fluidity/python/fluidity/regressiontest.py", line 143, in run assert(ret == 0)

AssertionError

Thank you in advance for your help.

[Patol75]

Hi,

Why are you assuming that the wrong version of Python is called? Judging from the traceback you attached

Traceback (most recent call last):
File "/softs/python/3.9.6-shared/lib/python3.9/site-packages/numpy/core/init.py", line 22, in
from . import multiarray
File "/softs/python/3.9.6-shared/lib/python3.9/site-packages/numpy/core/multiarray.py", line 12, in
from . import overrides
File "/softs/python/3.9.6-shared/lib/python3.9/site-packages/numpy/core/overrides.py", line 7, in
from numpy.core._multiarray_umath import (
ModuleNotFoundError: No module named 'numpy.core._multiarray_umath'

it definitely looks like it is using the Python version you expect. Nevertheless, you run into a Numpy failure. I would tend to think this is an actual problem of the Python version you compiled rather than anything related to Fluidity.

[100souci]

Hello,

I don't understand why it says The Python version is: Python3.6 from "/usr/bin/python3" when it should be v3.9 from /softs/python/3.9.6-shared

I also get these messages: ERROR \nFluidity python support, required by this example, was not found on your\nsystem. Please either (a) install the Fluidity binary package if you are on\nUbuntu, or (b) get hold of a Fluidity source tree and set your PYTHONPATH to\nrefer to this, as described in the Fluidity manual.

make[1]: Entering directory /softs/Fluidity/tests/annulus-diffusive' make[1]: warning: jobserver unavailable: using -j1. Add+' to parent make rule. tar -xvjf src/annulus-structured-tanh-23x64.tar.bz2 annulus-structured-tanh-23x64.ele annulus-structured-tanh-23x64.face annulus-structured-tanh-23x64.node ../../bin/triangle2gmsh annulus-structured-tanh-23x64 Warning: Failed to import numpy module Warning: Failed to import scipy module Warning: Failed to import scipy.interpolate module Warning: Failed to import numpy module Warning: Failed to import scipy.linalg module Warning: Failed to import scipy.stats module make[1]: *** [input] Segmentation fault make[1]: Leaving directory `/softs/Fluidity/tests/annulus-diffusive' annulus-diffusive: Running annulus-diffusive: Calling 'make input': Error: annulus-diffusive.xml raised an exception while running: Traceback (most recent call last):

or:

make[1]: Entering directory /softs/Fluidity/tests/biology_6_component' make[1]: warning: jobserver unavailable: using -j1. Add+' to parent make rule. rm -f .ele .edge .face .msh .node .poly .vtu .s .d.1 .stat *.detectors \ matrixdump matrixdump.info ../../bin/create_aligned_mesh column-3D 10 10 -- -50 2 2 10 Warning: Failed to import numpy module Warning: Failed to import numpy module Warning: Failed to import scipy.linalg module Warning: Failed to import scipy.stats module make[1]: *** [input] Segmentation fault make[1]: Leaving directory `/softs/Fluidity/tests/biology_6_component' biology_6_component: Running biology_6_component: Calling 'make input': Error: biology_6_component.xml raised an exception while running: Traceback (most recent call last):

File "../bin/testharness", line 437, in threadrun runtime=test.run(dir)

File "/softs/Fluidity/python/fluidity/regressiontest.py", line 143, in run assert(ret == 0)

AssertionError

I also tried setting PYTHONPATH=/softs/python/3.9.6-shared/lib/python3.9/site-packages:/softs/Fluidity/python:/softs/Fluidity/tools But it didn't help.

Our Python 3.9.6 was compiled as follows: ./configure --prefix=/softs/python/3.9.6-shared --with-ensurepip=install --enable-shared make -j16 make altinstall

And I'm able to import numpy when using the Python shell. We also have scipy. Both were installed using: pip3.9 install scipy/numpy

[stephankramer]

At the top of the interval script you can see what it uses to invoke python:

#!/usr/bin/env python3

This should normally just pick the first python3 it finds in the PATH (and thus works with python3 venvs and conda). So you have two choices: 1) prepend your path such that the right python3 is found first 2) use some sed magic to go through all script and replace the python3 shebang with a hard-coded path of your preference; this is a bit messy: the python scripts in bin/ are easy, but there's quite a few python3 references in the tests suite as well. The python you specify during the configure is not used to change which python binary is used for the scripts, rather it determines which python interpreter fluidity links to, i.e. the python that gets executed at run-time of fluidity (fluidity's in-built python extension). It looks like your PYTHONPATH is what it should be, which is why it has /softs/python/3.9.6-shared/lib/python3.9/site-packages in the backtrace: it's picking up the python modules from your python3.9 which of course refuse to be imported in python3.6. So fixing your PATH should be enough to get the pre and post-processing scripts working. If you haven't used "make install" of fluidity you may also need to add the python/ directory in the fluidity source to PYTHONPATH in order to make the python at fluidity run-time work. If you did use "make install" it should have put it in python3.9/site-packages already but if you didn't have the right python3 in your PATH when you ran "make install" it may have screwed that up...

[stephankramer]

Oh I see you specifically have /softs/python/3.9.6-shared/bin/python3.9: in addition to making sure that /softs/python/3.9.6-shared/bin/ is in front of your PATH, make sure that it actually has a python3 in that directory (sym-linked to python3.9)

[100souci]

Thank you, adding a python3 sym-link to python3.9 solved the problem.