Tox improvements on local machine

nico-canta commented 1 year ago

I tried to run tox locally. Since locally the venv is managed by anaconda, I used tox-conda. The following two tests failed:

test_petsc_installation. (reason: tox environment not recognized)
test_solve_linear. (reason: solve_linear obtains a different solution, which is still correct)

With this PR we are solving these 2 errors.

Interesting question:
Why within the same tox environment, solve_linear calculates 2 different results?

bbpbuildbot commented 1 year ago

Logfiles from GitLab pipeline #158523 (:white_check_mark:) have been uploaded here!

Status and direct links:

nico-canta commented 1 year ago

todo: add tox and tox-conda in the setup.py (only in the conda installation)

nico-canta commented 1 year ago

In tox.ini I added some features based on tox-conda. Since the CI is all green, I assume tox-conda is ignored when running on BB5. Is this the case?

bbpbuildbot commented 1 year ago

Logfiles from GitLab pipeline #158666 (:white_check_mark:) have been uploaded here!

Status and direct links:

tristan0x commented 1 year ago

Can you elaborate on the issue you encounter with test_petsc_installation? To me it looks like a conda issue, not related to tox.

bbpbuildbot commented 1 year ago

Logfiles from GitLab pipeline #158877 (:white_check_mark:) have been uploaded here!

Status and direct links:

bbpbuildbot commented 1 year ago

Logfiles from GitLab pipeline #158886 (:white_check_mark:) have been uploaded here!

Status and direct links:

nico-canta commented 1 year ago

At the moment there are two things I'm not very convinced:

running tox on bb5 doesn't work because it cannot install petsc4py using pip. EDIT: It works
in the CI we are using the
```
EXTRA_MODULES:
  unstable:gcc
  unstable:hpe-mpi
  unstable:petsc
  unstable:py-petsc4py
```
but importing modules inside a virtual environment doesn't seem a great idea.

So... overall, I'm not sure we should still keep using tox. Or maybe we could move to tox-conda and use it on bb5 and for our CI. I wanted to discuss this in next meeting. Of course, if this is not a priority we can keep it as it is, as far as it is working fine.

nico-canta commented 1 year ago

@tristan0x please disregard the previous comments. I found the problem. For some unknown reasons pass_env = * in tox.ini is not working (in my venv, in my local machine). Next week I'm gonna investigate why I have this problem.

nico-canta commented 1 year ago

@tristan0x please disregard the previous comments. I found the problem. For some unknown reasons pass_env = * in tox.ini is not working (in my venv, in my local machine). Next week I'm gonna investigate why I have this problem.

I found the reason for the strange behavior. Activating a conda environment on top of another environment messes everything up.

Anyway. The error persists:

E   ModuleNotFoundError: No module named 'mpi4py'

Honestly, I expected this error, since mpi4py is not in the dependencies. But now I'm not sure I did everything the way you did. @tristan0x Can you please let me know all the steps you did?

In any case, this work will be useful because the errors that I'm experiencing right now are potential errors for the Astrovascpy users. So, I will improve the readme file.

tristan0x commented 1 year ago

Here is the list of conda packages I have installed:

$ conda list
# Name                    Version                   Build  Channel
blosc2                    2.2.9                    pypi_0    pypi
boltons                   23.0.0          py311hca03da5_0  
brotlipy                  0.7.0           py311h80987f9_1002  
bzip2                     1.0.8                h620ffc9_4  
c-ares                    1.19.0               h80987f9_0  
ca-certificates           2023.08.22           hca03da5_0  
cached-property           1.5.2                    pypi_0    pypi
cachetools                5.3.1                    pypi_0    pypi
certifi                   2023.7.22       py311hca03da5_0  
cffi                      1.15.1          py311h80987f9_3  
chardet                   5.2.0                    pypi_0    pypi
charset-normalizer        2.0.4              pyhd3eb1b0_0  
click                     8.1.7                    pypi_0    pypi
colorama                  0.4.6                    pypi_0    pypi
conda                     23.9.0          py311hca03da5_0  
conda-content-trust       0.1.3           py311hca03da5_0  
conda-libmamba-solver     23.7.0             pyhd8ed1ab_0    conda-forge
conda-package-handling    2.2.0              pyh38be061_0    conda-forge
conda-package-streaming   0.9.0              pyhd8ed1ab_0    conda-forge
contourpy                 1.1.1                    pypi_0    pypi
coverage                  7.3.2                    pypi_0    pypi
cryptography              39.0.1          py311h834c97f_2  
cycler                    0.12.0                   pypi_0    pypi
cython                    3.0.3                    pypi_0    pypi
distlib                   0.3.7                    pypi_0    pypi
fftw                      3.3.10          mpi_openmpi_haef8dc3_8    conda-forge
filelock                  3.12.4                   pypi_0    pypi
fmt                       9.1.0                h48ca7d4_0  
fonttools                 4.43.1                   pypi_0    pypi
gmp                       6.2.1                h9f76cd9_0    conda-forge
h5py                      3.9.0                    pypi_0    pypi
hdf5                      1.14.2          mpi_openmpi_ha832a9f_0    conda-forge
hypre                     2.28.0          mpi_openmpi_haba3941_0    conda-forge
icu                       68.1                 hc377ac9_0  
idna                      3.4             py311hca03da5_0  
iniconfig                 2.0.0                    pypi_0    pypi
jinja2                    3.1.2                    pypi_0    pypi
jsonpatch                 1.32               pyhd3eb1b0_0  
jsonpointer               2.1                pyhd3eb1b0_0  
kiwisolver                1.4.5                    pypi_0    pypi
krb5                      1.20.1               hf3e1bf2_1  
libaec                    1.1.2                h13dd4ca_1    conda-forge
libarchive                3.6.2                h62fee54_2  
libblas                   3.9.0           18_osxarm64_openblas    conda-forge
libcblas                  3.9.0           18_osxarm64_openblas    conda-forge
libcurl                   8.2.1                h3e2b118_0  
libcxx                    16.0.6               h4653b0c_0    conda-forge
libedit                   3.1.20221030         h80987f9_0  
libev                     4.33                 h1a28f6b_1  
libexpat                  2.5.0                hb7217d7_1    conda-forge
libffi                    3.4.4                hca03da5_0  
libgfortran               5.0.0           13_2_0_hd922786_1    conda-forge
libgfortran5              13.2.0               hf226fd6_1    conda-forge
libiconv                  1.16                 h1a28f6b_2  
liblapack                 3.9.0           18_osxarm64_openblas    conda-forge
libmamba                  1.4.1                h1c5506f_1  
libmambapy                1.4.1           py311h1c5506f_1  
libnghttp2                1.52.0               h62f6fdd_1  
libopenblas               0.3.24          openmp_hd76b1f2_0    conda-forge
libsolv                   0.7.22               h98b2900_0  
libsonata                 0.1.23                   pypi_0    pypi
libsqlite                 3.43.0               hb31c410_0    conda-forge
libssh2                   1.10.0               h02f6b3c_2  
libxml2                   2.10.3               h372ba2a_0  
libzlib                   1.2.13               h53f4e23_5    conda-forge
llvm-openmp               17.0.2               h1c12783_0    conda-forge
lz4-c                     1.9.4                h313beb8_0  
markupsafe                2.1.3                    pypi_0    pypi
matplotlib                3.8.0                    pypi_0    pypi
metis                     5.1.0             h13dd4ca_1007    conda-forge
mock                      5.1.0                    pypi_0    pypi
morphio                   3.3.6                    pypi_0    pypi
mpfr                      4.2.0                he09a6ba_0    conda-forge
mpi                       1.0                     openmpi    conda-forge
mpi4py                    3.1.4           py311h01356f8_1    conda-forge
msgpack                   1.0.7                    pypi_0    pypi
mumps-include             5.2.1               hce30654_11    conda-forge
mumps-mpi                 5.2.1               h51c74a9_11    conda-forge
ncurses                   6.4                  h313beb8_0  
ndindex                   1.7                      pypi_0    pypi
networkx                  3.1                      pypi_0    pypi
numexpr                   2.8.7                    pypi_0    pypi
numpy                     1.26.0          py311hb8f3215_0    conda-forge
openmpi                   4.1.6              h4971d84_100    conda-forge
openssl                   3.1.3                h53f4e23_0    conda-forge
packaging                 23.2                     pypi_0    pypi
pandas                    1.5.3                    pypi_0    pypi
parmetis                  4.0.3             h6eb5794_1005    conda-forge
pcre2                     10.37                h37e8eca_1  
petsc                     3.20.0          real_h0603ae5_100    conda-forge
petsc4py                  3.20.0          real_h01487f0_100    conda-forge
pillow                    10.0.1                   pypi_0    pypi
pip                       23.2.1          py311hca03da5_0  
platformdirs              3.11.0                   pypi_0    pypi
pluggy                    1.3.0                    pypi_0    pypi
psutil                    5.9.5                    pypi_0    pypi
ptscotch                  6.0.9                h1aec651_2    conda-forge
py-cpuinfo                9.0.0                    pypi_0    pypi
pybind11-abi              4                    hd3eb1b0_1  
pycosat                   0.6.4           py311h80987f9_0  
pycparser                 2.21               pyhd3eb1b0_0  
pyopenssl                 23.0.0          py311hca03da5_0  
pyparsing                 3.1.1                    pypi_0    pypi
pyproject-api             1.6.1                    pypi_0    pypi
pysocks                   1.7.1           py311hca03da5_0  
pytest                    7.4.2                    pypi_0    pypi
pytest-click              1.1.0                    pypi_0    pypi
pytest-cov                4.1.0                    pypi_0    pypi
pytest-html               4.0.2                    pypi_0    pypi
pytest-metadata           3.0.0                    pypi_0    pypi
python                    3.11.5          h47c9636_0_cpython    conda-forge
python-dateutil           2.8.2                    pypi_0    pypi
python.app                3               py311h80987f9_0  
python_abi                3.11                    4_cp311    conda-forge
pytz                      2023.3.post1             pypi_0    pypi
pyyaml                    6.0.1                    pypi_0    pypi
readline                  8.2                  h1a28f6b_0  
reproc                    14.2.4               hc377ac9_1  
reproc-cpp                14.2.4               hc377ac9_1  
requests                  2.29.0          py311hca03da5_0  
ruamel.yaml               0.17.21         py311h80987f9_0  
scalapack                 2.2.0                h515df86_1    conda-forge
scipy                     1.11.3                   pypi_0    pypi
scotch                    6.0.9                h7537618_2    conda-forge
seaborn                   0.13.0                   pypi_0    pypi
setuptools                67.8.0          py311hca03da5_0  
six                       1.16.0             pyhd3eb1b0_1  
sqlite                    3.41.2               h80987f9_0  
suitesparse               5.10.1               h7cd81ec_1    conda-forge
superlu                   5.2.2                hc615359_0    conda-forge
superlu_dist              7.2.0                hfdb8677_0    conda-forge
tables                    3.9.1.dev0               pypi_0    pypi
tbb                       2021.10.0            h1995070_1    conda-forge
tk                        8.6.12               hb8d0fd4_0  
toolz                     0.12.0          py311hca03da5_0  
tox                       4.11.3                   pypi_0    pypi
tqdm                      4.65.0          py311hb6e6a13_0  
trimesh                   3.23.5                   pypi_0    pypi
truststore                0.8.0              pyhd8ed1ab_0    conda-forge
tzdata                    2023c                h04d1e81_0  
urllib3                   1.26.16         py311hca03da5_0  
vascpy                    0.1.1                    pypi_0    pypi
virtualenv                20.24.5                  pypi_0    pypi
vtk                       9.2.6                    pypi_0    pypi
wheel                     0.38.4          py311hca03da5_0  
xz                        5.4.2                h80987f9_0  
yaml                      0.2.5                h3422bc3_2    conda-forge
yaml-cpp                  0.7.0                hc377ac9_1  
zlib                      1.2.13               h53f4e23_5    conda-forge
zstandard                 0.19.0          py311h80987f9_0  
zstd                      1.5.5                hd90d995_0

And the Python packages installed with pip from conda:

$ $CONDA_PREFIX/bin/pip
Package                 Version
----------------------- ------------
blosc2                  2.2.9
boltons                 23.0.0
brotlipy                0.7.0
cached-property         1.5.2
cachetools              5.3.1
certifi                 2023.7.22
cffi                    1.15.1
chardet                 5.2.0
charset-normalizer      2.0.4
click                   8.1.7
colorama                0.4.6
conda                   23.9.0
conda-content-trust     0.1.3
conda-libmamba-solver   23.7.0
conda-package-handling  2.2.0
conda_package_streaming 0.9.0
contourpy               1.1.1
coverage                7.3.2
cryptography            39.0.1
cycler                  0.12.0
Cython                  3.0.3
distlib                 0.3.7
filelock                3.12.4
fonttools               4.43.1
h5py                    3.9.0
idna                    3.4
iniconfig               2.0.0
Jinja2                  3.1.2
jsonpatch               1.32
jsonpointer             2.1
kiwisolver              1.4.5
libmambapy              1.4.1
libsonata               0.1.23
MarkupSafe              2.1.3
matplotlib              3.8.0
mock                    5.1.0
MorphIO                 3.3.6
mpi4py                  3.1.4
msgpack                 1.0.7
ndindex                 1.7
networkx                3.1
numexpr                 2.8.7
numpy                   1.26.0
packaging               23.2
pandas                  1.5.3
petsc4py                3.20.0
Pillow                  10.0.1
pip                     23.2.1
platformdirs            3.11.0
pluggy                  1.3.0
psutil                  5.9.5
py-cpuinfo              9.0.0
pycosat                 0.6.4
pycparser               2.21
pyOpenSSL               23.0.0
pyparsing               3.1.1
pyproject-api           1.6.1
PySocks                 1.7.1
pytest                  7.4.2
pytest-click            1.1.0
pytest-cov              4.1.0
pytest-html             4.0.2
pytest-metadata         3.0.0
python-dateutil         2.8.2
pytz                    2023.3.post1
PyYAML                  6.0.1
requests                2.29.0
ruamel.yaml             0.17.21
scipy                   1.11.3
seaborn                 0.13.0
setuptools              67.8.0
six                     1.16.0
tables                  3.9.1.dev0
toolz                   0.12.0
tox                     4.11.3
tqdm                    4.65.0
trimesh                 3.23.5
truststore              0.8.0
urllib3                 1.26.16
vascpy                  0.1.1
virtualenv              20.24.5
vtk                     9.2.6
wheel                   0.38.4
zstandard               0.19.0

tristan0x commented 1 year ago

A way to force tox to see the conda Python packages is to force the PYTHONPATH environment variable.

For instance, you can execute in your shell the output of the following command:

$CONDA_PREFIX/bin/python3 -c "import sys; print('export PYTHONPATH={}'.format(';'.join(sys.path)))"

then, you may clean the previous tox environments (rm -rf .tox), and run tox again

bbpbuildbot commented 1 year ago

Logfiles from GitLab pipeline #159423 (:white_check_mark:) have been uploaded here!

Status and direct links:

bbpbuildbot commented 1 year ago

Logfiles from GitLab pipeline #159435 (:white_check_mark:) have been uploaded here!

Status and direct links:

bbpbuildbot commented 1 year ago

Logfiles from GitLab pipeline #159444 (:no_entry:) have been uploaded here!

Status and direct links:

bbpbuildbot commented 1 year ago

Logfiles from GitLab pipeline #159473 (:white_check_mark:) have been uploaded here!

Status and direct links:

bbpbuildbot commented 1 year ago

Logfiles from GitLab pipeline #159476 (:white_check_mark:) have been uploaded here!

Status and direct links:

BlueBrain / AstroVascPy

Tox improvements on local machine #8