search

haasad / PyPardiso

Python interface to the Intel MKL Pardiso library to solve large sparse linear systems of equations
BSD 3-Clause "New" or "Revised" License
133 stars 20 forks source link

[Windows] The Pardiso solver failed with error code -3 #36

Closed haasad closed 2 years ago

haasad commented 2 years ago

Issue was first reported in https://github.com/LCA-ActivityBrowser/activity-browser/issues/694.

I can reproduce it on one of my windows installations (didn't occur on the other one) by running the calculation step in the activity-browser:

Traceback (most recent call last):
  File "C:\Users\Adrian\miniconda3\envs\ab\lib\site-packages\activity_browser\layouts\tabs\LCA_results_tab.py", line 60, in generate_setup
    new_tab = LCAResultsSubTab(cs_name, presamples, self)
  File "C:\Users\Adrian\miniconda3\envs\ab\lib\site-packages\activity_browser\layouts\tabs\LCA_results_tabs.py", line 101, in __init__
    self.do_calculations()
  File "C:\Users\Adrian\miniconda3\envs\ab\lib\site-packages\activity_browser\layouts\tabs\LCA_results_tabs.py", line 128, in do_calculations
    self.mlca = MLCA(self.cs_name)
  File "C:\Users\Adrian\miniconda3\envs\ab\lib\site-packages\activity_browser\bwutils\multilca.py", line 124, in __init__
    self.lca.lci(factorize=True)
  File "C:\Users\Adrian\miniconda3\envs\ab\lib\site-packages\bw2calc\lca.py", line 342, in lci
    self.lci_calculation()
  File "C:\Users\Adrian\miniconda3\envs\ab\lib\site-packages\bw2calc\lca.py", line 350, in lci_calculation
    self.supply_array = self.solve_linear_system()
  File "C:\Users\Adrian\miniconda3\envs\ab\lib\site-packages\bw2calc\lca.py", line 314, in solve_linear_system
    return self.solver(self.demand_array)
  File "C:\Users\Adrian\miniconda3\envs\ab\lib\site-packages\pypardiso\scipy_aliases.py", line 46, in spsolve
    solver.factorize(A)
  File "C:\Users\Adrian\miniconda3\envs\ab\lib\site-packages\pypardiso\pardiso_wrapper.py", line 150, in factorize
    self._call_pardiso(A, b)
  File "C:\Users\Adrian\miniconda3\envs\ab\lib\site-packages\pypardiso\pardiso_wrapper.py", line 281, in _call_pardiso
    raise PyPardisoError(pardiso_error.value)
pypardiso.pardiso_wrapper.PyPardisoError: The Pardiso solver failed with error code -3. See Pardiso documentation for details.
haasad commented 2 years ago

Minimal example to reproduce:

python -c "import pypardiso as pp; import numpy as np; import scipy.sparse as sp; A = sp.rand(5000, 5000, density=0.05, format='csr'); b = np.random.rand(5000); x = pp.spsolve(A, b); print(x.sum())"

Traceback:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "C:\Users\Adrian\miniconda3\envs\ab\lib\site-packages\pypardiso\scipy_aliases.py", line 46, in spsolve
    solver.factorize(A)
  File "C:\Users\Adrian\miniconda3\envs\ab\lib\site-packages\pypardiso\pardiso_wrapper.py", line 150, in factorize
    self._call_pardiso(A, b)
  File "C:\Users\Adrian\miniconda3\envs\ab\lib\site-packages\pypardiso\pardiso_wrapper.py", line 281, in _call_pardiso
    raise PyPardisoError(pardiso_error.value)
pypardiso.pardiso_wrapper.PyPardisoError: The Pardiso solver failed with error code -3. See Pardiso documentation for details.

The minimal example does not fail on my other windows installation and on linux.

haasad commented 2 years ago

Output of the minimal example with pp.ps.set_statistical_info_on():

*** Error in PARDISO  (        reordering_phase) error_num= -180
*** error PARDISO: reordering, symbolic factorization

=== PARDISO: solving a real nonsymmetric system ===
1-based array indexing is turned ON
PARDISO double precision computation is turned ON
METIS algorithm at reorder step is turned ON
Scaling is turned ON
Matching is turned ON

Summary: ( starting phase is reordering, ending phase is factorization )
================

Times:
======
Time spent in calculations of symmetric matrix portrait (fulladj): 0.023025 s
Time spent in reordering of the initial matrix (reorder)         : 0.216925 s
Time spent in symbolic factorization (symbfct)                   : 0.019414 s
Time spent in allocation of internal data structures (malloc)    : 0.015355 s
Time spent in matching/scaling                                   : 0.144890 s
Time spent in additional calculations                            : 0.013778 s
Total time spent                                                 : 0.433386 s

Statistics:
===========
Parallel Direct Factorization is running on 4 OpenMP

< Linear system Ax = b >
             number of equations:           5000
             number of non-zeros in A:      1250000
             number of non-zeros in A (%): 5.000000

             number of right-hand sides:    1

< Factors L and U >
             number of columns for each panel: 96
             number of independent subgraphs:  0
< Preprocessing with state of the art partitioning metis>
             number of supernodes:                    761
             size of largest supernode:               1726
             number of non-zeros in L:                10483331
             number of non-zeros in U:                10083149
             number of non-zeros in L+U:              20566480
             gflop   for the numerical factorization: 0.000000
haasad commented 2 years ago

Apparently the error depends on the size of the matrices:

import pypardiso as pp
import numpy as np
import scipy.sparse as sp

for i in range(10):
    n = 1000+500*i
    A = sp.rand(n, n, density=1, format='csr')
    b = np.random.rand(n)
    print(f'shape: {A.shape}, nnz: {A.nnz}')
    x=pp.spsolve(A,b)
shape: (1000, 1000), nnz: 1000000
shape: (1500, 1500), nnz: 2250000
shape: (2000, 2000), nnz: 4000000
shape: (2500, 2500), nnz: 6250000
shape: (3000, 3000), nnz: 9000000
shape: (3500, 3500), nnz: 12250000
Traceback (most recent call last):
<...>
pypardiso.pardiso_wrapper.PyPardisoError: The Pardiso solver failed with error code -3. See Pardiso documentation for details.

With a lower density of A, the solver fails earlier:

import pypardiso as pp
import numpy as np
import scipy.sparse as sp

for i in range(10):
    n = 1000+500*i
    A = sp.rand(n, n, density=0.01, format='csr')
    b = np.random.rand(n)
    print(f'shape: {A.shape}, nnz: {A.nnz}')
    x=pp.spsolve(A,b)
shape: (1000, 1000), nnz: 10000
shape: (1500, 1500), nnz: 22500
shape: (2000, 2000), nnz: 40000
Traceback (most recent call last):
<...>
pypardiso.pardiso_wrapper.PyPardisoError: The Pardiso solver failed with error code -3. See Pardiso documentation for details.

However it doesn't consistently fail at the same size of A, even when using a seed. It fails somewhere between 1800 and 2200 using the code below:

import pypardiso as pp
import numpy as np
import scipy.sparse as sp

np.random.seed(0)
for i in range(10):
    n = 1500+100*i
    A = sp.rand(n, n, density=0.01, format='csr')
    b = np.random.rand(n)
    print(f'shape: {A.shape}, nnz: {A.nnz}')
    x=pp.spsolve(A,b)
    print(x.sum())
haasad commented 2 years ago

I was able to narrow down the differences between my two windows installations, it turns out the channel priority between defaults and conda-forge was different.

"Broken" environment:

conda create -n pp-broken -c conda-forge pypardiso python=3.9
conda list
# packages in environment at C:\Users\Adrian\miniconda3\envs\pp-broken:
#
# Name                    Version                   Build  Channel
bzip2                     1.0.8                h8ffe710_4    conda-forge
ca-certificates           2021.10.8            h5b45459_0    conda-forge
intel-openmp              2022.0.0          h57928b3_3663    conda-forge
libblas                   3.9.0              12_win64_mkl    conda-forge
libcblas                  3.9.0              12_win64_mkl    conda-forge
libffi                    3.4.2                h8ffe710_5    conda-forge
liblapack                 3.9.0              12_win64_mkl    conda-forge
libzlib                   1.2.11            h8ffe710_1013    conda-forge
m2w64-gcc-libgfortran     5.3.0                         6    conda-forge
m2w64-gcc-libs            5.3.0                         7    conda-forge
m2w64-gcc-libs-core       5.3.0                         7    conda-forge
m2w64-gmp                 6.1.0                         2    conda-forge
m2w64-libwinpthread-git   5.0.0.4634.697f757               2    conda-forge
mkl                       2021.4.0           h0e2418a_729    conda-forge
msys2-conda-epoch         20160418                      1    conda-forge
numpy                     1.22.0           py39h6331f09_0    conda-forge
openssl                   3.0.0                h8ffe710_2    conda-forge
pip                       21.3.1             pyhd8ed1ab_0    conda-forge
pypardiso                 0.4.0              pyhd8ed1ab_0    conda-forge
python                    3.9.9           hcf16a7b_0_cpython    conda-forge
python_abi                3.9                      2_cp39    conda-forge
scipy                     1.7.3            py39hc0c34ad_0    conda-forge
setuptools                60.5.0           py39hcbf5309_0    conda-forge
sqlite                    3.37.0               h8ffe710_0    conda-forge
tbb                       2021.5.0             h2d74725_0    conda-forge
tk                        8.6.11               h8ffe710_1    conda-forge
tzdata                    2021e                he74cb21_0    conda-forge
ucrt                      10.0.20348.0         h57928b3_0    conda-forge
vc                        14.2                 hb210afc_5    conda-forge
vs2015_runtime            14.29.30037          h902a5da_5    conda-forge
wheel                     0.37.1             pyhd8ed1ab_0    conda-forge
xz                        5.2.5                h62dcd97_1    conda-forge

Working environment:

conda create -n pp-working -c defaults -c conda-forge pypardiso python=3.9
conda list
# packages in environment at C:\Users\Adrian\miniconda3\envs\pp-working:
#
# Name                    Version                   Build  Channel
blas                      1.0                         mkl
ca-certificates           2021.10.26           haa95532_2
certifi                   2021.10.8        py39haa95532_0
icc_rt                    2019.0.0             h0cc432a_1
intel-openmp              2021.4.0          haa95532_3556
libblas                   3.9.0              12_win64_mkl    conda-forge
mkl                       2021.4.0           h0e2418a_729    conda-forge
mkl-service               2.4.0            py39h2bbff1b_0
mkl_fft                   1.3.1            py39h277e83a_0
mkl_random                1.2.2            py39hf11a4ad_0
numpy                     1.21.2           py39hfca59bb_0
numpy-base                1.21.2           py39h0829f74_0
openssl                   1.1.1l               h2bbff1b_0
pip                       21.2.4           py39haa95532_0
pypardiso                 0.4.0              pyhd8ed1ab_0    conda-forge
python                    3.9.7                h6244533_1
scipy                     1.7.3            py39h0a974cb_0
setuptools                58.0.4           py39haa95532_0
six                       1.16.0             pyhd3eb1b0_0
sqlite                    3.37.0               h2bbff1b_0
tbb                       2021.4.0             h59b6b97_0
tzdata                    2021e                hda174b7_0
vc                        14.2                 h21ff451_1
vs2015_runtime            14.27.29016          h5e58377_2
wheel                     0.37.0             pyhd3eb1b0_1
wincertstore              0.2              py39haa95532_2
haasad commented 2 years ago

Other ways of creating an environment where the error doesn't appear:

haasad commented 2 years ago

After some more tests, I'm pretty certain that the issue is caused because mkl version 2021.4.0 is not compatible with intel-openmp 2022.0.0. The mkl 2021.4.0 package from both the conda defaults channel and pypi have pinned their dependency on intel-openmp to version 2021.*.

haasad commented 2 years ago

If I install pypardiso from the haasad channel (which doesn't require an mkl build of libblas like on conda-forge (see here), I can also install mkl 2022.0.0 with intel-openmp 2022.0.0, which is yet another way to get an environment where the error doesn't happen.

conda create -n ppmkl22 -c haasad -c conda-forge pypardiso mkl=2022.0
conda list 
# packages in environment at C:\Users\adrian\anaconda3\envs\ppmkl22:
#
# Name                    Version                   Build  Channel
bzip2                     1.0.8                h8ffe710_4    conda-forge
ca-certificates           2021.10.8            h5b45459_0    conda-forge
intel-openmp              2022.0.0          h57928b3_3663    conda-forge
libblas                   3.9.0           5_hd5c7e75_netlib    conda-forge
libcblas                  3.9.0           5_hd5c7e75_netlib    conda-forge
libffi                    3.4.2                h8ffe710_5    conda-forge
liblapack                 3.9.0           5_hd5c7e75_netlib    conda-forge
libzlib                   1.2.11            h8ffe710_1013    conda-forge
m2w64-gcc-libgfortran     5.3.0                         6    conda-forge
m2w64-gcc-libs            5.3.0                         7    conda-forge
m2w64-gcc-libs-core       5.3.0                         7    conda-forge
m2w64-gmp                 6.1.0                         2    conda-forge
m2w64-libwinpthread-git   5.0.0.4634.697f757               2    conda-forge
mkl                       2022.0.0           h0e2418a_796    conda-forge
msys2-conda-epoch         20160418                      1    conda-forge
numpy                     1.22.0          py310hcae7c84_0    conda-forge
openssl                   3.0.0                h8ffe710_2    conda-forge
pip                       21.3.1             pyhd8ed1ab_0    conda-forge
pypardiso                 0.4.0                      py_0    haasad
python                    3.10.1          hcf16a7b_2_cpython    conda-forge
python_abi                3.10                    2_cp310    conda-forge
scipy                     1.7.3           py310h33db832_0    conda-forge
setuptools                60.5.0          py310h5588dad_0    conda-forge
sqlite                    3.37.0               h8ffe710_0    conda-forge
tbb                       2021.5.0             h2d74725_0    conda-forge
tk                        8.6.11               h8ffe710_1    conda-forge
tzdata                    2021e                he74cb21_0    conda-forge
ucrt                      10.0.20348.0         h57928b3_0    conda-forge
vc                        14.2                 hb210afc_6    conda-forge
vs2015_runtime            14.29.30037          h902a5da_6    conda-forge
wheel                     0.37.1             pyhd8ed1ab_0    conda-forge
xz                        5.2.5                h62dcd97_1    conda-forge
haasad commented 2 years ago

I also opened https://github.com/conda-forge/blas-feedstock/pull/81, which should fix the issue

haasad commented 2 years ago

Summary of the current situation:

Possible solutions:

  1. new release of libblas with a mkl 2022.0 build, as the 2022.0 versions of mkl and intel-openmp appear to be compatible -> PR: https://github.com/conda-forge/blas-feedstock/pull/81
  2. correct pins for intel-openmp in the conda-forge mkl package -> issue: https://github.com/conda-forge/intel_repack-feedstock/issues/33
  3. switch pypardiso from a noarch build to separate builds for each OS with a pinning for compatible mkl and intel-openmp on windows -> this is a lot of effort with the potential to break other things on linux and osx, last resort
  4. remove the requirement for a mkl-build of libblas and pin mkl to >=2022.0 -> this is probably the way to go short-term if there is no progress on 1. and 2., but it could lead to changes in performance on all operating systems for non-pardiso linalg workloads, because of the diferent libblas-build
haasad commented 2 years ago

Ping @cmutel, so you're aware of this issue in case you get reports of problems with brightway on windows

cmutel commented 2 years ago

Indeed

christian-cahig commented 2 years ago

Summary of the current situation:

  • mkl 2021.4 is not fully compatible with intel-openmp 2022.0, this only affects Windows
  • If you experience problems with pypardiso installed from conda-forge, downgrading intel-openmp is a temporary solution to fix it:
conda install -c conda-forge intel-openmp=2021.4
  • Because intel-openmp is only a Windows dependency (mkl conda-forge packages for linux and osx both use llvm-openmp) and pypardiso is a noarch package, there is no straightforward way to fix this with pypardiso's own dependency pins

Possible solutions:

  1. new release of libblas with a mkl 2022.0 build, as the 2022.0 versions of mkl and intel-openmp appear to be compatible -> PR: Update MKL to 2022.0 build conda-forge/blas-feedstock#81
  2. correct pins for intel-openmp in the conda-forge mkl package -> issue: Potential compatibility issue between mkl 2021.4.0 and intel-openmp 2022.0.0 on Windows conda-forge/intel_repack-feedstock#33
  3. switch pypardiso from a noarch build to separate builds for each OS with a pinning for compatible mkl and intel-openmp on windows -> this is a lot of effort with the potential to break other things on linux and osx, last resort
  4. remove the requirement for a mkl-build of libblas and pin mkl to >=2022.0 -> this is probably the way to go short-term if there is no progress on 1. and 2., but it could lead to changes in performance on all operating systems for non-pardiso linalg workloads, because of the diferent libblas-build

Thanks for this summary, @haasad. Just to clarify: there should not be a problem with mkl 2021.4 and intel-omp 2021.4 on Windows, right?

haasad commented 2 years ago

Thanks for this summary, @haasad. Just to clarify: there should not be a problem with mkl 2021.4 and intel-omp 2021.4 on Windows, right?

That's correct, according to my tests the problem only occurs with the combination of mkl 2021.4 and intel-openmp 2022.0.

astudillo20lca commented 2 years ago

I tried to downgrade to 2021.4 but I am getting the same error as reported here :-/

imagen

haasad commented 2 years ago

Hi @astudillo20lca,

libblas with the mkl 2022.0 build has been released in the meantime, so downgrading intel-openmp shouldn't be necessary anymore.

Would you mind posting the output of conda list of your environment?

astudillo20lca commented 2 years ago

sure thing

Name Version Build Channel

alabaster 0.7.12 pyhd3eb1b0_0 anyio 3.5.0 py39haa95532_0 appdirs 1.4.4 pyhd3eb1b0_0 argon2-cffi 21.3.0 pyhd3eb1b0_0 argon2-cffi-bindings 21.2.0 py39h2bbff1b_0 asgiref 3.4.1 pyhd3eb1b0_0 asteval 0.9.23 pyhd8ed1ab_0 conda-forge astunparse 1.6.3 py_0 async_generator 1.10 pyhd3eb1b0_0 attrs 21.4.0 pyhd3eb1b0_0 babel 2.9.1 pyhd3eb1b0_0 backcall 0.2.0 pyhd3eb1b0_0 blas 1.0 mkl bleach 4.1.0 pyhd3eb1b0_0 brightway25 1.0.6 py_0 cmutel
brotli 1.0.9 ha925a31_2 brotlipy 0.7.0 py39h2bbff1b_1003 bw-migrations 0.2 pypi_0 pypi
bw2analyzer 0.11.1 py_0 cmutel bw2calc 2.0.dev5 py_0 cmutel bw2data 4.0.dev11 py_0 cmutel bw2io 0.9.dev7 py_0 cmutel bw2parameters 0.7 py_0 cmutel bw_migrations 0.1 py_0 cmutel bw_processing 0.7.1 py_0 cmutel bzip2 1.0.8 he774522_0 ca-certificates 2021.10.8 h5b45459_0 conda-forge carculator 2021.05.31 py_0 romainsacchi carculator_bus 2021.04.12 py_0 romainsacchi carculator_truck 2021.07.08 py_0 romainsacchi certifi 2021.10.8 py39hcbf5309_1 conda-forge cffi 1.15.0 py39h2bbff1b_1 cfitsio 3.470 he774522_6 chardet 4.0.0 pypi_0 pypi charset-normalizer 2.0.4 pyhd3eb1b0_0 click 7.1.2 pyhd3eb1b0_0 click-plugins 1.1.1 pyhd3eb1b0_0 cligj 0.7.2 py39haa95532_0 colorama 0.4.4 pyhd3eb1b0_0 configupdater 3.0.1 pyhd8ed1ab_0 conda-forge constructive_geometries 0.7 py_0 cmutel country_converter 0.7.4 pyhd8ed1ab_0 conda-forge cryptography 36.0.0 py39h21b164f_0 curl 7.80.0 h2bbff1b_0 cycler 0.11.0 pyhd3eb1b0_0 danishcrown-pyexcel 0.0.post1.dev23+g5a374c8.d20220111 dev_0 debugpy 1.5.1 py39hd77b12b_0 decorator 5.1.1 pyhd3eb1b0_0 defusedxml 0.7.1 pyhd3eb1b0_0 django 3.2.9 pyhd8ed1ab_0 conda-forge docopt 0.6.2 py39haa95532_0 docutils 0.17.1 py39haa95532_1 entrypoints 0.3 py39haa95532_0 et_xmlfile 1.1.0 py39haa95532_0 expat 2.4.4 h6c2663c_0 fasteners 0.16.3 pyhd3eb1b0_0 fiona 1.8.13.post1 py39h758c064_0 fonttools 4.25.0 pyhd3eb1b0_0 freetype 2.10.4 hd328e21_0 freexl 1.0.6 h2bbff1b_0 frictionless 4.22.3 pypi_0 pypi fs 2.4.11 py39hcbf5309_3 conda-forge gdal 3.0.2 py39hb978731_2 geos 3.8.0 h33f27b4_0 geotiff 1.7.0 h4545760_0 hdf4 4.2.13 h712560f_2 hdf5 1.10.6 h7ebc959_0 icc_rt 2019.0.0 h0cc432a_1 icu 58.2 ha925a31_3 idna 3.3 pyhd3eb1b0_0 imagesize 1.3.0 pyhd3eb1b0_0 importlib-metadata 4.8.2 py39haa95532_0 importlib_metadata 4.8.2 hd3eb1b0_0 intel-openmp 2021.4.0 haa95532_3556 ipykernel 6.4.1 py39haa95532_1 ipython 7.31.1 py39haa95532_0 ipython_genutils 0.2.0 pyhd3eb1b0_1 isodate 0.6.1 pypi_0 pypi jedi 0.18.1 py39haa95532_1 jinja2 3.0.2 pyhd3eb1b0_0 jpeg 9d h2bbff1b_0 json5 0.9.6 pyhd3eb1b0_0 jsonschema 3.2.0 pyhd3eb1b0_2 jupyter_client 7.1.2 pyhd3eb1b0_0 jupyter_core 4.9.1 py39haa95532_0 jupyter_server 1.13.5 pyhd3eb1b0_0 jupyterlab 3.2.9 pyhd8ed1ab_0 conda-forge jupyterlab_pygments 0.1.2 py_0 jupyterlab_server 2.10.3 pyhd3eb1b0_1 kealib 1.4.14 hde4a422_1 kiwisolver 1.3.2 py39hd77b12b_0 klausen 0.1.1 py_0 cmutel krb5 1.19.2 h5b6d351_0 libblas 3.9.0 12_win64_mkl conda-forge libcurl 7.80.0 h86230a5_0 libgdal 3.0.2 h3d7c30d_2 libiconv 1.15 h1df5818_7 libnetcdf 4.8.1 h6685c40_1 libpng 1.6.37 h2a8f88b_0 libpq 12.9 hb652d5d_1 libspatialite 4.3.0a h14feca5_20 libssh2 1.9.0 h7a1dbc1_1 libtiff 4.2.0 hd0e1b90_0 libwebp 1.2.0 h2bbff1b_0 libxml2 2.9.12 h0ad7f3c_0 libxslt 1.1.34 he774522_0 libzip 1.5.1 h0ff8eda_1001 lxml 4.7.1 py39h1985fb9_1 lz4-c 1.9.3 h2bbff1b_1 m2w64-expat 2.1.1 2 m2w64-gcc-libgfortran 5.3.0 6 m2w64-gcc-libs 5.3.0 7 m2w64-gcc-libs-core 5.3.0 7 m2w64-gettext 0.19.7 2 m2w64-gmp 6.1.0 2 m2w64-libiconv 1.14 6 m2w64-libwinpthread-git 5.0.0.4634.697f757 2 m2w64-xz 5.2.2 2 markdown-it-py 2.0.1 pyhd8ed1ab_0 conda-forge marko 1.1.0 pypi_0 pypi markupsafe 2.0.1 py39h2bbff1b_0 matplotlib 3.4.3 py39hcbf5309_1 conda-forge matplotlib-base 3.4.3 py39h49ac443_0 matplotlib-inline 0.1.2 pyhd3eb1b0_2 matrix_utils 0.2.3 py_0 cmutel mdit-py-plugins 0.3.0 pyhd8ed1ab_0 conda-forge mdurl 0.1.0 pyhd8ed1ab_0 conda-forge mistune 0.8.4 py39h2bbff1b_1000
mkl 2021.4.0 h0e2418a_729 conda-forge mkl-service 2.4.0 py39h2bbff1b_0 mkl_fft 1.3.1 py39h277e83a_0 mkl_random 1.2.2 py39hf11a4ad_0 mrio-common-metadata 0.2 pypi_0 pypi mrio_common_metadata 0.1.1 py_0 cmutel msys2-conda-epoch 20160418 1 munch 2.5.0 pyhd3eb1b0_0 munkres 1.1.4 py_0 myst-parser 0.16.1 pyhd8ed1ab_0 conda-forge nbclassic 0.3.5 pyhd3eb1b0_0 nbclient 0.5.3 pyhd3eb1b0_0 nbconvert 6.1.0 py39haa95532_0 nbformat 5.1.3 pyhd3eb1b0_0 nest-asyncio 1.5.1 pyhd3eb1b0_0 networkx 2.6.2 pyhd3eb1b0_0 notebook 6.4.8 py39haa95532_0 numexpr 2.8.1 py39hb80d3ca_0 numpy 1.21.5 py39ha4e8547_0 numpy-base 1.21.5 py39hc2deb75_0 olefile 0.46 pyhd3eb1b0_0 openjpeg 2.4.0 h4fc8c34_0 openpyxl 3.0.9 pyhd3eb1b0_0 openssl 1.1.1l h8ffe710_0 conda-forge packaging 21.3 pyhd3eb1b0_0 pandas 1.4.0 py39h2e25243_0 conda-forge pandocfilters 1.5.0 pyhd3eb1b0_0 parso 0.8.3 pyhd3eb1b0_0 peewee 3.14.9 py39h539719c_0 conda-forge petl 1.7.4 pypi_0 pypi pickleshare 0.7.5 pyhd3eb1b0_1003 pillow 8.4.0 py39hd45dc43_0 pip 21.2.4 py39haa95532_0 prettytable 3.1.0 pyhd8ed1ab_0 conda-forge proj 6.2.1 h3758d61_0 prometheus_client 0.13.1 pyhd3eb1b0_0 prompt-toolkit 3.0.20 pyhd3eb1b0_0 psutil 5.8.0 py39h2bbff1b_1 pycountry 20.7.3 pyh9f0ad1d_0 conda-forge pycparser 2.21 pyhd3eb1b0_0 pygments 2.11.2 pyhd3eb1b0_0 pyopenssl 22.0.0 pyhd3eb1b0_0 pypardiso 0.4.0 pyhd8ed1ab_0 conda-forge pyparsing 3.0.4 pyhd3eb1b0_0 pyprind 2.11.2 py39hcbf5309_1002 conda-forge pyqt 5.9.2 py39hd77b12b_6 pyrsistent 0.18.0 py39h196d8e1_0 pyscaffold 4.1.1 pyhd8ed1ab_0 conda-forge pysocks 1.7.1 py39haa95532_0 python 3.9.7 h6244533_1 python-dateutil 2.8.2 pyhd3eb1b0_0 python-json-logger 2.0.1 py_0 python-slugify 5.0.2 pypi_0 pypi python_abi 3.9 2_cp39 conda-forge pytools20 0.1.0 dev_0 pytz 2021.3 pyhd3eb1b0_0 pywin32 302 py39h827c3e9_1 pywinpty 2.0.2 py39h5da7b33_0 pyxlsb 1.0.9 pypi_0 pypi pyyaml 6.0 py39h2bbff1b_1 pyzmq 22.3.0 py39hd77b12b_2 qt 5.9.7 vc14h73c81de_0 requests 2.27.1 pyhd3eb1b0_0 rfc3986 1.5.0 pypi_0 pypi scipy 1.7.3 py39h0a974cb_0 seaborn 0.11.2 pyhd3eb1b0_0 send2trash 1.8.0 pyhd3eb1b0_1 setuptools 58.0.4 py39haa95532_0 setuptools-scm 6.3.2 pyhd3eb1b0_0 shapely 1.7.1 py39h06580b3_0 shellingham 1.4.0 pypi_0 pypi simpleeval 0.9.11 pypi_0 pypi sip 4.19.13 py39hd77b12b_0 six 1.16.0 pyhd3eb1b0_1 sniffio 1.2.0 py39haa95532_1 snowballstemmer 2.2.0 pyhd3eb1b0_0 sphinx 4.4.0 pyhd3eb1b0_0 sphinxcontrib-applehelp 1.0.2 pyhd3eb1b0_0 sphinxcontrib-devhelp 1.0.2 pyhd3eb1b0_0 sphinxcontrib-htmlhelp 2.0.0 pyhd3eb1b0_0 sphinxcontrib-jsmath 1.0.1 pyhd3eb1b0_0 sphinxcontrib-qthelp 1.0.3 pyhd3eb1b0_0 sphinxcontrib-serializinghtml 1.1.5 pyhd3eb1b0_0 sqlite 3.37.2 h2bbff1b_0 sqlparse 0.4.1 py_0 stats_arrays 0.6.5 py_2 cmutel stringcase 1.2.0 pypi_0 pypi tabulate 0.8.9 py39haa95532_0 tbb 2021.5.0 h59b6b97_0 terminado 0.13.1 py39haa95532_0 testpath 0.5.0 pyhd3eb1b0_0 text-unidecode 1.3 pypi_0 pypi tiledb 2.2.9 hf84e3da_0 conda-forge tk 8.6.11 h2bbff1b_0 tomli 1.2.2 pyhd3eb1b0_0 tomlkit 0.9.2 pyhd3eb1b0_0 toolz 0.11.2 pyhd3eb1b0_0 tornado 6.1 py39h2bbff1b_0 tqdm 4.62.2 pyhd3eb1b0_1 traitlets 5.1.1 pyhd3eb1b0_0 typer 0.4.0 pypi_0 pypi typing-extensions 3.10.0.2 hd3eb1b0_0 typing_extensions 3.10.0.2 pyh06a4308_0 tzdata 2021e hda174b7_0 ulcarchetype 0.2 dev_0 unidecode 1.2.0 pyhd3eb1b0_0 urllib3 1.26.8 pyhd3eb1b0_0 validators 0.18.2 pypi_0 pypi vc 14.2 h21ff451_1 voluptuous 0.12.2 pyhd8ed1ab_1 conda-forge vs2015_runtime 14.27.29016 h5e58377_2 wcwidth 0.2.5 pyhd3eb1b0_0 webencodings 0.5.1 py39haa95532_1 websocket-client 0.58.0 py39haa95532_4 wheel 0.37.1 pyhd3eb1b0_0 whoosh 2.7.4 pyhd3eb1b0_1 win_inet_pton 1.1.0 py39haa95532_0 wincertstore 0.2 py39haa95532_2 winpty 0.4.3 4 wrapt 1.13.3 py39h2bbff1b_2 wurst 0.3 py_0 cmutel xarray 0.19.0 pyhd3eb1b0_1 xerces-c 3.2.3 ha925a31_0 xlrd 2.0.1 pyhd3eb1b0_0 xlsxwriter 3.0.2 pyhd3eb1b0_0 xlwings 0.24.7 py39haa95532_0 xz 5.2.5 h62dcd97_0 yaml 0.2.5 he774522_0 zipp 3.7.0 pyhd3eb1b0_0 zlib 1.2.11 h8cc25b3_4 zstd 1.4.9 h19a0ad4_0

haasad commented 2 years ago

It looks like you have two different BLAS packages in your environment. blas from the defaults channel and libblas from conda-forge. I could imagine that they interfere with each other. I would generally advise you against mixing packages from the defaults channel and conda-forge in the same environment. Especially when working with numpy et al.

In my experience you'll run into compatibility issues rather sooner than later when working with a single conda environment with a lot of different packages installed. I'd suggest to have separate environments for e.g. GIS stuff (gdal and friends) and LCA (brightway etc.).

Can you check if you still get the Pardiso error in a "fresh" environment? Something like:

conda create -n issue36 -c conda-forge -c cmutel brightway25 python=3.9
haasad commented 2 years ago

Also you're mixing intel-openmp from the defaults and mkl from conda-forge, not sure if they're compatible. Maybe this could fix the problem in your existing env, but no guarantees:

conda install -c conda-forge --override-channels mkl=2022 intel-openmp=2022
astudillo20lca commented 2 years ago

thanks Adrian! with a fresh installation it does indeed work. I'll follow your advice

haasad commented 2 years ago

With https://github.com/conda-forge/blas-feedstock/pull/81 being merged, the underlying problem shouldn't appear anymore.

SELTENEERDEN commented 2 years ago

After updating Anaconda, Brightway2 and Activity-Browser the work around "conda create -n ab_fixed activity-browser-dev intel-openmp=2021.4" mentioned here https://github.com/LCA-ActivityBrowser/activity-browser/issues/694#issuecomment-1015326029 does not work anymore for me. Could you please explain for a real DAU what the current workaround for the pypardiso issue is?

cmutel commented 2 years ago

@SELTENEERDEN Can you give some input on your system and library version? E.g. OS and version, conda or pip, brightway 2 or 2.5? And what is the exact error when you create a new environment?

tngTUDOR commented 1 year ago

I was also bitten by this issue, under Linux. The issue appeared on an install that was created with:

conda install -c conda-forge -c cmutel brightway25
# and then, I forgot to keep the conda-forge channel as the main source by doing:
conda update conda

The snippet above (the second conda execution in particular) pulled intel-openmp dependency and trashed my original environment.

TIP: keep conda-forge as the main source to get packages

a) Don't forget to use it as the first channel in your conda commands (update, install, etc.) or b) [nuke option] update your $HOME/.condarc to look like follows:

channel_priority: strict
channels:
  - conda-forge
  - defaults

ensure that all the dependencies come from the conda-forge channel unless they exist only on defaults

ref here from conda-forge

and never forget you did this !