ImportError: DLL load failed while importing _scs_direct

[hanyucc]

I am building scs for Python 3.8 on Windows, and I keep getting this error when trying to import scs.

Traceback (most recent call last):
  File "D:/xxx", line 4, in <module>
    import _scs_direct
ImportError: DLL load failed while importing _scs_direct: The specified module could not be found.

I linked NumPy to OpenBLAS, so scs should be building with it as well. I'm not sure what DLL it is looking for, but I did put C:/OpenBLAS/bin in my Path variable. I also looked inside the site-packages folder, and there does exist a file named _scs_direct.cp38-win_amd64.pyd, not sure if it's related though.

This is what NumPy outputs currently:

>>> import numpy
>>> from numpy.distutils.system_info import get_info
>>> blas_info = get_info('blas_opt')
>>> print(blas_info)
{'libraries': ['libopenblas', 'libopenblas'], 'library_dirs': ['C:/OpenBLAS/lib'], 'language': 'c', 'define_macros': [('HAVE_CBLAS', None)], 'runtime_library_dirs': ['C:/OpenBLAS/lib']}
>>> lapack_info = get_info('lapack_opt')
>>> print(lapack_info)
{'libraries': ['libopenblas', 'libopenblas'], 'library_dirs': ['C:/OpenBLAS/lib'], 'language': 'c', 'define_macros': [('HAVE_CBLAS', None)], 'runtime_library_dirs': ['C:/OpenBLAS/lib']}

[bodono]

It looks to me like this can't find the actual scs dll, rather than the blas dll. Can you try pointing all the PATH variables (PYTHON_PATH etc) to wherever the _scs_direct dll library has been installed?

[hgroll]

I got that error regarding _scs_direct when conda pulled the pypy build of scs in instead of the cpython version which is suitable for my conda installation. Reinstalling scs with the specific build solved the problem for me. The specific cpython build was scs=3.0.0=py37hd9fb7ff_0.

[h-vetinari]

I got that error regarding _scs_direct when conda pulled the pypy build of scs in instead of the cpython version

Hi, I'm helping to package scs for conda-forge. What you describe shouldn't happen, can you explain how you arrived in such a state?

[hgroll]

I don't really know how I arrived in that state. My conda is version 4.10.3 and win64. First, conda couldn't resolve the environment, i.e. conda install .. hanged in "Solving environment /", but this is not an scs issue. I could not install anything anymore. Then I've tried to resolve the issue with some hints in https://github.com/conda/conda/issues/7690 or https://github.com/conda/conda/issues/8051 which didn't really help. After some commenters complaining about memory size during package resolving, I've tried to install scs with mamba. Mamba allowed me to install packages, but somehow pulled in the pypy version of scs. This could be a mamba issue also.

[afugls]

I am experiencing the same problem when installing scs from source on Windows. I cloned the lastest version of scs-python from git. Because I am using the "x86_x64 Cross Tools Command Prompt for VS 2017" I then edited line 172 in setup.py, such it reads extra_compile_args = ["/O2"] instead of extra_compile_args = ["-O3"]. I then installed scs using the "x86_x64 Cross Tools Command Prompt for VS 2017" by

cd scs-python 
python setup.py install

The output was

Namespace(scs=False, gpu=False, mkl=False, float32=False, extraverbose=False, gpu_atrans=True, int32=False, blas64=False)
running install
C:\Users\ajf\AppData\Local\Programs\Python\Python310\lib\site-packages\setuptools\command\install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
  warnings.warn(
C:\Users\ajf\AppData\Local\Programs\Python\Python310\lib\site-packages\setuptools\command\easy_install.py:144: EasyInstallDeprecationWarning: easy_install command is deprecated. Use build and pip and other standards-based tools.
  warnings.warn(
running bdist_egg
running egg_info
writing scs.egg-info\PKG-INFO
writing dependency_links to scs.egg-info\dependency_links.txt
writing requirements to scs.egg-info\requires.txt
writing top-level names to scs.egg-info\top_level.txt
C:\Users\ajf\git\scs-python\setup.py:93: DeprecationWarning:

  `numpy.distutils` is deprecated since NumPy 1.23.0, as a result
  of the deprecation of `distutils` itself. It will be removed for
  Python >= 3.12. For older Python versions it will remain present.
  It is recommended to use `setuptools < 60.0` for those Python versions.
  For more details, see:
    https://numpy.org/devdocs/reference/distutils_status_migration.html

  from numpy.distutils.system_info import get_info
openblas64__info:
    library_dirs = ['D:\\a\\numpy\\numpy\\build\\openblas64__info']
    libraries = ['openblas64__info']
    language = f77
    define_macros = [('HAVE_CBLAS', None), ('BLAS_SYMBOL_SUFFIX', '64_'), ('HAVE_BLAS_ILP64', None)]
blas_ilp64_opt_info:
    library_dirs = ['D:\\a\\numpy\\numpy\\build\\openblas64__info']
    libraries = ['openblas64__info']
    language = f77
    define_macros = [('HAVE_CBLAS', None), ('BLAS_SYMBOL_SUFFIX', '64_'), ('HAVE_BLAS_ILP64', None)]
openblas64__lapack_info:
    library_dirs = ['D:\\a\\numpy\\numpy\\build\\openblas64__lapack_info']
    libraries = ['openblas64__lapack_info']
    language = f77
    define_macros = [('HAVE_CBLAS', None), ('BLAS_SYMBOL_SUFFIX', '64_'), ('HAVE_BLAS_ILP64', None), ('HAVE_LAPACKE', None)]
lapack_ilp64_opt_info:
    library_dirs = ['D:\\a\\numpy\\numpy\\build\\openblas64__lapack_info']
    libraries = ['openblas64__lapack_info']
    language = f77
    define_macros = [('HAVE_CBLAS', None), ('BLAS_SYMBOL_SUFFIX', '64_'), ('HAVE_BLAS_ILP64', None), ('HAVE_LAPACKE', None)]
Supported SIMD extensions in this NumPy install:
    baseline = SSE,SSE2,SSE3
    found = SSSE3,SSE41,POPCNT,SSE42,AVX,F16C,FMA3,AVX2
    not found = AVX512F,AVX512CD,AVX512_SKX,AVX512_CLX,AVX512_CNL,AVX512_ICL
WARN: Could not locate executable g77
WARN: Could not locate executable f77
WARN: Could not locate executable ifort
WARN: Could not locate executable ifl
WARN: Could not locate executable f90
WARN: Could not locate executable DF
WARN: Could not locate executable efl
WARN: Could not locate executable gfortran
WARN: Could not locate executable f95
WARN: Could not locate executable g95
WARN: Could not locate executable efort
WARN: Could not locate executable efc
WARN: Could not locate executable flang
WARN: don't know how to compile Fortran code on platform 'nt'
C:\Users\ajf\AppData\Local\Programs\Python\Python310\lib\site-packages\numpy\distutils\system_info.py:2077: UserWarning:
    Optimized (vendor) Blas libraries are not found.
    Falls back to netlib Blas library which has worse performance.
    A better performance should be easily gained by switching
    Blas library.
  if self._calc_info(blas):
Blas info:
{'define_macros': [('NO_ATLAS_INFO', 1)], 'libraries': ['blas', 'blas'], 'library_dirs': ['C:\\blaslapack'], 'include_dirs': ['C:\\Users\\ajf\\AppData\\Local\\Programs\\Python\\Python310\\include'], 'language': 'f77'}
Lapack info:
{'libraries': ['lapack', 'lapack', 'blas', 'blas'], 'library_dirs': ['C:\\blaslapack'], 'language': 'f77', 'define_macros': [('NO_ATLAS_INFO', 1)], 'include_dirs': ['C:\\Users\\ajf\\AppData\\Local\\Programs\\Python\\Python310\\include']}
Adding scs 3.2.1 to easy-install.pth file

Installed c:\users\ajf\appdata\local\programs\python\python310\lib\site-packages\scs-3.2.1-py3.10-win-amd64.egg
Processing dependencies for scs==3.2.1
Searching for scipy==1.9.1
Best match: scipy 1.9.1
Adding scipy 1.9.1 to easy-install.pth file

Using c:\users\ajf\appdata\local\programs\python\python310\lib\site-packages
Searching for numpy==1.23.3
Best match: numpy 1.23.3
Adding numpy 1.23.3 to easy-install.pth file
Installing f2py-script.py script to C:\Users\ajf\AppData\Local\Programs\Python\Python310\Scripts
Installing f2py.exe script to C:\Users\ajf\AppData\Local\Programs\Python\Python310\Scripts

Using c:\users\ajf\appdata\local\programs\python\python310\lib\site-packages
Finished processing dependencies for scs==3.2.1

However when importing scs I get the following error

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Users\ajf\AppData\Local\Programs\Python\Python310\lib\site-packages\scs-3.2.1-py3.10-win-amd64.egg\scs\__init__.py", line 4, in <module>
    import _scs_direct
ImportError: DLL load failed while importing _scs_direct: The specified module could not be found.

I can see that scs is installed in the following structure

site-packages\
     scs-3.2.1-py3.10-win-amd64.egg\
                    _scs_direct.cp310-win_amd64.pyd
                    _scs_direct.py
                    _scs_indirect.cp310-win_amd64.pyd
                    _scs_indirect.py
                     scs\
                             __init__.py
                    EGG-INFO\
                    __pyache__\    

When I check sys.path I get the following output.

['', 'C:\\Users\\ajf\\AppData\\Local\\Programs\\Python\\Python310\\python310.zip', 'C:\\Users\\ajf\\AppData\\Local\\Programs\\Python\\Python310\\DLLs', 'C:\\Users\\ajf\\AppData\\Local\\Programs\\Python\\Python310\\lib', 'C:\\Users\\ajf\\AppData\\Local\\Programs\\Python\\Python310', 'C:\\Users\\ajf\\AppData\\Roaming\\Python\\Python310\\site-packages', 'C:\\Users\\ajf\\AppData\\Roaming\\Python\\Python310\\site-packages\\win32', 'C:\\Users\\ajf\\AppData\\Roaming\\Python\\Python310\\site-packages\\win32\\lib', 'C:\\Users\\ajf\\AppData\\Roaming\\Python\\Python310\\site-packages\\Pythonwin', 'C:\\Users\\ajf\\AppData\\Local\\Programs\\Python\\Python310\\lib\\site-packages', 'C:\\Users\\ajf\\AppData\\Local\\Programs\\Python\\Python310\\lib\\site-packages\\pyrirgen-0.0.0-py3.10-win-amd64.egg', 'C:\\Users\\ajf\\AppData\\Local\\Programs\\Python\\Python310\\lib\\site-packages\\scs-3.2.1-py3.10-win-amd64.egg']

So far I have not been able to make it work

[bodono]

Have you made sure your PATH variables are pointing to the SCS install directory?

[afugls]

Have you made sure your PATH variables are pointing to the SCS install directory?

I am not sure what you mean by your question. I am a bit new to installing modules from source. Do you mean that the system and user PATH variables should point to the C:(...)\python(...)\site-packages\scs directory? Or is the scs-python\ directory that I cloned from github?

[bodono]

Yeah the DLL loader needs to know where the libraries are installed, looks like it's somewhere inside c:\users\ajf\appdata\local\programs\python\python310\lib\site-packages, make sure the _scs_direct.dll (or whatever) is inside the directory then update your PATH variables (and maybe PYTHONPATH too, not sure exactly on windows).

[afugls]

I managed to fix the problem. I think, I had already set my PATH correctly. However, I had a lot of missing .dll files. It was just not clear from the error message.

Firstly, I did a re-install of scs where i called python setup.py install --scs. This changed the directory structure a bit and added _scs_direct.py to site-packages. Then the error seemed to indicate that _scs_direct.cpy310-win_amd64.pyd was missing from the site-packages\ directory, so I copied it there.

There error continued, but I then used Process Monitor as suggested here to check the name of DLLs Python was looking for. I needed to move libblas and liblapack to the site-packages\ directory (even though I had them in a directory added to PATH), and also download and add libgcc_s_seh_61-1, libgcc_s_seh-1, libgfortran_64-3, libquadmath-0, and libwinpthread-1 to the site-packages\ directory. I think, I could also just add all the missing .dll files to C:\Windows\System32\, since Python seemed also to search there.

Also thank you @bodono for the quick responses.