search

SpikeInterface / spikeinterface

A Python-based module for creating flexible and robust spike sorting pipelines.
https://spikeinterface.readthedocs.io
MIT License
526 stars 187 forks source link

si_env issue #553

Closed vncntprvst closed 2 years ago

vncntprvst commented 2 years ago

When activating the si_env created with full_spikeinterface_environment_windows.yml, I get the error below. Removing msmpi and its .dlls then re-creating the environment did not resolve the issue. Visual Studio and C++ Build SKU are installed. 

V:\>conda activate si_env

V:\>set "_LIB="

V:\>set "_CPATH="

V:\>if "V:\ProgramData\Anaconda3\envs\si_env" == "" set "LIB=;\Library\lib"

V:\>if "V:\ProgramData\Anaconda3\envs\si_env" == "" set "CPATH=;\Library\include"

V:\>if not "V:\ProgramData\Anaconda3\envs\si_env" == "" set "LIB=;V:\ProgramData\Anaconda3\envs\si_env\Library\lib"

V:\>if not "V:\ProgramData\Anaconda3\envs\si_env" == "" set "CPATH=;V:\ProgramData\Anaconda3\envs\si_env\Library\include"

V:\>if defined CONDA_BUILD_STATE ()

V:\>if exist "C:\Windows\System32\msmpi.dll" (
echo "You seem to have a system wide installation of MSMPI. "
 echo "Due to the way DLL loading works on windows, system wide installation "
 echo "will probably overshadow the conda installation. Uninstalling "
 echo "the system wide installation and forced deleting C:\Windows\System32\msmpi*.dll"
 echo "will help, but may break other software using the system wide installation."
)

V:\>if defined MSMPI_BIN (set "MSMPI_BIN_CONDA_BACKUP=" )

V:\>if defined MSMPI_INC (set "MSMPI_INC_CONDA_BACKUP=" )

V:\>if defined MSMPI_LIB64 (set "MSMPI_LIB64_CONDA_BACKUP=" )

V:\>if defined MSMPI_LIB32 (set "MSMPI_LIB32_CONDA_BACKUP=" )

V:\>set MSMPI_BIN=\Library\bin

V:\>set MSMPI_INC=\Library\include

V:\>set MSMPI_LIB64=\Library\lib

V:\>set MSMPI_LIB32=""

V:\>SET DISTUTILS_USE_SDK=1

V:\>SET MSSdk=1

V:\>SET "VS_VERSION=15.0"

V:\>SET "VS_MAJOR=15"

V:\>SET "VS_YEAR=2017"

V:\>set "MSYS2_ARG_CONV_EXCL=/AI;/AL;/OUT;/out"

V:\>set "MSYS2_ENV_CONV_EXCL=CL"

V:\>set "PY_VCRUNTIME_REDIST=\bin\vcruntime140.dll"

V:\>set "CXX=cl.exe"

V:\>set "CC=cl.exe"

V:\>set "VSINSTALLDIR="

V:\>set "NEWER_VS_WITH_OLDER_VC=0"

V:\>for /F "usebackq tokens=*" %i in (`vswhere.exe -nologo -products * -version [15.0,16.0) -property installationPath`) do (set "VSINSTALLDIR=%i\" )

V:\>(set "VSINSTALLDIR=C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\" )

V:\>if not exist "C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\" (for /F "usebackq tokens=*" %i in (`vswhere.exe -nologo -products * -requires Microsoft.VisualStudio.Component.VC.v141.x86.x64 -property installationPath`) do (
set "VSINSTALLDIR=%i\"
 set "NEWER_VS_WITH_OLDER_VC=1"
) )

V:\>if not exist "C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\" (set "VSINSTALLDIR=C:\Program Files (x86)\Microsoft Visual Studio\2017\Professional\" )

V:\>if not exist "C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\" (set "VSINSTALLDIR=C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\" )

V:\>if not exist "C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\" (set "VSINSTALLDIR=C:\Program Files (x86)\Microsoft Visual Studio\2017\BuildTools\" )

V:\>if not exist "C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\" (set "VSINSTALLDIR=C:\Program Files (x86)\Microsoft Visual Studio\2017\Enterprise\" )

V:\>IF NOT "" == "" (
set "INCLUDE=;"
 set "LIB=;;V:\ProgramData\Anaconda3\envs\si_env\Library\lib"
 set "CMAKE_PREFIX_PATH=;"
)

V:\>call :GetWin10SdkDir

V:\>call :GetWin10SdkDirHelper HKLM\SOFTWARE\Wow6432Node  1>nul 2>&1

V:\>if errorlevel 1 call :GetWin10SdkDirHelper HKCU\SOFTWARE\Wow6432Node  1>nul 2>&1

V:\>if errorlevel 1 call :GetWin10SdkDirHelper HKLM\SOFTWARE  1>nul 2>&1

V:\>if errorlevel 1 call :GetWin10SdkDirHelper HKCU\SOFTWARE  1>nul 2>&1

V:\>if errorlevel 1 exit /B 1

V:\>exit /B 0

V:\>for /F %i in ('dir /ON /B "C:\Program Files (x86)\Windows Kits\10\\include\10.*"') DO (SET WindowsSDKVer=%~i )

V:\>(SET WindowsSDKVer=10.0.10150.0 )

V:\>(SET WindowsSDKVer=10.0.10240.0 )

V:\>(SET WindowsSDKVer=10.0.14393.0 )

V:\>if errorlevel 1 (echo "Didn't find any windows 10 SDK. I'm not sure if things will work, but let's try..." )  else (echo Windows SDK version found as: "10.0.14393.0" )
Windows SDK version found as: "10.0.14393.0"

V:\>IF "win-64" == "win-64" (
set "BITS=64"
 set "CMAKE_PLAT=x64"
)  ELSE (
set "BITS=32"
 set "CMAKE_PLAT=Win32"
)

V:\>IF 2017 GEQ 2019 (
set "CMAKE_GEN=Visual Studio 15 2017"
 set "USE_NEW_CMAKE_GEN_SYNTAX=1"
)  ELSE (
IF "win-64" == "win-64" (set "CMAKE_GEN=Visual Studio 15 2017 Win64" )  else (set "CMAKE_GEN=Visual Studio 15 2017" )
 set "USE_NEW_CMAKE_GEN_SYNTAX=0"
)

V:\>echo "NEWER_VS_WITH_OLDER_VC=0"
"NEWER_VS_WITH_OLDER_VC=0"

V:\>IF "0" == "1" (
set "CMAKE_GEN=Visual Studio 16 2019"
 set "USE_NEW_CMAKE_GEN_SYNTAX=1"
)

V:\>IF "" == "" SET "CMAKE_GENERATOR=Visual Studio 15 2017 Win64"

V:\>IF "0" == "1" (
IF "" == "" SET "CMAKE_GENERATOR_PLATFORM=x64"
 IF "" == "" SET "CMAKE_GENERATOR_TOOLSET=v141"
)

V:\>pushd C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\

C:\Program Files (x86)\Microsoft Visual Studio\2017\Community>CALL "VC\Auxiliary\Build\vcvars64.bat" -vcvars_ver=14.16 10.0.14393.0
[ERROR:vcvarsall.bat] Invalid argument found : -vcvars_ver=14.16
[ERROR:~nx0] Error in script usage. The correct usage is:
Syntax:
    vcvarsall.bat [arch]
  or
    vcvarsall.bat [arch] [version]
  or
    vcvarsall.bat [arch] [platform_type] [version]
where :
    [arch]: x86 | amd64 | x86_amd64 | x86_arm | x86_arm64 | amd64_x86 | amd64_arm | amd64_arm64
    [platform_type]: {empty} | store | uwp
    [version] : full Windows 10 SDK number (e.g. 10.0.10240.0) or "8.1" to use the Windows 8.1 SDK.

The store parameter sets environment variables to support Universal Windows Platform application
development and is an alias for 'uwp'.

For example:
    vcvarsall.bat x86_amd64
    vcvarsall.bat x86_amd64 10.0.10240.0
    vcvarsall.bat x86_arm uwp 10.0.10240.0
    vcvarsall.bat x86_arm onecore 10.0.10240.0
    vcvarsall.bat x64 8.1
    vcvarsall.bat x64 store 8.1

Please make sure either Visual Studio or C++ Build SKU is installed.
samuelgarcia commented 2 years ago

Hi Vincent, this looks a bad installation related to compilers. This yaml file use to work some month ago. I would try to create an empty env and install all list one by one to track the problem.

vncntprvst commented 2 years ago

Hi Samuel, Thanks for the reply. Yes, it's likely an issue with mpi4py, that is conflicting with a previous install (even though I uninstalled it). If anyone gets the same issue, here's a working recipe:

conda create --name si_env python=3.8
conda install pip>=21.0 numpy=1.20.3 joblib tqdm scipy h5py scikit-learn matplotlib pandas networkx
conda install -c conda-forge datalad

pip install PyQt5 ephyviewer neo>=0.10.0 elephant>=0.10.0 probeinterface MEArec>=1.7.1
pip install spikeinterface[full] spikeinterface-gui
pip install tridesclous>=1.6.3 phy==2.0b5
samuelgarcia commented 2 years ago

Thank you for the feedboack. Maybe we will remove this mpi from env in windows it is a bit too complicated.

vncntprvst commented 2 years ago

I was about to say that I'll work on creating a containerized version, so that people can run spikeinterface without having to troubleshoot any installation issue, but then I saw that docker containers are already available. I'll play with those first :)

samuelgarcia commented 2 years ago

Hi Vincent, we already have container for several sorter. See this:

The best way to avoid this installation nightmare is to install only spikeinterface on the host and then run sorter in docker/singularity. So no more need to install then. But this don't work on windows host.

vncntprvst commented 2 years ago

I checked those, and that's a great resource. As you mentioned, and documented in this issue, spikeinterface can't run those on Windows. That's a clear limitation. Not every lab can have a big GPU computing station and dedicate it to spike sorting, and use Ubuntu for that purpose. Some will opt to have it run on Windows, as this is still the OS most people are familiar with. That said, WSL2 is a really good compromise now for that user case. Alternatively, one could build a container with SI + a few spike sorters, and simply mount the data/current directory as a volume in the container.

samuelgarcia commented 2 years ago

We plan soon to make a big container with spikeinetrface + many sorters inside. We would update this lets says every 3 month.

vncntprvst commented 2 years ago

That's great! Looking forward to it. I made a working version in the meantime. See commit here. For Windows users, one can just drag and drop their data folder on the batch file to start Jupyterlab with Spike Interface and a bunch of sorters installed.