search

castorini / rank_llm

RankLLM is a Python toolkit for reproducible information retrieval research using rerankers, with a focus on listwise reranking.
http://rankllm.ai
Apache License 2.0
365 stars 43 forks source link

Error in Installation using pip in AWS Workspace #129

Open nikithakriz opened 3 months ago

nikithakriz commented 3 months ago

I tried to install rank_llm using pip install rank_llm in my AWS workspace(windows), but it's throwing below error, tried after installing 'wheel' and 'nmslib ' still same error. 


Building wheels for collected packages: nmslib
  Building wheel for nmslib (setup.py): started
  Building wheel for nmslib (setup.py): finished with status 'error'
  Running setup.py clean for nmslib
Failed to build nmslib
Installing collected packages: nmslib, lightgbm, fschat, Cython, python-dotenv, pyserini, ftfy, faiss-cpu, dacite, rank-llm
    Running setup.py install for nmslib: started
    Running setup.py install for nmslib: finished with status 'error'
  ERROR: Command errored out with exit status 1:
   command: 'D:\Users\\Scripts\python.exe' -u -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'D:\\Users\\\\AppData\\Local\\Temp\\1\\pip-install-947nxa0d\\nmslib_92d8237ca22a4706b8abb6aafa30140e\\setup.py'"'"'; __file__='"'"'D:\\Users\\\\AppData\\Local\\Temp\\1\\pip-install-947nxa0d\\nmslib_92d8237ca22a4706b8abb6aafa30140e\\setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' bdist_wheel -d 'D:\Users\AppData\Local\Temp\1\pip-wheel-iu4ldw8x'
       cwd: [D:\Users\AppData\Local\Temp\1\pip-install-947nxa0d\nmslib_92d8237ca22a4706b8abb6aafa30140e\](file:///D:/Users/AppData/Local/Temp/1/pip-install-947nxa0d/nmslib_92d8237ca22a4706b8abb6aafa30140e/)
  Complete output (514 lines):
  Dependence list: ['pybind11>=2.2.3', 'psutil', 'numpy>=1.10.0']
  [D:\Users\lib\site-packages\setuptools\dist.py:697](file:///D:/Users/lib/site-packages/setuptools/dist.py:697): UserWarning: Usage of dash-separated 'description-file' will not be supported in future versions. Please use the underscore name 'description_file' instead
    warnings.warn(
  running bdist_wheel
  running build
  running build_ext
  Extra compilation arguments: ['/EHsc', '/openmp', '/O2', '/DVERSION_INFO=\\"2.0.6\\"']
  building 'nmslib' extension
  creating build
  creating build\temp.win-amd64-3.10
  creating build\temp.win-amd64-3.10\Release
  creating build\temp.win-amd64-3.10\Release\similarity_search
  creating build\temp.win-amd64-3.10\Release\similarity_search\src
  creating build\temp.win-amd64-3.10\Release\similarity_search\src\method
  creating build\temp.win-amd64-3.10\Release\similarity_search\src\space
  creating build\temp.win-amd64-3.10\Release\tensorflow
  [C:\Program](file:///C:/Program) Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.39.33519\bin\HostX86\x64\cl.exe /c /nologo /Ox /W3 /GL /DNDEBUG /MD -I.\similarity_search\include -Itensorflow -ID:\Users\\lib\site-packages\pybind11\include -ID:\Users\lib\site-packages\pybind11\include -ID:\Users\lib\site-packages\pybind11\include -ID:\Users\lib\site-packages\pybind11\include -ID:\Users\lib\site-packages\numpy\core\include -ID:\Users\include -IC:\Program Files\Python310\include -IC:\Program Files\Python310\Include -IC:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.39.33519\include -IC:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.39.33519\ATLMFC\include -IC:\Program Files\Microsoft Visual Studio\2022\Community\VC\Auxiliary\VS\include -IC:\Program Files (x86)\Windows Kits\10\include\10.0.22621.0\ucrt -IC:\Program Files (x86)\Windows Kits\10\\include\10.0.22621.0\\um -IC:\Program Files (x86)\Windows Kits\10\\include\10.0.22621.0\\shared -IC:\Program Files (x86)\Windows Kits\10\\include\10.0.22621.0\\winrt -IC:\Program Files (x86)\Windows Kits\10\\include\10.0.22621.0\\cppwinrt /EHsc /Tp.\similarity_search\src\distcomp_bregman.cc /Fobuild\temp.win-amd64-3.10\Release\.\similarity_search\src\distcomp_bregman.obj /EHsc /openmp /O2 /DVERSION_INFO=\"2.0.6\"
  distcomp_bregman.cc
  [.\similarity_search\include\distcomp.h](https://file+.vscode-resource.vscode-cdn.net/d%3A/Users/similarity_search/include/distcomp.h)(260): warning C4244: 'initializing': conversion from 'size_t' to 'float', possible loss of data
  [C:\Program](file:///C:/Program) Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.39.33519\bin\HostX86\x64\cl.exe /c /nologo /Ox /W3 /GL /DNDEBUG /MD -I.\similarity_search\include -Itensorflow -ID: -IC:\Program Files\Python310\include -IC:\Program Files\Python310\Include -IC:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.39.33519\include -IC:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.39.33519\ATLMFC\include -IC:\Program Files\Microsoft Visual Studio\2022\Community\VC\Auxiliary\VS\include -IC:\Program Files (x86)\Windows Kits\10\include\10.0.22621.0\ucrt -IC:\Program Files (x86)\Windows Kits\10\\include\10.0.22621.0\\um -IC:\Program Files (x86)\Windows Kits\10\\include\10.0.22621.0\\shared -IC:\Program Files (x86)\Windows Kits\10\\include\10.0.22621.0\\winrt -IC:\Program Files (x86)\Windows Kits\10\\include\10.0.22621.0\\cppwinrt /EHsc /Tp.\similarity_search\src\distcomp_diverg.cc /Fobuild\temp.win-amd64-3.10\Release\.\similarity_search\src\distcomp_diverg.obj /EHsc /openmp /O2 /DVERSION_INFO=\"2.0.6\"
  distcomp_diverg.cc
  [.\similarity_search\include\distcomp.h](https://file+.vscode-resource.vscode-cdn.net/d%3A/Users/similarity_search/include/distcomp.h)(260): warning C4244: 'initializing': conversion from 'size_t' to 'float', possible loss of data
  [.\similarity_search\src\distcomp_diverg.cc](https://file+.vscode-resource.vscode-cdn.net/d%3A/Users//similarity_search/src/distcomp_diverg.cc)(61): warning C4244: '+=': conversion from 'double' to 'T', possible loss of data
          with
          [
              T=float
          ]
  [.\similarity_search\src\distcomp_diverg.cc](https://file+.vscode-resource.vscode-cdn.net/d%3A/Users/similarity_search/src/distcomp_diverg.cc)(61): note: the template instantiation context (the oldest one first) is
  [.\similarity_search\src\distcomp_diverg.cc](https://file+.vscode-resource.vscode-cdn.net/d%3A/Users/similarity_search/src/distcomp_diverg.cc)(66): note: see reference to function template instantiation 'float similarity::alphaBetaDivergenceSlowProxy<float>(const float *,const float *,const int,float,float)' being compiled
  [.\similarity_search\src\distcomp_diverg.cc](https://file+.vscode-resource.vscode-cdn.net/d%3A/Users//similarity_search/src/distcomp_diverg.cc)(73): warning C4244: '+=': conversion from 'double' to 'T', possible loss of data
          with
          [
              T=float
          ]
  [.\similarity_search\src\distcomp_diverg.cc](https://file+.vscode-resource.vscode-cdn.net/d%3A/Users//similarity_search/src/distcomp_diverg.cc)(73): note: the template instantiation context (the oldest one first) is
  [.\similarity_search\src\distcomp_diverg.cc](https://file+.vscode-resource.vscode-cdn.net/d%3A/Users//similarity_search/src/distcomp_diverg.cc)(79): note: see reference to function template instantiation 'float similarity::alphaBetaDivergenceFastProxy<float>(const float *,const float *,const int,float,float)' being compiled
  [.\similarity_search\src\distcomp_diverg.cc](https://file+.vscode-resource.vscode-cdn.net/d%3A/Users/similarity_search/src/distcomp_diverg.cc)(84): warning C4305: 'initializing': truncation from 'double' to 'T'

....

so on

sahel-sh commented 3 months ago

Unfortunately, I don't have access to a windows machine to try, but I just confirmed successful installation on Linux: conda create -n rankllm python=3.10 conda activate rankllm pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu118 pip install rank-llm Your issue seems to be related to building the mnslib wheel on windows, I would recommend looking into fixing that.

andrewxucs commented 1 month ago

Hi, I encountered a similar issue when running pip install -r requirements.txt so I manually installed the NMSLIB library with conda install -c conda-forge nmslib, then I runned pip install -r requirements.txt again, which successfully resolved the error on my side. Please let us know if other issues arise!