search

RUC-NLPIR / FlashRAG

⚡FlashRAG: A Python Toolkit for Efficient RAG Research
https://arxiv.org/abs/2405.13576
MIT License
1.33k stars 108 forks source link

Torch not compiled with CUDA enabled #49

Closed codecodebear closed 4 months ago

codecodebear commented 4 months ago

Hello, got Torch not compiled with CUDA enabled error while running demo_en.py, I am not familiar with CUDA's relationship with torch, so I am asking any approach to fix it. Btw, I am running on M2 Max chip MacOS 14.5, don't think I have Nvidia GPU in my mac. Do I need to have Nvidia GPUs to run code? If not, what torch version/configurations should I use when installing (or other ways to fix this)? Below is my error log:

Setting num_proc from 4 back to 1 for the train split to disable multiprocessing as it only contains one shard.
Generating train split: 15000 examples [00:00, 1158391.52 examples/s]
config.json: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████| 650/650 [00:00<00:00, 2.73MB/s]
model.safetensors: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████| 438M/438M [00:31<00:00, 13.7MB/s]

File "/opt/miniconda3/envs/flashrag-env/lib/python3.11/site-packages/torch/nn/modules/module.py", line 915, in <lambda>
    return self._apply(lambda t: t.cuda(device))
                                 ^^^^^^^^^^^^^^
  File "/opt/miniconda3/envs/flashrag-env/lib/python3.11/site-packages/torch/cuda/__init__.py", line 284, in _lazy_init
    raise AssertionError("Torch not compiled with CUDA enabled")
AssertionError: Torch not compiled with CUDA enabled

I installed packages in a conda env, below are all my packages in conda.

# packages in environment at /opt/miniconda3/envs/flashrag-env:
#
# Name                    Version                   Build  Channel
accelerate                0.32.1                   pypi_0    pypi
aiohttp                   3.9.5                    pypi_0    pypi
aiolimiter                1.1.0                    pypi_0    pypi
aiosignal                 1.3.1                    pypi_0    pypi
altair                    5.3.0                    pypi_0    pypi
annotated-types           0.7.0                    pypi_0    pypi
anyio                     4.4.0                    pypi_0    pypi
attrs                     23.2.0                   pypi_0    pypi
base58                    2.1.1                    pypi_0    pypi
blas                      1.0                    openblas  
blinker                   1.8.2                    pypi_0    pypi
blis                      0.7.11                   pypi_0    pypi
boto3                     1.34.140                 pypi_0    pypi
botocore                  1.34.140                 pypi_0    pypi
bzip2                     1.0.8                h93a5062_5    conda-forge
ca-certificates           2024.7.4             hf0a4a13_0    conda-forge
cachetools                5.3.3                    pypi_0    pypi
catalogue                 2.0.10                   pypi_0    pypi
certifi                   2024.7.4                 pypi_0    pypi
charset-normalizer        3.3.2                    pypi_0    pypi
click                     8.1.7                    pypi_0    pypi
cloudpathlib              0.18.1                   pypi_0    pypi
cohere                    5.5.8                    pypi_0    pypi
coloredlogs               15.0.1                   pypi_0    pypi
confection                0.1.5                    pypi_0    pypi
cymem                     2.0.8                    pypi_0    pypi
cython                    3.0.10                   pypi_0    pypi
datasets                  2.20.0                   pypi_0    pypi
dill                      0.3.8                    pypi_0    pypi
distro                    1.9.0                    pypi_0    pypi
dnspython                 2.6.1                    pypi_0    pypi
email-validator           2.2.0                    pypi_0    pypi
faiss-cpu                 1.8.0           py3.11_hbe593ad_0_cpu    pytorch
fastapi                   0.111.0                  pypi_0    pypi
fastapi-cli               0.0.4                    pypi_0    pypi
fastavro                  1.9.4                    pypi_0    pypi
filelock                  3.15.4                   pypi_0    pypi
flashrag                  0.0.1                    pypi_0    pypi
flatbuffers               24.3.25                  pypi_0    pypi
frozenlist                1.4.1                    pypi_0    pypi
fschat                    0.2.36                   pypi_0    pypi
fsspec                    2024.5.0                 pypi_0    pypi
gitdb                     4.0.11                   pypi_0    pypi
gitpython                 3.1.43                   pypi_0    pypi
h11                       0.14.0                   pypi_0    pypi
httpcore                  1.0.5                    pypi_0    pypi
httptools                 0.6.1                    pypi_0    pypi
httpx                     0.27.0                   pypi_0    pypi
httpx-sse                 0.4.0                    pypi_0    pypi
huggingface-hub           0.23.4                   pypi_0    pypi
humanfriendly             10.0                     pypi_0    pypi
idna                      3.7                      pypi_0    pypi
jinja2                    3.1.4                    pypi_0    pypi
jmespath                  1.0.1                    pypi_0    pypi
joblib                    1.4.2                    pypi_0    pypi
jsonschema                4.22.0                   pypi_0    pypi
jsonschema-specifications 2023.12.1                pypi_0    pypi
langcodes                 3.4.0                    pypi_0    pypi
language-data             1.2.0                    pypi_0    pypi
libcxx                    17.0.6               h0812c0d_3    conda-forge
libexpat                  2.6.2                hebf3989_0    conda-forge
libfaiss                  1.8.0            hcb8d3e5_0_cpu    pytorch
libffi                    3.4.2                h3422bc3_5    conda-forge
libgfortran               5.0.0           13_2_0_hd922786_3    conda-forge
libgfortran5              13.2.0               hf226fd6_3    conda-forge
libopenblas               0.3.21               h269037a_0  
libsqlite                 3.46.0               hfb93653_0    conda-forge
libzlib                   1.3.1                hfb2fe0b_1    conda-forge
lightgbm                  4.4.0                    pypi_0    pypi
llvm-openmp               18.1.8               hde57baf_0    conda-forge
marisa-trie               1.2.0                    pypi_0    pypi
markdown-it-py            3.0.0                    pypi_0    pypi
markdown2                 2.4.13                   pypi_0    pypi
markupsafe                2.1.5                    pypi_0    pypi
mdurl                     0.1.2                    pypi_0    pypi
mpmath                    1.3.0                    pypi_0    pypi
multidict                 6.0.5                    pypi_0    pypi
multiprocess              0.70.16                  pypi_0    pypi
murmurhash                1.0.10                   pypi_0    pypi
ncurses                   6.5                  hb89a1cb_0    conda-forge
networkx                  3.3                      pypi_0    pypi
nh3                       0.2.17                   pypi_0    pypi
nltk                      3.8.1                    pypi_0    pypi
nmslib                    2.1.1           py311h30ceab6_0  
nomkl                     3.0                           0  
numpy                     1.26.4          py311he598dae_0  
numpy-base                1.26.4          py311hfbfe69c_0  
onnxruntime               1.18.1                   pypi_0    pypi
openai                    1.35.10                  pypi_0    pypi
openblas                  0.3.21               hca03da5_0  
openblas-devel            0.3.21               hca03da5_0  
openssl                   3.3.1                hfb2fe0b_1    conda-forge
orjson                    3.10.6                   pypi_0    pypi
packaging                 24.1            py311hca03da5_0  
pandas                    2.2.2                    pypi_0    pypi
parameterized             0.9.0                    pypi_0    pypi
peft                      0.11.1                   pypi_0    pypi
pillow                    10.4.0                   pypi_0    pypi
pip                       24.0               pyhd8ed1ab_0    conda-forge
preshed                   3.0.9                    pypi_0    pypi
prompt-toolkit            3.0.47                   pypi_0    pypi
protobuf                  5.27.2                   pypi_0    pypi
psutil                    6.0.0           py311hd3f4193_0    conda-forge
pyarrow                   16.1.0                   pypi_0    pypi
pyarrow-hotfix            0.6                      pypi_0    pypi
pybind11                  2.13.1          py311h6bde47b_0    conda-forge
pybind11-global           2.13.1          py311h6bde47b_0    conda-forge
pydantic                  2.8.2                    pypi_0    pypi
pydantic-core             2.20.1                   pypi_0    pypi
pydeck                    0.9.1                    pypi_0    pypi
pygments                  2.18.0                   pypi_0    pypi
pyjnius                   1.6.1                    pypi_0    pypi
pyserini                  0.36.0                   pypi_0    pypi
python                    3.11.9          h932a869_0_cpython    conda-forge
python-dateutil           2.9.0.post0              pypi_0    pypi
python-dotenv             1.0.1                    pypi_0    pypi
python-multipart          0.0.9                    pypi_0    pypi
python_abi                3.11                    4_cp311    conda-forge
pytz                      2024.1                   pypi_0    pypi
pyyaml                    6.0.1                    pypi_0    pypi
rank-bm25                 0.2.2                    pypi_0    pypi
readline                  8.2                  h92ec313_1    conda-forge
referencing               0.35.1                   pypi_0    pypi
regex                     2024.5.15                pypi_0    pypi
requests                  2.32.3                   pypi_0    pypi
rich                      13.7.1                   pypi_0    pypi
rouge                     1.0.1                    pypi_0    pypi
rpds-py                   0.18.1                   pypi_0    pypi
s3transfer                0.10.2                   pypi_0    pypi
safetensors               0.4.3                    pypi_0    pypi
scikit-learn              1.5.1                    pypi_0    pypi
scipy                     1.14.0                   pypi_0    pypi
sentence-transformers     3.0.1                    pypi_0    pypi
sentencepiece             0.2.0                    pypi_0    pypi
setuptools                70.1.1             pyhd8ed1ab_0    conda-forge
shellingham               1.5.4                    pypi_0    pypi
shortuuid                 1.0.13                   pypi_0    pypi
six                       1.16.0                   pypi_0    pypi
smart-open                7.0.4                    pypi_0    pypi
smmap                     5.0.1                    pypi_0    pypi
sniffio                   1.3.1                    pypi_0    pypi
spacy                     3.7.5                    pypi_0    pypi
spacy-legacy              3.0.12                   pypi_0    pypi
spacy-loggers             1.0.5                    pypi_0    pypi
srsly                     2.4.8                    pypi_0    pypi
starlette                 0.37.2                   pypi_0    pypi
streamlit                 1.36.0                   pypi_0    pypi
svgwrite                  1.4.3                    pypi_0    pypi
sympy                     1.12.1                   pypi_0    pypi
tenacity                  8.5.0                    pypi_0    pypi
thinc                     8.2.5                    pypi_0    pypi
threadpoolctl             3.5.0                    pypi_0    pypi
tiktoken                  0.7.0                    pypi_0    pypi
tk                        8.6.13               h5083fa2_1    conda-forge
tokenizers                0.19.1                   pypi_0    pypi
toml                      0.10.2                   pypi_0    pypi
toolz                     0.12.1                   pypi_0    pypi
torch                     2.3.1                    pypi_0    pypi
tornado                   6.4.1                    pypi_0    pypi
tqdm                      4.66.4                   pypi_0    pypi
transformers              4.42.3                   pypi_0    pypi
typer                     0.12.3                   pypi_0    pypi
types-requests            2.32.0.20240622          pypi_0    pypi
typing-extensions         4.12.2                   pypi_0    pypi
tzdata                    2024.1                   pypi_0    pypi
ujson                     5.10.0                   pypi_0    pypi
urllib3                   2.2.2                    pypi_0    pypi
uvicorn                   0.30.1                   pypi_0    pypi
uvloop                    0.19.0                   pypi_0    pypi
voyageai                  0.2.3                    pypi_0    pypi
wasabi                    1.1.3                    pypi_0    pypi
watchfiles                0.22.0                   pypi_0    pypi
wavedrom                  2.0.3.post3              pypi_0    pypi
wcwidth                   0.2.13                   pypi_0    pypi
weasel                    0.4.1                    pypi_0    pypi
websockets                12.0                     pypi_0    pypi
wheel                     0.43.0             pyhd8ed1ab_1    conda-forge
wrapt                     1.16.0                   pypi_0    pypi
xxhash                    3.4.1                    pypi_0    pypi
xz                        5.2.6                h57fd34a_0    conda-forge
yarl                      1.9.4                    pypi_0    pypi

==== update ==== more traceback below, If I understand correctly, there is a code directly calling cuda, model.cuda() in flashrag/retriever/utils.py which may be related to my issue?

File "/Users/smalldog/Desktop/rag/FlashRAG/flashrag-env/lib/python3.11/site-packages/streamlit/runtime/scriptrunner/script_runner.py", line 589, in _run_script
    exec(code, module.__dict__)
File "/Users/smalldog/Desktop/rag/FlashRAG/examples/quick_start/demo_en.py", line 44, in <module>
    retriever = load_retriever(config)
                ^^^^^^^^^^^^^^^^^^^^^^
File "/Users/smalldog/Desktop/rag/FlashRAG/flashrag-env/lib/python3.11/site-packages/streamlit/runtime/caching/cache_utils.py", line 168, in wrapper
    return cached_func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/smalldog/Desktop/rag/FlashRAG/flashrag-env/lib/python3.11/site-packages/streamlit/runtime/caching/cache_utils.py", line 197, in __call__
    return self._get_or_create_cached_value(args, kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/smalldog/Desktop/rag/FlashRAG/flashrag-env/lib/python3.11/site-packages/streamlit/runtime/caching/cache_utils.py", line 224, in _get_or_create_cached_value
    return self._handle_cache_miss(cache, value_key, func_args, func_kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/smalldog/Desktop/rag/FlashRAG/flashrag-env/lib/python3.11/site-packages/streamlit/runtime/caching/cache_utils.py", line 280, in _handle_cache_miss
    computed_value = self._info.func(*func_args, **func_kwargs)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/smalldog/Desktop/rag/FlashRAG/examples/quick_start/demo_en.py", line 16, in load_retriever
    return get_retriever(_config)
           ^^^^^^^^^^^^^^^^^^^^^^
File "/Users/smalldog/Desktop/rag/FlashRAG/flashrag/utils/utils.py", line 76, in get_retriever
    return getattr(
           ^^^^^^^^
File "/Users/smalldog/Desktop/rag/FlashRAG/flashrag/retriever/retriever.py", line 247, in __init__
    self.encoder = Encoder(
                   ^^^^^^^^
File "/Users/smalldog/Desktop/rag/FlashRAG/flashrag/retriever/encoder.py", line 49, in __init__
    self.model, self.tokenizer = load_model(model_path=model_path,
                                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/smalldog/Desktop/rag/FlashRAG/flashrag/retriever/utils.py", line 13, in load_model
    model.cuda()

Thank you!

ignorejjj commented 4 months ago

Some parts of the code require CUDA, so you need to use NVIDIA GPU.

There are currently no plans to support FlashRAG on Mac. We welcome you to make some modifications to the code and provide a pull request.