search

shmsw25 / FActScore

A package to evaluate factuality of long-form generation. Original implementation of our EMNLP 2023 paper "FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation"
https://arxiv.org/abs/2305.14251
MIT License
238 stars 32 forks source link

Issue running get_score() on M1 Mac due to Torch build #33

Closed jcblsn closed 3 months ago

jcblsn commented 7 months ago

I'm having trouble running get_score() on an M1 Macbook Pro.

I tried toggling instances in the source code from "cuda" to "mps" for the functions loaded locally but haven't had any luck with the following error: 

    [222](file:///.../lib/python3.10/site-packages/torch/cuda/__init__.py?line=221) if _cudart is None:
    [223](file:///...env/lib/python3.10/site-packages/torch/cuda/__init__.py?line=222)     raise AssertionError(
    [224](file:///...env/lib/python3.10/site-packages/torch/cuda/__init__.py?line=223)         "libcudart functions unavailable. It looks like you have a broken build?")

AssertionError: Torch not compiled with CUDA enabled

Do you have any suggestions?

I have verified MPS support. My environment follows:

# Name                    Version                   Build  Channel
aiohttp                   3.9.1                    pypi_0    pypi
aiosignal                 1.3.1                    pypi_0    pypi
annotated-types           0.6.0                    pypi_0    pypi
anyio                     3.7.1                    pypi_0    pypi
appnope                   0.1.3              pyhd8ed1ab_0    conda-forge
argon2-cffi               23.1.0             pyhd8ed1ab_0    conda-forge
argon2-cffi-bindings      21.2.0          py310h1a28f6b_0  
arrow                     1.3.0              pyhd8ed1ab_0    conda-forge
asttokens                 2.4.1              pyhd8ed1ab_0    conda-forge
async-lru                 2.0.4           py310hca03da5_0  
async-timeout             4.0.3                    pypi_0    pypi
attrs                     23.1.0          py310hca03da5_0  
babel                     2.13.1             pyhd8ed1ab_0    conda-forge
beautifulsoup4            4.12.2          py310hca03da5_0  
bleach                    6.1.0              pyhd8ed1ab_0    conda-forge
blis                      0.7.11                   pypi_0    pypi
brotli-python             1.0.9           py310hc377ac9_7  
bzip2                     1.0.8                h620ffc9_4  
ca-certificates           2023.11.17           hf0a4a13_0    conda-forge
cached-property           1.5.2                      py_0  
catalogue                 2.0.10                   pypi_0    pypi
certifi                   2023.11.17      py310hca03da5_0  
cffi                      1.16.0          py310h80987f9_0  
charset-normalizer        3.3.2              pyhd8ed1ab_0    conda-forge
click                     8.1.7                    pypi_0    pypi
cloudpathlib              0.16.0                   pypi_0    pypi
comm                      0.1.4              pyhd8ed1ab_0    conda-forge
confection                0.1.4                    pypi_0    pypi
cryptography              41.0.3          py310hd4332d6_0  
cymem                     2.0.8                    pypi_0    pypi
cyrus-sasl                2.1.28               h9131b1a_1  
dataclasses-json          0.6.3                    pypi_0    pypi
debugpy                   1.6.7           py310h313beb8_0  
decorator                 5.1.1              pyhd3eb1b0_0  
defusedxml                0.7.1              pyhd3eb1b0_0  
en-core-web-sm            3.7.1                    pypi_0    pypi
entrypoints               0.4             py310hca03da5_0  
exceptiongroup            1.2.0              pyhd8ed1ab_0    conda-forge
executing                 2.0.1              pyhd8ed1ab_0    conda-forge
factscore                 0.2.0                    pypi_0    pypi
fqdn                      1.5.1              pyhd8ed1ab_0    conda-forge
frozenlist                1.4.0                    pypi_0    pypi
fsspec                    2023.10.0                pypi_0    pypi
gettext                   0.21.1               h0186832_0    conda-forge
glib                      2.78.1               h9e231a4_1    conda-forge
glib-tools                2.78.1               h9e231a4_1    conda-forge
gst-plugins-base          1.14.1               h313beb8_1  
gstreamer                 1.14.1               h80987f9_1  
huggingface-hub           0.19.4                   pypi_0    pypi
icu                       73.2                 hc8870d7_0    conda-forge
idna                      3.6                pyhd8ed1ab_0    conda-forge
importlib-metadata        6.9.0              pyha770c72_0    conda-forge
importlib_metadata        6.9.0                hd8ed1ab_0    conda-forge
importlib_resources       6.1.1              pyhd8ed1ab_0    conda-forge
ipykernel                 6.26.0             pyh3cd1d5f_0    conda-forge
ipython                   8.18.1             pyh31011fe_1    conda-forge
ipywidgets                8.1.1              pyhd8ed1ab_0    conda-forge
isoduration               20.11.0            pyhd8ed1ab_0    conda-forge
jedi                      0.19.1             pyhd8ed1ab_0    conda-forge
jinja2                    3.1.2           py310hca03da5_0  
joblib                    1.3.2                    pypi_0    pypi
jpeg                      9e                   h80987f9_1  
json5                     0.9.14             pyhd8ed1ab_0    conda-forge
jsonpatch                 1.33                     pypi_0    pypi
jsonpointer               2.1                pyhd3eb1b0_0  
jsonschema                4.20.0             pyhd8ed1ab_0    conda-forge
jsonschema-specifications 2023.11.2          pyhd8ed1ab_0    conda-forge
jsonschema-with-format-nongpl 4.20.0             pyhd8ed1ab_0    conda-forge
jupyter                   1.0.0           py310hca03da5_8  
jupyter-lsp               2.2.1              pyhd8ed1ab_0    conda-forge
jupyter_client            8.6.0           py310hca03da5_0  
jupyter_console           6.6.3           py310hca03da5_0  
jupyter_core              5.5.0           py310hca03da5_0  
jupyter_events            0.9.0              pyhd8ed1ab_0    conda-forge
jupyter_server            2.11.1             pyhd8ed1ab_0    conda-forge
jupyter_server_terminals  0.4.4           py310hca03da5_1  
jupyterlab                4.0.9              pyhd8ed1ab_0    conda-forge
jupyterlab_pygments       0.3.0              pyhd8ed1ab_0    conda-forge
jupyterlab_server         2.25.2             pyhd8ed1ab_0    conda-forge
jupyterlab_widgets        3.0.9           py310hca03da5_0  
krb5                      1.20.1               hf3e1bf2_1  
langchain                 0.0.344                  pypi_0    pypi
langchain-core            0.0.8                    pypi_0    pypi
langcodes                 3.3.0                    pypi_0    pypi
langsmith                 0.0.68                   pypi_0    pypi
libclang                  14.0.6          default_h1b80db6_1  
libclang13                14.0.6          default_h24352ff_1  
libcxx                    16.0.6               h4653b0c_0    conda-forge
libedit                   3.1.20221030         h80987f9_0  
libffi                    3.4.4                hca03da5_0  
libglib                   2.78.1               hb438215_1    conda-forge
libiconv                  1.17                 he4db4b2_0    conda-forge
libllvm14                 14.0.6               h7ec7a93_3  
libpng                    1.6.39               h80987f9_0  
libpq                     12.15                h02f6b3c_1  
libsodium                 1.0.18               h1a28f6b_0  
libsqlite                 3.44.2               h091b4b1_0    conda-forge
libzlib                   1.2.13               h53f4e23_5    conda-forge
lz4-c                     1.9.4                h313beb8_0  
markupsafe                2.1.1           py310h1a28f6b_0  
marshmallow               3.20.1                   pypi_0    pypi
matplotlib-inline         0.1.6           py310hca03da5_0  
mistune                   3.0.2              pyhd8ed1ab_0    conda-forge
multidict                 6.0.4                    pypi_0    pypi
murmurhash                1.0.10                   pypi_0    pypi
mypy-extensions           1.0.0                    pypi_0    pypi
mysql                     5.7.24               ha71a6ea_2  
nbclient                  0.8.0           py310hca03da5_0  
nbconvert                 7.11.0             pyhd8ed1ab_0    conda-forge
nbconvert-core            7.11.0             pyhd8ed1ab_0    conda-forge
nbconvert-pandoc          7.11.0             pyhd8ed1ab_0    conda-forge
nbformat                  5.9.2           py310hca03da5_0  
ncurses                   6.4                  h313beb8_0  
nest-asyncio              1.5.8              pyhd8ed1ab_0    conda-forge
nltk                      3.8.1                    pypi_0    pypi
notebook                  7.0.6           py310hca03da5_0  
notebook-shim             0.2.3           py310hca03da5_0  
numpy                     1.26.2                   pypi_0    pypi
openai                    0.27.10                  pypi_0    pypi
openssl                   3.2.0                h0d3ecfb_1    conda-forge
overrides                 7.4.0           py310hca03da5_0  
packaging                 23.2               pyhd8ed1ab_0    conda-forge
pandoc                    3.1.3                hce30654_0    conda-forge
pandocfilters             1.5.0              pyhd3eb1b0_0  
parso                     0.8.3              pyhd3eb1b0_0  
pcre2                     10.42                hb066dcc_0  
pexpect                   4.8.0              pyhd3eb1b0_3  
pickleshare               0.7.5           pyhd3eb1b0_1003  
pillow                    10.1.0                   pypi_0    pypi
pip                       23.3.1          py310hca03da5_0  
pkgutil-resolve-name      1.3.10             pyhd8ed1ab_1    conda-forge
platformdirs              4.0.0              pyhd8ed1ab_0    conda-forge
ply                       3.11            py310hca03da5_0  
preshed                   3.0.9                    pypi_0    pypi
prometheus_client         0.19.0             pyhd8ed1ab_0    conda-forge
prompt-toolkit            3.0.41             pyha770c72_0    conda-forge
prompt_toolkit            3.0.41               hd8ed1ab_0    conda-forge
psutil                    5.9.0           py310h1a28f6b_0  
ptyprocess                0.7.0              pyhd3eb1b0_2  
pure_eval                 0.2.2              pyhd3eb1b0_0  
pycparser                 2.21               pyhd3eb1b0_0  
pydantic                  2.5.2                    pypi_0    pypi
pydantic-core             2.14.5                   pypi_0    pypi
pygments                  2.17.2             pyhd8ed1ab_0    conda-forge
pyopenssl                 23.2.0          py310hca03da5_0  
pyqt                      5.15.10         py310h313beb8_0  
pyqt5-sip                 12.13.0         py310h80987f9_0  
pysocks                   1.7.1           py310hca03da5_0  
pysqlite-binary           0.5.1.3380300            pypi_0    pypi
python                    3.10.13              hb885b13_0  
python-dateutil           2.8.2              pyhd3eb1b0_0  
python-dotenv             1.0.0              pyhd8ed1ab_1    conda-forge
python-fastjsonschema     2.19.0             pyhd8ed1ab_0    conda-forge
python-json-logger        2.0.7           py310hca03da5_0  
pytorch                   1.13.1                 py3.10_0    pytorch
pytz                      2023.3.post1    py310hca03da5_0  
pyyaml                    6.0.1           py310h80987f9_0  
pyzmq                     25.1.0          py310h313beb8_0  
qt-main                   5.15.2              h0917680_10  
qtconsole                 5.5.1              pyhd8ed1ab_0    conda-forge
qtconsole-base            5.5.1              pyha770c72_0    conda-forge
qtpy                      2.4.1           py310hca03da5_0  
rank-bm25                 0.2.2                    pypi_0    pypi
readline                  8.2                  h1a28f6b_0  
referencing               0.31.1             pyhd8ed1ab_0    conda-forge
regex                     2023.10.3                pypi_0    pypi
requests                  2.31.0          py310hca03da5_0  
rfc3339-validator         0.1.4           py310hca03da5_0  
rfc3986-validator         0.1.1           py310hca03da5_0  
rpds-py                   0.10.6          py310hf0e4da2_0  
safetensors               0.4.1                    pypi_0    pypi
scikit-learn              1.3.2                    pypi_0    pypi
scipy                     1.11.4                   pypi_0    pypi
send2trash                1.8.2           py310hca03da5_0  
sentence-transformers     2.2.2                    pypi_0    pypi
sentencepiece             0.1.99                   pypi_0    pypi
setuptools                68.2.2             pyhd8ed1ab_0    conda-forge
sip                       6.7.12          py310h313beb8_0  
six                       1.16.0             pyhd3eb1b0_1  
smart-open                6.4.0                    pypi_0    pypi
sniffio                   1.3.0              pyhd8ed1ab_0    conda-forge
soupsieve                 2.5             py310hca03da5_0  
spacy                     3.7.2                    pypi_0    pypi
spacy-legacy              3.0.12                   pypi_0    pypi
spacy-loggers             1.0.5                    pypi_0    pypi
sqlalchemy                2.0.23                   pypi_0    pypi
sqlite                    3.44.2               hf2abe2d_0    conda-forge
srsly                     2.4.8                    pypi_0    pypi
stack_data                0.6.2              pyhd8ed1ab_0    conda-forge
tenacity                  8.2.3                    pypi_0    pypi
terminado                 0.18.0             pyh31c8845_0    conda-forge
thinc                     8.2.1                    pypi_0    pypi
threadpoolctl             3.2.0                    pypi_0    pypi
tinycss2                  1.2.1           py310hca03da5_0  
tk                        8.6.13               h5083fa2_1    conda-forge
tokenizers                0.15.0                   pypi_0    pypi
tomli                     2.0.1           py310hca03da5_0  
torchvision               0.14.1                   pypi_0    pypi
tornado                   6.3.3           py310h80987f9_0  
tqdm                      4.66.1                   pypi_0    pypi
traitlets                 5.14.0             pyhd8ed1ab_0    conda-forge
transformers              4.35.2                   pypi_0    pypi
typer                     0.9.0                    pypi_0    pypi
types-python-dateutil     2.8.19.14          pyhd8ed1ab_0    conda-forge
typing-extensions         4.8.0                hd8ed1ab_0    conda-forge
typing-inspect            0.9.0                    pypi_0    pypi
typing_extensions         4.8.0              pyha770c72_0    conda-forge
tzdata                    2023c                h04d1e81_0  
uri-template              1.3.0              pyhd8ed1ab_0    conda-forge
urllib3                   1.26.18         py310hca03da5_0  
wasabi                    1.1.2                    pypi_0    pypi
wcwidth                   0.2.12             pyhd8ed1ab_0    conda-forge
weasel                    0.3.4                    pypi_0    pypi
webcolors                 1.13               pyhd8ed1ab_0    conda-forge
webencodings              0.5.1           py310hca03da5_1  
websocket-client          1.6.4              pyhd8ed1ab_0    conda-forge
wheel                     0.42.0             pyhd8ed1ab_0    conda-forge
widgetsnbextension        4.0.9              pyhd8ed1ab_0    conda-forge
xz                        5.4.2                h80987f9_0  
yaml                      0.2.5                h1a28f6b_0  
yarl                      1.9.3                    pypi_0    pypi
zeromq                    4.3.5                h965bd2d_0    conda-forge
zipp                      3.17.0             pyhd8ed1ab_0    conda-forge
zlib                      1.2.13               h53f4e23_5    conda-forge
zstd                      1.5.5                hd90d995_0
martiansideofthemoon commented 3 months ago

Hi @jcblsn, did you have any luck fixing this? Sorry we didn't do any testing on M1 Macbook Pro. I suspect this maybe more of a torch issue and it maybe best to ask on the PyTorch Github repository, so I'll close this for now.