Closed lilyminium closed 5 months ago
See https://github.com/conda-forge/snappy-feedstock/issues/35, though I am not sure if it will be faster to mark the new snappy package as broken, or if the pyarrow feedstock needs to pin the version of snappy
The snappy 1.2.0 package has been marked as broken, and people are working on a correct update. But that should mean that creating a new fresh environment should work again now.
Thanks for marking the build as broken @jorisvandenbossche!
I see this error also in the new build that is not marked as broken:
snappy 1.2.0 hdb0a2a9_1 conda-forge
I see same error also
See https://github.com/conda-forge/snappy-feedstock/issues/35#issuecomment-2043486477. Should be resolved quickly once the change has been picked up by the CDN
Still getting error _undefined symbol ZN6snappy11RawCompressEPKcmPcPm, is this still a CDN rollout issue, any workaround? @jorisvandenbossche . Thanks
@corticalstack can you show the output of conda list
? It should be resolved now. Where do you get this, locally or on CI? (on CI you are maybe using some caching mechanism?)
@jorisvandenbossche just built env now, still same symbol issue, conda env created with conda install --no-cache
# Name Version Build Channel
_libgcc_mutex 0.1 conda_forge conda-forge
_openmp_mutex 4.5 2_gnu conda-forge
abseil-cpp 20211102.0 h93e1e8c_3 conda-forge
accelerate 0.29.2 pypi_0 pypi
aiohttp 3.9.3 py311h459d7ec_1 conda-forge
aiosignal 1.3.1 pyhd8ed1ab_0 conda-forge
appdirs 1.4.4 pyh9f0ad1d_0 conda-forge
arrow-cpp 14.0.2 h374c478_1
asttokens 2.0.5 pyhd3eb1b0_0
attrs 23.2.0 pyh71513ae_0 conda-forge
aws-c-auth 0.6.21 h3cb7b9d_0 conda-forge
aws-c-cal 0.5.20 hff2c3d7_3 conda-forge
aws-c-common 0.8.5 h166bdaf_0 conda-forge
aws-c-compression 0.2.16 hf5f93bc_0 conda-forge
aws-c-event-stream 0.2.15 h2c1f3d0_11 conda-forge
aws-c-http 0.6.27 hb11a807_3 conda-forge
aws-c-io 0.13.11 h4f448d1_2 conda-forge
aws-c-mqtt 0.7.13 h93e60df_9 conda-forge
aws-c-s3 0.2.1 h0fab5a8_0 conda-forge
aws-c-sdkutils 0.1.7 hf5f93bc_0 conda-forge
aws-checksums 0.1.13 hf5f93bc_5 conda-forge
aws-crt-cpp 0.18.16 h55da666_2 conda-forge
aws-sdk-cpp 1.10.55 h721c034_0
bitsandbytes 0.43.0 py311he54c9da_0 conda-forge
blas 1.1 openblas conda-forge
boost-cpp 1.78.0 he72f1d9_0 conda-forge
brotli-python 1.0.9 py311ha362b79_9 conda-forge
bzip2 1.0.8 h5eee18b_5
c-ares 1.28.1 hd590300_0 conda-forge
ca-certificates 2024.3.11 h06a4308_0
certifi 2024.2.2 pyhd8ed1ab_0 conda-forge
cffi 1.16.0 py311hb3a22ac_0 conda-forge
chardet 5.2.0 pypi_0 pypi
charset-normalizer 3.3.2 pyhd8ed1ab_0 conda-forge
click 8.1.7 unix_pyh707e725_0 conda-forge
colorama 0.4.6 pyhd8ed1ab_0 conda-forge
comm 0.2.1 py311h06a4308_0
cudatoolkit 11.8.0 h4ba93d1_13 conda-forge
cudatoolkit-dev 11.7.0 h1de0b5d_6 conda-forge
datasets 2.18.0 pypi_0 pypi
debugpy 1.6.7 py311h6a678d5_0
decorator 5.1.1 pyhd3eb1b0_0
dill 0.3.7 pyhd8ed1ab_0 conda-forge
docker-pycreds 0.4.0 py_0 conda-forge
docstring-parser 0.16 pypi_0 pypi
einops 0.7.0 pypi_0 pypi
executing 0.8.3 pyhd3eb1b0_0
filelock 3.13.4 pyhd8ed1ab_0 conda-forge
flash-attn 2.5.7 pypi_0 pypi
frozenlist 1.4.1 py311h459d7ec_0 conda-forge
fsspec 2024.2.0 pypi_0 pypi
gflags 2.2.2 he1b5a44_1004 conda-forge
git-lfs 3.5.1 ha770c72_0 conda-forge
gitdb 4.0.11 pyhd8ed1ab_0 conda-forge
gitpython 3.1.43 pyhd8ed1ab_0 conda-forge
glog 0.5.0 h48cff8f_0 conda-forge
grpc-cpp 1.48.2 he1ff14a_1
huggingface_hub 0.22.2 pyhd8ed1ab_0 conda-forge
icu 70.1 h27087fc_0 conda-forge
idna 3.6 pyhd8ed1ab_0 conda-forge
importlib-metadata 7.1.0 pyha770c72_0 conda-forge
ipykernel 6.28.0 py311h06a4308_0
ipython 8.20.0 py311h06a4308_0
ipywidgets 8.1.2 pyhd8ed1ab_0 conda-forge
jedi 0.18.1 py311h06a4308_1
jinja2 3.1.3 pyhd8ed1ab_0 conda-forge
jupyter_client 8.6.0 py311h06a4308_0
jupyter_core 5.5.0 py311h06a4308_0
jupyterlab_widgets 3.0.10 pyhd8ed1ab_0 conda-forge
keyutils 1.6.1 h166bdaf_0 conda-forge
krb5 1.20.1 h81ceb04_0 conda-forge
ld_impl_linux-64 2.38 h1181459_1
libabseil 20211102.0 cxx17_h48a1fff_3 conda-forge
libblas 3.9.0 22_linux64_openblas conda-forge
libbrotlicommon 1.0.9 h166bdaf_9 conda-forge
libbrotlidec 1.0.9 h166bdaf_9 conda-forge
libbrotlienc 1.0.9 h166bdaf_9 conda-forge
libcblas 3.9.0 22_linux64_openblas conda-forge
libcurl 8.5.0 h251f7ec_0
libedit 3.1.20191231 he28a2e2_2 conda-forge
libev 4.33 hd590300_2 conda-forge
libevent 2.1.12 hf998b51_1 conda-forge
libffi 3.4.4 h6a678d5_0
libgcc-ng 13.2.0 h807b86a_5 conda-forge
libgfortran-ng 13.2.0 h69a702a_5 conda-forge
libgfortran5 13.2.0 ha4646dd_5 conda-forge
libgomp 13.2.0 h807b86a_5 conda-forge
liblapack 3.9.0 22_linux64_openblas conda-forge
libnghttp2 1.57.0 h2d74bed_0
libopenblas 0.3.27 pthreads_h413a1c8_0 conda-forge
libprotobuf 3.20.3 he621ea3_0
libsodium 1.0.18 h7b6447c_0
libssh2 1.10.0 ha35d2d1_2 conda-forge
libstdcxx-ng 13.2.0 h7e041cc_5 conda-forge
libthrift 0.15.0 h1795dd8_2
libuuid 1.41.5 h5eee18b_0
lz4-c 1.9.4 hcb278e6_0 conda-forge
markdown-it-py 3.0.0 pypi_0 pypi
markupsafe 2.1.5 py311h459d7ec_0 conda-forge
matplotlib-inline 0.1.6 py311h06a4308_0
mdurl 0.1.2 pypi_0 pypi
mpmath 1.3.0 pyhd8ed1ab_0 conda-forge
multidict 6.0.5 py311h459d7ec_0 conda-forge
multiprocess 0.70.15 py311h459d7ec_1 conda-forge
mypy_extensions 1.0.0 pyha770c72_0 conda-forge
ncurses 6.4 h6a678d5_0
nest-asyncio 1.6.0 py311h06a4308_0
networkx 3.3 pyhd8ed1ab_1 conda-forge
ninja 1.11.1.1 pypi_0 pypi
numpy 1.26.4 py311h64a7726_0 conda-forge
openblas 0.3.27 pthreads_h7a3da1a_0 conda-forge
openssl 3.2.1 hd590300_1 conda-forge
orc 1.7.4 hb3bc3d3_1
packaging 23.2 py311h06a4308_0
pandas 2.2.2 py311h320fe9a_0 conda-forge
parso 0.8.3 pyhd3eb1b0_0
pathtools 0.1.2 py_1 conda-forge
peft 0.10.0 pypi_0 pypi
pexpect 4.8.0 pyhd3eb1b0_3
pip 23.3.1 py311h06a4308_0
platformdirs 3.10.0 py311h06a4308_0
prompt-toolkit 3.0.43 py311h06a4308_0
prompt_toolkit 3.0.43 hd3eb1b0_0
protobuf 3.20.3 py311hcafe171_1 conda-forge
psutil 5.9.0 py311h5eee18b_0
ptyprocess 0.7.0 pyhd3eb1b0_2
pure_eval 0.2.2 pyhd3eb1b0_0
pyarrow 14.0.2 py311hb6e97c4_0
pyarrow-hotfix 0.6 pypi_0 pypi
pycparser 2.22 pyhd8ed1ab_0 conda-forge
pygments 2.15.1 py311h06a4308_1
pyre-extensions 0.0.29 pyhd8ed1ab_0 conda-forge
pysocks 1.7.1 pyha2e5f31_6 conda-forge
python 3.11.7 h955ad1f_0
python-dateutil 2.8.2 pyhd3eb1b0_0
python-tzdata 2024.1 pyhd8ed1ab_0 conda-forge
python-xxhash 3.4.1 py311h459d7ec_0 conda-forge
python_abi 3.11 2_cp311 conda-forge
pytz 2024.1 pyhd8ed1ab_0 conda-forge
pyyaml 6.0.1 py311h459d7ec_1 conda-forge
pyzmq 25.1.2 py311h6a678d5_0
re2 2022.04.01 h27087fc_0 conda-forge
readline 8.2 h5eee18b_0
regex 2023.12.25 py311h459d7ec_0 conda-forge
requests 2.31.0 pyhd8ed1ab_0 conda-forge
rich 13.7.1 pypi_0 pypi
s2n 1.3.30 h3358134_0 conda-forge
safetensors 0.4.2 py311h46250e7_0 conda-forge
scipy 1.13.0 py311h64a7726_0 conda-forge
sentencepiece 0.2.0 pypi_0 pypi
sentry-sdk 1.45.0 pyhd8ed1ab_0 conda-forge
setproctitle 1.3.3 py311h459d7ec_0 conda-forge
setuptools 68.2.2 py311h06a4308_0
shtab 1.7.1 pypi_0 pypi
six 1.16.0 pyhd3eb1b0_1
smmap 5.0.0 pyhd8ed1ab_0 conda-forge
snappy 1.2.0 hdb0a2a9_1 conda-forge
sqlite 3.41.2 h5eee18b_0
stack_data 0.2.0 pyhd3eb1b0_0
sympy 1.12 pyh04b8f61_3 conda-forge
tk 8.6.12 h1ccaba5_0
tokenizers 0.15.2 py311h6640629_0 conda-forge
torch 2.1.0+cu121 pypi_0 pypi
tornado 6.3.3 py311h5eee18b_0
tqdm 4.66.2 pyhd8ed1ab_0 conda-forge
traitlets 5.7.1 py311h06a4308_0
transformers 4.39.3 pyhd8ed1ab_0 conda-forge
triton 2.1.0 pypi_0 pypi
trl 0.8.1 pypi_0 pypi
typing-extensions 4.11.0 hd8ed1ab_0 conda-forge
typing_extensions 4.11.0 pyha770c72_0 conda-forge
typing_inspect 0.9.0 pyhd8ed1ab_0 conda-forge
tyro 0.8.3 pypi_0 pypi
tzdata 2024a h04d1e81_0
unsloth 2024.4 pypi_0 pypi
urllib3 2.2.1 pyhd8ed1ab_0 conda-forge
utf8proc 2.6.1 h5eee18b_1
wandb 0.16.5 pyhd8ed1ab_0 conda-forge
wcwidth 0.2.5 pyhd3eb1b0_0
wheel 0.43.0 pypi_0 pypi
widgetsnbextension 4.0.10 pyhd8ed1ab_0 conda-forge
xformers 0.0.21 cpu_py311h0028418_1001 conda-forge
xxhash 0.8.2 hd590300_0 conda-forge
xz 5.4.6 h5eee18b_0
yaml 0.2.5 h7f98852_2 conda-forge
yarl 1.9.4 py311h459d7ec_0 conda-forge
zeromq 4.3.5 h6a678d5_0
zipp 3.17.0 pyhd8ed1ab_0 conda-forge
zlib 1.2.13 h5eee18b_0
zstd 1.5.5 hc292b87_0
Ah, you have installed an older version of pyarrow, I was testing with pyarrow 15.0. Wondering if that might be related (although it still shouldn't happen)
Tried with mamba create -n test-pyarrow-14 python=3.11 pyarrow=14
and that works fine for me as well. But I see that I get a (py)arrow with build number 16, while you have build number 0, so the version that was built almost half a year ago when pyarrow 14 was released.
Can you show the full command that you used to recreate the fresh env? Do you have other libraries / constraints that you are also installing?
@jorisvandenbossche
# Update conda
conda update -n base -c conda-forge conda -y
# Define Conda environment name and kernel display name
export CONDA_ENV_NAME="py311_pt_unsloth"
export KERNEL_DISPLAY_NAME="Python 3.11 - LLM tune"
echo $CONDA_ENV_NAME
echo $KERNEL_DISPLAY_NAME
# List existing Conda environments
conda env list
# Create a new Conda environment with Python 3.11.7
conda create --name $CONDA_ENV_NAME python=3.11.7 -y
# List Conda environments to verify the new one is created
conda env list
# Install ipykernel and other necessary packages into the new Conda environment
conda install -no-cache -n $CONDA_ENV_NAME ipykernel -y
conda install -no-cache -n $CONDA_ENV_NAME -c conda-forge cudatoolkit xformers bitsandbytes xformers -y
conda install -no-cache -n $CONDA_ENV_NAME -c conda-forge cudatoolkit-dev -y
conda install -no-cache -n $CONDA_ENV_NAME -c conda-forge ipywidgets widgetsnbextension transformers -y
conda install -no-cache -n $CONDA_ENV_NAME -c conda-forge git-lfs wandb
# Activate the new Conda environment
conda activate $CONDA_ENV_NAME
# Install additional packages via pip into the Conda environment
pip install chardet accelerate "unsloth[conda] @ git+https://github.com/unslothai/unsloth.git"
pip install torch==2.1.0+cu121 -f https://download.pytorch.org/whl/torch_stable.html
pip install flash-attn
@jorisvandenbossche thanks to your hints'I've just updated pyarrow with below, and now all resolved. Thanks so much for your support!
conda install -c conda-forge pyarrow=15.0
@jorisvandenbossche thanks to your hints'I've just updated pyarrow with below, and now all resolved. Thanks so much for your support!
conda install -c conda-forge pyarrow=15.0
Resolved too! Thanks for your post!
Describe the bug, including details regarding any error messages, version, and platform.
Hopefully this is the right place to raise this issue! When creating new environments, I've been seeing that installation of pyarrow from conda-forge pulls in the recently released
snappy=1.2.0
, which results in a missing symbol_ZN6snappy11RawCompressEPKcmPcPm
in both Mac and Linux. Downgrading to snappy 1.1.10 fixes the issue.Reproducing on linux (I am seeing this across Pythons 3.9 to 3.12):
Reproducing on Mac:
Component(s)
Python