search

conda-forge / transformers-feedstock

A conda-smithy repository for transformers.
BSD 3-Clause "New" or "Revised" License
3 stars 13 forks source link

transformers > 4.16 and < 4.23 can't use tokenizers >=0.13 but this restriction is not imposed by conda-forge packages #106

Open ksteimel opened 2 years ago

ksteimel commented 2 years ago

Solution to issue cannot be found in the documentation.

Issue

Tokenizers versions 0.13.0 and 0.13.1 are not compatible with transformers 4.18 - 4.22.2. But the current packages in conda forge do not impose these restrictions (which results in errors when importing transformers). Is there any way to update recipes with new build numbers for old package versions? I can open the branch if so. With the installed packages below, import transformers fails

Installed packages

# packages in environment at /home/kenneth/miniforge3/envs/tokenizers_test:
#
# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                 conda_forge    conda-forge
_openmp_mutex             4.5                  2_kmp_llvm    conda-forge
aiohttp                   3.8.3           py310h5764c6d_0    conda-forge
aiosignal                 1.2.0              pyhd8ed1ab_0    conda-forge
arrow-cpp                 9.0.0           py310h9c91e85_7_cpu    conda-forge
async-timeout             4.0.2              pyhd8ed1ab_0    conda-forge
attrs                     22.1.0             pyh71513ae_1    conda-forge
aws-c-cal                 0.5.11               h95a6274_0    conda-forge
aws-c-common              0.6.2                h7f98852_0    conda-forge
aws-c-event-stream        0.2.7               h3541f99_13    conda-forge
aws-c-io                  0.10.5               hfb6a706_0    conda-forge
aws-checksums             0.1.11               ha31a3da_7    conda-forge
aws-sdk-cpp               1.8.186              hb4091e7_3    conda-forge
brotlipy                  0.7.0           py310h5764c6d_1004    conda-forge
bzip2                     1.0.8                h7f98852_4    conda-forge
c-ares                    1.18.1               h7f98852_0    conda-forge
ca-certificates           2022.9.24            ha878542_0    conda-forge
certifi                   2022.9.24          pyhd8ed1ab_0    conda-forge
cffi                      1.15.1          py310h255011f_1    conda-forge
charset-normalizer        2.1.1              pyhd8ed1ab_0    conda-forge
click                     8.1.3           py310hff52083_0    conda-forge
colorama                  0.4.5              pyhd8ed1ab_0    conda-forge
cryptography              38.0.2          py310h597c629_0    conda-forge
cudatoolkit               11.7.0              hd8887f6_10    conda-forge
cudnn                     8.4.1.50             hed8a83a_0    conda-forge
dataclasses               0.8                pyhc8e2a94_3    conda-forge
datasets                  2.6.1              pyhd8ed1ab_0    conda-forge
dill                      0.3.5.1            pyhd8ed1ab_0    conda-forge
filelock                  3.8.0              pyhd8ed1ab_0    conda-forge
frozenlist                1.3.1           py310h5764c6d_0    conda-forge
fsspec                    2022.10.0          pyhd8ed1ab_0    conda-forge
gflags                    2.2.2             he1b5a44_1004    conda-forge
glog                      0.6.0                h6f12383_0    conda-forge
grpc-cpp                  1.47.1               h05bd8bd_6    conda-forge
huggingface_hub           0.10.1             pyhd8ed1ab_0    conda-forge
idna                      3.4                pyhd8ed1ab_0    conda-forge
importlib-metadata        4.11.4          py310hff52083_0    conda-forge
importlib_metadata        4.11.4               hd8ed1ab_0    conda-forge
joblib                    1.2.0              pyhd8ed1ab_0    conda-forge
keyutils                  1.6.1                h166bdaf_0    conda-forge
krb5                      1.19.3               h3790be6_0    conda-forge
ld_impl_linux-64          2.39                 hc81fddc_0    conda-forge
libabseil                 20220623.0      cxx17_h48a1fff_4    conda-forge
libblas                   3.9.0           16_linux64_openblas    conda-forge
libbrotlicommon           1.0.9                h166bdaf_7    conda-forge
libbrotlidec              1.0.9                h166bdaf_7    conda-forge
libbrotlienc              1.0.9                h166bdaf_7    conda-forge
libcblas                  3.9.0           16_linux64_openblas    conda-forge
libcrc32c                 1.1.2                h9c3ff4c_0    conda-forge
libcurl                   7.85.0               h7bff187_0    conda-forge
libedit                   3.1.20191231         he28a2e2_2    conda-forge
libev                     4.33                 h516909a_1    conda-forge
libevent                  2.1.10               h9b69904_4    conda-forge
libffi                    3.4.2                h7f98852_5    conda-forge
libgcc-ng                 12.2.0              h65d4601_18    conda-forge
libgfortran-ng            12.2.0              h69a702a_18    conda-forge
libgfortran5              12.2.0              h337968e_18    conda-forge
libgoogle-cloud           2.2.0                h838d150_1    conda-forge
liblapack                 3.9.0           16_linux64_openblas    conda-forge
libnghttp2                1.47.0               hdcd2b5c_1    conda-forge
libnsl                    2.0.0                h7f98852_0    conda-forge
libopenblas               0.3.21          pthreads_h78a6416_3    conda-forge
libprotobuf               3.21.8               h6239696_0    conda-forge
libsqlite                 3.39.4               h753d276_0    conda-forge
libssh2                   1.10.0               haa6b8db_3    conda-forge
libstdcxx-ng              12.2.0              h46fd767_18    conda-forge
libthrift                 0.16.0               h491838f_2    conda-forge
libutf8proc               2.7.0                h7f98852_0    conda-forge
libuuid                   2.32.1            h7f98852_1000    conda-forge
libzlib                   1.2.13               h166bdaf_4    conda-forge
llvm-openmp               14.0.4               he0ac6c6_0    conda-forge
lz4-c                     1.9.3                h9c3ff4c_1    conda-forge
magma                     2.5.4                hc72dce7_3    conda-forge
mkl                       2022.1.0           h84fe81f_915    conda-forge
multidict                 6.0.2           py310h5764c6d_1    conda-forge
multiprocess              0.70.13         py310h5764c6d_1    conda-forge
nccl                      2.14.3.1             h0800d71_0    conda-forge
ncurses                   6.3                  h27087fc_1    conda-forge
ninja                     1.11.0               h924138e_0    conda-forge
numpy                     1.23.4          py310h53a5b5f_0    conda-forge
openssl                   1.1.1q               h166bdaf_1    conda-forge
orc                       1.8.0                h09e0d61_0    conda-forge
packaging                 21.3               pyhd8ed1ab_0    conda-forge
pandas                    1.5.1           py310h769672d_0    conda-forge
parquet-cpp               1.5.1                         2    conda-forge
pip                       22.3               pyhd8ed1ab_0    conda-forge
pyarrow                   9.0.0           py310h225c066_7_cpu    conda-forge
pycparser                 2.21               pyhd8ed1ab_0    conda-forge
pyopenssl                 22.1.0             pyhd8ed1ab_0    conda-forge
pyparsing                 3.0.9              pyhd8ed1ab_0    conda-forge
pysocks                   1.7.1              pyha2e5f31_6    conda-forge
python                    3.10.6          h582c2e5_0_cpython    conda-forge
python-dateutil           2.8.2              pyhd8ed1ab_0    conda-forge
python-xxhash             3.0.0           py310h5764c6d_1    conda-forge
python_abi                3.10                    2_cp310    conda-forge
pytorch                   1.12.1          cuda112py310he33e0d6_201    conda-forge
pytz                      2022.5             pyhd8ed1ab_0    conda-forge
pyyaml                    6.0             py310h5764c6d_4    conda-forge
re2                       2022.06.01           h27087fc_0    conda-forge
readline                  8.1.2                h0f457ee_0    conda-forge
regex                     2022.9.13       py310h5764c6d_0    conda-forge
requests                  2.28.1             pyhd8ed1ab_1    conda-forge
responses                 0.18.0             pyhd8ed1ab_0    conda-forge
s2n                       1.0.10               h9b69904_0    conda-forge
sacremoses                0.0.53             pyhd8ed1ab_0    conda-forge
setuptools                65.5.0             pyhd8ed1ab_0    conda-forge
six                       1.16.0             pyh6c4a22f_0    conda-forge
sleef                     3.5.1                h9b69904_2    conda-forge
snappy                    1.1.9                hbd366e4_1    conda-forge
tbb                       2021.6.0             h924138e_0    conda-forge
tk                        8.6.12               h27826a3_0    conda-forge
tokenizers                0.13.1          py310h3143118_0    conda-forge
tqdm                      4.64.1             pyhd8ed1ab_0    conda-forge
transformers              4.22.2             pyhd8ed1ab_0    conda-forge
typing-extensions         4.4.0                hd8ed1ab_0    conda-forge
typing_extensions         4.4.0              pyha770c72_0    conda-forge
tzdata                    2022e                h191b570_0    conda-forge
urllib3                   1.26.11            pyhd8ed1ab_0    conda-forge
wheel                     0.37.1             pyhd8ed1ab_0    conda-forge
xxhash                    0.8.0                h7f98852_3    conda-forge
xz                        5.2.6                h166bdaf_0    conda-forge
yaml                      0.2.5                h7f98852_2    conda-forge
yarl                      1.7.2           py310h5764c6d_2    conda-forge
zipp                      3.9.0              pyhd8ed1ab_0    conda-forge
zlib                      1.2.13               h166bdaf_4    conda-forge
zstd                      1.5.2                h6239696_4    conda-forge

Environment info

active environment : tokenizers_test
    active env location : /home/kenneth/miniforge3/envs/tokenizers_test
            shell level : 2
       user config file : /home/kenneth/.condarc
 populated config files : /home/kenneth/miniforge3/.condarc
          conda version : 22.9.0
    conda-build version : not installed
         python version : 3.9.13.final.0
       virtual packages : __cuda=11.7=0
                          __linux=6.0.1=0
                          __glibc=2.36=0
                          __unix=0=0
                          __archspec=1=x86_64
       base environment : /home/kenneth/miniforge3  (writable)
      conda av data dir : /home/kenneth/miniforge3/etc/conda
  conda av metadata url : None
           channel URLs : https://conda.anaconda.org/conda-forge/linux-64
                          https://conda.anaconda.org/conda-forge/noarch
          package cache : /home/kenneth/miniforge3/pkgs
                          /home/kenneth/.conda/pkgs
       envs directories : /home/kenneth/miniforge3/envs
                          /home/kenneth/.conda/envs
               platform : linux-64
             user-agent : conda/22.9.0 requests/2.28.1 CPython/3.9.13 Linux/6.0.1-1-default opensuse-tumbleweed/20221016 glibc/2.36
                UID:GID : 1000:1000
             netrc file : None
           offline mode : False
setu4993 commented 2 years ago

Hey @ksteimel, thanks for opening this issue. Makes sense.

If you want to open branches from the tags for the previous versions (with bumped build versions), I'm happy to review those PRs. That should fix the previous versions.

bollwyvl commented 1 year ago

Also saw this on https://github.com/conda-forge/spacy-models-feedstock/pull/6.

Unfortunately, the "greedy" (or "opportunistic"?) conda/mamba solvers might pull in the old package so they can have the newer (incorrect) dependency.

However, this can be fixed for the old packages by making a PR here:

https://github.com/conda-forge/conda-forge-repodata-patches-feedstock

If it's not clear how to proceed there from previous PRs, one might be able to open an issue to ask for help.

xhochy commented 1 year ago

Submitted a PR to patch this: https://github.com/conda-forge/conda-forge-repodata-patches-feedstock/pull/458