Fail to import torchtext KeyError: 'SP_DIR'

[cecilialee]

❓ Questions and Help

Description

I failed to import torchtext with the following error. I tried it with a fresh conda env install (under a different python version) and still got the same issue.

Originally I was able to use torchtext (I remember installed from pip) in an env of python 3.11, but then it raised error with the dataset module, so I updated torchtext with pip and started getting kernel crush for pytorch import. So I did some uninstall and install of the pytorch and torchtext packages from different sources (conda or pip) and couldn't fix the issue. Even a new conda env using python 3.10 raised the same error. I don't know what is messed up.

---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
Cell In[3], line 1
----> 1 import torchtext

File ~/miniconda3/envs/ml2/lib/python3.10/site-packages/torchtext/__init__.py:6
      3 from torch.hub import _get_torch_home
      5 # the following import has to happen first in order to load the torchtext C++ library
----> 6 from torchtext import _extension  # noqa: F401
      8 _TEXT_BUCKET = \"https://download.pytorch.org/models/text/\"
     10 _CACHE_DIR = os.path.expanduser(os.path.join(_get_torch_home(), \"text\"))

File ~/miniconda3/envs/ml2/lib/python3.10/site-packages/torchtext/_extension.py:7
      4 import torch
      5 from torchtext._internal import module_utils as _mod_utils
----> 7 _LIB_DIR = Path(os.environ[\"SP_DIR\"]) / \"torch\" / \"lib\"
     10 def _get_lib_path(lib: str):
     11     suffix = \"pyd\" if os.name == \"nt\" else \"so\"

File ~/miniconda3/envs/ml2/lib/python3.10/os.py:680, in _Environ.__getitem__(self, key)
    677     value = self._data[self.encodekey(key)]
    678 except KeyError:
    679     # raise KeyError with the original key value
--> 680     raise KeyError(key) from None
    681 return self.decodevalue(value)

KeyError: 'SP_DIR'

# packages in environment at /Users/cecilia/miniconda3/envs/ml2:
#
# Name                    Version                   Build  Channel
annotated-types           0.6.0              pyhd8ed1ab_0    conda-forge
appnope                   0.1.3              pyhd8ed1ab_0    conda-forge
asttokens                 2.4.1              pyhd8ed1ab_0    conda-forge
brotli-python             1.1.0           py310h9e9d8ca_1    conda-forge
bzip2                     1.0.8                h10d778d_5    conda-forge
ca-certificates           2023.11.17           h8857fd0_0    conda-forge
catalogue                 2.0.10          py310h2ec42d9_0    conda-forge
certifi                   2023.11.17         pyhd8ed1ab_0    conda-forge
charset-normalizer        3.3.2              pyhd8ed1ab_0    conda-forge
click                     8.1.7           unix_pyh707e725_0    conda-forge
cloudpathlib              0.16.0             pyhd8ed1ab_0    conda-forge
colorama                  0.4.6              pyhd8ed1ab_0    conda-forge
comm                      0.2.1              pyhd8ed1ab_0    conda-forge
confection                0.1.4           py310h1cef2ca_0    conda-forge
cymem                     2.0.8           py310h9e9d8ca_1    conda-forge
cython-blis               0.7.10          py310hf0b6da5_2    conda-forge
debugpy                   1.8.0           py310h9e9d8ca_1    conda-forge
decorator                 5.1.1              pyhd8ed1ab_0    conda-forge
double-conversion         3.3.0                he965462_0    conda-forge
exceptiongroup            1.2.0              pyhd8ed1ab_2    conda-forge
executing                 2.0.1              pyhd8ed1ab_0    conda-forge
filelock                  3.13.1             pyhd8ed1ab_0    conda-forge
fsspec                    2023.12.2          pyhca7485f_0    conda-forge
gmp                       6.3.0                h93d8f39_0    conda-forge
gmpy2                     2.1.2           py310hb691cb2_1    conda-forge
icu                       73.2                 hf5e326d_0    conda-forge
idna                      3.6                pyhd8ed1ab_0    conda-forge
importlib-metadata        7.0.1              pyha770c72_0    conda-forge
importlib_metadata        7.0.1                hd8ed1ab_0    conda-forge
ipykernel                 6.29.0             pyh3cd1d5f_0    conda-forge
ipython                   8.20.0             pyh707e725_0    conda-forge
jedi                      0.19.1             pyhd8ed1ab_0    conda-forge
jinja2                    3.1.3              pyhd8ed1ab_0    conda-forge
joblib                    1.3.2              pyhd8ed1ab_0    conda-forge
jupyter_client            8.6.0              pyhd8ed1ab_0    conda-forge
jupyter_core              5.7.1           py310h2ec42d9_0    conda-forge
langcodes                 3.3.0              pyhd8ed1ab_0    conda-forge
libabseil                 20230802.1      cxx17_h048a20a_0    conda-forge
libblas                   3.9.0           21_osx64_openblas    conda-forge
libcblas                  3.9.0           21_osx64_openblas    conda-forge
libcxx                    16.0.6               hd57cbcb_0    conda-forge
libffi                    3.4.2                h0d85af4_5    conda-forge
libgfortran               5.0.0           13_2_0_h97931a8_2    conda-forge
libgfortran5              13.2.0               h2873a65_2    conda-forge
libhwloc                  2.9.3           default_h24e0189_1009    conda-forge
libiconv                  1.17                 hd75f5a5_2    conda-forge
liblapack                 3.9.0           21_osx64_openblas    conda-forge
libopenblas               0.3.26          openmp_hfef2a42_0    conda-forge
libprotobuf               4.24.4               hc4f2305_0    conda-forge
libre2-11                 2023.06.02           h4694dbf_0    conda-forge
libsentencepiece          0.1.99               ha269934_5    conda-forge
libsodium                 1.0.18               hbcb3906_1    conda-forge
libsqlite                 3.44.2               h92b6c6a_0    conda-forge
libtorch                  2.1.0           cpu_mkl_hc49ff94_103    conda-forge
libutf8proc               2.8.0                hb7f2c08_0    conda-forge
libuv                     1.46.0               h0c2f820_0    conda-forge
libxml2                   2.12.4               hc0ae0f7_1    conda-forge
libzlib                   1.2.13               h8a1eda9_5    conda-forge
llvm-openmp               17.0.6               hb6ac08f_0    conda-forge
markdown-it-py            3.0.0              pyhd8ed1ab_0    conda-forge
markupsafe                2.1.4           py310hb372a2b_0    conda-forge
matplotlib-inline         0.1.6              pyhd8ed1ab_0    conda-forge
mdurl                     0.1.2              pyhd8ed1ab_0    conda-forge
mkl                       2023.2.0         h54c2260_50500    conda-forge
mpc                       1.3.1                h81bd1dd_0    conda-forge
mpfr                      4.2.1                h0c69b56_0    conda-forge
mpmath                    1.3.0              pyhd8ed1ab_0    conda-forge
murmurhash                1.0.10          py310h9e9d8ca_1    conda-forge
ncurses                   6.4                  h93d8f39_2    conda-forge
nest-asyncio              1.6.0              pyhd8ed1ab_0    conda-forge
networkx                  3.2.1              pyhd8ed1ab_0    conda-forge
nltk                      3.8.1              pyhd8ed1ab_0    conda-forge
numpy                     1.26.3          py310h4bfa8fc_0    conda-forge
openssl                   3.2.0                hd75f5a5_1    conda-forge
packaging                 23.2               pyhd8ed1ab_0    conda-forge
parso                     0.8.3              pyhd8ed1ab_0    conda-forge
pathy                     0.10.3          py310hecd8cb5_0
pexpect                   4.9.0              pyhd8ed1ab_0    conda-forge
pickleshare               0.7.5                   py_1003    conda-forge
pip                       23.3.2             pyhd8ed1ab_0    conda-forge
platformdirs              4.1.0              pyhd8ed1ab_0    conda-forge
preshed                   3.0.9           py310h9e9d8ca_1    conda-forge
prompt-toolkit            3.0.42             pyha770c72_0    conda-forge
psutil                    5.9.8           py310hb372a2b_0    conda-forge
ptyprocess                0.7.0              pyhd3deb0d_0    conda-forge
pure_eval                 0.2.2              pyhd8ed1ab_0    conda-forge
pydantic                  2.6.0              pyhd8ed1ab_0    conda-forge
pydantic-core             2.16.1          py310h54baaa9_0    conda-forge
pygments                  2.17.2             pyhd8ed1ab_0    conda-forge
pysocks                   1.7.1              pyha2e5f31_6    conda-forge
python                    3.10.13         h00d2728_1_cpython    conda-forge
python-dateutil           2.8.2              pyhd8ed1ab_0    conda-forge
python_abi                3.10                    4_cp310    conda-forge
pytorch                   2.1.0           cpu_mkl_py310h1822dd0_103    conda-forge
pyzmq                     25.1.2          py310h6b67f7f_0    conda-forge
re2                       2023.06.02           hd34609a_0    conda-forge
readline                  8.2                  h9e318b2_1    conda-forge
regex                     2023.12.25      py310hb372a2b_0    conda-forge
requests                  2.31.0             pyhd8ed1ab_0    conda-forge
revtok                    0.0.3.1            pyhd8ed1ab_0    conda-forge
rich                      13.7.0             pyhd8ed1ab_0    conda-forge
sacremoses                0.0.53             pyhd8ed1ab_0    conda-forge
setuptools                69.0.3             pyhd8ed1ab_0    conda-forge
shellingham               1.5.4              pyhd8ed1ab_0    conda-forge
six                       1.16.0             pyh6c4a22f_0    conda-forge
sleef                     3.5.1                h6db0672_2    conda-forge
smart_open                6.4.0              pyhd8ed1ab_0    conda-forge
spacy                     3.7.2           py310h65d09f4_0    conda-forge
spacy-legacy              3.0.12             pyhd8ed1ab_0    conda-forge
spacy-loggers             1.0.5              pyhd8ed1ab_0    conda-forge
srsly                     2.4.8           py310h9e9d8ca_1    conda-forge
stack_data                0.6.2              pyhd8ed1ab_0    conda-forge
sympy                     1.12            pypyh9d50eac_103    conda-forge
tbb                       2021.11.0            h7728843_1    conda-forge
thinc                     8.2.2           py310h076e4b7_0    conda-forge
tk                        8.6.13               h1abcd95_1    conda-forge
torch                     2.1.0.post103            pypi_0    pypi
torchtext                 0.15.2          py310h5de3785_4    conda-forge
tornado                   6.3.3           py310h6729b98_1    conda-forge
tqdm                      4.66.1             pyhd8ed1ab_0    conda-forge
traitlets                 5.14.1             pyhd8ed1ab_0    conda-forge
typer                     0.9.0              pyhd8ed1ab_0    conda-forge
typing-extensions         4.9.0                hd8ed1ab_0    conda-forge
typing_extensions         4.9.0              pyha770c72_0    conda-forge
tzdata                    2023d                h0c530f3_0    conda-forge
urllib3                   2.1.0              pyhd8ed1ab_0    conda-forge
wasabi                    1.1.2           py310h2ec42d9_0    conda-forge
wcwidth                   0.2.13             pyhd8ed1ab_0    conda-forge
weasel                    0.3.4              pyhd8ed1ab_0    conda-forge
wheel                     0.42.0             pyhd8ed1ab_0    conda-forge
xz                        5.2.6                h775f41a_0    conda-forge
zeromq                    4.3.5                h93d8f39_0    conda-forge
zipp                      3.17.0             pyhd8ed1ab_0    conda-forge
❯ conda list torch
# packages in environment at /Users/cecilia/miniconda3/envs/ml2:
#
# Name                    Version                   Build  Channel
libtorch                  2.1.0           cpu_mkl_hc49ff94_103    conda-forge
pytorch                   2.1.0           cpu_mkl_py310h1822dd0_103    conda-forge
torch                     2.1.0.post103            pypi_0    pypi
torchtext                 0.15.2          py310h5de3785_4    conda-forge
❯ conda list numpy
# packages in environment at /Users/cecilia/miniconda3/envs/ml2:
#
# Name                    Version                   Build  Channel
numpy                     1.26.3          py310h4bfa8fc_0    conda-forge
❯ conda list
# packages in environment at /Users/cecilia/miniconda3/envs/ml2:
#
# Name                    Version                   Build  Channel
annotated-types           0.6.0              pyhd8ed1ab_0    conda-forge
appnope                   0.1.3              pyhd8ed1ab_0    conda-forge
asttokens                 2.4.1              pyhd8ed1ab_0    conda-forge
brotli-python             1.1.0           py310h9e9d8ca_1    conda-forge
bzip2                     1.0.8                h10d778d_5    conda-forge
ca-certificates           2023.11.17           h8857fd0_0    conda-forge
catalogue                 2.0.10          py310h2ec42d9_0    conda-forge
certifi                   2023.11.17         pyhd8ed1ab_0    conda-forge
charset-normalizer        3.3.2              pyhd8ed1ab_0    conda-forge
click                     8.1.7           unix_pyh707e725_0    conda-forge
cloudpathlib              0.16.0             pyhd8ed1ab_0    conda-forge
colorama                  0.4.6              pyhd8ed1ab_0    conda-forge
comm                      0.2.1              pyhd8ed1ab_0    conda-forge
confection                0.1.4           py310h1cef2ca_0    conda-forge
cymem                     2.0.8           py310h9e9d8ca_1    conda-forge
cython-blis               0.7.10          py310hf0b6da5_2    conda-forge
debugpy                   1.8.0           py310h9e9d8ca_1    conda-forge
decorator                 5.1.1              pyhd8ed1ab_0    conda-forge
double-conversion         3.3.0                he965462_0    conda-forge
exceptiongroup            1.2.0              pyhd8ed1ab_2    conda-forge
executing                 2.0.1              pyhd8ed1ab_0    conda-forge
filelock                  3.13.1             pyhd8ed1ab_0    conda-forge
fsspec                    2023.12.2          pyhca7485f_0    conda-forge
gmp                       6.3.0                h93d8f39_0    conda-forge
gmpy2                     2.1.2           py310hb691cb2_1    conda-forge
icu                       73.2                 hf5e326d_0    conda-forge
idna                      3.6                pyhd8ed1ab_0    conda-forge
importlib-metadata        7.0.1              pyha770c72_0    conda-forge
importlib_metadata        7.0.1                hd8ed1ab_0    conda-forge
ipykernel                 6.29.0             pyh3cd1d5f_0    conda-forge
ipython                   8.20.0             pyh707e725_0    conda-forge
jedi                      0.19.1             pyhd8ed1ab_0    conda-forge
jinja2                    3.1.3              pyhd8ed1ab_0    conda-forge
joblib                    1.3.2              pyhd8ed1ab_0    conda-forge
jupyter_client            8.6.0              pyhd8ed1ab_0    conda-forge
jupyter_core              5.7.1           py310h2ec42d9_0    conda-forge
langcodes                 3.3.0              pyhd8ed1ab_0    conda-forge
libabseil                 20230802.1      cxx17_h048a20a_0    conda-forge
libblas                   3.9.0           21_osx64_openblas    conda-forge
libcblas                  3.9.0           21_osx64_openblas    conda-forge
libcxx                    16.0.6               hd57cbcb_0    conda-forge
libffi                    3.4.2                h0d85af4_5    conda-forge
libgfortran               5.0.0           13_2_0_h97931a8_2    conda-forge
libgfortran5              13.2.0               h2873a65_2    conda-forge
libhwloc                  2.9.3           default_h24e0189_1009    conda-forge
libiconv                  1.17                 hd75f5a5_2    conda-forge
liblapack                 3.9.0           21_osx64_openblas    conda-forge
libopenblas               0.3.26          openmp_hfef2a42_0    conda-forge
libprotobuf               4.24.4               hc4f2305_0    conda-forge
libre2-11                 2023.06.02           h4694dbf_0    conda-forge
libsentencepiece          0.1.99               ha269934_5    conda-forge
libsodium                 1.0.18               hbcb3906_1    conda-forge
libsqlite                 3.44.2               h92b6c6a_0    conda-forge
libtorch                  2.1.0           cpu_mkl_hc49ff94_103    conda-forge
libutf8proc               2.8.0                hb7f2c08_0    conda-forge
libuv                     1.46.0               h0c2f820_0    conda-forge
libxml2                   2.12.4               hc0ae0f7_1    conda-forge
libzlib                   1.2.13               h8a1eda9_5    conda-forge
llvm-openmp               17.0.6               hb6ac08f_0    conda-forge
markdown-it-py            3.0.0              pyhd8ed1ab_0    conda-forge
markupsafe                2.1.4           py310hb372a2b_0    conda-forge
matplotlib-inline         0.1.6              pyhd8ed1ab_0    conda-forge
mdurl                     0.1.2              pyhd8ed1ab_0    conda-forge
mkl                       2023.2.0         h54c2260_50500    conda-forge
mpc                       1.3.1                h81bd1dd_0    conda-forge
mpfr                      4.2.1                h0c69b56_0    conda-forge
mpmath                    1.3.0              pyhd8ed1ab_0    conda-forge
murmurhash                1.0.10          py310h9e9d8ca_1    conda-forge
ncurses                   6.4                  h93d8f39_2    conda-forge
nest-asyncio              1.6.0              pyhd8ed1ab_0    conda-forge
networkx                  3.2.1              pyhd8ed1ab_0    conda-forge
nltk                      3.8.1              pyhd8ed1ab_0    conda-forge
numpy                     1.26.3          py310h4bfa8fc_0    conda-forge
openssl                   3.2.0                hd75f5a5_1    conda-forge
packaging                 23.2               pyhd8ed1ab_0    conda-forge
parso                     0.8.3              pyhd8ed1ab_0    conda-forge
pathy                     0.10.3          py310hecd8cb5_0
pexpect                   4.9.0              pyhd8ed1ab_0    conda-forge
pickleshare               0.7.5                   py_1003    conda-forge
pip                       23.3.2             pyhd8ed1ab_0    conda-forge
platformdirs              4.1.0              pyhd8ed1ab_0    conda-forge
preshed                   3.0.9           py310h9e9d8ca_1    conda-forge
prompt-toolkit            3.0.42             pyha770c72_0    conda-forge
psutil                    5.9.8           py310hb372a2b_0    conda-forge
ptyprocess                0.7.0              pyhd3deb0d_0    conda-forge
pure_eval                 0.2.2              pyhd8ed1ab_0    conda-forge
pydantic                  2.6.0              pyhd8ed1ab_0    conda-forge
pydantic-core             2.16.1          py310h54baaa9_0    conda-forge
pygments                  2.17.2             pyhd8ed1ab_0    conda-forge
pysocks                   1.7.1              pyha2e5f31_6    conda-forge
python                    3.10.13         h00d2728_1_cpython    conda-forge
python-dateutil           2.8.2              pyhd8ed1ab_0    conda-forge
python_abi                3.10                    4_cp310    conda-forge
pytorch                   2.1.0           cpu_mkl_py310h1822dd0_103    conda-forge
pyzmq                     25.1.2          py310h6b67f7f_0    conda-forge
re2                       2023.06.02           hd34609a_0    conda-forge
readline                  8.2                  h9e318b2_1    conda-forge
regex                     2023.12.25      py310hb372a2b_0    conda-forge
requests                  2.31.0             pyhd8ed1ab_0    conda-forge
revtok                    0.0.3.1            pyhd8ed1ab_0    conda-forge
rich                      13.7.0             pyhd8ed1ab_0    conda-forge
sacremoses                0.0.53             pyhd8ed1ab_0    conda-forge
setuptools                69.0.3             pyhd8ed1ab_0    conda-forge
shellingham               1.5.4              pyhd8ed1ab_0    conda-forge
six                       1.16.0             pyh6c4a22f_0    conda-forge
sleef                     3.5.1                h6db0672_2    conda-forge
smart_open                6.4.0              pyhd8ed1ab_0    conda-forge
spacy                     3.7.2           py310h65d09f4_0    conda-forge
spacy-legacy              3.0.12             pyhd8ed1ab_0    conda-forge
spacy-loggers             1.0.5              pyhd8ed1ab_0    conda-forge
srsly                     2.4.8           py310h9e9d8ca_1    conda-forge
stack_data                0.6.2              pyhd8ed1ab_0    conda-forge
sympy                     1.12            pypyh9d50eac_103    conda-forge
tbb                       2021.11.0            h7728843_1    conda-forge
thinc                     8.2.2           py310h076e4b7_0    conda-forge
tk                        8.6.13               h1abcd95_1    conda-forge
torch                     2.1.0.post103            pypi_0    pypi
torchtext                 0.15.2          py310h5de3785_4    conda-forge
tornado                   6.3.3           py310h6729b98_1    conda-forge
tqdm                      4.66.1             pyhd8ed1ab_0    conda-forge
traitlets                 5.14.1             pyhd8ed1ab_0    conda-forge
typer                     0.9.0              pyhd8ed1ab_0    conda-forge
typing-extensions         4.9.0                hd8ed1ab_0    conda-forge
typing_extensions         4.9.0              pyha770c72_0    conda-forge
tzdata                    2023d                h0c530f3_0    conda-forge
urllib3                   2.1.0              pyhd8ed1ab_0    conda-forge
wasabi                    1.1.2           py310h2ec42d9_0    conda-forge
wcwidth                   0.2.13             pyhd8ed1ab_0    conda-forge
weasel                    0.3.4              pyhd8ed1ab_0    conda-forge
wheel                     0.42.0             pyhd8ed1ab_0    conda-forge
xz                        5.2.6                h775f41a_0    conda-forge
zeromq                    4.3.5                h93d8f39_0    conda-forge
zipp                      3.17.0             pyhd8ed1ab_0    conda-forge

     active environment : ml2
    active env location : /Users/cecilia/miniconda3/envs/ml2
            shell level : 2
       user config file : /Users/cecilia/.condarc
 populated config files : /Users/cecilia/.condarc
          conda version : 23.11.0
    conda-build version : not installed
         python version : 3.11.6.final.0
                 solver : libmamba (default)
       virtual packages : __archspec=1=skylake
                          __conda=23.11.0=0
                          __osx=14.2.1=0
                          __unix=0=0
       base environment : /Users/cecilia/miniconda3  (writable)
      conda av data dir : /Users/cecilia/miniconda3/etc/conda
  conda av metadata url : None
           channel URLs : https://conda.anaconda.org/conda-forge/osx-64
                          https://conda.anaconda.org/conda-forge/noarch
                          https://repo.anaconda.com/pkgs/main/osx-64
                          https://repo.anaconda.com/pkgs/main/noarch
                          https://repo.anaconda.com/pkgs/r/osx-64
                          https://repo.anaconda.com/pkgs/r/noarch
          package cache : /Users/cecilia/miniconda3/pkgs
                          /Users/cecilia/.conda/pkgs
       envs directories : /Users/cecilia/miniconda3/envs
                          /Users/cecilia/.conda/envs
               platform : osx-64
             user-agent : conda/23.11.0 requests/2.31.0 CPython/3.11.6 Darwin/23.2.0 OSX/14.2.1 solver/libmamba conda-libmamba-solver/23.11.1 libmambapy/1.5.3
                UID:GID : 501:20
             netrc file : None
           offline mode : False

[joecummings]

Hi @cecilialee - unfortunately, TorchText is no longer actively maintained by PyTorch.

For NLP model implementations, data transforms, and datasets, I recommend HuggingFace's Transformers, Tokenizers, and Datasets libraries.