installing on mac silicon py 3.11.4 - sentencepiece building

[willwade]

I cant install espnet_model_zoo as I get this sentencepiece building problem. What version of sentencepiece is this using? I cant figure it out.. as setup.py doesnt have a fixed version.

I have successfully installed pip install sentencepiece - so its installed. All I can think is this requires a different version..

pip install espnet_model_zoo
Collecting espnet_model_zoo
  Using cached espnet_model_zoo-0.1.7-py3-none-any.whl.metadata (10 kB)
Requirement already satisfied: pandas in /Users/willwade/.pyenv/versions/3.11.4/lib/python3.11/site-packages (from espnet_model_zoo) (2.0.3)
Requirement already satisfied: requests in /Users/willwade/.pyenv/versions/3.11.4/lib/python3.11/site-packages (from espnet_model_zoo) (2.31.0)
Requirement already satisfied: tqdm in /Users/willwade/.pyenv/versions/3.11.4/lib/python3.11/site-packages (from espnet_model_zoo) (4.66.1)
Requirement already satisfied: numpy in /Users/willwade/.pyenv/versions/3.11.4/lib/python3.11/site-packages (from espnet_model_zoo) (1.24.3)
Collecting espnet (from espnet_model_zoo)
  Using cached espnet-202402-py3-none-any.whl.metadata (68 kB)
Requirement already satisfied: huggingface-hub in /Users/willwade/.pyenv/versions/3.11.4/lib/python3.11/site-packages (from espnet_model_zoo) (0.22.2)
Requirement already satisfied: filelock in /Users/willwade/.pyenv/versions/3.11.4/lib/python3.11/site-packages (from espnet_model_zoo) (3.12.2)
Requirement already satisfied: setuptools>=38.5.1 in /Users/willwade/.pyenv/versions/3.11.4/lib/python3.11/site-packages (from espnet->espnet_model_zoo) (69.5.1)
Requirement already satisfied: packaging in /Users/willwade/.pyenv/versions/3.11.4/lib/python3.11/site-packages (from espnet->espnet_model_zoo) (23.1)
Collecting configargparse>=1.2.1 (from espnet->espnet_model_zoo)
  Using cached ConfigArgParse-1.7-py3-none-any.whl.metadata (23 kB)
Collecting typeguard==2.13.3 (from espnet->espnet_model_zoo)
  Using cached typeguard-2.13.3-py3-none-any.whl.metadata (3.6 kB)
Collecting humanfriendly (from espnet->espnet_model_zoo)
  Using cached humanfriendly-10.0-py2.py3-none-any.whl.metadata (9.2 kB)
Requirement already satisfied: scipy>=1.4.1 in /Users/willwade/.pyenv/versions/3.11.4/lib/python3.11/site-packages (from espnet->espnet_model_zoo) (1.11.2)
Collecting librosa==0.9.2 (from espnet->espnet_model_zoo)
  Using cached librosa-0.9.2-py3-none-any.whl.metadata (8.2 kB)
Requirement already satisfied: jamo==0.4.1 in /Users/willwade/.pyenv/versions/3.11.4/lib/python3.11/site-packages (from espnet->espnet_model_zoo) (0.4.1)
Requirement already satisfied: PyYAML>=5.1.2 in /Users/willwade/.pyenv/versions/3.11.4/lib/python3.11/site-packages (from espnet->espnet_model_zoo) (6.0.1)
Requirement already satisfied: soundfile>=0.10.2 in /Users/willwade/.pyenv/versions/3.11.4/lib/python3.11/site-packages (from espnet->espnet_model_zoo) (0.12.1)
Collecting h5py>=2.10.0 (from espnet->espnet_model_zoo)
  Using cached h5py-3.11.0-cp311-cp311-macosx_11_0_arm64.whl.metadata (2.5 kB)
Collecting kaldiio>=2.18.0 (from espnet->espnet_model_zoo)
  Using cached kaldiio-2.18.0-py3-none-any.whl.metadata (13 kB)
Requirement already satisfied: torch>=1.11.0 in /Users/willwade/.pyenv/versions/3.11.4/lib/python3.11/site-packages (from espnet->espnet_model_zoo) (2.1.0)
Collecting torch-complex (from espnet->espnet_model_zoo)
  Using cached torch_complex-0.4.3-py3-none-any.whl.metadata (3.0 kB)
Requirement already satisfied: nltk>=3.4.5 in /Users/willwade/.pyenv/versions/3.11.4/lib/python3.11/site-packages (from espnet->espnet_model_zoo) (3.8.1)
Collecting numpy (from espnet_model_zoo)
  Using cached numpy-1.23.5-cp311-cp311-macosx_11_0_arm64.whl.metadata (2.3 kB)
Requirement already satisfied: protobuf in /Users/willwade/.pyenv/versions/3.11.4/lib/python3.11/site-packages (from espnet->espnet_model_zoo) (4.24.1)
Collecting hydra-core (from espnet->espnet_model_zoo)
  Using cached hydra_core-1.3.2-py3-none-any.whl.metadata (5.5 kB)
Collecting opt-einsum (from espnet->espnet_model_zoo)
  Using cached opt_einsum-3.3.0-py3-none-any.whl.metadata (6.5 kB)
Collecting sentencepiece==0.1.97 (from espnet->espnet_model_zoo)
  Using cached sentencepiece-0.1.97.tar.gz (524 kB)
  Preparing metadata (setup.py) ... done
Collecting ctc-segmentation>=1.6.6 (from espnet->espnet_model_zoo)
  Using cached ctc_segmentation-1.7.4-cp311-cp311-macosx_13_0_arm64.whl
Collecting pyworld>=0.3.4 (from espnet->espnet_model_zoo)
  Using cached pyworld-0.3.4-cp311-cp311-macosx_13_0_arm64.whl
Collecting pypinyin<=0.44.0 (from espnet->espnet_model_zoo)
  Using cached pypinyin-0.44.0-py2.py3-none-any.whl.metadata (10 kB)
Collecting espnet-tts-frontend (from espnet->espnet_model_zoo)
  Using cached espnet_tts_frontend-0.0.3-py3-none-any.whl.metadata (3.4 kB)
Collecting ci-sdr (from espnet->espnet_model_zoo)
  Using cached ci_sdr-0.0.2-py3-none-any.whl
Collecting fast-bss-eval==0.1.3 (from espnet->espnet_model_zoo)
  Using cached fast_bss_eval-0.1.3-py3-none-any.whl
Requirement already satisfied: asteroid-filterbanks==0.4.0 in /Users/willwade/.pyenv/versions/3.11.4/lib/python3.11/site-packages (from espnet->espnet_model_zoo) (0.4.0)
Collecting editdistance (from espnet->espnet_model_zoo)
  Using cached editdistance-0.8.1-cp311-cp311-macosx_11_0_arm64.whl.metadata (3.9 kB)
Collecting importlib-metadata<5.0 (from espnet->espnet_model_zoo)
  Using cached importlib_metadata-4.13.0-py3-none-any.whl.metadata (4.9 kB)
Requirement already satisfied: typing-extensions in /Users/willwade/.pyenv/versions/3.11.4/lib/python3.11/site-packages (from asteroid-filterbanks==0.4.0->espnet->espnet_model_zoo) (4.9.0)
Requirement already satisfied: audioread>=2.1.9 in /Users/willwade/.pyenv/versions/3.11.4/lib/python3.11/site-packages (from librosa==0.9.2->espnet->espnet_model_zoo) (3.0.0)
Requirement already satisfied: scikit-learn>=0.19.1 in /Users/willwade/.pyenv/versions/3.11.4/lib/python3.11/site-packages (from librosa==0.9.2->espnet->espnet_model_zoo) (1.3.0)
Requirement already satisfied: joblib>=0.14 in /Users/willwade/.pyenv/versions/3.11.4/lib/python3.11/site-packages (from librosa==0.9.2->espnet->espnet_model_zoo) (1.3.2)
Requirement already satisfied: decorator>=4.0.10 in /Users/willwade/.pyenv/versions/3.11.4/lib/python3.11/site-packages (from librosa==0.9.2->espnet->espnet_model_zoo) (4.4.2)
Collecting resampy>=0.2.2 (from librosa==0.9.2->espnet->espnet_model_zoo)
  Using cached resampy-0.4.3-py3-none-any.whl.metadata (3.0 kB)
Requirement already satisfied: numba>=0.45.1 in /Users/willwade/.pyenv/versions/3.11.4/lib/python3.11/site-packages (from librosa==0.9.2->espnet->espnet_model_zoo) (0.57.0)
Requirement already satisfied: pooch>=1.0 in /Users/willwade/.pyenv/versions/3.11.4/lib/python3.11/site-packages (from librosa==0.9.2->espnet->espnet_model_zoo) (1.6.0)
Requirement already satisfied: fsspec>=2023.5.0 in /Users/willwade/.pyenv/versions/3.11.4/lib/python3.11/site-packages (from huggingface-hub->espnet_model_zoo) (2023.6.0)
Requirement already satisfied: python-dateutil>=2.8.2 in /Users/willwade/.pyenv/versions/3.11.4/lib/python3.11/site-packages (from pandas->espnet_model_zoo) (2.8.2)
Requirement already satisfied: pytz>=2020.1 in /Users/willwade/.pyenv/versions/3.11.4/lib/python3.11/site-packages (from pandas->espnet_model_zoo) (2023.3)
Requirement already satisfied: tzdata>=2022.1 in /Users/willwade/.pyenv/versions/3.11.4/lib/python3.11/site-packages (from pandas->espnet_model_zoo) (2023.3)
Requirement already satisfied: charset-normalizer<4,>=2 in /Users/willwade/.pyenv/versions/3.11.4/lib/python3.11/site-packages (from requests->espnet_model_zoo) (3.1.0)
Requirement already satisfied: idna<4,>=2.5 in /Users/willwade/.pyenv/versions/3.11.4/lib/python3.11/site-packages (from requests->espnet_model_zoo) (3.4)
Requirement already satisfied: urllib3<3,>=1.21.1 in /Users/willwade/.pyenv/versions/3.11.4/lib/python3.11/site-packages (from requests->espnet_model_zoo) (2.2.1)
Requirement already satisfied: certifi>=2017.4.17 in /Users/willwade/.pyenv/versions/3.11.4/lib/python3.11/site-packages (from requests->espnet_model_zoo) (2023.7.22)
Requirement already satisfied: Cython in /Users/willwade/.pyenv/versions/3.11.4/lib/python3.11/site-packages (from ctc-segmentation>=1.6.6->espnet->espnet_model_zoo) (0.29.30)
Collecting zipp>=0.5 (from importlib-metadata<5.0->espnet->espnet_model_zoo)
  Using cached zipp-3.18.1-py3-none-any.whl.metadata (3.5 kB)
Requirement already satisfied: click in /Users/willwade/.pyenv/versions/3.11.4/lib/python3.11/site-packages (from nltk>=3.4.5->espnet->espnet_model_zoo) (8.1.7)
Requirement already satisfied: regex>=2021.8.3 in /Users/willwade/.pyenv/versions/3.11.4/lib/python3.11/site-packages (from nltk>=3.4.5->espnet->espnet_model_zoo) (2023.8.8)
Requirement already satisfied: six>=1.5 in /Users/willwade/.pyenv/versions/3.11.4/lib/python3.11/site-packages (from python-dateutil>=2.8.2->pandas->espnet_model_zoo) (1.16.0)
Requirement already satisfied: cffi>=1.0 in /Users/willwade/.pyenv/versions/3.11.4/lib/python3.11/site-packages (from soundfile>=0.10.2->espnet->espnet_model_zoo) (1.15.1)
Requirement already satisfied: sympy in /Users/willwade/.pyenv/versions/3.11.4/lib/python3.11/site-packages (from torch>=1.11.0->espnet->espnet_model_zoo) (1.12)
Requirement already satisfied: networkx in /Users/willwade/.pyenv/versions/3.11.4/lib/python3.11/site-packages (from torch>=1.11.0->espnet->espnet_model_zoo) (2.8.8)
Requirement already satisfied: jinja2 in /Users/willwade/.pyenv/versions/3.11.4/lib/python3.11/site-packages (from torch>=1.11.0->espnet->espnet_model_zoo) (3.1.2)
Requirement already satisfied: einops in /Users/willwade/.pyenv/versions/3.11.4/lib/python3.11/site-packages (from ci-sdr->espnet->espnet_model_zoo) (0.6.1)
Collecting unidecode>=1.0.22 (from espnet-tts-frontend->espnet->espnet_model_zoo)
  Using cached Unidecode-1.3.8-py3-none-any.whl.metadata (13 kB)
Requirement already satisfied: inflect>=1.0.0 in /Users/willwade/.pyenv/versions/3.11.4/lib/python3.11/site-packages (from espnet-tts-frontend->espnet->espnet_model_zoo) (5.6.0)
Collecting jaconv (from espnet-tts-frontend->espnet->espnet_model_zoo)
  Using cached jaconv-0.3.4-py3-none-any.whl
Collecting g2p-en (from espnet-tts-frontend->espnet->espnet_model_zoo)
  Using cached g2p_en-2.1.0-py3-none-any.whl.metadata (4.5 kB)
Requirement already satisfied: omegaconf<2.4,>=2.2 in /Users/willwade/.pyenv/versions/3.11.4/lib/python3.11/site-packages (from hydra-core->espnet->espnet_model_zoo) (2.3.0)
Requirement already satisfied: antlr4-python3-runtime==4.9.* in /Users/willwade/.pyenv/versions/3.11.4/lib/python3.11/site-packages (from hydra-core->espnet->espnet_model_zoo) (4.9.3)
Requirement already satisfied: pycparser in /Users/willwade/.pyenv/versions/3.11.4/lib/python3.11/site-packages (from cffi>=1.0->soundfile>=0.10.2->espnet->espnet_model_zoo) (2.21)
Requirement already satisfied: llvmlite<0.41,>=0.40.0dev0 in /Users/willwade/.pyenv/versions/3.11.4/lib/python3.11/site-packages (from numba>=0.45.1->librosa==0.9.2->espnet->espnet_model_zoo) (0.40.1)
Requirement already satisfied: appdirs>=1.3.0 in /Users/willwade/.pyenv/versions/3.11.4/lib/python3.11/site-packages (from pooch>=1.0->librosa==0.9.2->espnet->espnet_model_zoo) (1.4.4)
Requirement already satisfied: threadpoolctl>=2.0.0 in /Users/willwade/.pyenv/versions/3.11.4/lib/python3.11/site-packages (from scikit-learn>=0.19.1->librosa==0.9.2->espnet->espnet_model_zoo) (3.2.0)
Collecting distance>=0.1.3 (from g2p-en->espnet-tts-frontend->espnet->espnet_model_zoo)
  Using cached Distance-0.1.3-py3-none-any.whl
Requirement already satisfied: MarkupSafe>=2.0 in /Users/willwade/.pyenv/versions/3.11.4/lib/python3.11/site-packages (from jinja2->torch>=1.11.0->espnet->espnet_model_zoo) (2.1.3)
Requirement already satisfied: mpmath>=0.19 in /Users/willwade/.pyenv/versions/3.11.4/lib/python3.11/site-packages (from sympy->torch>=1.11.0->espnet->espnet_model_zoo) (1.3.0)
Using cached espnet_model_zoo-0.1.7-py3-none-any.whl (19 kB)
Using cached espnet-202402-py3-none-any.whl (1.8 MB)
Using cached librosa-0.9.2-py3-none-any.whl (214 kB)
Using cached typeguard-2.13.3-py3-none-any.whl (17 kB)
Using cached numpy-1.23.5-cp311-cp311-macosx_11_0_arm64.whl (13.3 MB)
Using cached ConfigArgParse-1.7-py3-none-any.whl (25 kB)
Using cached h5py-3.11.0-cp311-cp311-macosx_11_0_arm64.whl (2.9 MB)
Using cached importlib_metadata-4.13.0-py3-none-any.whl (23 kB)
Using cached kaldiio-2.18.0-py3-none-any.whl (28 kB)
Using cached pypinyin-0.44.0-py2.py3-none-any.whl (1.3 MB)
Using cached editdistance-0.8.1-cp311-cp311-macosx_11_0_arm64.whl (79 kB)
Using cached espnet_tts_frontend-0.0.3-py3-none-any.whl (11 kB)
Using cached humanfriendly-10.0-py2.py3-none-any.whl (86 kB)
Using cached hydra_core-1.3.2-py3-none-any.whl (154 kB)
Using cached opt_einsum-3.3.0-py3-none-any.whl (65 kB)
Using cached torch_complex-0.4.3-py3-none-any.whl (9.1 kB)
Using cached resampy-0.4.3-py3-none-any.whl (3.1 MB)
Using cached Unidecode-1.3.8-py3-none-any.whl (235 kB)
Using cached zipp-3.18.1-py3-none-any.whl (8.2 kB)
Using cached g2p_en-2.1.0-py3-none-any.whl (3.1 MB)
Building wheels for collected packages: sentencepiece
  Building wheel for sentencepiece (setup.py) ... error
  error: subprocess-exited-with-error

  × python setup.py bdist_wheel did not run successfully.
  │ exit code: 1
  ╰─> [88 lines of output]
      running bdist_wheel
      running build
      running build_py
      creating build
      creating build/lib.macosx-13.4-arm64-cpython-311
      creating build/lib.macosx-13.4-arm64-cpython-311/sentencepiece
      copying src/sentencepiece/__init__.py -> build/lib.macosx-13.4-arm64-cpython-311/sentencepiece
      copying src/sentencepiece/_version.py -> build/lib.macosx-13.4-arm64-cpython-311/sentencepiece
      copying src/sentencepiece/sentencepiece_model_pb2.py -> build/lib.macosx-13.4-arm64-cpython-311/sentencepiece
      copying src/sentencepiece/sentencepiece_pb2.py -> build/lib.macosx-13.4-arm64-cpython-311/sentencepiece
      running build_ext
      Package sentencepiece was not found in the pkg-config search path.
      Perhaps you should add the directory containing `sentencepiece.pc'
      to the PKG_CONFIG_PATH environment variable
      No package 'sentencepiece' found
      Cloning into 'sentencepiece'...
      Note: switching to '58f256cf6f01bb86e6fa634a5cc560de5bd1667d'.

      You are in 'detached HEAD' state. You can look around, make experimental
      changes and commit them, and you can discard any commits you make in this
      state without impacting any branches by switching back to a branch.

      If you want to create a new branch to retain commits you create, you may
      do so (now or later) by using -c with the switch command. Example:

        git switch -c <new-branch-name>

      Or undo this operation with:

        git switch -

      Turn off this advice by setting config variable advice.detachedHead to false

      ./build_bundled.sh: line 19: cmake: command not found
      ./build_bundled.sh: line 20: nproc: command not found
      ./build_bundled.sh: line 20: cmake: command not found
      Traceback (most recent call last):
        File "<string>", line 2, in <module>
        File "<pip-setuptools-caller>", line 34, in <module>
        File "/private/var/folders/y8/jgtt0gmx5z54mbjd0zx_hb8m0000gn/T/pip-install-2zqoh_nv/sentencepiece_ad33f7f96f2a4cc08ae00c731de66ee4/setup.py", line 136, in <module>
          setup(
        File "/Users/willwade/.pyenv/versions/3.11.4/lib/python3.11/site-packages/setuptools/__init__.py", line 104, in setup
          return distutils.core.setup(**attrs)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/Users/willwade/.pyenv/versions/3.11.4/lib/python3.11/site-packages/setuptools/_distutils/core.py", line 184, in setup
          return run_commands(dist)
                 ^^^^^^^^^^^^^^^^^^
        File "/Users/willwade/.pyenv/versions/3.11.4/lib/python3.11/site-packages/setuptools/_distutils/core.py", line 200, in run_commands
          dist.run_commands()
        File "/Users/willwade/.pyenv/versions/3.11.4/lib/python3.11/site-packages/setuptools/_distutils/dist.py", line 969, in run_commands
          self.run_command(cmd)
        File "/Users/willwade/.pyenv/versions/3.11.4/lib/python3.11/site-packages/setuptools/dist.py", line 967, in run_command
          super().run_command(command)
        File "/Users/willwade/.pyenv/versions/3.11.4/lib/python3.11/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
          cmd_obj.run()
        File "/Users/willwade/.pyenv/versions/3.11.4/lib/python3.11/site-packages/wheel/bdist_wheel.py", line 368, in run
          self.run_command("build")
        File "/Users/willwade/.pyenv/versions/3.11.4/lib/python3.11/site-packages/setuptools/_distutils/cmd.py", line 316, in run_command
          self.distribution.run_command(command)
        File "/Users/willwade/.pyenv/versions/3.11.4/lib/python3.11/site-packages/setuptools/dist.py", line 967, in run_command
          super().run_command(command)
        File "/Users/willwade/.pyenv/versions/3.11.4/lib/python3.11/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
          cmd_obj.run()
        File "/Users/willwade/.pyenv/versions/3.11.4/lib/python3.11/site-packages/setuptools/_distutils/command/build.py", line 132, in run
          self.run_command(cmd_name)
        File "/Users/willwade/.pyenv/versions/3.11.4/lib/python3.11/site-packages/setuptools/_distutils/cmd.py", line 316, in run_command
          self.distribution.run_command(command)
        File "/Users/willwade/.pyenv/versions/3.11.4/lib/python3.11/site-packages/setuptools/dist.py", line 967, in run_command
          super().run_command(command)
        File "/Users/willwade/.pyenv/versions/3.11.4/lib/python3.11/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
          cmd_obj.run()
        File "/Users/willwade/.pyenv/versions/3.11.4/lib/python3.11/site-packages/setuptools/command/build_ext.py", line 91, in run
          _build_ext.run(self)
        File "/Users/willwade/.pyenv/versions/3.11.4/lib/python3.11/site-packages/Cython/Distutils/old_build_ext.py", line 186, in run
          _build_ext.build_ext.run(self)
        File "/Users/willwade/.pyenv/versions/3.11.4/lib/python3.11/site-packages/setuptools/_distutils/command/build_ext.py", line 359, in run
          self.build_extensions()
        File "/Users/willwade/.pyenv/versions/3.11.4/lib/python3.11/site-packages/Cython/Distutils/old_build_ext.py", line 195, in build_extensions
          _build_ext.build_ext.build_extensions(self)
        File "/Users/willwade/.pyenv/versions/3.11.4/lib/python3.11/site-packages/setuptools/_distutils/command/build_ext.py", line 479, in build_extensions
          self._build_extensions_serial()
        File "/Users/willwade/.pyenv/versions/3.11.4/lib/python3.11/site-packages/setuptools/_distutils/command/build_ext.py", line 505, in _build_extensions_serial
          self.build_extension(ext)
        File "/private/var/folders/y8/jgtt0gmx5z54mbjd0zx_hb8m0000gn/T/pip-install-2zqoh_nv/sentencepiece_ad33f7f96f2a4cc08ae00c731de66ee4/setup.py", line 89, in build_extension
          subprocess.check_call(['./build_bundled.sh', __version__])
        File "/Users/willwade/.pyenv/versions/3.11.4/lib/python3.11/subprocess.py", line 413, in check_call
          raise CalledProcessError(retcode, cmd)
      subprocess.CalledProcessError: Command '['./build_bundled.sh', '0.1.97']' returned non-zero exit status 127.
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for sentencepiece
  Running setup.py clean for sentencepiece
Failed to build sentencepiece
ERROR: Could not build wheels for sentencepiece, which is required to install pyproject.toml-based projects

[retoreQAQ]

try to down your python version to 3.10, I met the same problem and found it's because python3.11 so I created a new conda env and solved it.