Helsinki-NLP / OpusFilter

OpusFilter - Parallel corpus processing toolkit
MIT License
102 stars 18 forks source link

Issue with opus-fast-mosestokenizer dep for ARM-macs #69

Open rggdmonk opened 8 months ago

rggdmonk commented 8 months ago

Hi!

I am currently experiencing an issue with the installation on M-series Macs (via pip install opusfilter).

Unfortunately, the installation process fails due to a problem with the opus-fast-mosestokenizer dependency.

Tested with Python 3.8/3.9/3.10/3.11

Some output:


  Building wheel for opus-fast-mosestokenizer (setup.py) ... error
  error: subprocess-exited-with-error

  × python setup.py bdist_wheel did not run successfully.
  │ exit code: 1
  ╰─> [117 lines of output]
      running bdist_wheel
      running build
      running build_py
      creating build
      creating build/lib.macosx-11.1-arm64-cpython-311
      creating build/lib.macosx-11.1-arm64-cpython-311/mosestokenizer
      copying bindings/python/mosestokenizer/__init__.py -> build/lib.macosx-11.1-arm64-cpython-311/mosestokenizer
      creating build/lib.macosx-11.1-arm64-cpython-311/mosestokenizer/share
      copying bindings/python/mosestokenizer/share/README.txt -> build/lib.macosx-11.1-arm64-cpython-311/mosestokenizer/share
      creating build/lib.macosx-11.1-arm64-cpython-311/mosestokenizer/share/nonbreaking_prefixes
      copying bindings/python/mosestokenizer/share/nonbreaking_prefixes/nonbreaking_prefix.hi -> build/lib.macosx-11.1-arm64-cpython-311/mosestokenizer/share/nonbreaking_prefixes
      copying bindings/python/mosestokenizer/share/nonbreaking_prefixes/nonbreaking_prefix.as -> build/lib.macosx-11.1-arm64-cpython-311/mosestokenizer/share/nonbreaking_prefixes
      copying bindings/python/mosestokenizer/share/nonbreaking_prefixes/nonbreaking_prefix.de -> build/lib.macosx-11.1-arm64-cpython-311/mosestokenizer/share/nonbreaking_prefixes
      copying bindings/python/mosestokenizer/share/nonbreaking_prefixes/nonbreaking_prefix.nl -> build/lib.macosx-11.1-arm64-cpython-311/mosestokenizer/share/nonbreaking_prefixes
      copying bindings/python/mosestokenizer/share/nonbreaking_prefixes/nonbreaking_prefix.hu -> build/lib.macosx-11.1-arm64-cpython-311/mosestokenizer/share/nonbreaking_prefixes
      copying bindings/python/mosestokenizer/share/nonbreaking_prefixes/nonbreaking_prefix.bn -> build/lib.macosx-11.1-arm64-cpython-311/mosestokenizer/share/nonbreaking_prefixes
      copying bindings/python/mosestokenizer/share/nonbreaking_prefixes/nonbreaking_prefix.kn -> build/lib.macosx-11.1-arm64-cpython-311/mosestokenizer/share/nonbreaking_prefixes
      copying bindings/python/mosestokenizer/share/nonbreaking_prefixes/nonbreaking_prefix.ml -> build/lib.macosx-11.1-arm64-cpython-311/mosestokenizer/share/nonbreaking_prefixes
      copying bindings/python/mosestokenizer/share/nonbreaking_prefixes/nonbreaking_prefix.yue -> build/lib.macosx-11.1-arm64-cpython-311/mosestokenizer/share/nonbreaking_prefixes
      copying bindings/python/mosestokenizer/share/nonbreaking_prefixes/nonbreaking_prefix.mni -> build/lib.macosx-11.1-arm64-cpython-311/mosestokenizer/share/nonbreaking_prefixes
      copying bindings/python/mosestokenizer/share/nonbreaking_prefixes/nonbreaking_prefix.pa -> build/lib.macosx-11.1-arm64-cpython-311/mosestokenizer/share/nonbreaking_prefixes
      copying bindings/python/mosestokenizer/share/nonbreaking_prefixes/nonbreaking_prefix.pt -> build/lib.macosx-11.1-arm64-cpython-311/mosestokenizer/share/nonbreaking_prefixes
      copying bindings/python/mosestokenizer/share/nonbreaking_prefixes/nonbreaking_prefix.zh -> build/lib.macosx-11.1-arm64-cpython-311/mosestokenizer/share/nonbreaking_prefixes
      copying bindings/python/mosestokenizer/share/nonbreaking_prefixes/nonbreaking_prefix.ta -> build/lib.macosx-11.1-arm64-cpython-311/mosestokenizer/share/nonbreaking_prefixes
      copying bindings/python/mosestokenizer/share/nonbreaking_prefixes/nonbreaking_prefix.en -> build/lib.macosx-11.1-arm64-cpython-311/mosestokenizer/share/nonbreaking_prefixes
      copying bindings/python/mosestokenizer/share/nonbreaking_prefixes/nonbreaking_prefix.lt -> build/lib.macosx-11.1-arm64-cpython-311/mosestokenizer/share/nonbreaking_prefixes
      copying bindings/python/mosestokenizer/share/nonbreaking_prefixes/nonbreaking_prefix.or -> build/lib.macosx-11.1-arm64-cpython-311/mosestokenizer/share/nonbreaking_prefixes
      copying bindings/python/mosestokenizer/share/nonbreaking_prefixes/nonbreaking_prefix.fr -> build/lib.macosx-11.1-arm64-cpython-311/mosestokenizer/share/nonbreaking_prefixes
      copying bindings/python/mosestokenizer/share/nonbreaking_prefixes/nonbreaking_prefix.es -> build/lib.macosx-11.1-arm64-cpython-311/mosestokenizer/share/nonbreaking_prefixes
      copying bindings/python/mosestokenizer/share/nonbreaking_prefixes/nonbreaking_prefix.et -> build/lib.macosx-11.1-arm64-cpython-311/mosestokenizer/share/nonbreaking_prefixes
      copying bindings/python/mosestokenizer/share/nonbreaking_prefixes/nonbreaking_prefix.fi -> build/lib.macosx-11.1-arm64-cpython-311/mosestokenizer/share/nonbreaking_prefixes
      copying bindings/python/mosestokenizer/share/nonbreaking_prefixes/nonbreaking_prefix.gu -> build/lib.macosx-11.1-arm64-cpython-311/mosestokenizer/share/nonbreaking_prefixes
      copying bindings/python/mosestokenizer/share/nonbreaking_prefixes/nonbreaking_prefix.mr -> build/lib.macosx-11.1-arm64-cpython-311/mosestokenizer/share/nonbreaking_prefixes
      copying bindings/python/mosestokenizer/share/nonbreaking_prefixes/nonbreaking_prefix.ga -> build/lib.macosx-11.1-arm64-cpython-311/mosestokenizer/share/nonbreaking_prefixes
      copying bindings/python/mosestokenizer/share/nonbreaking_prefixes/nonbreaking_prefix.sv -> build/lib.macosx-11.1-arm64-cpython-311/mosestokenizer/share/nonbreaking_prefixes
      copying bindings/python/mosestokenizer/share/nonbreaking_prefixes/nonbreaking_prefix.pl -> build/lib.macosx-11.1-arm64-cpython-311/mosestokenizer/share/nonbreaking_prefixes
      copying bindings/python/mosestokenizer/share/nonbreaking_prefixes/nonbreaking_prefix.sl -> build/lib.macosx-11.1-arm64-cpython-311/mosestokenizer/share/nonbreaking_prefixes
      copying bindings/python/mosestokenizer/share/nonbreaking_prefixes/nonbreaking_prefix.sk -> build/lib.macosx-11.1-arm64-cpython-311/mosestokenizer/share/nonbreaking_prefixes
      copying bindings/python/mosestokenizer/share/nonbreaking_prefixes/README.txt -> build/lib.macosx-11.1-arm64-cpython-311/mosestokenizer/share/nonbreaking_prefixes
      copying bindings/python/mosestokenizer/share/nonbreaking_prefixes/nonbreaking_prefix.ro -> build/lib.macosx-11.1-arm64-cpython-311/mosestokenizer/share/nonbreaking_prefixes
      copying bindings/python/mosestokenizer/share/nonbreaking_prefixes/nonbreaking_prefix.te -> build/lib.macosx-11.1-arm64-cpython-311/mosestokenizer/share/nonbreaking_prefixes
      copying bindings/python/mosestokenizer/share/nonbreaking_prefixes/nonbreaking_prefix.ru -> build/lib.macosx-11.1-arm64-cpython-311/mosestokenizer/share/nonbreaking_prefixes
      copying bindings/python/mosestokenizer/share/nonbreaking_prefixes/nonbreaking_prefix.cs -> build/lib.macosx-11.1-arm64-cpython-311/mosestokenizer/share/nonbreaking_prefixes
      copying bindings/python/mosestokenizer/share/nonbreaking_prefixes/nonbreaking_prefix.it -> build/lib.macosx-11.1-arm64-cpython-311/mosestokenizer/share/nonbreaking_prefixes
      copying bindings/python/mosestokenizer/share/nonbreaking_prefixes/nonbreaking_prefix.is -> build/lib.macosx-11.1-arm64-cpython-311/mosestokenizer/share/nonbreaking_prefixes
      copying bindings/python/mosestokenizer/share/nonbreaking_prefixes/nonbreaking_prefix.ca -> build/lib.macosx-11.1-arm64-cpython-311/mosestokenizer/share/nonbreaking_prefixes
      copying bindings/python/mosestokenizer/share/nonbreaking_prefixes/nonbreaking_prefix.el -> build/lib.macosx-11.1-arm64-cpython-311/mosestokenizer/share/nonbreaking_prefixes
      copying bindings/python/mosestokenizer/share/nonbreaking_prefixes/nonbreaking_prefix.lv -> build/lib.macosx-11.1-arm64-cpython-311/mosestokenizer/share/nonbreaking_prefixes
      running build_ext
      CMake Deprecation Warning at CMakeLists.txt:1 (cmake_minimum_required):
        Compatibility with CMake < 3.5 will be removed from a future version of
        CMake.

        Update the VERSION argument <min> value or use a ...<max> suffix to tell
        CMake that the project does not need compatibility with older versions.

      -- The CXX compiler identification is AppleClang 15.0.0.15000100
      -- Detecting CXX compiler ABI info
      -- Detecting CXX compiler ABI info - done
      -- Check for working CXX compiler: /Library/Developer/CommandLineTools/usr/bin/c++ - skipped
      -- Detecting CXX compile features
      -- Detecting CXX compile features - done
      -- Performing Test CMAKE_HAVE_LIBC_PTHREAD
      -- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Success
      -- Found Threads: TRUE
      CMake Error at /opt/homebrew/Cellar/cmake/3.28.3/share/cmake/Modules/FindPackageHandleStandardArgs.cmake:230 (message):
        Could NOT find Boost (missing: Boost_INCLUDE_DIR program_options thread)
      Call Stack (most recent call first):
        /opt/homebrew/Cellar/cmake/3.28.3/share/cmake/Modules/FindPackageHandleStandardArgs.cmake:600 (_FPHSA_FAILURE_MESSAGE)
        /opt/homebrew/Cellar/cmake/3.28.3/share/cmake/Modules/FindBoost.cmake:2393 (find_package_handle_standard_args)
        CMakeLists.txt:70 (find_package)

      -- Configuring incomplete, errors occurred!
      Traceback (most recent call last):
        File "<string>", line 2, in <module>
        File "<pip-setuptools-caller>", line 34, in <module>
        File "/private/var/folders/tg/b_nprk2142s5bh07pw240f3r0000gp/T/pip-install-sd9_6q0a/opus-fast-mosestokenizer_86290359b51c412aaad6b283997c5351/setup.py", line 107, in <module>
          setup(
        File "/Users/pstepach/miniconda3/envs/clear/lib/python3.11/site-packages/setuptools/__init__.py", line 103, in setup
          return distutils.core.setup(**attrs)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/Users/pstepach/miniconda3/envs/clear/lib/python3.11/site-packages/setuptools/_distutils/core.py", line 185, in setup
          return run_commands(dist)
                 ^^^^^^^^^^^^^^^^^^
        File "/Users/pstepach/miniconda3/envs/clear/lib/python3.11/site-packages/setuptools/_distutils/core.py", line 201, in run_commands
          dist.run_commands()
        File "/Users/pstepach/miniconda3/envs/clear/lib/python3.11/site-packages/setuptools/_distutils/dist.py", line 969, in run_commands
          self.run_command(cmd)
        File "/Users/pstepach/miniconda3/envs/clear/lib/python3.11/site-packages/setuptools/dist.py", line 989, in run_command
          super().run_command(command)
        File "/Users/pstepach/miniconda3/envs/clear/lib/python3.11/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
          cmd_obj.run()
        File "/Users/pstepach/miniconda3/envs/clear/lib/python3.11/site-packages/wheel/bdist_wheel.py", line 364, in run
          self.run_command("build")
        File "/Users/pstepach/miniconda3/envs/clear/lib/python3.11/site-packages/setuptools/_distutils/cmd.py", line 318, in run_command
          self.distribution.run_command(command)
        File "/Users/pstepach/miniconda3/envs/clear/lib/python3.11/site-packages/setuptools/dist.py", line 989, in run_command
          super().run_command(command)
        File "/Users/pstepach/miniconda3/envs/clear/lib/python3.11/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
          cmd_obj.run()
        File "/Users/pstepach/miniconda3/envs/clear/lib/python3.11/site-packages/setuptools/_distutils/command/build.py", line 131, in run
          self.run_command(cmd_name)
        File "/Users/pstepach/miniconda3/envs/clear/lib/python3.11/site-packages/setuptools/_distutils/cmd.py", line 318, in run_command
          self.distribution.run_command(command)
        File "/Users/pstepach/miniconda3/envs/clear/lib/python3.11/site-packages/setuptools/dist.py", line 989, in run_command
          super().run_command(command)
        File "/Users/pstepach/miniconda3/envs/clear/lib/python3.11/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
          cmd_obj.run()
        File "/private/var/folders/tg/b_nprk2142s5bh07pw240f3r0000gp/T/pip-install-sd9_6q0a/opus-fast-mosestokenizer_86290359b51c412aaad6b283997c5351/setup.py", line 53, in run
          self.build_extension(ext)
        File "/private/var/folders/tg/b_nprk2142s5bh07pw240f3r0000gp/T/pip-install-sd9_6q0a/opus-fast-mosestokenizer_86290359b51c412aaad6b283997c5351/setup.py", line 91, in build_extension
          subprocess.check_call(cmake_args, cwd=self.build_temp, env=env)
        File "/Users/pstepach/miniconda3/envs/clear/lib/python3.11/subprocess.py", line 413, in check_call
          raise CalledProcessError(retcode, cmd)
      subprocess.CalledProcessError: Command '['cmake', '/private/var/folders/tg/b_nprk2142s5bh07pw240f3r0000gp/T/pip-install-sd9_6q0a/opus-fast-mosestokenizer_86290359b51c412aaad6b283997c5351', '-DCMAKE_LIBRARY_OUTPUT_DIRECTORY=/private/var/folders/tg/b_nprk2142s5bh07pw240f3r0000gp/T/pip-install-sd9_6q0a/opus-fast-mosestokenizer_86290359b51c412aaad6b283997c5351/build/lib.macosx-11.1-arm64-cpython-311/mosestokenizer/lib/', '-DPYTHON_EXECUTABLE=/Users/pstepach/miniconda3/envs/clear/bin/python', '-DBUILD_CLI:BOOL=OFF', '-DBUILD_PYTHON:BOOL=ON', '-DBUILD_SHARED_LIBS:BOOL=ON', '-DCMAKE_BUILD_TYPE=Release']' returned non-zero exit status 1.
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for opus-fast-mosestokenizer
  Running setup.py clean for opus-fast-mosestokenizer
  Building wheel for langid (setup.py) ... done
  Created wheel for langid: filename=langid-1.1.6-py3-none-any.whl size=1941173 sha256=fb9479a4d2fdb43a32e92a248da54a802ec26086bea4c2a25ba98c612f77ac7a
  Stored in directory: /Users/pstepach/Library/Caches/pip/wheels/32/6a/b6/b7eb43a6ad55b139c15c5daa29f3707659cfa6944d3c696f5b
  Building wheel for pycld2 (setup.py) ... done
  Created wheel for pycld2: filename=pycld2-0.41-cp311-cp311-macosx_11_0_arm64.whl size=4816791 sha256=16ddfa06735a6e70169b323d9f67f81eebbee850b8fa524a5a8aa9d87140c69c
  Stored in directory: /Users/pstepach/Library/Caches/pip/wheels/9a/71/a1/89cdc6ae2ecd7bb3bf67baee27a134f761b3e2922ea7fdfbcb
Successfully built langid pycld2
Failed to build opus-fast-mosestokenizer
ERROR: Could not build wheels for opus-fast-mosestokenizer, which is required to install pyproject.toml-based projects```
svirpioj commented 8 months ago

There was a pending pull request on opus-fast-mosestokenizer related to this. It's now merged to the master branch, can you test if the change helps?

rggdmonk commented 8 months ago

Sure. I plan to test it later this week.

rggdmonk commented 8 months ago

Still issues. I tried https://github.com/Helsinki-NLP/opus-fast-mosestokenizer/blob/master/INSTALL.md#from-source

(arm_fix) ➜  opus-fast-mosestokenizer git:(master) python -c "import pybind11; print(pybind11.get_cmake_dir())"
/Users/pstepach/miniconda3/envs/arm_fix/lib/python3.10/site-packages/pybind11/share/cmake/pybind11

opus-fast-mosestokenizer git:(master) python setup.py build_ext install
running build_ext
CMake Deprecation Warning at CMakeLists.txt:1 (cmake_minimum_required):
  Compatibility with CMake < 3.5 will be removed from a future version of
  CMake.

  Update the VERSION argument <min> value or use a ...<max> suffix to tell
  CMake that the project does not need compatibility with older versions.

CMake Error at CMakeLists.txt:154 (find_package):
  By not providing "Findpybind11.cmake" in CMAKE_MODULE_PATH this project has
  asked CMake to find a package configuration file provided by "pybind11",
  but CMake did not find one.

  Could not find a package configuration file provided by "pybind11" with any
  of the following names:

    pybind11Config.cmake
    pybind11-config.cmake

  Add the installation prefix of "pybind11" to CMAKE_PREFIX_PATH or set
  "pybind11_DIR" to a directory containing one of the above files.  If
  "pybind11" provides a separate development package or SDK, be sure it has
  been installed.

-- Configuring incomplete, errors occurred!
Traceback (most recent call last):
  File "/Users/pstepach/Desktop/WORKING_FOLDER/arm_fix/opus-fast-mosestokenizer/setup.py", line 107, in <module>
    setup(
  File "/Users/pstepach/miniconda3/envs/arm_fix/lib/python3.10/site-packages/setuptools/__init__.py", line 103, in setup
    return distutils.core.setup(**attrs)
  File "/Users/pstepach/miniconda3/envs/arm_fix/lib/python3.10/site-packages/setuptools/_distutils/core.py", line 185, in setup
    return run_commands(dist)
  File "/Users/pstepach/miniconda3/envs/arm_fix/lib/python3.10/site-packages/setuptools/_distutils/core.py", line 201, in run_commands
    dist.run_commands()
  File "/Users/pstepach/miniconda3/envs/arm_fix/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 969, in run_commands
    self.run_command(cmd)
  File "/Users/pstepach/miniconda3/envs/arm_fix/lib/python3.10/site-packages/setuptools/dist.py", line 989, in run_command
    super().run_command(command)
  File "/Users/pstepach/miniconda3/envs/arm_fix/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
    cmd_obj.run()
  File "/Users/pstepach/Desktop/WORKING_FOLDER/arm_fix/opus-fast-mosestokenizer/setup.py", line 53, in run
    self.build_extension(ext)
  File "/Users/pstepach/Desktop/WORKING_FOLDER/arm_fix/opus-fast-mosestokenizer/setup.py", line 91, in build_extension
    subprocess.check_call(cmake_args, cwd=self.build_temp, env=env)
  File "/Users/pstepach/miniconda3/envs/arm_fix/lib/python3.10/subprocess.py", line 369, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['cmake', '/Users/pstepach/Desktop/WORKING_FOLDER/arm_fix/opus-fast-mosestokenizer', '-DCMAKE_LIBRARY_OUTPUT_DIRECTORY=/Users/pstepach/Desktop/WORKING_FOLDER/arm_fix/opus-fast-mosestokenizer/build/lib.macosx-11.1-arm64-cpython-310/mosestokenizer/lib/', '-DPYTHON_EXECUTABLE=/Users/pstepach/miniconda3/envs/arm_fix/bin/python', '-DBUILD_CLI:BOOL=OFF', '-DBUILD_PYTHON:BOOL=ON', '-DBUILD_SHARED_LIBS:BOOL=ON', '-DCMAKE_BUILD_TYPE=Release']' returned non-zero exit status 1.```