search

kylebgorman / pynini

Read-only mirror of Pynini
http://pynini.opengrm.org
Apache License 2.0
118 stars 27 forks source link

pip install fails on Mac OS 13.5.1 #69

Closed schar closed 3 months ago

schar commented 11 months ago

Hi Kyle! I'm trying to install Pynini 2.1.4. I'm working on something that requires OpenFST 1.8.1, so 2.1.5 is out (though I can install 2.1.5 fine from conda-forge). I'm running into difficulty with pip install pynini==2.1.4. I don't see a way to install 2.1.4 from conda-forge.

This (pip installing 2.1.4) used to work on our M1 macs, so I'm wondering if something has changed in Cython or elsewhere that causes this issue.

(slu) ➜  pynini-2.1.4 pip install pynini==2.1.4
Collecting pynini==2.1.4
  Using cached pynini-2.1.4.tar.gz (621 kB)
  Preparing metadata (setup.py) ... done
Requirement already satisfied: Cython>=0.29 in /Users/Simon.Charlow/opt/miniconda3/envs/slu/lib/python3.8/site-packages (from pynini==2.1.4) (3.0.2)
Building wheels for collected packages: pynini
  Building wheel for pynini (setup.py) ... error
  error: subprocess-exited-with-error

  × python setup.py bdist_wheel did not run successfully.
  │ exit code: 1
  ╰─> [117 lines of output]
      running bdist_wheel
      running build
      running build_py
      creating build
      creating build/lib.macosx-11.0-arm64-cpython-38
      creating build/lib.macosx-11.0-arm64-cpython-38/pywrapfst
      copying pywrapfst/__init__.py -> build/lib.macosx-11.0-arm64-cpython-38/pywrapfst
      creating build/lib.macosx-11.0-arm64-cpython-38/pynini
      copying pynini/__init__.py -> build/lib.macosx-11.0-arm64-cpython-38/pynini
      creating build/lib.macosx-11.0-arm64-cpython-38/pynini/examples
      copying pynini/examples/chatspeak.py -> build/lib.macosx-11.0-arm64-cpython-38/pynini/examples
      copying pynini/examples/chatspeak_model.py -> build/lib.macosx-11.0-arm64-cpython-38/pynini/examples
      copying pynini/examples/weather.py -> build/lib.macosx-11.0-arm64-cpython-38/pynini/examples
      copying pynini/examples/__init__.py -> build/lib.macosx-11.0-arm64-cpython-38/pynini/examples
      copying pynini/examples/numbers.py -> build/lib.macosx-11.0-arm64-cpython-38/pynini/examples
      copying pynini/examples/plurals.py -> build/lib.macosx-11.0-arm64-cpython-38/pynini/examples
      copying pynini/examples/t9.py -> build/lib.macosx-11.0-arm64-cpython-38/pynini/examples
      copying pynini/examples/case.py -> build/lib.macosx-11.0-arm64-cpython-38/pynini/examples
      copying pynini/examples/dates.py -> build/lib.macosx-11.0-arm64-cpython-38/pynini/examples
      copying pynini/examples/g2p.py -> build/lib.macosx-11.0-arm64-cpython-38/pynini/examples
      creating build/lib.macosx-11.0-arm64-cpython-38/pynini/lib
      copying pynini/lib/edit_transducer.py -> build/lib.macosx-11.0-arm64-cpython-38/pynini/lib
      copying pynini/lib/utf8.py -> build/lib.macosx-11.0-arm64-cpython-38/pynini/lib
      copying pynini/lib/__init__.py -> build/lib.macosx-11.0-arm64-cpython-38/pynini/lib
      copying pynini/lib/features.py -> build/lib.macosx-11.0-arm64-cpython-38/pynini/lib
      copying pynini/lib/pynutil.py -> build/lib.macosx-11.0-arm64-cpython-38/pynini/lib
      copying pynini/lib/paradigms.py -> build/lib.macosx-11.0-arm64-cpython-38/pynini/lib
      copying pynini/lib/tagger.py -> build/lib.macosx-11.0-arm64-cpython-38/pynini/lib
      copying pynini/lib/rule_cascade.py -> build/lib.macosx-11.0-arm64-cpython-38/pynini/lib
      copying pynini/lib/rewrite.py -> build/lib.macosx-11.0-arm64-cpython-38/pynini/lib
      copying pynini/lib/byte.py -> build/lib.macosx-11.0-arm64-cpython-38/pynini/lib
      creating build/lib.macosx-11.0-arm64-cpython-38/pynini/export
      copying pynini/export/__init__.py -> build/lib.macosx-11.0-arm64-cpython-38/pynini/export
      copying pynini/export/export.py -> build/lib.macosx-11.0-arm64-cpython-38/pynini/export
      copying pynini/export/multi_grm_example.py -> build/lib.macosx-11.0-arm64-cpython-38/pynini/export
      copying pynini/export/multi_grm.py -> build/lib.macosx-11.0-arm64-cpython-38/pynini/export
      copying pynini/export/grm.py -> build/lib.macosx-11.0-arm64-cpython-38/pynini/export
      copying pynini/export/grm_example.py -> build/lib.macosx-11.0-arm64-cpython-38/pynini/export
      copying pywrapfst/__init__.pyi -> build/lib.macosx-11.0-arm64-cpython-38/pywrapfst
      copying pywrapfst/py.typed -> build/lib.macosx-11.0-arm64-cpython-38/pywrapfst
      copying pynini/__init__.pyi -> build/lib.macosx-11.0-arm64-cpython-38/pynini
      copying pynini/py.typed -> build/lib.macosx-11.0-arm64-cpython-38/pynini
      copying pynini/examples/py.typed -> build/lib.macosx-11.0-arm64-cpython-38/pynini/examples
      copying pynini/lib/py.typed -> build/lib.macosx-11.0-arm64-cpython-38/pynini/lib
      copying pynini/export/py.typed -> build/lib.macosx-11.0-arm64-cpython-38/pynini/export
      running build_ext
      building '_pywrapfst' extension
      creating build/temp.macosx-11.0-arm64-cpython-38
      creating build/temp.macosx-11.0-arm64-cpython-38/extensions
      clang -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -fwrapv -O2 -Wall -fPIC -O2 -isystem /Users/Simon.Charlow/opt/miniconda3/envs/slu/include -arch arm64 -fPIC -O2 -isystem /Users/Simon.Charlow/opt/miniconda3/envs/slu/include -arch arm64 -I/Users/Simon.Charlow/opt/miniconda3/envs/slu/include/python3.8 -c extensions/_pywrapfst.cpp -o build/temp.macosx-11.0-arm64-cpython-38/extensions/_pywrapfst.o -std=c++17 -Wno-register -Wno-deprecated-declarations -Wno-unused-function -Wno-unused-local-typedefs -funsigned-char -stdlib=libc++ -mmacosx-version-min=10.7
      In file included from extensions/_pywrapfst.cpp:1220:
      In file included from /Users/Simon.Charlow/work/slu/fst_tools/include/fst/fstlib.h:48:
      /Users/Simon.Charlow/work/slu/fst_tools/include/fst/const-fst.h:405:25: warning: comparison of integers of different signs: 'int64' (aka 'long long') and 'size_t' (aka 'unsigned long') [-Wsign-compare]
          if (hdr.NumStates() != num_states) {
              ~~~~~~~~~~~~~~~ ^  ~~~~~~~~~~
      /Users/Simon.Charlow/work/slu/fst_tools/include/fst/const-fst.h:409:23: warning: comparison of integers of different signs: 'int64' (aka 'long long') and 'size_t' (aka 'unsigned long') [-Wsign-compare]
          if (hdr.NumArcs() != num_arcs) {
              ~~~~~~~~~~~~~ ^  ~~~~~~~~
      extensions/_pywrapfst.cpp:11274:40: error: cannot cast from type 'std::string' (aka 'basic_string<char, char_traits<char>, allocator<char>>') to pointer type 'PyObject *' (aka '_object *')
        __pyx_t_2 = __Pyx_PyObject_AsDouble(((PyObject *)__pyx_t_1)); if (unlikely(__pyx_t_2 == ((double)((double)-1)) && PyErr_Occurred())) __PYX_ERR(0, 436, __pyx_L1_error)
                                             ^~~~~~~~~~~~~~~~~~~~~
      extensions/_pywrapfst.cpp:3302:29: note: expanded from macro '__Pyx_PyObject_AsDouble'
      ((likely(PyFloat_CheckExact(obj))) ?  PyFloat_AS_DOUBLE(obj) :\
                                  ^~~
      /Users/Simon.Charlow/opt/miniconda3/envs/slu/include/python3.8/floatobject.h:24:41: note: expanded from macro 'PyFloat_CheckExact'
      #define PyFloat_CheckExact(op) (Py_TYPE(op) == &PyFloat_Type)
                                              ^~
      /Users/Simon.Charlow/opt/miniconda3/envs/slu/include/python3.8/object.h:122:49: note: expanded from macro 'Py_TYPE'
      #define Py_TYPE(ob)             (_PyObject_CAST(ob)->ob_type)
                                                      ^~
      /Users/Simon.Charlow/opt/miniconda3/envs/slu/include/python3.8/object.h:111:41: note: expanded from macro '_PyObject_CAST'
      #define _PyObject_CAST(op) ((PyObject*)(op))
                                              ^~
      extensions/_pywrapfst.cpp:1490:43: note: expanded from macro 'likely'
        #define likely(x)   __builtin_expect(!!(x), 1)
                                                ^
      extensions/_pywrapfst.cpp:11274:40: error: cannot cast from type 'std::string' (aka 'basic_string<char, char_traits<char>, allocator<char>>') to pointer type 'PyObject *' (aka '_object *')
        __pyx_t_2 = __Pyx_PyObject_AsDouble(((PyObject *)__pyx_t_1)); if (unlikely(__pyx_t_2 == ((double)((double)-1)) && PyErr_Occurred())) __PYX_ERR(0, 436, __pyx_L1_error)
                                             ^~~~~~~~~~~~~~~~~~~~~
      extensions/_pywrapfst.cpp:3302:57: note: expanded from macro '__Pyx_PyObject_AsDouble'
      ((likely(PyFloat_CheckExact(obj))) ?  PyFloat_AS_DOUBLE(obj) :\
                                                              ^~~
      /Users/Simon.Charlow/opt/miniconda3/envs/slu/include/python3.8/floatobject.h:51:51: note: expanded from macro 'PyFloat_AS_DOUBLE'
      #define PyFloat_AS_DOUBLE(op) (((PyFloatObject *)(op))->ob_fval)
                                                        ^~
      extensions/_pywrapfst.cpp:11274:40: error: cannot cast from type 'std::string' (aka 'basic_string<char, char_traits<char>, allocator<char>>') to pointer type 'PyObject *' (aka '_object *')
        __pyx_t_2 = __Pyx_PyObject_AsDouble(((PyObject *)__pyx_t_1)); if (unlikely(__pyx_t_2 == ((double)((double)-1)) && PyErr_Occurred())) __PYX_ERR(0, 436, __pyx_L1_error)
                                             ^~~~~~~~~~~~~~~~~~~~~
      extensions/_pywrapfst.cpp:3303:27: note: expanded from macro '__Pyx_PyObject_AsDouble'
       likely(PyLong_CheckExact(obj)) ?\
                                ^~~
      /Users/Simon.Charlow/opt/miniconda3/envs/slu/include/python3.8/longobject.h:16:40: note: expanded from macro 'PyLong_CheckExact'
      #define PyLong_CheckExact(op) (Py_TYPE(op) == &PyLong_Type)
                                             ^~
      /Users/Simon.Charlow/opt/miniconda3/envs/slu/include/python3.8/object.h:122:49: note: expanded from macro 'Py_TYPE'
      #define Py_TYPE(ob)             (_PyObject_CAST(ob)->ob_type)
                                                      ^~
      /Users/Simon.Charlow/opt/miniconda3/envs/slu/include/python3.8/object.h:111:41: note: expanded from macro '_PyObject_CAST'
      #define _PyObject_CAST(op) ((PyObject*)(op))
                                              ^~
      extensions/_pywrapfst.cpp:1490:43: note: expanded from macro 'likely'
        #define likely(x)   __builtin_expect(!!(x), 1)
                                                ^
      extensions/_pywrapfst.cpp:11274:40: error: cannot cast from type 'std::string' (aka 'basic_string<char, char_traits<char>, allocator<char>>') to pointer type 'PyObject *' (aka '_object *')
        __pyx_t_2 = __Pyx_PyObject_AsDouble(((PyObject *)__pyx_t_1)); if (unlikely(__pyx_t_2 == ((double)((double)-1)) && PyErr_Occurred())) __PYX_ERR(0, 436, __pyx_L1_error)
                                             ^~~~~~~~~~~~~~~~~~~~~
      extensions/_pywrapfst.cpp:3304:18: note: expanded from macro '__Pyx_PyObject_AsDouble'
       PyLong_AsDouble(obj) : __Pyx__PyObject_AsDouble(obj))
                       ^~~
      extensions/_pywrapfst.cpp:11274:40: error: cannot cast from type 'std::string' (aka 'basic_string<char, char_traits<char>, allocator<char>>') to pointer type 'PyObject *' (aka '_object *')
        __pyx_t_2 = __Pyx_PyObject_AsDouble(((PyObject *)__pyx_t_1)); if (unlikely(__pyx_t_2 == ((double)((double)-1)) && PyErr_Occurred())) __PYX_ERR(0, 436, __pyx_L1_error)
                                             ^~~~~~~~~~~~~~~~~~~~~
      extensions/_pywrapfst.cpp:3304:50: note: expanded from macro '__Pyx_PyObject_AsDouble'
       PyLong_AsDouble(obj) : __Pyx__PyObject_AsDouble(obj))
                                                       ^~~
      2 warnings and 5 errors generated.
      error: command '/usr/bin/clang' failed with exit code 1
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for pynini
  Running setup.py clean for pynini
Failed to build pynini
ERROR: Could not build wheels for pynini, which is required to install pyproject.toml-based projects
kylebgorman commented 11 months ago

I am actually not sure, haven't seen this before, but building (or debugging) from source on a Mac is going to be difficult...hard to get an isolated environment that doesn't have something in the way. However, I did just test this and I believe it'll work on Mac too?

conda create -n simon pynini=2.1.4 -c conda-forge

I don't see a M1-specific version for that release, but I thought it could fall back to the OS X version---let me know if I'm wrong.

FWIW, Pynini 2.1.4 is pretty old, and the file serializations are pretty stable so they can be used across versions.

schar commented 11 months ago

Thanks for checking, and the quick reply! Unfortunately, it doesn't fall back:

(slu) ➜  ~ conda create -n simon pynini=2.1.4 -c conda-forge
Collecting package metadata (current_repodata.json): done
Solving environment: unsuccessful attempt using repodata from current_repodata.json, retrying with next repodata source.
Collecting package metadata (repodata.json): done
Solving environment: failed

PackagesNotFoundError: The following packages are not available from current channels:

  - pynini=2.1.4

Current channels:

  - https://conda.anaconda.org/conda-forge/osx-arm64
  - https://conda.anaconda.org/conda-forge/noarch
  - https://repo.anaconda.com/pkgs/main/osx-arm64
  - https://repo.anaconda.com/pkgs/main/noarch
  - https://repo.anaconda.com/pkgs/r/osx-arm64
  - https://repo.anaconda.com/pkgs/r/noarch

To search for alternate channels that may provide the conda package you're
looking for, navigate to

    https://anaconda.org

and use the search bar at the top of the page.
kylebgorman commented 11 months ago

Yeah, then I'm stuck for easy solutions. I would consider just using Pynini 2.1.5 in one environment and OpenFst 1.8.1 in the other? They can talk to each other through the file system. Is that feasible for your application?

On Conda-Forge's setup, I can always add M1 support to a package that doesn't have it (cf. this), but I don't think there's any way to add M1 support to older versions of packages that didn't have it at the time that version was superceded.

schar commented 11 months ago

There's definitely things that we can do, just wanted to see if this might be one of them. Thanks again for your quick and clear help.

kylebgorman commented 3 months ago

Okay if I close this @schar?

schar commented 3 months ago

Sure, thanks for your help