search

kylebgorman / pynini

Read-only mirror of Pynini
http://pynini.opengrm.org
Apache License 2.0
118 stars 27 forks source link

Adds a minimal pyproject.toml file #62

Closed cdeln closed 9 months ago

cdeln commented 1 year ago

Solves #30 Using pyproject.toml solves the bootstrap problem with Cython and makes the project build in isolation, as required by certain build systems (e.g. poetry).

kylebgorman commented 1 year ago

Thanks @cdeln. I have never been able to actually replicate #30 so this does help. I also don't really understand the PEP 518 rationale, but I guess that's the way things are headed.

We can't accept outside contributions here (this is just a mirror of code developed internally) but I will study your solution and work something similar into the next release.

cdeln commented 1 year ago

Yeah, the Python distribution ecosystem is a bit of a mess, with multiple systems supported at the same time. I see, I'll keep an eye open for a fix into the next release, and use my fork in the meantime to make my project build. Ping me if you want testing of the package (as seen from a package consumer, I can try it out in my project), prior to next release. Btw it's pretty easy to replicate #30, just create a new virtual env and run pip install . in the repo. In the fresh virtual env you wouldn't have Cython installed, so it fails when trying to import it in setup.py.

kylebgorman commented 1 year ago

Yeah, the Python distribution ecosystem is a bit of a mess, with multiple systems supported at the same time. I see, I'll keep an eye open for a fix into the next release, and use my fork in the meantime to make my project build. Ping me if you want testing of the package (as seen from a package consumer, I can try it out in my project), prior to next release.

Will do.

alealv commented 1 year ago

It still fails for me because it doesn't downloads openFST

╰─ pip install pynini
Collecting pynini
  Using cached pynini-2.1.5.tar.gz (627 kB)
  Preparing metadata (setup.py) ... done
Requirement already satisfied: Cython>=0.29 in /Users/aalvarez/.pyenv/versions/test/lib/python3.11/site-packages (from pynini) (0.29.33)
Installing collected packages: pynini
  DEPRECATION: pynini is being installed using the legacy 'setup.py install' method, because it does not have a 'pyproject.toml' and the 'wheel' package is not installed. pip 23.1 will enforce this behaviour change. A possible replacement is to enable the '--use-pep517' option. Discussion can be found at https://github.com/pypa/pip/issues/8559
  Running setup.py install for pynini ... error
  error: subprocess-exited-with-error

  × Running setup.py install for pynini did not run successfully.
  │ exit code: 1
  ╰─> [57 lines of output]
      running install
      /Users/aalvarez/.pyenv/versions/test/lib/python3.11/site-packages/setuptools/command/install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
        warnings.warn(
      running build
      running build_py
      creating build
      creating build/lib.macosx-13.2-x86_64-cpython-311
      creating build/lib.macosx-13.2-x86_64-cpython-311/pywrapfst
      copying pywrapfst/__init__.py -> build/lib.macosx-13.2-x86_64-cpython-311/pywrapfst
      creating build/lib.macosx-13.2-x86_64-cpython-311/pynini
      copying pynini/__init__.py -> build/lib.macosx-13.2-x86_64-cpython-311/pynini
      creating build/lib.macosx-13.2-x86_64-cpython-311/pynini/examples
      copying pynini/examples/chatspeak.py -> build/lib.macosx-13.2-x86_64-cpython-311/pynini/examples
      copying pynini/examples/chatspeak_model.py -> build/lib.macosx-13.2-x86_64-cpython-311/pynini/examples
      copying pynini/examples/weather.py -> build/lib.macosx-13.2-x86_64-cpython-311/pynini/examples
      copying pynini/examples/__init__.py -> build/lib.macosx-13.2-x86_64-cpython-311/pynini/examples
      copying pynini/examples/numbers.py -> build/lib.macosx-13.2-x86_64-cpython-311/pynini/examples
      copying pynini/examples/plurals.py -> build/lib.macosx-13.2-x86_64-cpython-311/pynini/examples
      copying pynini/examples/t9.py -> build/lib.macosx-13.2-x86_64-cpython-311/pynini/examples
      copying pynini/examples/case.py -> build/lib.macosx-13.2-x86_64-cpython-311/pynini/examples
      copying pynini/examples/dates.py -> build/lib.macosx-13.2-x86_64-cpython-311/pynini/examples
      copying pynini/examples/g2p.py -> build/lib.macosx-13.2-x86_64-cpython-311/pynini/examples
      creating build/lib.macosx-13.2-x86_64-cpython-311/pynini/lib
      copying pynini/lib/edit_transducer.py -> build/lib.macosx-13.2-x86_64-cpython-311/pynini/lib
      copying pynini/lib/utf8.py -> build/lib.macosx-13.2-x86_64-cpython-311/pynini/lib
      copying pynini/lib/__init__.py -> build/lib.macosx-13.2-x86_64-cpython-311/pynini/lib
      copying pynini/lib/features.py -> build/lib.macosx-13.2-x86_64-cpython-311/pynini/lib
      copying pynini/lib/pynutil.py -> build/lib.macosx-13.2-x86_64-cpython-311/pynini/lib
      copying pynini/lib/paradigms.py -> build/lib.macosx-13.2-x86_64-cpython-311/pynini/lib
      copying pynini/lib/tagger.py -> build/lib.macosx-13.2-x86_64-cpython-311/pynini/lib
      copying pynini/lib/rule_cascade.py -> build/lib.macosx-13.2-x86_64-cpython-311/pynini/lib
      copying pynini/lib/rewrite.py -> build/lib.macosx-13.2-x86_64-cpython-311/pynini/lib
      copying pynini/lib/byte.py -> build/lib.macosx-13.2-x86_64-cpython-311/pynini/lib
      creating build/lib.macosx-13.2-x86_64-cpython-311/pynini/export
      copying pynini/export/__init__.py -> build/lib.macosx-13.2-x86_64-cpython-311/pynini/export
      copying pynini/export/export.py -> build/lib.macosx-13.2-x86_64-cpython-311/pynini/export
      copying pynini/export/multi_grm_example.py -> build/lib.macosx-13.2-x86_64-cpython-311/pynini/export
      copying pynini/export/multi_grm.py -> build/lib.macosx-13.2-x86_64-cpython-311/pynini/export
      copying pynini/export/grm.py -> build/lib.macosx-13.2-x86_64-cpython-311/pynini/export
      copying pynini/export/grm_example.py -> build/lib.macosx-13.2-x86_64-cpython-311/pynini/export
      copying pywrapfst/__init__.pyi -> build/lib.macosx-13.2-x86_64-cpython-311/pywrapfst
      copying pywrapfst/py.typed -> build/lib.macosx-13.2-x86_64-cpython-311/pywrapfst
      copying pynini/__init__.pyi -> build/lib.macosx-13.2-x86_64-cpython-311/pynini
      copying pynini/py.typed -> build/lib.macosx-13.2-x86_64-cpython-311/pynini
      copying pynini/examples/py.typed -> build/lib.macosx-13.2-x86_64-cpython-311/pynini/examples
      copying pynini/lib/py.typed -> build/lib.macosx-13.2-x86_64-cpython-311/pynini/lib
      copying pynini/export/py.typed -> build/lib.macosx-13.2-x86_64-cpython-311/pynini/export
      running build_ext
      building '_pywrapfst' extension
      creating build/temp.macosx-13.2-x86_64-cpython-311
      creating build/temp.macosx-13.2-x86_64-cpython-311/extensions
      clang -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -I/Applications/Xcode-14.2.0.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include -I/Applications/Xcode-14.2.0.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include -I/Users/aalvarez/.pyenv/versions/test/include -I/Users/aalvarez/.pyenv/versions/3.11.2/include/python3.11 -c extensions/_pywrapfst.cpp -o build/temp.macosx-13.2-x86_64-cpython-311/extensions/_pywrapfst.o -std=c++17 -Wno-register -Wno-deprecated-declarations -Wno-unused-function -Wno-unused-local-typedefs -funsigned-char -stdlib=libc++ -mmacosx-version-min=10.7
      extensions/_pywrapfst.cpp:815:10: fatal error: 'fst/util.h' file not found
      #include <fst/util.h>
               ^~~~~~~~~~~~
      1 error generated.
      error: command '/usr/bin/clang' failed with exit code 1
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: legacy-install-failure

× Encountered error while trying to install package.
╰─> pynini

note: This is an issue with the package mentioned above, not pip.
hint: See above for output from the failure.
kylebgorman commented 1 year ago

@ Alejandro: It is extremely unlikely that we can ever make it actually install OpenFst (vendored or otherwise) via setup.py or pyproject.toml with any reliability. That is "Will Not Fix". I don't think it's possible without a Herculean effort. If you really want to install from source (and there is no good reason unless you're on an exotic platform), you just have to install OpenFst first using the normal ./configure; make -j; sudo make -install thing first.

I have provided mitigations: install via Conda on Linux, MacOS, or even Windows, or on Linux x86-64, use the binary wheel we provide on PyPI. I already put over several engineer-weeks into getting those two to work, I think what we have already represents adequate solutions, and since you're on MacOS, just use Conda.

I am not sure why you are posting abouit this here: your issue (you're trying to install from source on a platform where we explicitly discourage source installation and provide pre-packaged binaries via Conda) is unrelated to the bug.

On Mon, Mar 20, 2023 at 11:02 AM Alejandro Gastón Alvarez < @.***> wrote:

It still fails for me because it doesn't downloads openFST

╰─ pip install pynini Collecting pynini Using cached pynini-2.1.5.tar.gz (627 kB) Preparing metadata (setup.py) ... done Requirement already satisfied: Cython>=0.29 in /Users/aalvarez/.pyenv/versions/test/lib/python3.11/site-packages (from pynini) (0.29.33) Installing collected packages: pynini DEPRECATION: pynini is being installed using the legacy 'setup.py install' method, because it does not have a 'pyproject.toml' and the 'wheel' package is not installed. pip 23.1 will enforce this behaviour change. A possible replacement is to enable the '--use-pep517' option. Discussion can be found at https://github.com/pypa/pip/issues/8559 Running setup.py install for pynini ... error error: subprocess-exited-with-error

× Running setup.py install for pynini did not run successfully. │ exit code: 1 ╰─> [57 lines of output] running install /Users/aalvarez/.pyenv/versions/test/lib/python3.11/site-packages/setuptools/command/install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools. warnings.warn( running build running build_py creating build creating build/lib.macosx-13.2-x86_64-cpython-311 creating build/lib.macosx-13.2-x86_64-cpython-311/pywrapfst copying pywrapfst/init.py -> build/lib.macosx-13.2-x86_64-cpython-311/pywrapfst creating build/lib.macosx-13.2-x86_64-cpython-311/pynini copying pynini/init.py -> build/lib.macosx-13.2-x86_64-cpython-311/pynini creating build/lib.macosx-13.2-x86_64-cpython-311/pynini/examples copying pynini/examples/chatspeak.py -> build/lib.macosx-13.2-x86_64-cpython-311/pynini/examples copying pynini/examples/chatspeak_model.py -> build/lib.macosx-13.2-x86_64-cpython-311/pynini/examples copying pynini/examples/weather.py -> build/lib.macosx-13.2-x86_64-cpython-311/pynini/examples copying pynini/examples/init.py -> build/lib.macosx-13.2-x86_64-cpython-311/pynini/examples copying pynini/examples/numbers.py -> build/lib.macosx-13.2-x86_64-cpython-311/pynini/examples copying pynini/examples/plurals.py -> build/lib.macosx-13.2-x86_64-cpython-311/pynini/examples copying pynini/examples/t9.py -> build/lib.macosx-13.2-x86_64-cpython-311/pynini/examples copying pynini/examples/case.py -> build/lib.macosx-13.2-x86_64-cpython-311/pynini/examples copying pynini/examples/dates.py -> build/lib.macosx-13.2-x86_64-cpython-311/pynini/examples copying pynini/examples/g2p.py -> build/lib.macosx-13.2-x86_64-cpython-311/pynini/examples creating build/lib.macosx-13.2-x86_64-cpython-311/pynini/lib copying pynini/lib/edit_transducer.py -> build/lib.macosx-13.2-x86_64-cpython-311/pynini/lib copying pynini/lib/utf8.py -> build/lib.macosx-13.2-x86_64-cpython-311/pynini/lib copying pynini/lib/init.py -> build/lib.macosx-13.2-x86_64-cpython-311/pynini/lib copying pynini/lib/features.py -> build/lib.macosx-13.2-x86_64-cpython-311/pynini/lib copying pynini/lib/pynutil.py -> build/lib.macosx-13.2-x86_64-cpython-311/pynini/lib copying pynini/lib/paradigms.py -> build/lib.macosx-13.2-x86_64-cpython-311/pynini/lib copying pynini/lib/tagger.py -> build/lib.macosx-13.2-x86_64-cpython-311/pynini/lib copying pynini/lib/rule_cascade.py -> build/lib.macosx-13.2-x86_64-cpython-311/pynini/lib copying pynini/lib/rewrite.py -> build/lib.macosx-13.2-x86_64-cpython-311/pynini/lib copying pynini/lib/byte.py -> build/lib.macosx-13.2-x86_64-cpython-311/pynini/lib creating build/lib.macosx-13.2-x86_64-cpython-311/pynini/export copying pynini/export/init.py -> build/lib.macosx-13.2-x86_64-cpython-311/pynini/export copying pynini/export/export.py -> build/lib.macosx-13.2-x86_64-cpython-311/pynini/export copying pynini/export/multi_grm_example.py -> build/lib.macosx-13.2-x86_64-cpython-311/pynini/export copying pynini/export/multi_grm.py -> build/lib.macosx-13.2-x86_64-cpython-311/pynini/export copying pynini/export/grm.py -> build/lib.macosx-13.2-x86_64-cpython-311/pynini/export copying pynini/export/grm_example.py -> build/lib.macosx-13.2-x86_64-cpython-311/pynini/export copying pywrapfst/init.pyi -> build/lib.macosx-13.2-x86_64-cpython-311/pywrapfst copying pywrapfst/py.typed -> build/lib.macosx-13.2-x86_64-cpython-311/pywrapfst copying pynini/init.pyi -> build/lib.macosx-13.2-x86_64-cpython-311/pynini copying pynini/py.typed -> build/lib.macosx-13.2-x86_64-cpython-311/pynini copying pynini/examples/py.typed -> build/lib.macosx-13.2-x86_64-cpython-311/pynini/examples copying pynini/lib/py.typed -> build/lib.macosx-13.2-x86_64-cpython-311/pynini/lib copying pynini/export/py.typed -> build/lib.macosx-13.2-x86_64-cpython-311/pynini/export running build_ext building '_pywrapfst' extension creating build/temp.macosx-13.2-x86_64-cpython-311 creating build/temp.macosx-13.2-x86_64-cpython-311/extensions clang -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -I/Applications/Xcode-14.2.0.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include -I/Applications/Xcode-14.2.0.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include -I/Users/aalvarez/.pyenv/versions/test/include -I/Users/aalvarez/.pyenv/versions/3.11.2/include/python3.11 -c extensions/_pywrapfst.cpp -o build/temp.macosx-13.2-x86_64-cpython-311/extensions/_pywrapfst.o -std=c++17 -Wno-register -Wno-deprecated-declarations -Wno-unused-function -Wno-unused-local-typedefs -funsigned-char -stdlib=libc++ -mmacosx-version-min=10.7 extensions/_pywrapfst.cpp:815:10: fatal error: 'fst/util.h' file not found

include <fst/util.h>

           ^~~~~~~~~~~~
  1 error generated.
  error: command '/usr/bin/clang' failed with exit code 1
  [end of output]

note: This error originates from a subprocess, and is likely not a problem with pip. error: legacy-install-failure

× Encountered error while trying to install package. ╰─> pynini

note: This is an issue with the package mentioned above, not pip. hint: See above for output from the failure.

— Reply to this email directly, view it on GitHub https://github.com/kylebgorman/pynini/pull/62#issuecomment-1476395077, or unsubscribe https://github.com/notifications/unsubscribe-auth/AABG4OKAJRZRFZ7AQCG2MDTW5BWPHANCNFSM6AAAAAASX6EWYM . You are receiving this because you commented.Message ID: @.***>

alealv commented 1 year ago

Update

Just some hours ago I was able to installed from wheel. Problem was that python version isn't specified in project.toml My environment had python 3.11.2 and there is no wheel yet for that version, so it was trying to build from source.

Sorry for the confusion

kylebgorman commented 9 months ago

Addressed in #71.