pip install fails because cython is imported in setup directly

[bertsky]

When I install via pip from current master, I get this:

Collecting pynini@ git+https://github.com/kylebgorman/pynini.git#egg=pynini
  Cloning https://github.com/kylebgorman/pynini.git to /tmp/pip-install-477ckk7a/pynini
    ERROR: Command errored out with exit status 1:
     command: /usr/bin/python3 -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-477ckk7a/pynini/setup.py'"'"'; __file__='"'"'/tmp/pip-install-477ckk7a/pynini/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-pip-egg-info-vy_k3j1z
         cwd: /tmp/pip-install-477ckk7a/pynini/
    Complete output (5 lines):
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-install-477ckk7a/pynini/setup.py", line 22, in <module>
        from Cython.Build import cythonize
    ModuleNotFoundError: No module named 'Cython'

(I do not recall seeing this problem in the past, but maybe I did things differently then).

The cause is simple: setup.py tries to import Cython directly, before it gets the chance to follow install_requires. Traditionally, people have coped with this by all kinds of tricks (like calling the pip cmdline from setup).

But there's good news: we can simply use pyproject.toml instead!

[kylebgorman]

I have never tried to install this way (pip+git)---the only way that is likely to work in a wide variety is via Conda-Forge, since you need big non-Python dependencies---but I can replicate this in a totally clean client. Would you be willing to submit a PR?

(We probably should do this against the current Pynini 2.1.2---this is just an out of date mirror, though I'll update it when I get a chance.)

[bertsky]

I have never tried to install this way (pip+git)---the only way that is likely to work in a wide variety is via Conda-Forge, since you need big non-Python dependencies

I know, it only works when manually compiling OpenFST, since Debian/Ubuntu are still at 1.6.3 :-/ Nevertheless, this setup at least encapsulates the Python part, and Conda is no option for me unfortunately.

Would you be willing to submit a PR?

I could work it, yes.

We probably should do this against the current Pynini 2.1.2---this is just an out of date mirror, though I'll update it when I get a chance.

Definitely. I am already running into #29 with 1.7.9 and cannot compile OpenFST 1.7.5 with recent g++ (internal compiler error) – so if you could upgrade this would help a lot.

[bertsky]

I am already running into #29 with 1.7.9 and cannot compile OpenFST 1.7.5 with recent g++ (internal compiler error) – so if you could upgrade this would help a lot.

correction: I cannot compile pynini (both the version here and upstream) due to internal compiler error. So this is a completely independent issue (#31), and I should not wait for an update with my PR.

[cdeln]

Hi, the build system bootstrap with Cython and setuptools can now be solved with using pyproject.toml files. See #62 for a fix.

[alealv]

JFR I'm also experiencing the same issue. Installing with pip and poetry (which uses pip underneath)

Installing in a fresh environment

╰─ pip install pynini
Collecting pynini
  Using cached pynini-2.1.5.tar.gz (627 kB)
  Preparing metadata (setup.py) ... error
  error: subprocess-exited-with-error

  × python setup.py egg_info did not run successfully.
  │ exit code: 1
  ╰─> [6 lines of output]
      Traceback (most recent call last):
        File "<string>", line 2, in <module>
        File "<pip-setuptools-caller>", line 34, in <module>
        File "/private/var/folders/qp/8v0z3y5n3y5_g0ftmdrr0b800000gs/T/pip-install-y62o9qws/pynini_3f8387eb1916426ba4ad0b9434a0e531/setup.py", line 22, in <module>
          from Cython.Build import cythonize
      ModuleNotFoundError: No module named 'Cython'
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed

× Encountered error while generating package metadata.
╰─> See above for output.

note: This is an issue with the package mentioned above, not pip.
hint: See above for details.

And this is testing whether is PEP517 compatible

╰─ pip install pynini --use-pep517
Collecting pynini
  Using cached pynini-2.1.5.tar.gz (627 kB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... error
  error: subprocess-exited-with-error

  × Getting requirements to build wheel did not run successfully.
  │ exit code: 1
  ╰─> [21 lines of output]
      Traceback (most recent call last):
        File "/Users/aalvarez/.pyenv/versions/test/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 353, in <module>
          main()
        File "/Users/aalvarez/.pyenv/versions/test/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 335, in main
          json_out['return_val'] = hook(**hook_input['kwargs'])
                                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/Users/aalvarez/.pyenv/versions/test/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 118, in get_requires_for_build_wheel
          return hook(config_settings)
                 ^^^^^^^^^^^^^^^^^^^^^
        File "/private/var/folders/qp/8v0z3y5n3y5_g0ftmdrr0b800000gs/T/pip-build-env-_pjzuka3/overlay/lib/python3.11/site-packages/setuptools/build_meta.py", line 338, in get_requires_for_build_wheel
          return self._get_build_requires(config_settings, requirements=['wheel'])
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/private/var/folders/qp/8v0z3y5n3y5_g0ftmdrr0b800000gs/T/pip-build-env-_pjzuka3/overlay/lib/python3.11/site-packages/setuptools/build_meta.py", line 320, in _get_build_requires
          self.run_setup()
        File "/private/var/folders/qp/8v0z3y5n3y5_g0ftmdrr0b800000gs/T/pip-build-env-_pjzuka3/overlay/lib/python3.11/site-packages/setuptools/build_meta.py", line 485, in run_setup
          self).run_setup(setup_script=setup_script)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/private/var/folders/qp/8v0z3y5n3y5_g0ftmdrr0b800000gs/T/pip-build-env-_pjzuka3/overlay/lib/python3.11/site-packages/setuptools/build_meta.py", line 335, in run_setup
          exec(code, locals())
        File "<string>", line 22, in <module>
      ModuleNotFoundError: No module named 'Cython'
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error

× Getting requirements to build wheel did not run successfully.
│ exit code: 1
╰─> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip.

Error even with Cython

If I add Cython before installing if fails to compile because it doesn't downloads openFST

╰─ pip install pynini
Collecting pynini
  Using cached pynini-2.1.5.tar.gz (627 kB)
  Preparing metadata (setup.py) ... done
Requirement already satisfied: Cython>=0.29 in /Users/aalvarez/.pyenv/versions/test/lib/python3.11/site-packages (from pynini) (0.29.33)
Installing collected packages: pynini
  DEPRECATION: pynini is being installed using the legacy 'setup.py install' method, because it does not have a 'pyproject.toml' and the 'wheel' package is not installed. pip 23.1 will enforce this behaviour change. A possible replacement is to enable the '--use-pep517' option. Discussion can be found at https://github.com/pypa/pip/issues/8559
  Running setup.py install for pynini ... error
  error: subprocess-exited-with-error

  × Running setup.py install for pynini did not run successfully.
  │ exit code: 1
  ╰─> [57 lines of output]
      running install
      /Users/aalvarez/.pyenv/versions/test/lib/python3.11/site-packages/setuptools/command/install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
        warnings.warn(
      running build
      running build_py
      creating build
      creating build/lib.macosx-13.2-x86_64-cpython-311
      creating build/lib.macosx-13.2-x86_64-cpython-311/pywrapfst
      copying pywrapfst/__init__.py -> build/lib.macosx-13.2-x86_64-cpython-311/pywrapfst
      creating build/lib.macosx-13.2-x86_64-cpython-311/pynini
      copying pynini/__init__.py -> build/lib.macosx-13.2-x86_64-cpython-311/pynini
      creating build/lib.macosx-13.2-x86_64-cpython-311/pynini/examples
      copying pynini/examples/chatspeak.py -> build/lib.macosx-13.2-x86_64-cpython-311/pynini/examples
      copying pynini/examples/chatspeak_model.py -> build/lib.macosx-13.2-x86_64-cpython-311/pynini/examples
      copying pynini/examples/weather.py -> build/lib.macosx-13.2-x86_64-cpython-311/pynini/examples
      copying pynini/examples/__init__.py -> build/lib.macosx-13.2-x86_64-cpython-311/pynini/examples
      copying pynini/examples/numbers.py -> build/lib.macosx-13.2-x86_64-cpython-311/pynini/examples
      copying pynini/examples/plurals.py -> build/lib.macosx-13.2-x86_64-cpython-311/pynini/examples
      copying pynini/examples/t9.py -> build/lib.macosx-13.2-x86_64-cpython-311/pynini/examples
      copying pynini/examples/case.py -> build/lib.macosx-13.2-x86_64-cpython-311/pynini/examples
      copying pynini/examples/dates.py -> build/lib.macosx-13.2-x86_64-cpython-311/pynini/examples
      copying pynini/examples/g2p.py -> build/lib.macosx-13.2-x86_64-cpython-311/pynini/examples
      creating build/lib.macosx-13.2-x86_64-cpython-311/pynini/lib
      copying pynini/lib/edit_transducer.py -> build/lib.macosx-13.2-x86_64-cpython-311/pynini/lib
      copying pynini/lib/utf8.py -> build/lib.macosx-13.2-x86_64-cpython-311/pynini/lib
      copying pynini/lib/__init__.py -> build/lib.macosx-13.2-x86_64-cpython-311/pynini/lib
      copying pynini/lib/features.py -> build/lib.macosx-13.2-x86_64-cpython-311/pynini/lib
      copying pynini/lib/pynutil.py -> build/lib.macosx-13.2-x86_64-cpython-311/pynini/lib
      copying pynini/lib/paradigms.py -> build/lib.macosx-13.2-x86_64-cpython-311/pynini/lib
      copying pynini/lib/tagger.py -> build/lib.macosx-13.2-x86_64-cpython-311/pynini/lib
      copying pynini/lib/rule_cascade.py -> build/lib.macosx-13.2-x86_64-cpython-311/pynini/lib
      copying pynini/lib/rewrite.py -> build/lib.macosx-13.2-x86_64-cpython-311/pynini/lib
      copying pynini/lib/byte.py -> build/lib.macosx-13.2-x86_64-cpython-311/pynini/lib
      creating build/lib.macosx-13.2-x86_64-cpython-311/pynini/export
      copying pynini/export/__init__.py -> build/lib.macosx-13.2-x86_64-cpython-311/pynini/export
      copying pynini/export/export.py -> build/lib.macosx-13.2-x86_64-cpython-311/pynini/export
      copying pynini/export/multi_grm_example.py -> build/lib.macosx-13.2-x86_64-cpython-311/pynini/export
      copying pynini/export/multi_grm.py -> build/lib.macosx-13.2-x86_64-cpython-311/pynini/export
      copying pynini/export/grm.py -> build/lib.macosx-13.2-x86_64-cpython-311/pynini/export
      copying pynini/export/grm_example.py -> build/lib.macosx-13.2-x86_64-cpython-311/pynini/export
      copying pywrapfst/__init__.pyi -> build/lib.macosx-13.2-x86_64-cpython-311/pywrapfst
      copying pywrapfst/py.typed -> build/lib.macosx-13.2-x86_64-cpython-311/pywrapfst
      copying pynini/__init__.pyi -> build/lib.macosx-13.2-x86_64-cpython-311/pynini
      copying pynini/py.typed -> build/lib.macosx-13.2-x86_64-cpython-311/pynini
      copying pynini/examples/py.typed -> build/lib.macosx-13.2-x86_64-cpython-311/pynini/examples
      copying pynini/lib/py.typed -> build/lib.macosx-13.2-x86_64-cpython-311/pynini/lib
      copying pynini/export/py.typed -> build/lib.macosx-13.2-x86_64-cpython-311/pynini/export
      running build_ext
      building '_pywrapfst' extension
      creating build/temp.macosx-13.2-x86_64-cpython-311
      creating build/temp.macosx-13.2-x86_64-cpython-311/extensions
      clang -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -I/Applications/Xcode-14.2.0.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include -I/Applications/Xcode-14.2.0.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include -I/Users/aalvarez/.pyenv/versions/test/include -I/Users/aalvarez/.pyenv/versions/3.11.2/include/python3.11 -c extensions/_pywrapfst.cpp -o build/temp.macosx-13.2-x86_64-cpython-311/extensions/_pywrapfst.o -std=c++17 -Wno-register -Wno-deprecated-declarations -Wno-unused-function -Wno-unused-local-typedefs -funsigned-char -stdlib=libc++ -mmacosx-version-min=10.7
      extensions/_pywrapfst.cpp:815:10: fatal error: 'fst/util.h' file not found
      #include <fst/util.h>
               ^~~~~~~~~~~~
      1 error generated.
      error: command '/usr/bin/clang' failed with exit code 1
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: legacy-install-failure

× Encountered error while trying to install package.
╰─> pynini

note: This is an issue with the package mentioned above, not pip.
hint: See above for output from the failure.

[kylebgorman]

@ Alejandro this is a know issue and will be fixed in the next release, but the second error is unrelated.

[monim67]

Any update on this?

[kylebgorman]

It should be fixed by the migration to pyproject.toml in the next release. I can't make any promises when that'll be though I am working on it this fall. In the meantime, consider using one of the two sources for precompiled binaries of the library.

[kylebgorman]

Note to any readers: this is not platform-specific as far as I know.

[kylebgorman]

I put out a post-release because a lot of people were having trouble with this. I hope it solves the issue. It'll be on PyPI shortly.