taishi-i / nagisa

A Japanese tokenizer based on recurrent neural networks
https://huggingface.co/spaces/taishi-i/nagisa-demo
MIT License
379 stars 22 forks source link

Could not install nagisa with poetry (without complicated configurations) #32

Closed himkt closed 7 months ago

himkt commented 7 months ago

Hello @taishi-i, thank you for maintaining nagisa package! Let me report that I think we couldn't install nagisa with poetry. I'm not sure why but poetry tried to build DyNet even if the manifest explicitly requires Python 3.8+ (in this case, 3.10+).

Currently I have a workaround to specify the repository on GitHub so it's just to share with you. Maybe switch_install_requires doesn't work as expected. I'm sorry that currently I'm not sure which poetry or my config is bad.

From PyPI

It (might) use sdist.

> cat pyproject.toml
[tool.poetry]
name = "testtt"
version = "0.1.0"
description = ""
authors = ["himkt <himkt@klis.tsukuba.ac.jp>"]
readme = "README.md"

[tool.poetry.dependencies]
python = "^3.10"

[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"
> poetry add nagisa -vvv
Loading configuration file /Users/himkt/Library/Application Support/pypoetry/config.toml
Using virtualenv: /Users/himkt/Desktop/testtt/.venv
[keyring.backend] Loading KWallet
[keyring.backend] Loading SecretService
[keyring.backend] Loading Windows
[keyring.backend] Loading chainer
[keyring.backend] Loading libsecret
[keyring.backend] Loading macOS
Creating new session for pypi.org
Source (PyPI): 23 packages found for nagisa *
Using version ^0.2.10 for nagisa

Updating dependencies
Resolving dependencies...
   1: fact: testtt is 0.1.0
   1: derived: testtt
   1: fact: testtt depends on nagisa (^0.2.10)
   1: selecting testtt (0.1.0)
   1: derived: nagisa (>=0.2.10,<0.3.0)
Source (PyPI): 1 packages found for nagisa >=0.2.10,<0.3.0
   1: fact: nagisa (0.2.10) depends on six (*)
   1: fact: nagisa (0.2.10) depends on numpy (*)
   1: fact: nagisa (0.2.10) depends on DyNet (*)
   1: selecting nagisa (0.2.10)
   1: derived: DyNet
   1: derived: numpy
   1: derived: six
Source (PyPI): 12 packages found for dynet *
Source (PyPI): 103 packages found for numpy *
Source (PyPI): 27 packages found for six *
   1: selecting numpy (1.26.3)
   1: selecting six (1.16.0)
   1: fact: dynet (2.1.2) depends on cython (*)
   1: fact: dynet (2.1.2) depends on numpy (*)
   1: selecting dynet (2.1.2)
   1: derived: cython
Source (PyPI): 116 packages found for cython *
   1: selecting cython (3.0.8)
   1: Version solving took 0.172 seconds.
   1: Tried 1 solutions.

Finding the necessary packages for the current system
Source (PyPI): 1 packages found for nagisa >=0.2.10,<0.3.0
Source (PyPI): 1 packages found for dynet *
Source (PyPI): 1 packages found for numpy *
Source (PyPI): 1 packages found for six *
Source (PyPI): 1 packages found for cython *

Package operations: 2 installs, 0 updates, 0 removals, 3 skipped

  • Installing dynet (2.1.2): Pending...
Skipping wheel dyNET-2.1.2-cp27-cp27m-macosx_10_13_x86_64.whl as this is not supported by the current environment
Skipping wheel dyNET-2.1.2-cp27-cp27m-manylinux1_i686.whl as this is not supported by the current environment
Skipping wheel dyNET-2.1.2-cp27-cp27m-manylinux1_x86_64.whl as this is not supported by the current environment
Skipping wheel dyNET-2.1.2-cp27-cp27mu-manylinux1_i686.whl as this is not supported by the current environment
Skipping wheel dyNET-2.1.2-cp27-cp27mu-manylinux1_x86_64.whl as this is not supported by the current environment
Skipping wheel dyNET-2.1.2-cp35-cp35m-macosx_10_13_x86_64.whl as this is not supported by the current environment
Skipping wheel dyNET-2.1.2-cp35-cp35m-manylinux1_i686.whl as this is not supported by the current environment
Skipping wheel dyNET-2.1.2-cp35-cp35m-manylinux1_x86_64.whl as this is not supported by the current environment
Skipping wheel dyNET-2.1.2-cp36-cp36m-macosx_10_13_x86_64.whl as this is not supported by the current environment
Skipping wheel dyNET-2.1.2-cp36-cp36m-manylinux1_i686.whl as this is not supported by the current environment
Skipping wheel dyNET-2.1.2-cp36-cp36m-manylinux1_x86_64.whl as this is not supported by the current environment
Skipping wheel dyNET-2.1.2-cp36-cp36m-win_amd64.whl as this is not supported by the current environment
Skipping wheel dyNET-2.1.2-cp37-cp37m-macosx_10_13_x86_64.whl as this is not supported by the current environment
Skipping wheel dyNET-2.1.2-cp37-cp37m-manylinux1_i686.whl as this is not supported by the current environment
Skipping wheel dyNET-2.1.2-cp37-cp37m-manylinux1_x86_64.whl as this is not supported by the current environment
Skipping wheel dyNET-2.1.2-cp37-cp37m-win_amd64.whl as this is not supported by the current environment
Skipping wheel dyNET-2.1.2-cp38-cp38-macosx_10_13_x86_64.whl as this is not supported by the current environment
Skipping wheel dyNET-2.1.2-cp38-cp38-manylinux1_i686.whl as this is not supported by the current environment
Skipping wheel dyNET-2.1.2-cp38-cp38-manylinux1_x86_64.whl as this is not supported by the current environment
Skipping wheel dyNET-2.1.2-cp38-cp38-win_amd64.whl as this is not supported by the current environment
  • Installing dynet (2.1.2): Preparing...
  • Installing dynet (2.1.2): Failed

  Stack trace:

  8  ~/.pyenv/versions/3.10.12/lib/python3.10/site-packages/poetry/installation/executor.py:269 in _execute_operation
      267│
      268│             try:
    → 269│                 result = self._do_execute_operation(operation)
      270│             except EnvCommandError as e:
      271│                 if e.e.returncode == -2:
...
  /var/folders/v1/77c226vs4m7_bfwn8m9y7yzr0000gn/T/tmp0nxnyp_h/.venv/lib/python3.10/site-packages/setuptools/command/sdist.py:125: SetuptoolsDeprecationWarning: `build_py` command does not inherit from setuptools' `build_py`.
  !!

          ********************************************************************************
          Custom 'build_py' does not implement 'get_data_files_without_manifest'.
          Please extend command classes from setuptools instead of distutils.

          See https://peps.python.org/pep-0632/ for details.
          ********************************************************************************

  !!
    self._add_data_files(self._safe_data_files(build_py))
  INFO:root:reading manifest file 'dyNET.egg-info/SOURCES.txt'
  INFO:root:adding license file 'LICENSE.txt'
  INFO:root:writing manifest file 'dyNET.egg-info/SOURCES.txt'
  INFO:root:Copying dyNET.egg-info to build/bdist.macosx-13.5-x86_64/wheel/dyNET-2.1.2-py3.10.egg-info
  INFO:root:running install_scripts
  [WARNING] This wheel needs a higher macOS version than the version your Python interpreter is compiled against.  To silence this warning, set MACOSX_DEPLOYMENT_TARGET to at least 14_0 or recreate these files with lower MACOSX_DEPLOYMENT_TARGET:
  build/bdist.macosx-13.5-x86_64/wheel/libdynet.dyliberror: [Errno 2] No such file or directory: 'LICENSE.txt'

  at ~/.pyenv/versions/3.10.12/lib/python3.10/site-packages/poetry/installation/chef.py:164 in _prepare
      160│
      161│                 error = ChefBuildError("\n\n".join(message_parts))
      162│
      163│             if error is not None:
    → 164│                 raise error from None
      165│
      166│             return path
      167│
      168│     def _prepare_sdist(self, archive: Path, destination: Path | None = None) -> Path:

Note: This error originates from the build backend, and is likely not a problem with poetry but with dynet (2.1.2) not supporting PEP 517 builds. You can verify this by running 'pip wheel --no-cache-dir --use-pep517 "dynet (==2.1.2)"'.

Full log: https://gist.github.com/himkt/f714d3c6a1f1e21f268a0ba5f4b7ee8b

From GitHub (git repo)

It uses dyNet38 as expected.

> cat pyproject.toml
[tool.poetry]
name = "testtt"
version = "0.1.0"
description = ""
authors = ["himkt <himkt@klis.tsukuba.ac.jp>"]
readme = "README.md"

[tool.poetry.dependencies]
python = "^3.10"

[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"
> poetry add git+https://github.com/himkt/nagisa.git#7b760c5 -vvv
Loading configuration file /Users/himkt/Library/Application Support/pypoetry/config.toml
Using virtualenv: /Users/himkt/Desktop/testtt/.venv
[keyring.backend] Loading KWallet
[keyring.backend] Loading SecretService
[keyring.backend] Loading Windows
[keyring.backend] Loading chainer
[keyring.backend] Loading libsecret
[keyring.backend] Loading macOS
[urllib3.connectionpool] Starting new HTTPS connection (1): github.com:443
[urllib3.connectionpool] https://github.com:443 "GET /himkt/nagisa.git/info/refs?service=git-upload-pack HTTP/1.1" 200 None
[urllib3.connectionpool] Starting new HTTPS connection (2): github.com:443
[urllib3.connectionpool] https://github.com:443 "POST /himkt/nagisa.git/git-upload-pack HTTP/1.1" 200 None
Cloning https://github.com/himkt/nagisa.git at '7b760c5' to /Users/himkt/Desktop/testtt/.venv/src/nagisa

Updating dependencies
Resolving dependencies...
   1: fact: testtt is 0.1.0
   1: derived: testtt
   1: fact: testtt depends on nagisa (0.2.10)
   1: selecting testtt (0.1.0)
   1: derived: nagisa (0.2.10) @ git+https://github.com/himkt/nagisa.git@7b760c5
   1: fact: nagisa (0.2.10) depends on six (*)
   1: fact: nagisa (0.2.10) depends on numpy (*)
   1: fact: nagisa (0.2.10) depends on DyNet38 (*)
   1: selecting nagisa (0.2.10 7b760c5)
   1: derived: DyNet38
   1: derived: numpy
   1: derived: six
   1: selecting numpy (1.26.3)
   1: selecting six (1.16.0)
   1: fact: dynet38 (2.2) depends on cython (*)
   1: fact: dynet38 (2.2) depends on numpy (*)
   1: selecting dynet38 (2.2)
   1: derived: cython
   1: selecting cython (3.0.8)
   1: Version solving took 0.192 seconds.
   1: Tried 1 solutions.

Finding the necessary packages for the current system

Package operations: 0 installs, 1 update, 0 removals, 4 skipped

  • Installing cython (3.0.8): Pending...
  • Installing cython (3.0.8): Skipped for the following reason: Already installed
  • Installing dynet38 (2.2): Pending...
  • Installing dynet38 (2.2): Skipped for the following reason: Already installed
  • Installing six (1.16.0): Pending...
  • Installing six (1.16.0): Skipped for the following reason: Already installed
  • Installing numpy (1.26.3): Pending...
  • Installing numpy (1.26.3): Skipped for the following reason: Already installed
  • Updating nagisa (0.2.10 989a080 -> 0.2.10 7b760c5): Pending...
  • Updating nagisa (0.2.10 989a080 -> 0.2.10 7b760c5): Cloning...
  • Updating nagisa (0.2.10 989a080 -> 0.2.10 7b760c5): Preparing...
  • Updating nagisa (0.2.10 989a080 -> 0.2.10 7b760c5): Installing...
  • Updating nagisa (0.2.10 989a080 -> 0.2.10 7b760c5)

Writing lock file
taishi-i commented 7 months ago

Hi @himkt! Thank you for providing the report. I will try to conduct a similar verification myself. I haven't been using poetry regularly, so my verification was insufficient. Please wait a little. I will share the information reproduced in my environment.

taishi-i commented 7 months ago

Thank you @himkt! I have figured out the cause of the error.

The following poetry command retrieves dependency information from https://pypi.org/pypi/nagisa/json. In this command, the dependent libraries are ['six', 'numpy', 'DyNet'], and it seems that the conditions in install_requires are not functioning correctly.

poetry add nagisa

You can check with this code.

import requests

package_name = "nagisa"

url = f"https://pypi.org/pypi/{package_name}/json"
response = requests.get(url)
data = response.json()
dependencies = data.get("info", {}).get("requires_dist", [])
print(dependencies)

Also, when adding GitHub or wheel directly in the poetry command, it is possible to execute switch_install_requires directly in that environment, and it correctly specifies DyNet38.

poetry add git+https://github.com/taishi-i/nagisa.git

As a solution to this problem, it is necessary to correct the following part.

def switch_install_requires():
    major = sys.version_info.major
    minor = sys.version_info.minor
    if os.name == 'posix' and major == 3 and minor > 7:
        return ['six', 'numpy', 'DyNet38']
    else:
        return ['six', 'numpy', 'DyNet']

Since it involves handling dependencies, it is necessary to conduct sufficient testing and add a new wheel to PyPI. Therefore, once sufficient testing has been completed, I will release a version addressing this issue, so please wait a little longer.

himkt commented 7 months ago

Wow, thank you so much for your blazing fast investigation (did you read the codebase of poetry...?). 😮 And I understand the current status, I'll wait the next release!

Just for my curiosity: will you continue to support Python<=3.7 (if the minimum python requirements were updated to py38 simply, switch_install_requires would no longer necessary)?

taishi-i commented 7 months ago

Hi @himkt!

Yes. Just to be sure, I checked the contents of the poetry code! I referred to this part.

Currently, to maintain consistency and stability in code processing, this repository is adopting a policy of making only minimal changes. In the future, when significant changes become inevitable, I plan to discontinue the provision of Python libraries that have been deprecated.

Regarding the recent error, the issue was that the dependency library information in nagisa-0.2.10.tar.gz was not correctly included. This was due to my wheel build environment being outdated and unable to set the correct library dependencies.

I resolved the issue where Poetry could not be installed this time by first updating pip, wheel, and build to the latest versions, and then executing python -m build to construct a tar.gz that includes the correct dependencies of the library.

An updated wheel has been released (nagisa v0.2.11), and with it, I've managed to use nagisa with poetry in my Python 3.10 environment. Could you please verify if it works in your environment as well?

More detailed information about this error will be shared at nagisa v0.2.11 of release.

himkt commented 7 months ago

Thank you so much for your work and clarification! I understand the policy about dependencies. 👍

Regarding the recent error, the issue was that the dependency library information in nagisa-0.2.10.tar.gz was not correctly included. This was due to my wheel build environment being outdated and unable to set the correct library dependencies.

Aha, I see. I also succeeded to add nagisa==0.2.11 to dependencies in a fresh py310+ environment and konoha package as well (https://github.com/himkt/konoha/pull/203)! 🎉

taishi-i commented 7 months ago

Sorry for the inconvenience I have caused so far. Also, I haven't had the chance to thank you until now, but thank you for integrating nagisa into konoha!

This issue is resolved, so I close it. Thank you!