Use PEP 508 environment marker for `install_requires`

[andreoliwa]

Problem

The typing module is being installed on Python 3.8 despite the specifications in setup.py:

https://github.com/Doist/todoist-python/blob/605443c67a8e2f105071e0da001c9f6f2a89ef19/setup.py#L23-L26

It happens in this specific situation:

• Using Poetry to install dependencies.
• Installing todoist-python together with a local dependency: https://github.com/andreoliwa/dontforget/commit/885c7f52397dfa45b54f5b1adc33613a4509ce7f

Poetry doesn't read constraints from setup.py, which is not executed for security reasons. See https://github.com/python-poetry/poetry/issues/1287#issuecomment-519655153

(see "Details" of the problem below)

Solution

Using the recommended PEP 508 environment markers for install_requires in setup.cfg, instead of setup.py.

Then Poetry can resolve dependencies.

Details

poetry update currently raises AttributeError: type object 'Callable' has no attribute '_abc_registry', because the typing module is still being installed:

$ poetry update
(...)
  • Installing typing (3.7.4.3)
  • Installing tzlocal (2.1)
  • Installing uritemplate (3.0.1)
  • Installing apscheduler (3.6.3)
  • Installing autorepr (0.3.0)
  • Installing clib (0.10.0 /Users/waa/Code/python-clib): Failed

  EnvCommandError

  Command ['/Users/waa/Library/Caches/pypoetry/virtualenvs/dontforget-6lDh86rC-py3.8/bin/pip', 'install', '--no-deps', '-U', '/Users/waa/Code/pyth
on-clib'] errored with the following return code 1, and output:
  Processing /Users/waa/Code/python-clib
    Installing build dependencies: started
    Installing build dependencies: finished with status 'error'
    ERROR: Command errored out with exit status 1:
     command: /Users/waa/Library/Caches/pypoetry/virtualenvs/dontforget-6lDh86rC-py3.8/bin/python /Users/waa/Library/Caches/pypoetry/virtualenvs/d
ontforget-6lDh86rC-py3.8/lib/python3.8/site-packages/pip install --ignore-installed --no-user --prefix /private/var/folders/6k/jw7hpy9s6zg1vt2p6vb
2zvkw0000gp/T/pip-build-env-6c2hwime/overlay --no-warn-script-location --no-binary :none: --only-binary :none: -i https://pypi.org/simple -- 'poet
ry>=0.12'
         cwd: None
    Complete output (42 lines):
    Traceback (most recent call last):
      File "/Users/waa/.pyenv/versions/3.8.6/lib/python3.8/runpy.py", line 194, in _run_module_as_main
        return _run_code(code, main_globals, None,
      File "/Users/waa/.pyenv/versions/3.8.6/lib/python3.8/runpy.py", line 87, in _run_code
        exec(code, run_globals)
      File "/Users/waa/Library/Caches/pypoetry/virtualenvs/dontforget-6lDh86rC-py3.8/lib/python3.8/site-packages/pip/__main__.py", line 26, in <mo
dule>
        sys.exit(_main())
      File "/Users/waa/Library/Caches/pypoetry/virtualenvs/dontforget-6lDh86rC-py3.8/lib/python3.8/site-packages/pip/_internal/cli/main.py", line
73, in main
        command = create_command(cmd_name, isolated=("--isolated" in cmd_args))
      File "/Users/waa/Library/Caches/pypoetry/virtualenvs/dontforget-6lDh86rC-py3.8/lib/python3.8/site-packages/pip/_internal/commands/__init__.p
y", line 104, in create_command
        module = importlib.import_module(module_path)
      File "/Users/waa/.pyenv/versions/3.8.6/lib/python3.8/importlib/__init__.py", line 127, in import_module
        return _bootstrap._gcd_import(name[level:], package, level)
      File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
      File "<frozen importlib._bootstrap>", line 991, in _find_and_load
      File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
      File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
      File "<frozen importlib._bootstrap_external>", line 783, in exec_module
      File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
      File "/Users/waa/Library/Caches/pypoetry/virtualenvs/dontforget-6lDh86rC-py3.8/lib/python3.8/site-packages/pip/_internal/commands/install.py
", line 17, in <module>
        from pip._internal.cli.req_command import RequirementCommand, with_cleanup
      File "/Users/waa/Library/Caches/pypoetry/virtualenvs/dontforget-6lDh86rC-py3.8/lib/python3.8/site-packages/pip/_internal/cli/req_command.py"
, line 16, in <module>
        from pip._internal.index.collector import LinkCollector
      File "/Users/waa/Library/Caches/pypoetry/virtualenvs/dontforget-6lDh86rC-py3.8/lib/python3.8/site-packages/pip/_internal/index/collector.py"
, line 14, in <module>
        from pip._vendor import html5lib, requests
      File "/Users/waa/Library/Caches/pypoetry/virtualenvs/dontforget-6lDh86rC-py3.8/lib/python3.8/site-packages/pip/_vendor/requests/__init__.py"
, line 125, in <module>
        from . import utils
      File "/Users/waa/Library/Caches/pypoetry/virtualenvs/dontforget-6lDh86rC-py3.8/lib/python3.8/site-packages/pip/_vendor/requests/utils.py", l
ine 25, in <module>
        from . import certs
      File "/Users/waa/Library/Caches/pypoetry/virtualenvs/dontforget-6lDh86rC-py3.8/lib/python3.8/site-packages/pip/_vendor/requests/certs.py", l
ine 15, in <module>
        from pip._vendor.certifi import where
      File "/Users/waa/Library/Caches/pypoetry/virtualenvs/dontforget-6lDh86rC-py3.8/lib/python3.8/site-packages/pip/_vendor/certifi/__init__.py",
 line 1, in <module>
        from .core import contents, where
      File "/Users/waa/Library/Caches/pypoetry/virtualenvs/dontforget-6lDh86rC-py3.8/lib/python3.8/site-packages/pip/_vendor/certifi/core.py", lin
e 12, in <module>
        from importlib.resources import path as get_path, read_text
      File "/Users/waa/.pyenv/versions/3.8.6/lib/python3.8/importlib/resources.py", line 11, in <module>
        from typing import Iterable, Iterator, Optional, Set, Union   # noqa: F401
      File "/Users/waa/Library/Caches/pypoetry/virtualenvs/dontforget-6lDh86rC-py3.8/lib/python3.8/site-packages/typing.py", line 1359, in <module
>
        class Callable(extra=collections_abc.Callable, metaclass=CallableMeta):
      File "/Users/waa/Library/Caches/pypoetry/virtualenvs/dontforget-6lDh86rC-py3.8/lib/python3.8/site-packages/typing.py", line 1007, in __new__
        self._abc_registry = extra._abc_registry
    AttributeError: type object 'Callable' has no attribute '_abc_registry'
    ----------------------------------------
  ERROR: Command errored out with exit status 1: /Users/waa/Library/Caches/pypoetry/virtualenvs/dontforget-6lDh86rC-py3.8/bin/python /Users/waa/Li
brary/Caches/pypoetry/virtualenvs/dontforget-6lDh86rC-py3.8/lib/python3.8/site-packages/pip install --ignore-installed --no-user --prefix /private
/var/folders/6k/jw7hpy9s6zg1vt2p6vb2zvkw0000gp/T/pip-build-env-6c2hwime/overlay --no-warn-script-location --no-binary :none: --only-binary :none:
-i https://pypi.org/simple -- 'poetry>=0.12' Check the logs for full command output.
  WARNING: You are using pip version 20.2.4; however, version 20.3.1 is available.
  You should consider upgrading via the '/Users/waa/Library/Caches/pypoetry/virtualenvs/dontforget-6lDh86rC-py3.8/bin/python -m pip install --upgr
ade pip' command.

  at ~/.poetry/lib/poetry/utils/env.py:1074 in _run
      1070│                 output = subprocess.check_output(
      1071│                     cmd, stderr=subprocess.STDOUT, **kwargs
      1072│                 )
      1073│         except CalledProcessError as e:
    → 1074│             raise EnvCommandError(e, input=input_)
      1075│
      1076│         return decode(output)
      1077│
      1078│     def execute(self, bin, *args, **kwargs):

  • Installing deprecated (1.2.10)
  • Installing environs (9.2.0)
(...)

The error is fixed by this pull request. Now the typing module is removed, and my local Python lib is installed:

$ poetry update
Updating dependencies
Resolving dependencies... (2.2s)

Writing lock file

Package operations: 1 install, 0 updates, 1 removal

  • Removing typing (3.7.4.3)
  • Installing clib (0.10.0 /Users/waa/Code/python-clib)

[PotHix]

The current version should be working. It's OK to use setup.py or setup.cfg. The current master was not released for some reason and I just bumped the version and uploaded it to PyPI.

Can you please check if it works as expected now? In case it's not working I will review this PR again.

[andreoliwa]

It's working now, thanks. 👍🏻

I'm closing this, it's not needed anymore.