AcademySoftwareFoundation / rez

An integrated package configuration, build and deployment system for software
https://rez.readthedocs.io
Apache License 2.0
950 stars 341 forks source link

rez-pip install of black does not add tomli as a requirement. #1341

Open jopollack opened 2 years ago

jopollack commented 2 years ago

Rez Version 2.106.0

I'm not sure what's going on here, but when running rez-pip to install black, it leaves out one of the requirements, namely tomli. It installs tomli, but it doesn't add it to the package.py, so the resulting package is broken and I have to manually call tomli in the rez-env or add it into the package.py as a requirement.

rez-pip black --install -p /work/rez --python-version 3.7
Collecting black
  Downloading https://files.pythonhosted.org/packages/2b/70/1d0e33a4df4ed73e9f02f698a29b5d94ff58e39f029c939ecf96a10fb1f3/black-22.6.0-py3-none-any.whl (156kB)
     |████████████████████████████████| 163kB 25.6MB/s 
Collecting typing-extensions>=3.10.0.0; python_version < "3.10" (from black)
  Downloading https://files.pythonhosted.org/packages/75/e1/932e06004039dd670c9d5e1df0cd606bf46e29a28e65d5bb28e894ea29c9/typing_extensions-4.2.0-py3-none-any.whl
Collecting typed-ast>=1.4.2; python_version < "3.8" and implementation_name == "cpython" (from black)
  Downloading https://files.pythonhosted.org/packages/04/93/482d12fd3334b53ec4087e658ab161ab23affcf8b052166b4cf972ca673b/typed_ast-1.5.4-cp37-cp37m-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl (843kB)
     |████████████████████████████████| 849kB 34.4MB/s 
Collecting mypy-extensions>=0.4.3 (from black)
  Downloading https://files.pythonhosted.org/packages/5c/eb/975c7c080f3223a5cdaff09612f3a5221e4ba534f7039db34c35d95fa6a5/mypy_extensions-0.4.3-py2.py3-none-any.whl
Collecting pathspec>=0.9.0 (from black)
  Downloading https://files.pythonhosted.org/packages/42/ba/a9d64c7bcbc7e3e8e5f93a52721b377e994c22d16196e2b0f1236774353a/pathspec-0.9.0-py2.py3-none-any.whl
Collecting tomli>=1.1.0; python_full_version < "3.11.0a7" (from black)
  Downloading https://files.pythonhosted.org/packages/97/75/10a9ebee3fd790d20926a90a2547f0bf78f371b2f13aa822c759680ca7b9/tomli-2.0.1-py3-none-any.whl
Collecting platformdirs>=2 (from black)
  Downloading https://files.pythonhosted.org/packages/ed/22/967181c94c3a4063fe64e15331b4cb366bdd7dfbf46fcb8ad89650026fec/platformdirs-2.5.2-py3-none-any.whl
Collecting click>=8.0.0 (from black)
  Downloading https://files.pythonhosted.org/packages/c2/f1/df59e28c642d583f7dacffb1e0965d0e00b218e0186d7858ac5233dce840/click-8.1.3-py3-none-any.whl (96kB)
     |████████████████████████████████| 102kB 10.4MB/s 
Collecting importlib-metadata; python_version < "3.8" (from click>=8.0.0->black)
  Downloading https://files.pythonhosted.org/packages/d2/a2/8c239dc898138f208dd14b441b196e7b3032b94d3137d9d8453e186967fc/importlib_metadata-4.12.0-py3-none-any.whl
Collecting zipp>=0.5 (from importlib-metadata; python_version < "3.8"->click>=8.0.0->black)
  Using cached https://files.pythonhosted.org/packages/80/0e/16a7ee38617aab6a624e95948d314097cc2669edae9b02ded53309941cfc/zipp-3.8.0-py3-none-any.whl
Installing collected packages: typing-extensions, typed-ast, mypy-extensions, pathspec, tomli, platformdirs, zipp, importlib-metadata, click, black
Successfully installed black-22.6.0 click-8.1.3 importlib-metadata-4.12.0 mypy-extensions-0.4.3 pathspec-0.9.0 platformdirs-2.5.2 tomli-2.0.1 typed-ast-1.5.4 typing-extensions-4.2.0 zipp-3.8.0
WARNING: You are using pip version 19.2.3, however version 22.1.2 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
15:16:52 WARNING  Metadata: missing: ['Home-page', 'Author'], warnings: []
15:16:52 WARNING  Metadata: missing: ['Home-page', 'Author'], warnings: []
15:16:52 WARNING  Metadata: missing: ['Home-page', 'Author'], warnings: []
15:16:52 INFO     Installed [black-22.6.0] /work/rez/black/22.6.0/package.py (6e0a42ad67bd5da79eb94cd760314c7f48292a49)
15:16:52 INFO     Installed [click-8.1.3] /work/rez/click/8.1.3/package.py (f98bb122ce29a25ef03cd5ba3ebdfa16f3a2d0c5)
15:16:52 INFO     Installed [importlib_metadata-4.12.0] /work/rez/importlib_metadata/4.12.0/package.py (1ea170f09dad84a21d7e5918bcf56066e867f747)
15:16:52 INFO     Installed [mypy_extensions-0.4.3] /work/rez/mypy_extensions/0.4.3/package.py (88a70aca30cb79a278872594adf043dc6c40af99)
15:16:52 INFO     Installed [pathspec-0.9.0] /work/rez/pathspec/0.9.0/package.py (88a70aca30cb79a278872594adf043dc6c40af99)
15:16:53 INFO     Installed [platformdirs-2.5.2] /work/rez/platformdirs/2.5.2/package.py (88a70aca30cb79a278872594adf043dc6c40af99)
15:16:53 INFO     Installed [tomli-2.0.1] /work/rez/tomli/2.0.1/package.py (88a70aca30cb79a278872594adf043dc6c40af99)
15:16:53 INFO     Installed [typed_ast-1.5.4] /work/rez/typed_ast/1.5.4/package.py (d9e9d80193dcd9578844ec4c2c22c9366ef0b88a)
15:16:53 INFO     Installed [typing_extensions-4.2.0] /work/rez/typing_extensions/4.2.0/package.py (88a70aca30cb79a278872594adf043dc6c40af99)
15:16:53 INFO     Installed [zipp-3.8.0] /work/rez/zipp/3.8.0/package.py (88a70aca30cb79a278872594adf043dc6c40af99)
15:16:53 INFO     10 packages were installed.
# -*- coding: utf-8 -*-

name = 'black'

version = '22.6.0'

description = 'The uncompromising code formatter.'

authors = ['Łukasz Langa lukasz@langa.pl']

tools = [
    'black',
    'blackd'
]

requires = [
    'click-8.0.0+',
    'platformdirs-2+',
    'mypy_extensions-0.4.3+',
    'pathspec-0.9.0+'
]

variants = [['python-3.7', 'typing_extensions-3.10.0.0+', 'typed_ast-1.4.2+']]

def commands():
    env.PYTHONPATH.append('{root}/python')
    env.PATH.append('{root}/bin')

help = [['Home Page', 'https://github.com/psf/black']]

timestamp = 1656454612

hashed_variants = True

is_pure_python = True

from_pip = True

pip_name = 'black (22.6.0)'

format_version = 2
Traceback (most recent call last):
  File "/work/rez/black/22.6.0/6e0a42ad67bd5da79eb94cd760314c7f48292a49/bin/black", line 6, in <module>
    from black import patched_main
  File "/work/rez/black/22.6.0/6e0a42ad67bd5da79eb94cd760314c7f48292a49/python/black/__init__.py", line 53, in <module>
    from black.files import (
  File "/work/rez/black/22.6.0/6e0a42ad67bd5da79eb94cd760314c7f48292a49/python/black/files.py", line 31, in <module>
    import tomli as tomllib
ModuleNotFoundError: No module named 'tomli'

As you can see, it installs tomli, but it doesn't end up in the package.py, so attempting to run black causes an import error.

nerdvegas commented 2 years ago

Thanks, that does seem odd! A

On Wed, Jun 29, 2022 at 8:29 AM jopollack @.***> wrote:

Rez Version 2.106.0

I'm not sure what's going on here, but when running rez-pip to install black, it leaves out one of the requirements, namely tomli. It installs tomli, but it doesn't add it to the package.py, so the resulting package is broken and I have to manually call tomli in the rez-env or add it into the package.py as a requirement.

rez-pip black --install -p /work/rez --python-version 3.7

Collecting black

Downloading https://files.pythonhosted.org/packages/2b/70/1d0e33a4df4ed73e9f02f698a29b5d94ff58e39f029c939ecf96a10fb1f3/black-22.6.0-py3-none-any.whl (156kB)

 |████████████████████████████████| 163kB 25.6MB/s

Collecting typing-extensions>=3.10.0.0; python_version < "3.10" (from black)

Downloading https://files.pythonhosted.org/packages/75/e1/932e06004039dd670c9d5e1df0cd606bf46e29a28e65d5bb28e894ea29c9/typing_extensions-4.2.0-py3-none-any.whl

Collecting typed-ast>=1.4.2; python_version < "3.8" and implementation_name == "cpython" (from black)

Downloading https://files.pythonhosted.org/packages/04/93/482d12fd3334b53ec4087e658ab161ab23affcf8b052166b4cf972ca673b/typed_ast-1.5.4-cp37-cp37m-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl (843kB)

 |████████████████████████████████| 849kB 34.4MB/s

Collecting mypy-extensions>=0.4.3 (from black)

Downloading https://files.pythonhosted.org/packages/5c/eb/975c7c080f3223a5cdaff09612f3a5221e4ba534f7039db34c35d95fa6a5/mypy_extensions-0.4.3-py2.py3-none-any.whl

Collecting pathspec>=0.9.0 (from black)

Downloading https://files.pythonhosted.org/packages/42/ba/a9d64c7bcbc7e3e8e5f93a52721b377e994c22d16196e2b0f1236774353a/pathspec-0.9.0-py2.py3-none-any.whl

Collecting tomli>=1.1.0; python_full_version < "3.11.0a7" (from black)

Downloading https://files.pythonhosted.org/packages/97/75/10a9ebee3fd790d20926a90a2547f0bf78f371b2f13aa822c759680ca7b9/tomli-2.0.1-py3-none-any.whl

Collecting platformdirs>=2 (from black)

Downloading https://files.pythonhosted.org/packages/ed/22/967181c94c3a4063fe64e15331b4cb366bdd7dfbf46fcb8ad89650026fec/platformdirs-2.5.2-py3-none-any.whl

Collecting click>=8.0.0 (from black)

Downloading https://files.pythonhosted.org/packages/c2/f1/df59e28c642d583f7dacffb1e0965d0e00b218e0186d7858ac5233dce840/click-8.1.3-py3-none-any.whl (96kB)

 |████████████████████████████████| 102kB 10.4MB/s

Collecting importlib-metadata; python_version < "3.8" (from click>=8.0.0->black)

Downloading https://files.pythonhosted.org/packages/d2/a2/8c239dc898138f208dd14b441b196e7b3032b94d3137d9d8453e186967fc/importlib_metadata-4.12.0-py3-none-any.whl

Collecting zipp>=0.5 (from importlib-metadata; python_version < "3.8"->click>=8.0.0->black)

Using cached https://files.pythonhosted.org/packages/80/0e/16a7ee38617aab6a624e95948d314097cc2669edae9b02ded53309941cfc/zipp-3.8.0-py3-none-any.whl

Installing collected packages: typing-extensions, typed-ast, mypy-extensions, pathspec, tomli, platformdirs, zipp, importlib-metadata, click, black

Successfully installed black-22.6.0 click-8.1.3 importlib-metadata-4.12.0 mypy-extensions-0.4.3 pathspec-0.9.0 platformdirs-2.5.2 tomli-2.0.1 typed-ast-1.5.4 typing-extensions-4.2.0 zipp-3.8.0

WARNING: You are using pip version 19.2.3, however version 22.1.2 is available.

You should consider upgrading via the 'pip install --upgrade pip' command.

15:16:52 WARNING Metadata: missing: ['Home-page', 'Author'], warnings: []

15:16:52 WARNING Metadata: missing: ['Home-page', 'Author'], warnings: []

15:16:52 WARNING Metadata: missing: ['Home-page', 'Author'], warnings: []

15:16:52 INFO Installed [black-22.6.0] /work/rez/black/22.6.0/package.py (6e0a42ad67bd5da79eb94cd760314c7f48292a49)

15:16:52 INFO Installed [click-8.1.3] /work/rez/click/8.1.3/package.py (f98bb122ce29a25ef03cd5ba3ebdfa16f3a2d0c5)

15:16:52 INFO Installed [importlib_metadata-4.12.0] /work/rez/importlib_metadata/4.12.0/package.py (1ea170f09dad84a21d7e5918bcf56066e867f747)

15:16:52 INFO Installed [mypy_extensions-0.4.3] /work/rez/mypy_extensions/0.4.3/package.py (88a70aca30cb79a278872594adf043dc6c40af99)

15:16:52 INFO Installed [pathspec-0.9.0] /work/rez/pathspec/0.9.0/package.py (88a70aca30cb79a278872594adf043dc6c40af99)

15:16:53 INFO Installed [platformdirs-2.5.2] /work/rez/platformdirs/2.5.2/package.py (88a70aca30cb79a278872594adf043dc6c40af99)

15:16:53 INFO Installed [tomli-2.0.1] /work/rez/tomli/2.0.1/package.py (88a70aca30cb79a278872594adf043dc6c40af99)

15:16:53 INFO Installed [typed_ast-1.5.4] /work/rez/typed_ast/1.5.4/package.py (d9e9d80193dcd9578844ec4c2c22c9366ef0b88a)

15:16:53 INFO Installed [typing_extensions-4.2.0] /work/rez/typing_extensions/4.2.0/package.py (88a70aca30cb79a278872594adf043dc6c40af99)

15:16:53 INFO Installed [zipp-3.8.0] /work/rez/zipp/3.8.0/package.py (88a70aca30cb79a278872594adf043dc6c40af99)

15:16:53 INFO 10 packages were installed.

-- coding: utf-8 --

name = 'black'

version = '22.6.0'

description = 'The uncompromising code formatter.'

authors = ['Łukasz Langa @.***']

tools = [

'black',

'blackd'

]

requires = [

'click-8.0.0+',

'platformdirs-2+',

'mypy_extensions-0.4.3+',

'pathspec-0.9.0+'

]

variants = [['python-3.7', 'typing_extensions-3.10.0.0+', 'typed_ast-1.4.2+']]

def commands():

env.PYTHONPATH.append('{root}/python')

env.PATH.append('{root}/bin')

help = [['Home Page', 'https://github.com/psf/black']]

timestamp = 1656454612

hashed_variants = True

is_pure_python = True

from_pip = True

pip_name = 'black (22.6.0)'

format_version = 2

Traceback (most recent call last):

File "/work/rez/black/22.6.0/6e0a42ad67bd5da79eb94cd760314c7f48292a49/bin/black", line 6, in

from black import patched_main

File "/work/rez/black/22.6.0/6e0a42ad67bd5da79eb94cd760314c7f48292a49/python/black/init.py", line 53, in

from black.files import (

File "/work/rez/black/22.6.0/6e0a42ad67bd5da79eb94cd760314c7f48292a49/python/black/files.py", line 31, in

import tomli as tomllib

ModuleNotFoundError: No module named 'tomli'

As you can see, it installs tomli, but it doesn't end up in the package.py, so attempting to run black causes an import error.

— Reply to this email directly, view it on GitHub https://github.com/AcademySoftwareFoundation/rez/issues/1341, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAMOUSS5G5CCV3XI2EP2ZK3VRN4FPANCNFSM52DW52GA . You are receiving this because you are subscribed to this thread.Message ID: @.***>

instinct-vfx commented 2 years ago

My guess is that's because it is a conditional requirement and something likely breaks:

Requires-Dist: tomli (>=1.1.0) ; python_full_version < "3.11.0a7"

instinct-vfx commented 2 years ago

(the difference to the other requirements besides the alpha version is python_full_version instead of python_version

instinct-vfx commented 2 years ago

Also: Works fine on windows. It ends up in the variant as it is version specific:

name = 'black'

version = '22.6.0'

description = 'The uncompromising code formatter.'

authors = ['Łukasz Langa lukasz@langa.pl']

tools = [
    'black.exe',
    'blackd.exe'
]

requires = [
    'platformdirs-2+',
    'pathspec-0.9.0+',
    'click-8.0.0+',
    'mypy_extensions-0.4.3+'
]

variants = \
    [['platform-windows',
      'arch-AMD64',
      'python-3.7',
      'typed_ast-1.4.2+',
      'typing_extensions-3.10.0.0+',
      'tomli-1.1.0+']]
JeanChristopheMorinPerso commented 2 years ago

We do support python_full_version, see https://github.com/AcademySoftwareFoundation/rez/blob/master/src/rez/utils/pip.py#L391. And as you can see, we use the packaging library to evaluate the markers (which is the library also used by pip and plenty of other tools).

So we would have to debug that issue and see what's going on as it's supposed to work.

instinct-vfx commented 2 years ago

What distro are you on? As i can not repro it on windows

ColinKennedy commented 2 years ago

FWIW I reproduced this on Windows 3 days ago when the ticket was made but tried again last night and couldn't reproduce it. The only difference between then and now is that I added a python 3 Rez package via @instinct-vfx 's rezpy. Before that, pip was getting found from Rez's own python. That may be a contributing factor.

ColinKennedy commented 2 years ago

Yep, just confirmed again. I removed the python 3 packages that I'd installed via rezpy, Rez used its own python (rez-python --version == "3.7.9"), and then this happened.

variants = \
    [['platform-windows',
      'arch-AMD64',
      'python-3.7',
      'typing_extensions-3.10.0.0+',
      'typed_ast-1.4.2+']]

Note the missing tomli.

I then reinstalled a python Rez package (3.7.9) using rezpy

rezpy --python_version=3.7.9

Then re-ran rez-pip and got these variants

variants = \
    [['platform-windows',
      'arch-AMD64',
      'python-3.7',
      'typed_ast-1.4.2+',
      'tomli-1.1.0+',
      'typing_extensions-3.10.0.0+']]

I suspect @instinct-vfx uses a python package via rezpy, which is why he cannot reproduce it at the moment. But if you remove the python package and use Rez's own python, it should happen.

jopollack commented 2 years ago

In my case I discovered the bug on Linux using rez-pip in a default environment with rez 2.106.0, so the existing pip or python packages rez comes with may be a factor.

ColinKennedy commented 2 years ago

Your original reproduction doesn't show which python / pip was used (I'd thought rez-pip always outputs that at the start. Did you omit that information or was it truly not shown in your output?) but yes, likely Rez's own rez-python was chosen or a python package that may not be set up properly. In my personal experience, in Windows, rez-bind fails to build a proper Python package and then rez-pip detects that and falls back to rez-python.

jopollack commented 2 years ago

Yes, I omitted it. It found pip-19.2.3 in the python-3.7 I had installed.

rez-pip black --install -p /work/rez --python-version 3.7
15:16:46 INFO     Trying to use pip from python package
15:16:48 INFO     Found pip-19.2.3 inside /work/rez/python/3.7.7/package.py. Will use it via /work/rez/python/3.7.7/bin/python3.7
15:16:48 INFO     Installing 'black' with pip taken from '/work/rez/python/3.7.7/bin/python3.7'