search

JeanChristopheMorinPerso / rez-pip

PyPI/python package ingester/converter for the rez package manager
https://rez-pip.readthedocs.io/en/latest/
Apache License 2.0
23 stars 5 forks source link

Extras not translated to rez #66

Open brycegbrazen opened 10 months ago

brycegbrazen commented 10 months ago

When running a rez-pip2 on the boxsdk PyPi package with the extra "jwt" specified, the extra dependencies are not included in the resulting package.py file.

Environment

To Reproduce

  1. Install new rez-pip
  2. rez-pip2 boxsdk[jwt]==3.9.2 --python-version 3.7

Expected behavior "jwt" and "cryptography" packages listed in "extra_requires" on boxsdk's setup.py file are included in requires portion of resulting package.py.

Actual behavior "jwt" and "cryptography" packages are missing from the resulting requires portion of the package.py.

JeanChristopheMorinPerso commented 10 months ago

Hi @brycegbrazen , can you provide me with the full output logs please? Also, could you try with rez-pip2 'boxsdk[jwt]==3.9.2' --python-version 3.7 (quoting boxsdk[jwt]==3.9.2)? I remember seeing something like this while I was working on the early PoC...

brycegbrazen commented 10 months ago

@JeanChristopheMorinPerso Does it actually log to a file? Not seeing anything in the docs that mention that so I figured I'd ask. Otherwise I can just upload the output from my stdout.

JeanChristopheMorinPerso commented 10 months ago

No, it logs everything to stdout/stderr.

brycegbrazen commented 10 months ago

Looks like it includes those extra dependencies, but just doesn't include them as requirements in the package.py. I'll try quoting the package request next.

(Rez) λ rez-pip2 boxsdk[jwt]==3.9.2 --python-version 3.7
INFO     Installing requested packages for Python 3.7.9
INFO     Resolved 17 dependencies for python 3.7.9
INFO     Downloading...
INFO     Downloaded PyJWT-2.8.0 to 'C:\\Users\\BRYCE~1.GAT\\AppData\\Local\\Temp\\rez-pip-targeth7q9kw_c\\wheels\\PyJWT-2.8.0-py3-none-any.whl' (22591
         bytes)
INFO     Downloaded idna-3.4 to 'C:\\Users\\BRYCE~1.GAT\\AppData\\Local\\Temp\\rez-pip-targeth7q9kw_c\\wheels\\idna-3.4-py3-none-any.whl' (61538 bytes)
INFO     Downloaded boxsdk-3.9.2 to 'C:\\Users\\BRYCE~1.GAT\\AppData\\Local\\Temp\\rez-pip-targeth7q9kw_c\\wheels\\boxsdk-3.9.2-py2.py3-none-any.whl'
         (139209 bytes)
INFO     Downloaded typing_extensions-4.7.1 to
         'C:\\Users\\BRYCE~1.GAT\\AppData\\Local\\Temp\\rez-pip-targeth7q9kw_c\\wheels\\typing_extensions-4.7.1-py3-none-any.whl' (33232 bytes)
INFO     Downloaded requests-2.31.0 to 'C:\\Users\\BRYCE~1.GAT\\AppData\\Local\\Temp\\rez-pip-targeth7q9kw_c\\wheels\\requests-2.31.0-py3-none-any.whl'
         (62574 bytes)
INFO     Downloaded urllib3-2.0.7 to 'C:\\Users\\BRYCE~1.GAT\\AppData\\Local\\Temp\\rez-pip-targeth7q9kw_c\\wheels\\urllib3-2.0.7-py3-none-any.whl'
         (124213 bytes)
INFO     Downloaded python-dateutil-2.8.2 to
         'C:\\Users\\BRYCE~1.GAT\\AppData\\Local\\Temp\\rez-pip-targeth7q9kw_c\\wheels\\python_dateutil-2.8.2-py2.py3-none-any.whl' (247702 bytes)
INFO     Downloaded charset-normalizer-3.3.2 to
         'C:\\Users\\BRYCE~1.GAT\\AppData\\Local\\Temp\\rez-pip-targeth7q9kw_c\\wheels\\charset_normalizer-3.3.2-cp37-cp37m-win_amd64.whl' (98127 bytes)
INFO     Downloaded attrs-23.1.0 to 'C:\\Users\\BRYCE~1.GAT\\AppData\\Local\\Temp\\rez-pip-targeth7q9kw_c\\wheels\\attrs-23.1.0-py3-none-any.whl' (61160
         bytes)
INFO     Downloaded requests-toolbelt-1.0.0 to
         'C:\\Users\\BRYCE~1.GAT\\AppData\\Local\\Temp\\rez-pip-targeth7q9kw_c\\wheels\\requests_toolbelt-1.0.0-py2.py3-none-any.whl' (54481 bytes)
INFO     Downloaded six-1.16.0 to 'C:\\Users\\BRYCE~1.GAT\\AppData\\Local\\Temp\\rez-pip-targeth7q9kw_c\\wheels\\six-1.16.0-py2.py3-none-any.whl' (11053
         bytes)
INFO     Downloaded importlib-metadata-6.7.0 to
         'C:\\Users\\BRYCE~1.GAT\\AppData\\Local\\Temp\\rez-pip-targeth7q9kw_c\\wheels\\importlib_metadata-6.7.0-py3-none-any.whl' (22934 bytes)
INFO     Downloaded zipp-3.15.0 to 'C:\\Users\\BRYCE~1.GAT\\AppData\\Local\\Temp\\rez-pip-targeth7q9kw_c\\wheels\\zipp-3.15.0-py3-none-any.whl' (6758
         bytes)
INFO     Downloaded certifi-2023.7.22 to
         'C:\\Users\\BRYCE~1.GAT\\AppData\\Local\\Temp\\rez-pip-targeth7q9kw_c\\wheels\\certifi-2023.7.22-py3-none-any.whl' (158334 bytes)
INFO     Downloaded cffi-1.15.1 to 'C:\\Users\\BRYCE~1.GAT\\AppData\\Local\\Temp\\rez-pip-targeth7q9kw_c\\wheels\\cffi-1.15.1-cp37-cp37m-win_amd64.whl'
         (179301 bytes)
INFO     Downloaded pycparser-2.21 to 'C:\\Users\\BRYCE~1.GAT\\AppData\\Local\\Temp\\rez-pip-targeth7q9kw_c\\wheels\\pycparser-2.21-py2.py3-none-any.whl'
         (118697 bytes)
INFO     Downloaded cryptography-41.0.5 to
         'C:\\Users\\BRYCE~1.GAT\\AppData\\Local\\Temp\\rez-pip-targeth7q9kw_c\\wheels\\cryptography-41.0.5-cp37-abi3-win_amd64.whl' (2669522 bytes)
INFO     Downloaded 17 wheels
INFO     Installing boxsdk-3.9.2 wheel
INFO     Installing attrs-23.1.0 wheel
INFO     Installing cryptography-41.0.5 wheel
INFO     Installing PyJWT-2.8.0 wheel
INFO     Installing requests-2.31.0 wheel
INFO     Installing requests-toolbelt-1.0.0 wheel
INFO     Installing urllib3-2.0.7 wheel
INFO     Installing python-dateutil-2.8.2 wheel
INFO     Installing certifi-2023.7.22 wheel
INFO     Installing cffi-1.15.1 wheel
INFO     Installing charset-normalizer-3.3.2 wheel
INFO     Installing idna-3.4 wheel
INFO     Installing six-1.16.0 wheel
INFO     Installing importlib-metadata-6.7.0 wheel
INFO     Installing typing_extensions-4.7.1 wheel
INFO     Installing zipp-3.15.0 wheel
INFO     Installing pycparser-2.21 wheel
INFO     Creating rez package for boxsdk
INFO     Installing boxsdk-3.9.2 []
INFO     Created 1 variants and skipped 0
INFO     Creating rez package for attrs
INFO     Installing attrs-23.1.0 [importlib_metadata]
INFO     Created 1 variants and skipped 0
INFO     Creating rez package for cryptography
INFO     Installing cryptography-41.0.5 [platform-windows, arch-AMD64, python-3.7]
INFO     Created 1 variants and skipped 0
INFO     Creating rez package for PyJWT
INFO     Installing PyJWT-2.8.0 [typing_extensions]
INFO     Created 1 variants and skipped 0
INFO     Creating rez package for requests
INFO     Installing requests-2.31.0 []
INFO     Created 1 variants and skipped 0
INFO     Creating rez package for requests-toolbelt
INFO     Installing requests_toolbelt-1.0.0 []
INFO     Created 1 variants and skipped 0
INFO     Creating rez package for urllib3
INFO     Installing urllib3-2.0.7 []
INFO     Created 1 variants and skipped 0
INFO     Creating rez package for python-dateutil
INFO     Installing python_dateutil-2.8.2 []
INFO     Created 1 variants and skipped 0
INFO     Creating rez package for certifi
INFO     Installing certifi-2023.7.22 []
INFO     Created 1 variants and skipped 0
INFO     Creating rez package for cffi
INFO     Installing cffi-1.15.1 [platform-windows, arch-AMD64, python-3.7]
INFO     Created 1 variants and skipped 0
INFO     Creating rez package for charset-normalizer
INFO     Installing charset_normalizer-3.3.2 [platform-windows, arch-AMD64, python-3.7]
INFO     Created 1 variants and skipped 0
INFO     Creating rez package for idna
INFO     Installing idna-3.4 []
INFO     Created 1 variants and skipped 0
INFO     Creating rez package for six
INFO     Installing six-1.16.0 []
INFO     Created 1 variants and skipped 0
INFO     Creating rez package for importlib-metadata
INFO     Installing importlib_metadata-6.7.0 [typing_extensions-3.6.4+]
INFO     Created 1 variants and skipped 0
INFO     Creating rez package for typing_extensions
INFO     Installing typing_extensions-4.7.1 []
INFO     Created 1 variants and skipped 0
INFO     Creating rez package for zipp
INFO     Installing zipp-3.15.0 []
INFO     Created 1 variants and skipped 0
INFO     Creating rez package for pycparser
INFO     Installing pycparser-2.21 []
INFO     Created 1 variants and skipped 0
JeanChristopheMorinPerso commented 10 months ago

I just tried on my side and it's working. And from what I see in your log, it seems like it also worked for you. The extra is called jwt and it installs PyJWT and cryptography (https://github.com/box/box-python-sdk/blob/v3.9.2/setup.py#L63)

JeanChristopheMorinPerso commented 10 months ago

I see

INFO     Creating rez package for cryptography
INFO     Installing cryptography-41.0.5 [platform-windows, arch-AMD64, python-3.7]
INFO     Created 1 variants and skipped 0
INFO     Creating rez package for PyJWT
INFO     Installing PyJWT-2.8.0 [typing_extensions]
INFO     Created 1 variants and skipped 0

Which means it created a package cryptography-41.0.5 and PyJWT-2.8.0.

brycegbrazen commented 10 months ago

Yeah I see it's installing those two extra dependencies, but it doesn't actually include them in the resolve when I run a rez-env boxsdk. I thought it'd include PyJWT and cryptography as dependencies in the package.py somehow.

JeanChristopheMorinPerso commented 10 months ago

Ah yeah. Well, the boxsdk package will be boxsdk only... We don't try to merge things (as it would be had to know what you really want to do. If you pip install boxsdk[jwt], you will endup with everything on one env, but it doesn't modify dependencies. In other words

$ python
>>> import importlib.metadata
>>> dist = importlib.metadata.distribution('boxsdk') 
>>> dist.requires
['attrs (>=17.3.0)', 'urllib3', 'requests (<3,>=2.4.3)', 'requests-toolbelt (>=0.4.0)', 'python-dateutil', "coveralls ; extra == 'coveralls'", "tox (<=3.28.0) ; extra == 'coveralls'", "tox (<=3.28.0) ; extra == 'dev'", "tox-gh-actions ; extra == 'gh'", "tox (<=3.28.0) ; extra == 'gh'", "pyjwt (>=1.7.0) ; extra == 'jwt'", "cryptography (>=3) ; extra == 'jwt'", "redis (>=2.10.3) ; extra == 'redis'", "bottle ; extra == 'test'", "jsonpatch (>1.14) ; extra == 'test'", "sqlalchemy (<1.4.0) ; extra == 'test'", "pytest ; extra == 'test'", "pytest-timeout ; extra == 'test'", "pytest-cov ; extra == 'test'", "pytest-lazy-fixture ; extra == 'test'", "pytz ; extra == 'test'", "urllib3 (<2) ; extra == 'test'"]

The dependencies aren't merged. The xtras are still extras even after the install.

rez-pip does the same (both the builtin rez-pip and rez-pip2).

We additionally don't have a good way to translate boxsdk[jwt] to rez-env boxsdk[jwt]. See https://github.com/JeanChristopheMorinPerso/rez-pip/blob/0217569e14dff414ad3372dd3909536944a6c7c9/src/rez_pip/utils.py#L490-L493. We simply don't have a way to do that right now.

JeanChristopheMorinPerso commented 10 months ago

Maybe ephemerals could help, but it would need some investigation to see if they would work or not.