Missing 'archive_info' while converting pip package causes rez-pip crash

[brycegbrazen]

When running rez-pip2 on ShotGrid's latest version of their tk-core package, the process crashes trying to get a archive_info key in the package's dictionary.

Environment

• Windows 10
• Rez version 2.112.0
• Rez python version 3.7
• rez-pip 0.3.1

To Reproduce

1. Install new rez-pip
2. rez-pip2 git+https://github.com/shotgunsoftware/tk-core@v0.20.26 --python-version 3.7

Expected behavior Successful install.

Actual behavior

(Rez) λ rez-pip2 git+https://github.com/shotgunsoftware/tk-core@v0.20.26 --python-version 3.7
INFO     Installing requested packages for Python 3.7.9
Traceback (most recent call last):
  File "C:\Python37\lib\runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "C:\Python37\lib\runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "C:\Program Files\Rez\Scripts\rez-pip2.exe\__main__.py", line 7, in <module>
  File "c:\program files\rez\lib\site-packages\rez_pip\cli.py", line 318, in run
    _run(args, pipArgs, pipWorkArea)
  File "c:\program files\rez\lib\site-packages\rez_pip\cli.py", line 183, in _run
    pipArgs,
  File "c:\program files\rez\lib\site-packages\rez_pip\pip.py", line 138, in getPackages
    packageInfo = PackageInfo.from_dict(rawPackage)
  File "c:\program files\rez\lib\site-packages\dataclasses_json\api.py", line 70, in from_dict
    return _decode_dataclass(cls, kvs, infer_missing)
  File "c:\program files\rez\lib\site-packages\dataclasses_json\core.py", line 217, in _decode_dataclass
    infer_missing)
  File "c:\program files\rez\lib\site-packages\dataclasses_json\core.py", line 172, in _decode_dataclass
    field_value = kvs[field.name]
KeyError: 'archive_info'

[JeanChristopheMorinPerso]

Installing from a git repo isn't supported since we only support installing wheels. But rez-pip should give a nicer error. It shouldn't go that far.

You could pip wheel git+https://github.com/shotgunsoftware/tk-core@v0.20.26 and then rez-pip2 <path to wheel> I think. I'm on my phone so I can't confirm.

[JeanChristopheMorinPerso]

48

[JeanChristopheMorinPerso]

Umm, the workaround I proposed doesn't work. I'll have to add support for installing local wheels.

[JeanChristopheMorinPerso]

@brycegbrazen Can you try with https://github.com/JeanChristopheMorinPerso/rez-pip/pull/75?

You can install it and then do

pip wheel --no-deps git+https://github.com/shotgunsoftware/tk-core@v0.20.26

and then

rez-pip2 sgtk*.whl

or you can also use

rez-pip2 sgtk -- --find-links .

[brycegbrazen]

My immediate attempts failed. I tried first to activate the venv that is created by Rez. I installed Rez in C:/Program Files/Rez, and I am then running the venv's activate file in C:/Program Files/Rez/Scripts.

I was able to create the .whl file by running pip wheel --no-deps git+https://github.com/shotgunsoftware/tk-core@v0.20.26 as you mentioned which created the .whl file in my current directory, which was C:/Program Files/Rez/Scripts.

The problems came when I tried to run rez-pip2 as you mentioned above. Firstly, rez-pip2 sgtk*.whl didn't work, it gave this error:

(Rez) λ rez-pip2 sgtk*.whl
INFO     Installing requested packages for Python 3.7.9
WARNING: Requirement 'sgtk*.whl' looks like a filename, but the file does not exist
ERROR: sgtk*.whl is not a valid wheel filename.
rez_pip.exceptions.PipError: Failed to run pip command: 'python.exe c:\\program files\\rez\\lib\\site-packages\\rez_pip\\data\\pip.pyz install -q sgtk*.whl --disable-pip-version-check --dry-run --ignore-installed --python-version=3.7.9 --only-binary=:all: --disable-pip-version-check --report C:\\Users\\BRYCE~1.GAT\\AppData\\Local\\Temp\\pip-install-outputbhkdbqex'

Pip reported this:

WARNING: Requirement 'sgtk*.whl' looks like a filename, but the file does not exist
ERROR: sgtk*.whl is not a valid wheel filename.

However, after running either:

rez-pip2 sgtk-0.20.26-py3-none-any.whl

or

rez-pip2 sgtk -- --find-links .

gives me this error instead:

(Rez) λ rez-pip2 sgtk -- --find-links .
INFO     Installing requested packages for Python 3.7.9
INFO     Resolved 1 dependencies for python 3.7.9
INFO     Downloading...
INFO     Downloaded 1 wheels
INFO     Installing sgtk-0.20.26 wheel
Traceback (most recent call last):
  File "C:\Python37\lib\runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "C:\Python37\lib\runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "C:\Program Files\Rez\Scripts\rez-pip2.exe\__main__.py", line 7, in <module>
  File "c:\program files\rez\lib\site-packages\rez_pip\cli.py", line 329, in run
    _run(args, pipArgs, pipWorkArea)
  File "c:\program files\rez\lib\site-packages\rez_pip\cli.py", line 212, in _run
    package, pathlib.Path(wheel), installedWheelsDir
  File "c:\program files\rez\lib\site-packages\rez_pip\install.py", line 86, in installWheel
    with installer.sources.WheelFile.open(wheelPath) as source:
  File "C:\Python37\lib\contextlib.py", line 112, in __enter__
    return next(self.gen)
  File "c:\program files\rez\lib\site-packages\installer\sources.py", line 162, in open
    with zipfile.ZipFile(path) as f:
  File "C:\Python37\lib\zipfile.py", line 1240, in __init__
    self.fp = io.open(file, filemode)
FileNotFoundError: [Errno 2] No such file or directory: 'C:\\Program%20Files\\Rez\\Scripts\\sgtk-0.20.26-py3-none-any.whl'

Based on what I'm seeing, it's due to the space in between "Program" and "Files" in the path. Not sure if this is something we can deal with though. I did verify that the file does indeed exist at that location.

I can try putting the wheel in a location where the path does not have a space next.

[brycegbrazen]

Yep creating the wheel in a path with no spaces worked fine and was able to create the Rez package as expected 💯

[brycegbrazen]

@JeanChristopheMorinPerso Question: Isn't a pip install from a directory supposed to create a wheel? So that shouldn't cause any issues with using rez-pip with a Git URL? I saw this bit in the pip documentation, but it's not super clear to me if passing a Git URL has special behavior regarding wheels.

[JeanChristopheMorinPerso]

Thanks for testing! I'll take a note of the white space issue.

Question: Isn't a pip install from a directory supposed to create a wheel? So that shouldn't cause any issues with using rez-pip with a Git URL? I saw this bit in the pip documentation, but it's not super clear to me if passing a Git URL has special behavior regarding wheels.

You are right that when running pip install ., pip will indeed create a wheel. The only thing pip (and any other python installers) can install are wheels. When installing git+https://github.com/shotgunsoftware/tk-core@v0.20.26, pip will clone the repository, checkout the reference (v0.20.26) and it will do the equivalent of pip install . which will create a wheel and install it (note that it doesn't really calls pip install ., but it's easier to illustrate like this).

The thing is that when you rez-pip something, pip install isn't used to install the packages. It's only used to get the list of packages to download (something like pip install <package> --dry-run --quiet --report /tmp/asd). That's described in https://rez-pip.readthedocs.io/en/latest/faq.html#why-can-t-it-install-python-2-packages (which admittedly could be improved).

In other words, the issue you are seeing (minus the exception that shouldn't be raised in the first place) is https://github.com/JeanChristopheMorinPerso/rez-pip/issues/48. Here, we can replace sdist (source distribution, .tar.gz) with git. The result is the same.