AcademySoftwareFoundation / rez

An integrated package configuration, build and deployment system for software
https://rez.readthedocs.io
Apache License 2.0
945 stars 336 forks source link

rez-pkg-cache always stalls #1486

Closed Anti-Distinctlyminty closed 1 year ago

Anti-Distinctlyminty commented 1 year ago

When trying to cache SideFX labs, the copying always stalls

Environment

To Reproduce

  1. Download labs https://github.com/sideeffects/SideFXLabs/archive/refs/tags/19.0.426.zip
  2. Install the package (see below)
  3. Try to cache the sidefx labs package
  4. It fails with the shutil error:
    Traceback (most recent call last):
    File "C:\Program Files\Python39\lib\runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
    File "C:\Program Files\Python39\lib\runpy.py", line 87, in _run_code
    exec(code, run_globals)
    File "C:\opt\rez\Scripts\rez\rez-pkg-cache.exe\__main__.py", line 7, in <module>
    File "c:\opt\rez\lib\site-packages\rez\cli\_entry_points.py", line 190, in run_rez_pkg_cache
    return run("pkg-cache")
    File "c:\opt\rez\lib\site-packages\rez\cli\_main.py", line 191, in run
    returncode = run_cmd()
    File "c:\opt\rez\lib\site-packages\rez\cli\_main.py", line 183, in run_cmd
    return func(opts, opts.parser, extra_arg_groups)
    File "c:\opt\rez\lib\site-packages\rez\cli\pkg-cache.py", line 142, in command
    add_variant(pkgcache, uri, opts)
    File "c:\opt\rez\lib\site-packages\rez\cli\pkg-cache.py", line 74, in add_variant
    destpath, status = pkgcache.add_variant(variant, force=opts.force)
    File "c:\opt\rez\lib\site-packages\rez\package_cache.py", line 298, in add_variant
    shutil.copytree(variant_root, rootpath)
    File "C:\Program Files\Python39\lib\shutil.py", line 568, in copytree
    return _copytree(entries=entries, src=src, dst=dst, symlinks=symlinks,
    File "C:\Program Files\Python39\lib\shutil.py", line 522, in _copytree
    raise Error(errors)
    shutil.Error: [('\\\\truenas.nymus.local\\software\\packages\\sidefx_labs\\19.0.426\\platform-windows\\arch-AMD64\\SideFXLabs-19.0.426\\automation\\unittest\\baseline\\pivot_painter_demoscene\\ae1ad03442568b83f37bdf84004f926197bf56b817dddff422804f0c\\testfile__\\BrickWall_RGB_PivotPos_A_SelectionOrder_IntAsFloat_UV2.exr', 'C:\\ProgramData\\Rez\\cache\\sidefx_labs\\19.0.426\\6946\\a\\SideFXLabs-19.0.426\\automation\\unittest\\baseline\\pivot_painter_demoscene\\ae1ad03442568b83f37bdf84004f926197bf56b817dddff422804f0c\\testfile__\\BrickWall_RGB_PivotPos_A_SelectionOrder_IntAsFloat_UV2.exr', "[Errno 2] No such file or directory: '\\\\\\\\truenas.nymus.local\\\\software\\\\packages\\\\sidefx_labs\\\\19.0.426\\\\platform-windows\\\\arch-AMD64\\\\SideFXLabs-19.0.426\\\\automation\\\\unittest\\\\baseline\\\\pivot_painter_demoscene\\\\ae1ad03442568b83f37bdf84004f926197bf56b817dddff422804f0c\\\\testfile__\\\\BrickWall_RGB_PivotPos_A_SelectionOrder_IntAsFloat_UV2.exr'"), ('\\\\truenas.nymus.local\\software\\packages\\sidefx_labs\\19.0.426\\platform-windows\\arch-AMD64\\SideFXLabs-19.0.426\\automation\\unittest\\baseline\\pivot_painter_demoscene\\ae1ad03442568b83f37bdf84004f926197bf56b817dddff422804f0c\\testfile__\\BrickWall_RGB_XVector_A_Random0-1_UV2.tga', 'C:\\ProgramData\\Rez\\cache\\sidefx_labs\\19.0.426\\6946\\a\\SideFXLabs-19.0.426\\automation\\unittest\\baseline\\pivot_painter_demoscene\\ae1ad03442568b83f37bdf84004f926197bf56b817dddff422804f0c\\testfile__\\BrickWall_RGB_XVector_A_Random0-1_UV2.tga', "[Errno 2] No such file or directory: '\\\\\\\\truenas.nymus.local\\\\software\\\\packages\\\\sidefx_labs\\\\19.0.426\\\\platform-windows\\\\arch-AMD64\\\\SideFXLabs-19.0.426\\\\automation\\\\unittest\\\\baseline\\\\pivot_painter_demoscene\\\\ae1ad03442568b83f37bdf84004f926197bf56b817dddff422804f0c\\\\testfile__\\\\BrickWall_RGB_XVector_A_Random0-1_UV2.tga'")]
package.py ```python name = "sidefx_labs" version = "19.0.426" requires = ["houdini-19+"] variants = [["platform-windows", "arch-AMD64"]] build_requires = ["python-3"] build_command = "python {root}/build.py" uuid = "788291ea-56d4-11ed-8c87-e0d55e201446" def commands(): env.HOUDINI_PATH.prepend("{root}/SideFXLabs-19.0.426") ```
build.py ```python import os import zipfile from pathlib import Path if __name__ == "__main__": sourcePath = Path(os.environ["REZ_BUILD_SOURCE_PATH"]) if os.environ["REZ_BUILD_INSTALL"] == "1": binPath = Path(os.environ["REZ_BUILD_SOURCE_PATH"]) / "bin" installPath = Path(os.environ["REZ_BUILD_INSTALL_PATH"]) buildPath = Path(os.environ["REZ_BUILD_PATH"]) payload = "19.0.426.zip" targetFilePath = binPath / payload if not targetFilePath.exists(): raise FileExistsError() if not zipfile.is_zipfile(targetFilePath): raise RuntimeError(f"Invalid zip archive {targetFilePath}") with zipfile.ZipFile(targetFilePath) as z: z.extractall(path=installPath) ```

Expected behavior It should cache successfully

Actual behavior It fails with the above error on the same two files. All other files seem to copy successfully.

JeanChristopheMorinPerso commented 1 year ago

My first bet would be the length of these paths is too long.

Anti-Distinctlyminty commented 1 year ago

My first bet would be the length of these paths is too long.

Bingo. I really thought I had long paths enabled, but apparently not. I set the following registry key to 1: Computer\HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\FileSystem\LongPathsEnabled And in the future I'll be using hashed variants.

Thank you @JeanChristopheMorinPerso

JeanChristopheMorinPerso commented 1 year ago

You can also get rid of the SideFXLabs-19.0.426 folder.