rez-pip release flag does not make use of release hooks

After some investigation I was able to verify that the rez-pip command unfortunately does not call release hooks in the same fashion the rez-release command does.

Trying to release a normal package via rez-release calls configured release hooks as expected, this is done via the build process.

Please see the relevant excerpts below.

# release.py 
builder = create_build_process(opts.process,
                                   working_dir,
                                   package=package,
                                   build_system=buildsys,
                                   vcs=vcs,
                                   ensure_latest=(not opts.no_latest),
                                   skip_repo_errors=opts.skip_repo_errors,
                                   ignore_existing_tag=opts.ignore_existing_tag,
                                   verbose=True)
....

 # perform the release
 builder.release(release_message=release_msg or None,
                 variants=opts.variants){code}

# build_process.py

def run_hooks(self, hook_event, **kwargs):
        hook_names = self.package.config.release_hooks or []
        hooks = create_release_hooks(hook_names, self.working_dir)        for hook in hooks:
            debug_print("Running %s hook '%s'...",
                        hook_event.label, hook.name())

....

# local.py

def release(self, release_message=None, variants=None):
        self._print_header("Releasing %s..." % self.package.qualified_name)        # test that we're in a state to release
        self.pre_release()        release_path = self.package.config.release_packages_path
        release_data = self.get_release_data()
        changelog = release_data.get("changelog")
        previous_version = release_data.get("previous_version")
        previous_revision = release_data.get("previous_revision")       

# run pre/post-release hooks
...

❯ rez release 11:11:34 DEBUG searching plugin path /home/lambdaclan/.rez/lib/python3.8/site-packages/rezplugins/release_vcs... 11:11:34 DEBUG loading release_vcs plugin at /home/lambdaclan/.rez/lib/python3.8/site-packages/rezplugins/release_vcs: rezplugins.release_vcs.git... 11:11:34 DEBUG loading release_vcs plugin at /home/lambdaclan/.rez/lib/python3.8/site-packages/rezplugins/release_vcs: rezplugins.release_vcs.hg... 11:11:34 DEBUG loading release_vcs plugin at /home/lambdaclan/.rez/lib/python3.8/site-packages/rezplugins/release_vcs: rezplugins.release_vcs.stub... 11:11:34 DEBUG loading release_vcs plugin at /home/lambdaclan/.rez/lib/python3.8/site-packages/rezplugins/release_vcs: rezplugins.release_vcs.svn... 11:11:34 DEBUG Traceback (most recent call last): File "/home/lambdaclan/.rez/lib/python3.8/site-packages/rez/plugin_managers.py", line 140, in load_plugins module = loader.find_module(modname).load_module(modname) File "", line 462, in _check_name_wrapper File "", line 962, in load_module File "", line 787, in load_module File "", line 265, in _load_module_shim File "", line 702, in _load File "", line 671, in _load_unlocked File "", line 783, in exec_module File "", line 219, in _call_with_frames_removed File "/home/lambdaclan/.rez/lib/python3.8/site-packages/rezplugins/release_vcs/svn.py", line 9, in import pysvn ModuleNotFoundError: No module named 'pysvn'

11:11:34 DEBUG searching plugin path /home/lambdaclan/.rez/lib/python3.8/site-packages/rezplugins/build_process... 11:11:34 DEBUG loading build_process plugin at /home/lambdaclan/.rez/lib/python3.8/site-packages/rezplugins/build_process: rezplugins.build_process.local... 11:11:34 DEBUG loading build_process plugin at /home/lambdaclan/.rez/lib/python3.8/site-packages/rezplugins/build_process: rezplugins.build_process.remote... 11:11:34 DEBUG searching plugin path /home/lambdaclan/.rez/lib/python3.8/site-packages/rezplugins/package_repository... 11:11:34 DEBUG loading package_repository plugin at /home/lambdaclan/.rez/lib/python3.8/site-packages/rezplugins/package_repository: rezplugins.package_repository.filesystem... 11:11:34 DEBUG loading package_repository plugin at /home/lambdaclan/.rez/lib/python3.8/site-packages/rezplugins/package_repository: rezplugins.package_repository.memory... 11:11:34 DEBUG searching plugin path /home/lambdaclan/.rez/lib/python3.8/site-packages/rezplugins/build_system... 11:11:34 DEBUG loading build_system plugin at /home/lambdaclan/.rez/lib/python3.8/site-packages/rezplugins/build_system: rezplugins.build_system.bez... 11:11:34 DEBUG loading build_system plugin at /home/lambdaclan/.rez/lib/python3.8/site-packages/rezplugins/build_system: rezplugins.build_system.cmake... 11:11:34 DEBUG loading build_system plugin at /home/lambdaclan/.rez/lib/python3.8/site-packages/rezplugins/build_system: rezplugins.build_system.custom... 11:11:34 DEBUG loading build_system plugin at /home/lambdaclan/.rez/lib/python3.8/site-packages/rezplugins/build_system: rezplugins.build_system.make... 11:11:34 DEBUG loading build_system plugin at /home/lambdaclan/.rez/lib/python3.8/site-packages/rezplugins/build_system: rezplugins.build_system.rezconfig... 11:11:34 WARNING no 'register_plugin' function at /home/lambdaclan/.rez/lib/python3.8/site-packages/rezplugins/build_system: rezplugins.build_system.rezconfig 11:11:34 DEBUG Running command: /usr/bin/git rev-parse Releasing hello_world-1.0.0... Checking state of repository... 11:11:34 DEBUG Running command: /usr/bin/git rev-parse --is-bare-repository 11:11:34 DEBUG Running command: /usr/bin/git rev-parse --abbrev-ref --symbolic-full-name '@ Unknown macro: {u}' 11:11:34 DEBUG Running command: /usr/bin/git rev-parse --abbrev-ref HEAD 11:11:34 DEBUG Running command: /usr/bin/git diff-index --quiet HEAD 11:11:34 DEBUG command stdout: 11:11:34 DEBUG 11:11:34 DEBUG command stderr: 11:11:34 DEBUG 11:11:34 DEBUG Running command: /usr/bin/git diff-index --stat HEAD 11:11:34 DEBUG Running command: /usr/bin/git rev-parse HEAD 11:11:34 DEBUG Running command: /usr/bin/git rev-parse --abbrev-ref HEAD 11:11:34 DEBUG Running command: /usr/bin/git rev-parse --abbrev-ref --symbolic-full-name '@' 11:11:34 DEBUG Running command: /usr/bin/git remote -v 11:11:34 DEBUG Running command: /usr/bin/git remote -v 11:11:34 DEBUG Running command: /usr/bin/git log 11:11:34 DEBUG searching plugin path /home/lambdaclan/.rez/lib/python3.8/site-packages/rezplugins/release_hook... 11:11:34 DEBUG loading release_hook plugin at /home/lambdaclan/.rez/lib/python3.8/site-packages/rezplugins/release_hook: rezplugins.release_hook.amqp... 11:11:34 DEBUG loading release_hook plugin at /home/lambdaclan/.rez/lib/python3.8/site-packages/rezplugins/release_hook: rezplugins.release_hook.command... 11:11:34 DEBUG loading release_hook plugin at /home/lambdaclan/.rez/lib/python3.8/site-packages/rezplugins/release_hook: rezplugins.release_hook.emailer... ***Hook OK...**** 11:11:34 DEBUG Running pre-release hook 'emailer'... Resolving build environment: python resolved by lambdaclan@apollo, on Wed Apr 08 11:11:34 2020, using Rez v2.56.1

requested packages: python ~platform==linux (implicit) ~arch==x86_64 (implicit) ~os==Arch-rolling (implicit)

resolved packages: arch-x86_64 /home/lambdaclan/int/arch/x86_64 os-Arch-rolling /home/lambdaclan/int/os/Arch-rolling platform-linux /home/lambdaclan/int/platform/linux python-3.8.2 /home/lambdaclan/int/python/3.8.2/platform-linux/arch-x86_64/os-Arch-rolling

Invoking cmake build system... 11:11:34 DEBUG searching plugin path /home/lambdaclan/.rez/lib/python3.8/site-packages/rezplugins/shell... 11:11:34 DEBUG loading shell plugin at /home/lambdaclan/.rez/lib/python3.8/site-packages/rezplugins/shell: rezplugins.shell.bash... 11:11:34 DEBUG loading shell plugin at /home/lambdaclan/.rez/lib/python3.8/site-packages/rezplugins/shell: rezplugins.shell.cmd... 11:11:34 WARNING 'register_plugin' function at /home/lambdaclan/.rez/lib/python3.8/site-packages/rezplugins/shell: rezplugins.shell.cmd did not return a class. 11:11:34 DEBUG loading shell plugin at /home/lambdaclan/.rez/lib/python3.8/site-packages/rezplugins/shell: rezplugins.shell.csh... 11:11:34 DEBUG loading shell plugin at /home/lambdaclan/.rez/lib/python3.8/site-packages/rezplugins/shell: rezplugins.shell.powershell... 11:11:34 WARNING 'register_plugin' function at /home/lambdaclan/.rez/lib/python3.8/site-packages/rezplugins/shell: rezplugins.shell.powershell did not return a class. 11:11:34 DEBUG loading shell plugin at /home/lambdaclan/.rez/lib/python3.8/site-packages/rezplugins/shell: rezplugins.shell.pwsh... 11:11:34 DEBUG loading shell plugin at /home/lambdaclan/.rez/lib/python3.8/site-packages/rezplugins/shell: rezplugins.shell.sh... 11:11:34 DEBUG loading shell plugin at /home/lambdaclan/.rez/lib/python3.8/site-packages/rezplugins/shell: rezplugins.shell.tcsh... 11:11:34 DEBUG loading shell plugin at /home/lambdaclan/.rez/lib/python3.8/site-packages/rezplugins/shell: rezplugins.shell.zsh... Executing: /bin/cmake -d /home/lambdaclan/Workspace/GitHub/rez/example_packages/hello_world -Wno-dev -DCMAKE_ECLIPSE_GENERATE_SOURCE_PROJECT=TRUE -D_ECLIPSE_VERSION=4.3 --no-warn-unused-cli -DCMAKE_INSTALL_PREFIX=/home/lambdaclan/releases/hello_world/1.0.0 -DCMAKE_MODULE_PATH=${CMAKE_MODULE_PATH} -DCMAKE_BUILD_TYPE=Release -DREZ_BUILD_TYPE=central -DREZ_BUILD_INSTALL=1 -G Unix Makefiles -DCENTRAL=1 Not searching for unused variables given on the command line. – The C compiler identification is GNU 9.3.0 – The CXX compiler identification is GNU 9.3.0 – Check for working C compiler: /bin/cc – Check for working C compiler: /bin/cc - works – Detecting C compiler ABI info – Detecting C compiler ABI info - done – Detecting C compile features – Detecting C compile features - done – Check for working CXX compiler: /bin/c++ – Check for working CXX compiler: /bin/c++ - works – Detecting CXX compiler ABI info – Detecting CXX compiler ABI info - done – Detecting CXX compile features – Detecting CXX compile features - done – Found PkgConfig: /bin/pkg-config (found version "1.6.3") – Configuring done – Generating done – Build files have been written to: /home/lambdaclan/Workspace/GitHub/rez/example_packages/hello_world/build

Executing: make -j4 make[1]: Entering directory '/home/lambdaclan/Workspace/GitHub/rez/example_packages/hello_world/build' make[2]: Entering directory '/home/lambdaclan/Workspace/GitHub/rez/example_packages/hello_world/build' Scanning dependencies of target py make[2]: Leaving directory '/home/lambdaclan/Workspace/GitHub/rez/example_packages/hello_world/build' make[2]: Entering directory '/home/lambdaclan/Workspace/GitHub/rez/example_packages/hello_world/build' [100%] Generating py/python/hello_world.pyc make[2]: Leaving directory '/home/lambdaclan/Workspace/GitHub/rez/example_packages/hello_world/build' [100%] Built target py make[1]: Leaving directory '/home/lambdaclan/Workspace/GitHub/rez/example_packages/hello_world/build'

Executing: make -j4 install make[1]: Entering directory '/home/lambdaclan/Workspace/GitHub/rez/example_packages/hello_world/build' make[2]: Entering directory '/home/lambdaclan/Workspace/GitHub/rez/example_packages/hello_world/build' make[2]: Leaving directory '/home/lambdaclan/Workspace/GitHub/rez/example_packages/hello_world/build' [100%] Built target py make[1]: Leaving directory '/home/lambdaclan/Workspace/GitHub/rez/example_packages/hello_world/build' Install the project... – Install configuration: "Release" – Installing: /home/lambdaclan/releases/hello_world/1.0.0/./python/hello_world.py – Installing: /home/lambdaclan/releases/hello_world/1.0.0/./python/hello_world.pyc – Installing: /home/lambdaclan/releases/hello_world/1.0.0/./bin/hello 11:11:38 DEBUG Running command: /usr/bin/git rev-parse HEAD 11:11:38 DEBUG Running command: /usr/bin/git rev-parse --abbrev-ref HEAD 11:11:38 DEBUG Running command: /usr/bin/git rev-parse --abbrev-ref --symbolic-full-name '@ Unknown macro: {u} ' 11:11:38 DEBUG Running command: /usr/bin/git remote -v 11:11:38 DEBUG Running command: /usr/bin/git remote -v 11:11:38 DEBUG Running command: /usr/bin/git log Hook OK... **11:11:38 DEBUG Running post-release hook 'emailer'...*** Sending release email to:

lambdaclan@myemail.com 11:11:38 ERROR release email delivery failed: [Errno -2] Name or service not known 11:11:38 DEBUG Running command: /usr/bin/git tag Release Summary

1 of 1 releases were successful

Whereas rez-pip does not use the same build process as such no release hook seems to be ever called

❯ rez pip -i -r yapf 11:15:05 DEBUG searching plugin path /home/lambdaclan/.rez/lib/python3.8/site-packages/rezplugins/package_repository... 11:15:05 DEBUG loading package_repository plugin at /home/lambdaclan/.rez/lib/python3.8/site-packages/rezplugins/package_repository: rezplugins.package_repository.filesystem... 11:15:05 DEBUG loading package_repository plugin at /home/lambdaclan/.rez/lib/python3.8/site-packages/rezplugins/package_repository: rezplugins.package_repository.memory... 11:15:05 INFO Trying to use pip from python package 11:15:05 DEBUG searching plugin path /home/lambdaclan/.rez/lib/python3.8/site-packages/rezplugins/shell... 11:15:05 DEBUG loading shell plugin at /home/lambdaclan/.rez/lib/python3.8/site-packages/rezplugins/shell: rezplugins.shell.bash... 11:15:05 DEBUG loading shell plugin at /home/lambdaclan/.rez/lib/python3.8/site-packages/rezplugins/shell: rezplugins.shell.cmd... 11:15:05 WARNING 'register_plugin' function at /home/lambdaclan/.rez/lib/python3.8/site-packages/rezplugins/shell: rezplugins.shell.cmd did not return a class. 11:15:05 DEBUG loading shell plugin at /home/lambdaclan/.rez/lib/python3.8/site-packages/rezplugins/shell: rezplugins.shell.csh... 11:15:05 DEBUG loading shell plugin at /home/lambdaclan/.rez/lib/python3.8/site-packages/rezplugins/shell: rezplugins.shell.powershell... 11:15:05 WARNING 'register_plugin' function at /home/lambdaclan/.rez/lib/python3.8/site-packages/rezplugins/shell: rezplugins.shell.powershell did not return a class. 11:15:05 DEBUG loading shell plugin at /home/lambdaclan/.rez/lib/python3.8/site-packages/rezplugins/shell: rezplugins.shell.pwsh... 11:15:05 DEBUG loading shell plugin at /home/lambdaclan/.rez/lib/python3.8/site-packages/rezplugins/shell: rezplugins.shell.sh... 11:15:05 DEBUG loading shell plugin at /home/lambdaclan/.rez/lib/python3.8/site-packages/rezplugins/shell: rezplugins.shell.tcsh... 11:15:05 DEBUG loading shell plugin at /home/lambdaclan/.rez/lib/python3.8/site-packages/rezplugins/shell: rezplugins.shell.zsh... 11:15:06 INFO Found pip-20.0.2 inside /home/lambdaclan/packages/python/3.8.2/package.py. Will use it with /bin/python3 11:15:06 INFO Installing 'yapf' with pip taken from '/bin/python3' 11:15:06 DEBUG

package download environment: resolved by lambdaclan@apollo, on Wed Apr 08 11:15:05 2020, using Rez v2.56.1

requested packages: python-3.8
~platform==linux (implicit) ~arch==x86_64 (implicit) ~os==Arch-rolling (implicit)

resolved packages: arch-x86_64 /home/lambdaclan/packages/arch/x86_64 (local) os-Arch-rolling /home/lambdaclan/packages/os/Arch-rolling (local) platform-linux /home/lambdaclan/packages/platform/linux (local) python-3.8.2 /home/lambdaclan/packages/python/3.8.2/platform-linux/arch-x86_64/os-Arch-rolling (local)

11:15:06 DEBUG running: /bin/python3 -m pip install --use-pep517 --target=/tmp/pip-hpxiaohz-rez yapf ╭─ ~/Workspace/GitHub/rez/example_packages/hello_world
╰─❯ 11:15:08 DEBUG Found /tmp/pip-hpxiaohz-rez/yapf-0.29.0.dist-info 11:15:08 DEBUG Getting requirements from metadata {'name': 'yapf', 'version': '0.29.0', 'license': 'Apache License, Version 2.0', 'summary': 'A formatter for Python code.', 'classifiers': ['Development Status :: 4 - Beta', 'Environment :: Console', 'Intended Audience :: Developers', 'License :: OSI Approved :: Apache Software License', 'Operating System :: OS Independent', 'Programming Language :: Python', 'Programming Language :: Python :: 2', 'Programming Language :: Python :: 2.7', 'Programming Language :: Python :: 3', 'Programming Language :: Python :: 3.6', 'Topic :: Software Development :: Libraries :: Python Modules', 'Topic :: Software Development :: Quality Assurance']} 11:15:08 DEBUG Getting requirements from metadata {'name': 'yapf', 'version': '0.29.0', 'license': 'Apache License, Version 2.0', 'summary': 'A formatter for Python code.', 'classifiers': ['Development Status :: 4 - Beta', 'Environment :: Console', 'Intended Audience :: Developers', 'License :: OSI Approved :: Apache Software License', 'Operating System :: OS Independent', 'Programming Language :: Python', 'Programming Language :: Python :: 2', 'Programming Language :: Python :: 2.7', 'Programming Language :: Python :: 3', 'Programming Language :: Python :: 3.6', 'Topic :: Software Development :: Libraries :: Python Modules', 'Topic :: Software Development :: Quality Assurance']} 11:15:08 DEBUG Pip to rez requirements translation information for yapf (0.29.0): {'pip': {'run_requires': <map object at 0x7f8614f571c0>}, 'rez': {'metadata': {'is_pure_python': True}, 'requires': [], 'variant_requires': ['python-3.8']}} 11:15:08 INFO Installed [yapf-0.29.0] /home/lambdaclan/releases/yapf/0.29.0/package.py (11141ce2a13f3ba8344491dbd6c536267b83e386) 11:15:08 INFO 1 packages were installed.

The only thing that seems to change when passing the release flag (-r) to rez pip is the location at which the package is being stored:

# src/rez/pip.py
 if prefix is not None:
        packages_path = prefix
    else:
        packages_path = (config.release_packages_path if release
                         else config.local_packages_path)

rez pip does not make use of the local build process which in turn calls the release hooks. It uses make package

# src/rez/pip.py
# create the rez package
        name = pip_to_rez_package_name(distribution.name)
        version = pip_to_rez_version(distribution.version)
        requires = rez_requires["requires"]
        variant_requires = rez_requires["variant_requires"]
        metadata = rez_requires["metadata"]

        with make_package(name, packages_path, make_root=make_root) as pkg:
            # basics (version etc)
            pkg.version = version
            .....

Since release hooks can be very useful for all kinds of automation tasks such as CI we will need to add proper release hook support to rez pip command.

The two options proposed were the following:

a. Make rez-pip command use a custom build system instead of custom package building

b. Just load and fire the hooks directly using the existing functions

After a discussion with @nerdvegas I had on Slack, it seems like the second option is preferred at least for now.

a build system for pip would be something quite different to what rez-pip is currently doing (not incorrect, just different). Imo we should look at what it would take to call the hooks directly.

@nerdvegas

I have been looking into calling the release hooks directly from pip but there are some things that need to be clarified:

the run_hooks function is part of the BuildProcessHelper class in build_process.py which inherits from BuildProcess that is meant to be sub classed to create a new build system that can define its own release function having access to pre/post release hook checks and the actual execution of the hooks
- now since we are trying to avoid defining a new build system I have extracted the run_hooks function to make it available to pip command but what about the pre/post release checks that take place normally via the pre_release and post_release functions of the BuildProcessHelper? Do we define new checks specifically for pip packages?
- the hooks seem to be expecting a developer package that is not being generated during the pip package creation process, I am assuming we will need to make one?

14:24:46 WARNING Release hook 'emailer' is not available. 14:24:46 DEBUG Traceback (most recent call last): File "/home/lambdaclan/.rez/lib/python3.8/site-packages/rez/release_hook.py", line 24, in create_release_hooks hook = create_release_hook(name, source_path) File "/home/lambdaclan/.rez/lib/python3.8/site-packages/rez/release_hook.py", line 15, in create_release_hook return plugin_manager.create_instance('release_hook', File "/home/lambdaclan/.rez/lib/python3.8/site-packages/rez/plugin_managers.py", line 327, in create_instance return plugin_type.create_instance(plugin_name, instance_kwargs) File "/home/lambdaclan/.rez/lib/python3.8/site-packages/rez/plugin_managers.py", line 205, in create_instance return self.get_plugin_class(plugin)(instance_kwargs) File "/home/lambdaclan/.rez/lib/python3.8/site-packages/rezplugins/release_hook/emailer.py", line 37, in init super(EmailReleaseHook, self).init(source_path) File "/home/lambdaclan/.rez/lib/python3.8/site-packages/rez/release_hook.py", line 52, in init self.package = get_developer_package(source_path) File "/home/lambdaclan/.rez/lib/python3.8/site-packages/rez/packages.py", line 601, in get_developer_package return DeveloperPackage.from_path(path, format=format) File "/home/lambdaclan/.rez/lib/python3.8/site-packages/rez/developer_package.py", line 100, in from_path raise PackageMetadataError("No package definition file found at %s" % path) rez.exceptions.PackageMetadataError: No package definition file found at /tmp/pip-0yjl4gfj-rez

I managed to circumvent this problem by pointing to the resulting package directly

_run_hooks(
            ReleaseHookEvent.pre_release,
            os.path.join(packages_path, name, version),
            install_path=packages_path,
            variants=variant_requires,
            release_message="release_message")

❯ rez-pip -i -r yapf 15:19:30 DEBUG searching plugin path /home/lambdaclan/.rez/lib/python3.8/site-packages/rezplugins/package_repository... 15:19:30 DEBUG loading package_repository plugin at /home/lambdaclan/.rez/lib/python3.8/site-packages/rezplugins/package_repository: rezplugins.package_repository.filesystem... 15:19:30 DEBUG loading package_repository plugin at /home/lambdaclan/.rez/lib/python3.8/site-packages/rezplugins/package_repository: rezplugins.package_repository.memory... 15:19:30 DEBUG MISS: 1.59:2::('listdir', '/home/lambdaclan/packages/python', 26083329, 1586499377.2979412) 15:19:30 DEBUG SET: 1.59:2::('listdir', '/home/lambdaclan/packages/python', 26083329, 1586499377.2979412) 15:19:30 INFO Trying to use pip from python package 15:19:30 DEBUG Retrieving memcache key: "('resolve', ('python-3.8', '~platform==linux', '~arch==x86_64', '~os==Arch-rolling'), (('filesystem', '/home/lambdaclan/packages', 23860203), ('filesystem', '/home/lambdaclan/int', 23724552), ('filesystem', '/home/lambdaclan/ext', 23877305)), '', '', False, True)" 15:19:30 DEBUG MISS: 1.59:2::('resolve', ('python-3.8', '~platform==linux', '~arch==x86_64', '~os==Arch-rolling'), (('filesystem', '/home/lambdaclan/packages', 23860203), ('filesystem', '/home/lambdaclan/int', 23724552), ('filesystem', '/home/lambdaclan/ext', 23877305)), '', '', False, True) 15:19:30 DEBUG No cache key retrieved 15:19:30 DEBUG memcache get (resolve) took 0.0008952617645263672 15:19:30 DEBUG MISS: 1.59:2::('package_file', '/home/lambdaclan/packages/python/3.8.2/package.py', 'FileFormat.py', 'None', 26088035, 1586499377.2912745) 15:19:30 DEBUG Loading file: /home/lambdaclan/packages/python/3.8.2/package.py 15:19:30 DEBUG SET: 1.59:2::('package_file', '/home/lambdaclan/packages/python/3.8.2/package.py', 'FileFormat.py', 'None', 26088035, 1586499377.2912745) 15:19:30 DEBUG Loaded resource: filesystem.package{'location': '/home/lambdaclan/packages', 'name': 'python', 'version': '3.8.2', 'repository_type': 'filesystem'} 15:19:30 DEBUG MISS: 1.59:2::('listdir', '/home/lambdaclan/packages/platform', 23879150, 1586499376.9046068) 15:19:30 DEBUG SET: 1.59:2::('listdir', '/home/lambdaclan/packages/platform', 23879150, 1586499376.9046068) 15:19:30 DEBUG MISS: 1.59:2::('package_file', '/home/lambdaclan/packages/platform/linux/package.py', 'FileFormat.py', 'None', 23879609, 1586499376.8979402) 15:19:30 DEBUG Loading file: /home/lambdaclan/packages/platform/linux/package.py 15:19:30 DEBUG SET: 1.59:2::('package_file', '/home/lambdaclan/packages/platform/linux/package.py', 'FileFormat.py', 'None', 23879609, 1586499376.8979402) 15:19:30 DEBUG Loaded resource: filesystem.package{'location': '/home/lambdaclan/packages', 'name': 'platform', 'version': 'linux', 'repository_type': 'filesystem'} 15:19:30 DEBUG MISS: 1.59:2::('listdir', '/home/lambdaclan/packages/arch', 24513623, 1586499376.9279401) 15:19:30 DEBUG SET: 1.59:2::('listdir', '/home/lambdaclan/packages/arch', 24513623, 1586499376.9279401) 15:19:30 DEBUG MISS: 1.59:2::('package_file', '/home/lambdaclan/packages/arch/x86_64/package.py', 'FileFormat.py', 'None', 24513676, 1586499376.9212735) 15:19:30 DEBUG Loading file: /home/lambdaclan/packages/arch/x86_64/package.py 15:19:30 DEBUG SET: 1.59:2::('package_file', '/home/lambdaclan/packages/arch/x86_64/package.py', 'FileFormat.py', 'None', 24513676, 1586499376.9212735) 15:19:30 DEBUG Loaded resource: filesystem.package{'location': '/home/lambdaclan/packages', 'name': 'arch', 'version': 'x86_64', 'repository_type': 'filesystem'} 15:19:30 DEBUG MISS: 1.59:2::('listdir', '/home/lambdaclan/packages/os', 25956087, 1586499376.9512737) 15:19:30 DEBUG SET: 1.59:2::('listdir', '/home/lambdaclan/packages/os', 25956087, 1586499376.9512737) 15:19:30 DEBUG MISS: 1.59:2::('package_file', '/home/lambdaclan/packages/os/Arch-rolling/package.py', 'FileFormat.py', 'None', 25956105, 1586499376.9446068) 15:19:30 DEBUG Loading file: /home/lambdaclan/packages/os/Arch-rolling/package.py 15:19:30 DEBUG SET: 1.59:2::('package_file', '/home/lambdaclan/packages/os/Arch-rolling/package.py', 'FileFormat.py', 'None', 25956105, 1586499376.9446068) 15:19:30 DEBUG Loaded resource: filesystem.package{'location': '/home/lambdaclan/packages', 'name': 'os', 'version': 'Arch-rolling', 'repository_type': 'filesystem'} 15:19:30 DEBUG SET: 1.59:2::('resolve', ('python-3.8', '~platform==linux', '~arch==x86_64', '~os==Arch-rolling'), (('filesystem', '/home/lambdaclan/packages', 23860203), ('filesystem', '/home/lambdaclan/int', 23724552), ('filesystem', '/home/lambdaclan/ext', 23877305)), '', '', False, True) 15:19:30 DEBUG Sent memcache key: "('resolve', ('python-3.8', '~platform==linux', '~arch==x86_64', '~os==Arch-rolling'), (('filesystem', '/home/lambdaclan/packages', 23860203), ('filesystem', '/home/lambdaclan/int', 23724552), ('filesystem', '/home/lambdaclan/ext', 23877305)), '', '', False, True)" 15:19:30 DEBUG memcache set (resolve) took 0.00080108642578125 15:19:30 DEBUG searching plugin path /home/lambdaclan/.rez/lib/python3.8/site-packages/rezplugins/shell... 15:19:30 DEBUG loading shell plugin at /home/lambdaclan/.rez/lib/python3.8/site-packages/rezplugins/shell: rezplugins.shell.bash... 15:19:30 DEBUG loading shell plugin at /home/lambdaclan/.rez/lib/python3.8/site-packages/rezplugins/shell: rezplugins.shell.cmd... 15:19:30 WARNING 'register_plugin' function at /home/lambdaclan/.rez/lib/python3.8/site-packages/rezplugins/shell: rezplugins.shell.cmd did not return a class. 15:19:30 DEBUG loading shell plugin at /home/lambdaclan/.rez/lib/python3.8/site-packages/rezplugins/shell: rezplugins.shell.csh... 15:19:30 DEBUG loading shell plugin at /home/lambdaclan/.rez/lib/python3.8/site-packages/rezplugins/shell: rezplugins.shell.powershell... 15:19:30 WARNING 'register_plugin' function at /home/lambdaclan/.rez/lib/python3.8/site-packages/rezplugins/shell: rezplugins.shell.powershell did not return a class. 15:19:30 DEBUG loading shell plugin at /home/lambdaclan/.rez/lib/python3.8/site-packages/rezplugins/shell: rezplugins.shell.pwsh... 15:19:30 DEBUG loading shell plugin at /home/lambdaclan/.rez/lib/python3.8/site-packages/rezplugins/shell: rezplugins.shell.sh... 15:19:30 DEBUG loading shell plugin at /home/lambdaclan/.rez/lib/python3.8/site-packages/rezplugins/shell: rezplugins.shell.tcsh... 15:19:30 DEBUG loading shell plugin at /home/lambdaclan/.rez/lib/python3.8/site-packages/rezplugins/shell: rezplugins.shell.zsh... 15:19:30 INFO Found pip-20.0.2 inside /home/lambdaclan/packages/python/3.8.2/package.py. Will use it with /bin/python3 15:19:30 INFO Installing 'yapf' with pip taken from '/bin/python3' 15:19:30 DEBUG

package download environment: resolved by lambdaclan@zeus, on Fri Apr 10 15:19:30 2020, using Rez v2.56.1

requested packages: python-3.8
~platform==linux (implicit) ~arch==x86_64 (implicit) ~os==Arch-rolling (implicit)

15:19:30 DEBUG running: /bin/python3 -m pip install --use-pep517 --target=/tmp/pip-g9blyi3t-rez yapf ╭─ ~/Workspace/GitHub/rez/example_packages/hello_world
╰─❯ 15:19:33 DEBUG Found /tmp/pip-g9blyi3t-rez/yapf-0.29.0.dist-info 15:19:33 DEBUG Getting requirements from metadata {'name': 'yapf', 'version': '0.29.0', 'license': 'Apache License, Version 2.0', 'summary': 'A formatter for Python code.', 'classifiers': ['Development Status :: 4 - Beta', 'Environment :: Console', 'Intended Audience :: Developers', 'License :: OSI Approved :: Apache Software License', 'Operating System :: OS Independent', 'Programming Language :: Python', 'Programming Language :: Python :: 2', 'Programming Language :: Python :: 2.7', 'Programming Language :: Python :: 3', 'Programming Language :: Python :: 3.6', 'Topic :: Software Development :: Libraries :: Python Modules', 'Topic :: Software Development :: Quality Assurance']} 15:19:33 DEBUG Getting requirements from metadata {'name': 'yapf', 'version': '0.29.0', 'license': 'Apache License, Version 2.0', 'summary': 'A formatter for Python code.', 'classifiers': ['Development Status :: 4 - Beta', 'Environment :: Console', 'Intended Audience :: Developers', 'License :: OSI Approved :: Apache Software License', 'Operating System :: OS Independent', 'Programming Language :: Python', 'Programming Language :: Python :: 2', 'Programming Language :: Python :: 2.7', 'Programming Language :: Python :: 3', 'Programming Language :: Python :: 3.6', 'Topic :: Software Development :: Libraries :: Python Modules', 'Topic :: Software Development :: Quality Assurance']} 15:19:33 DEBUG Pip to rez requirements translation information for yapf (0.29.0): {'pip': {'run_requires': <map object at 0x7f3dcc56d070>}, 'rez': {'metadata': {'is_pure_python': True}, 'requires': [], 'variant_requires': ['python-3.8']}} 15:19:33 DEBUG Loaded resource: memory.package{'location': 'memory{0x7f3dcc505f40}', 'name': 'yapf', 'version': '0.29.0', 'repository_type': 'memory'} 15:19:33 DEBUG MISS: 1.59:2::('listdir', '/home/lambdaclan/releases/yapf', 16258769, 1586499573.4754102) 15:19:33 DEBUG SET: 1.59:2::('listdir', '/home/lambdaclan/releases/yapf', 16258769, 1586499573.4754102) 15:19:33 DEBUG Writing to /tmp/rez_write_i8dzr_7e/package.py (local cache of /home/lambdaclan/releases/yapf/0.29.0/package.py) 15:19:33 DEBUG MISS: 1.59:2::('listdir', '/home/lambdaclan/releases/yapf', 16258769, 1586499573.502077) 15:19:33 DEBUG SET: 1.59:2::('listdir', '/home/lambdaclan/releases/yapf', 16258769, 1586499573.502077) 15:19:33 DEBUG Loading file: /tmp/rez_write_i8dzr_7e/package.py (local cache of /home/lambdaclan/releases/yapf/0.29.0/package.py) 15:19:33 DEBUG Loaded resource: filesystem.package{'location': '/home/lambdaclan/releases', 'name': 'yapf', 'version': '0.29.0', 'repository_type': 'filesystem'} 15:19:33 DEBUG searching plugin path /home/lambdaclan/.rez/lib/python3.8/site-packages/rezplugins/release_hook... 15:19:33 DEBUG loading release_hook plugin at /home/lambdaclan/.rez/lib/python3.8/site-packages/rezplugins/release_hook: rezplugins.release_hook.amqp... 15:19:33 DEBUG loading release_hook plugin at /home/lambdaclan/.rez/lib/python3.8/site-packages/rezplugins/release_hook: rezplugins.release_hook.command... 15:19:33 DEBUG loading release_hook plugin at /home/lambdaclan/.rez/lib/python3.8/site-packages/rezplugins/release_hook: rezplugins.release_hook.emailer... 15:19:33 DEBUG Loading file: /tmp/rez_write_i8dzr_7e/package.py (local cache of /home/lambdaclan/releases/yapf/0.29.0/package.py) 15:19:33 DEBUG Loaded resource: memory.package{'location': 'memory{0x7f3dcc6ba1c0}', 'name': 'yapf', 'version': '0.29.0', 'repository_type': 'memory'} 15:19:33 DEBUG Running pre-release hook 'emailer'... 15:19:33 INFO Installed [yapf-0.29.0] /home/lambdaclan/releases/yapf/0.29.0/package.py (11141ce2a13f3ba8344491dbd6c536267b83e386) 15:19:33 INFO 1 packages were installed. ╭─ ~/Workspace/GitHub/rez/example_packages/hello_world feat/rez-pip-rel-hook !2 3s  rezenv3  rezenv3 ╰─❯

I am also a bit confused about the pre/post release timing. Currently the pip command release flag only changes the path where the package is deployed. Until that package is made (at the end of the make_package context no package exists that run_hooks requires forworking dir where the development package data is expected to be found.
- should the pre release hook run within the for loop of each distribution to be installed and the post release once after all distributions are installed?

Sorry for all the questions but trying to wrap my head around all this to make the PR easier to merge when the time comes. I will keep adding more questions as they arise, thanks for the feedback in advance.

Looking at the main release function I am curious at to which parts will be necessary to be ported over to the pip command

def release(self, release_message=None, variants=None):
        self._print_header("Releasing %s..." % self.package.qualified_name)

        # test that we're in a state to release
        self.pre_release()

        release_path = self.package.config.release_packages_path
        release_data = self.get_release_data()
        changelog = release_data.get("changelog")
        previous_version = release_data.get("previous_version")
        previous_revision = release_data.get("previous_revision")

        # run pre-release hooks
        self.run_hooks(ReleaseHookEvent.pre_release,
                       install_path=release_path,
                       variants=variants,
                       release_message=release_message,
                       changelog=changelog,
                       previous_version=previous_version,
                       previous_revision=previous_revision)

        # release variants
        num_visited, released_variants = self.visit_variants(
            self._release_variant,
            variants=variants,
            release_message=release_message)

        # ignore skipped variants
        released_variants = [x for x in released_variants if x is not None]
        num_released = len(released_variants)

        # run post-release hooks
        self.run_hooks(ReleaseHookEvent.post_release,
                       install_path=release_path,
                       variants=released_variants,
                       release_message=release_message,
                       changelog=changelog,
                       previous_version=previous_version,
                       previous_revision=previous_revision)

        # perform post-release actions: tag repo etc
        if released_variants:
            self.post_release(release_message=release_message)

        self._print_header("Release Summary")

        msg = "\n%d of %d releases were successful" % (num_released, num_visited)
        if num_released < num_visited:
            Printer()(msg, warning)
        else:
            self._print(msg)

        if self.all_test_results.num_tests:
            print('')
            self.all_test_results.print_summary()
            print('')

        return num_released

test that we're in a state to release? --> self.pre_release() [missing]
release_path = self.package.config.release_packages_path --> this is what -r does with rez pip command
release_data = self.get_release_data()--> [missing]
changelog = release_data.get("changelog") --> [missing]
previous_version = release_data.get("previous_version") [missing]
previous_revision = release_data.get("previous_revisio [missing]-
perform post-release actions: tag repo etc --> [missing]

for reference this is what we have for each distribution before the make package part

# create the rez package
        name = pip_to_rez_package_name(distribution.name)
        version = pip_to_rez_version(distribution.version)
        requires = rez_requires["requires"]
        variant_requires = rez_requires["variant_requires"]
        metadata = rez_requires["metadata"]

Another big difference is that each pip package can have multiple distributions so need to know when hooks are supposed to be called.

I can see that the hook dependency to the build system is quite big because of all this extra stuff. I am not sure which parts are essential. If all we care about is running hooks before and after a pip package release then we only need run hooks and whatever that depends on plus the point at which to call each.

Ok, I've taken a look and yes I see the issues. The problem here is that by adapting pre/post-release hooks for rez-pip, we are shoehorning it into a pretty different situation than it was designed for. The main problem is that the 'developer' package (ie the rez package definition, before it's built) does not exist.

It is unclear how some existing hooks could even work in this situation (ie without a developer package). For example, consider the 'command' hook, which runs arbitrary commands, and is able to expand references such as "{package.name}". 'Package' in this case refers to the developer package.

So what to do? Turning rez-pip into its own build system would avoid the issue, however it's also a completely different thing to rez-pip as it currently stands. To do that, rez would become more of an adaption layer on other packaging systems. We would not only have to have the ability to derive a rez package definition from a setup.py, but we'd also have to have a new package repository implementation that would also be able to interpret setup.py's, as well as fetch from Pypi (for eg). In other words, lots of work, and something else entirely. So let's shelf that.

Another option that might work would be to write out the created rez packages (one for each distribution) locally, then use that as the developer package, in order to run the pre-* hooks. We still won't have some attributes (such as release info like message, previous revision and so on) but I think we could live with that.

By the way, anything related to the developer package vcs repo (eg git) such as "am I in a state to release" etc, is simply not applicable here - rez-pip is not intended to follow the same workflow as rez-release (which is all about keeping a git/etc repo correctly in sync wrt the released packages).

In short, the only approach I see that is viable is to call the hooks directly, and to do something like this:

rez-pip -i [ -r ] is called
for each dependency downloaded:
- create a local rez package (ie use make_package as is done currently)
- run pre-build/release, using this as the developer package
- do the actual install, as is done currently
- run post-release if applicable

To answer your earlier question about timing (where these hooks go), they should be before+after every distribution. By specifying -r you're "releasing" not just the target pip package, but all it's dependencies, so I would consider those N separate releases.

I am also thinking that a normal rez package release, and rez-pip release, are different enough that they may require their own release_hooks setting (ie, we'd add a pip_release_hooks attribute and use that instead).

Thoughts? A

Hell Allan, I hope you are doing well and staying safe during this time.

Thank you very much for your invaluable feedback regarding this issue.

That is what I was thinking as well, it seems like retrofitting the build system based release hooks or rez-release into rez-pip will not only be an overkill but will have us trying to guesstimate how to fill the missing links such as the develop package and its attributes to make the hooks happy.

Your explanation in terms of becoming an adaption layer confirms my initial thoughts of the above being a bad idea and since it will make a part of rez depend on another package system I do agree that we should take a different approach in line with the scope of rez.

create rez packages (one for each distribution) locally, then use that as the developer package, in order to run the pre-* hooks. We still won't have some attributes (such as release info like message, previous revision and so on) but I think we could live with that.

The key point I will take from here is that we will have a rez package for each distribution and somehow emulate that to be something like the build system's developer package. I was worried about the missing attributes but if you say we can ignore them then fine by me but I was wondering since this will be a different release hook approach not binded to the build systems one can't we just define a new developer package for pip and then specify the attributes? Do we need to use the developer package as currently interpreted by the hook functions?

anything related to the developer package vcs repo (eg git) such as "am I in a state to release" etc, is simply not applicable here - rez-pip is not intended to follow the same workflow as rez-release (which is all about keeping a git/etc repo correctly in sync with the released packages).

Can you please elaborate a bit more on this? Why is rez-pip supposed to have a different workflow? Is that because the packages in this case are made by someone else so no internal tracking is need? @instinct-vfx Any comments on this? Do we have a need to emulate the rez-release workflow even for pip packages?

call the hooks directly, and to do something like this: rez-pip -i [ -r ] is called for each dependency downloaded: create a local rez package (ie use make_package as is done currently) run pre-build/release, using this as the developer package do the actual install, as is done currently run post-release if applicable

hooks should be called before+after every distribution. By specifying -r you're "releasing" not just the target pip package, but all it's dependencies, so I would consider those N separate releases.

Overall sounds good to me and this is what I already started doing. The points about having each dependency represent a developer package (result of make_package) as well as the timing at which to call each hook make things perfectly clear.

In terms of using the hooks directly I suppose you dont mean using the existing functions correct? This is what I am currently doing but I feel that we will need to implement pip specific pre/post hook call functions. Should they reside within the pip file itself?

I am also thinking that a normal rez package release, and rez-pip release, are different enough that they may require their own release_hooks setting (ie, we'd add a pip_release_hooks attribute and use that instead).

You are talking about the rezconfig below setting right?

The release hooks to run when a release occurs. Release hooks are plugins - if a plugin listed here is not present, a warning message is printed. Note that a release hook plugin being loaded does not mean it will run - it needs to be listed here as well. Several built-in release hooks are available, see rezplugins/release_hook. release_hooks = []

If that is the case yes makes sense, we will need a pip specific setting.

Thanks again for your feedback and sorry for the many follow up questions. Trying to clarify as much as possible so that I can get back into it.

Talk to you again soon.

It's been 3 years. Any new thoughts on this development since then? Is there a public roadmap for a better rez-pip?

I'm getting bit by bug #1431 where rez-pip tries to re-install existing packages in place, and due to some __pycache__ and *.pyc garbage that appeared in the release directories with the wrong permissions (totally ignoring parent directories' sticky bits, possibly due to being loaded from a Windows mount), now rez-pip is broken for some of my pip dependencies because it can't clean up the folders due to the pythonic garbage files that were written in them because they were not secured.

I wanted to correct it by making a release hook to write-protect the variant roots so the *.pyc/__pycache__ garbage would not appear, but now I've learnt rez-pip doesn't run release hooks, so.... I guess I'm gonna have to fork the whole project now? 😢

(And before someone says so: no I don't want to set PYTHONDONTWRITEBYTECODE=1 globally everywhere.)

AcademySoftwareFoundation / rez

rez-pip release flag does not make use of release hooks #874