Add ability to cache package payloads synchronously

[isohedronpipeline]

Fixes #1379

Carrying on the work from https://github.com/AcademySoftwareFoundation/rez/pull/1452

This adds support for synchronous/blocking package caching during the resolve process.

[linux-foundation-easycla[bot]]

The committers listed above are authorized under a signed CLA.

• :white_check_mark: login: JeanChristopheMorinPerso / name: Jean-Christophe Morin (2bb1ce646cbc92ffc0987e5e0f3dab4912aec165, b0309eb295942432ba6b40f02bbd488294b4b2d7, 355a891e2710e17585a35298a082d8b24b6f58d2, a8d0439e39c3f9048ca23e0bb4f8ff326e46c0be, 71e0c4a0f1f133d720e9145eb4324035fc8faf0a, b12651ef927f62923d301c0ca99aaa5af9028b63)
• :white_check_mark: login: ttrently (d6ede1b44a2dcc37e040fd57fe9f543182e2301c, 0d1e29cb5324891d0f833dc9ecddeb0d55e134df, 98bd61f08b589f73bf3cfb00f897f7cbee6c6c6c)
• :white_check_mark: login: isohedronpipeline / name: Ben Andersen (5a55f516ecaba3b6cc678d583aeb69f1b45f9ac4, 2253d76ad9d3c8551e5e9615d8c8504dc0c8f5df, 461adb1cd0b8728fe27bc369b1134aa74ad3a45a, 5ea25760d071a87b57e04c3906083fd12b071f77, b5950d64f8de25cecb4653ea1d8a0a4bf210b29b, 2251de8dd70bf6ea758acd51e6c37b2eb5a38d54, 1c6c94b26cee5718ecd22d931fd8887848cacd0f, d8dbe4eed2cc397583f5d4f92f01b21ac1b354d5, 72747ca418cb8f710fda474d6cf46b265f6fd8ef, a5997b94a412b7534eb4be219ca85bf12ea88352, 9463a3bc28888aa494dd13a6b2ca1359597be8be)

[codecov[bot]]

Codecov Report

Attention: Patch coverage is 49.09091% with 28 lines in your changes missing coverage. Please review.

Project coverage is 58.39%. Comparing base (153bd87) to head (355a891). Report is 38 commits behind head on main.

Files	Patch %	Lines
src/rez/package_cache.py	51.21%	17 Missing and 3 partials :warning:
src/rez/cli/env.py	25.00%	4 Missing and 2 partials :warning:
src/rez/resolved_context.py	80.00%	0 Missing and 1 partial :warning:
src/rez/rezconfig.py	0.00%	1 Missing :warning:

Additional details and impacted files

```diff @@ Coverage Diff @@ ## main #1679 +/- ## ========================================== + Coverage 58.03% 58.39% +0.36% ========================================== Files 127 126 -1 Lines 17069 17205 +136 Branches 3496 3519 +23 ========================================== + Hits 9906 10047 +141 + Misses 6496 6489 -7 - Partials 667 669 +2 ```

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

[JeanChristopheMorinPerso]

I took a deeper look, and as things are in this PR, it's doesn't work as you would expect. If you were to rez-env maya --pkg-cache-mode async and then rez-env maya --pkg-cache-mode sync and initial cache (when we ran with async) takes long enough, the second rez-env (in sync mode) will skip the maya package. See https://github.com/AcademySoftwareFoundation/rez/blob/main/src/rez/package_cache.py#L210-L219. Basically, the package cache will skip any variant that is marked as in-progress. This would result in a non-localized environment.

Additionally, I still have to verify, but I think that we should really avoid using --daemon when caching synchronously. If we use --daemon, it would mean that cancelling rez-env --pkg-cache-mode sync would not cancel the caching process. After all, keeping the process alive is the the reason to have a daemon mode in the first place. Or I'd go even further, we should not use a subprocess.

We should also consider adding a progress bar or some logs when using the sync mode. I anticipate that users will complain if rez-env maya houdini blocks for multiple minutes without clearly saying what's going on.

[isohedronpipeline]

I took a deeper look, and as things are in this PR, it's doesn't work as you would expect. If you were to rez-env maya --pkg-cache-mode async and then rez-env maya --pkg-cache-mode sync and initial cache (when we ran with async) takes long enough, the second rez-env (in sync mode) will skip the maya package. See https://github.com/AcademySoftwareFoundation/rez/blob/main/src/rez/package_cache.py#L210-L219. Basically, the package cache will skip any variant that is marked as in-progress. This would result in a non-localized environment.

I think it would be preferable to call add_variant(force=True) rather than poll until the .copying sentinel file has disappeared, but I'm not sure what to do about the file lock. We really don't want to be copying the same path at the same time. Perhaps we should modify this logic to copy to an intermediate location that does an atomic move operation once the copy has completed to avoid errors with multiple cache operations running concurrently. I'm not sure there is a great solution for mixing sync and async caching operations for the same files at the same time.

Additionally, I still have to verify, but I think that we should really avoid using --daemon when caching synchronously. If we use --daemon, it would mean that cancelling rez-env --pkg-cache-mode sync would not cancel the caching process. After all, keeping the process alive is the the reason to have a daemon mode in the first place. Or I'd go even further, we should not use a subprocess.

Yeah, I'm fine with that. I'll move some of the logic from the cli entry point into package_cache later today or tomorrow.

We should also consider adding a progress bar or some logs when using the sync mode. I anticipate that users will complain if rez-env maya houdini blocks for multiple minutes without clearly saying what's going on.

We can have something print out in the _while_copying() function that is threaded. Since we're using shutil.copytree I don't think we're going to have much by the way of stats about total size/amount transferred/estimated time to completion without hammering the disk, which I'd like to avoid.

[JeanChristopheMorinPerso]

I think it would be preferable to call add_variant(force=True) rather than poll until the .copying sentinel file has disappeared, but I'm not sure what to do about the file lock. We really don't want to be copying the same path at the same time. Perhaps we should modify this logic to copy to an intermediate location that does an atomic move operation once the copy has completed to avoid errors with multiple cache operations running concurrently. I'm not sure there is a great solution for mixing sync and async caching operations for the same files at the same time.

I'm not too sure why we would use force=True or why you don't want to poll. Can you explain your thought a bit more please? Like you say, we really don't want to copy twice or copy the same package multiple times in parallel. The code goes to great length to avoid this.

[isohedronpipeline]

I'm not too sure why we would use force=True or why you don't want to poll. Can you explain your thought a bit more please? Like you say, we really don't want to copy twice or copy the same package multiple times in parallel. The code goes to great length to avoid this.

You could be right. I am worried that a previous sync process could have failed, leaving behind the .copying file, but no further updates to it will have been made. I would imagine that one use case for doing a package cache syncronously would be to force all of the caching operations to happen right now as possibly a way to restart a locked or frozen cache operation.

I see that the copying file is updated every .2 seconds though, so we should be able to check that and continue to wait while that file is being updated.

My hesitation to hit the disk repeatedly is maybe misplaced since the cache is on the local disk. In studio environments, I hesitate to hit network drives with something that could be run many times in parallel. Of course you're right and starting multiple large file transfers instead is madness.

[JeanChristopheMorinPerso]

The package cache class already knows how to handle stalled packages. It should be able to handle the case where a variant is left in progress due to an error or whatever else and restart it. See https://github.com/AcademySoftwareFoundation/rez/blob/dc2c77757748e87d2abf0967924ee0195aa0f11d/src/rez/package_cache.py#L830-L835

As for polling, I agree that polling a shared file system is not wanted. Though, caching is usually done on local disks. We could maybe use inotify to optimize ans simplify the process.

[chadrik]

Hi @JeanChristopheMorinPerso I think all the outstanding issues have been addressed. Any other thoughts?

[ameliendeshams]

Hi @isohedronpipeline,

I'm delighted to see this pull request! We've been waiting for this feature, and it's great to see it implemented.

At Fortiche, we've tested this branch, and it works exactly as expected. The synchronous package caching is a game-changer for our studio environments, where we often need to cache packages quickly and reliably.

We're also happy to see that you've addressed the concerns about progress and logging in sync mode. It's essential to provide users with clear feedback when running long-running commands.

Thanks again for your work on this feature! We're excited to see it merged and start using it in our production environments.