Houdini: update frames in latest version

MustafaJafar commented 3 months ago

Changelog Description

Rendering is expensive, sometimes it is helpful only to re-render changed frames and reuse existing. Todo list:

[ ] Add profiles to CollectFramesFixDef settings
[x] ExtractLastPublished: A generic extractor to copy the files of the latest published version last_version_published_files . It only copies files that do not need to be fixed.
ExtractRender
- [ ] local render: overwrite them by rendering the frames to fix.. By looping over the frame ranges and setting the parms, rendering and finally returning it as it was.
- [ ] Farm render: update the job_info.Frames. luckily it expects the same format as frames_to_fix.

Additional info

This PR is the Houdini equivalent for https://github.com/ynput/OpenPype/pull/4362

Additional info 2

Unrelated to this PR. but, I'll just save some notes for the future.

this line in nuke/extrack_render_local can be replaced by frames_to_render = intspan(frames_to_fix).ranges() and remove the definition of _get_frames_to_render

Testing notes:

start with this step
follow this step

ynbot commented 3 months ago

Task linked: OP-7433 Houdini: overwrite existing cache

BigRoy commented 3 months ago

Oh yeah baby - as soon as we'll have a steady "workflow" for this I'll also be porting this to Maya and Fusion integration. However, few questions:

Avoid `intspan`?

Does it really warrant introducing a new dependency intspan? By the way, it seems the corresponding bitbucket homepage for intspan does not exist anymore and the latest release is from "Sep 24, 2019" which may indicate that it's dead? (Which doesn't necessarily mean it's not useful - just not maintained anymore?). In essence it's also a single file and could be vendorized? Or better, maybe we can already use clique for this?

import clique
collection = clique.parse("1,3-5,10", "{ranges}")
print(list(collection.indexes))
# [1, 3, 4, 5, 10]

Is the logic ignoring aovs?

I have a feeling the logic is assuming a single instance in houdini turns into one published file - but I believe, when e.g. submitting to the farm - there may be a single render instance+job that ends up publishing multiple separate AOVs/products? Is this logic allowing that?

Publishing into existing version?

I have a feeling that publishing into the previous version is the wrong and unsafe approach. Instead, I wonder whether we should instead publish a new version but just happen to re-use frames from the previous publish. So that the old (shorter) publish still exists as previous version and our new version just happens to be extended?

Note that similar logic was actually originally in OpenPype as well for Maya called "extend frames" where the new render frame range that you're publishing/submitting would be appended/replaced to the previous publish generating a new (potentially longer) version.

What I like about the current concept here however is that you can specify the extract "new" frames you want to generate or replace. So that e.g. you could say, add three frames to the end but also replace frames 15-20. Do note however, that whatever render logic we'd end up with that we ALSO avoid rendering the actual frames we DO NOT want to re-render but want to preserve from the previous render. I have a feeling that this PR Is lacking the logic that makes it skip existing frames? (Which you do seem to mention as "todo" on your PR description)

Extend frames / render new base logic

With Nuke having something in this area, apparently - and now Houdini also trying to have it. And Maya originally having "extend frames" logic as well (inheriting from avalon) I really feel it's worth to abstract this way into "base publishing logic" that hopefully works for all publish types that write out per-frame files.

It might be worth laying out/proposing a framework that we could use that way so that hopefully to support rendering only a subset of frames or extending an existing publish becomes trivial across each host integration without much effort on the development side. I feel like even this PR seems to be doing a lot of 'logic' that should be abstracted away and joined together with any feature in another integration also supporting it.

In essence it is:

previous_frames = get_last_publish_frames()
current_frames = get_current_frames()
publish frames = previous_frames.update(current_frames)

However it gets more complicated upon merging when deciding what to do with 'handles', etc. or what if new frame range is 1020-1030 and the previous was 1000-1010. What do we do with the gap? Currently nothing in AYON expects gaps in published frame ranges.

Until AYON supports products with 'custom frame ranges' (and thus also GAPS) we may just want to have an additional global validator that ensures upon extending that it's a continuous frame range.

MustafaJafar commented 3 months ago

@BigRoy @moonyuet @iLLiCiTiT This PR may take some time also, it affects ayon-core. should I close it and re-open after ayon-houdini craetion ?

BigRoy commented 3 months ago

Yes please.

iLLiCiTiT commented 2 months ago

This PR may take some time also, it affects ayon-core.

You mean modification of dependencies? That dependency should be added to ayon-houdini, not to ayon-core.

MustafaJafar commented 2 months ago

Re-created in ayon-houdini https://github.com/ynput/ayon-houdini/pull/45

ynput / ayon-core