Houdini - Split Render, allow Export Job to run locally

BigRoy commented 9 months ago

Is there an existing issue for this?

[X] I have searched the existing issues.

Please describe the feature you have in mind and explain what the current shortcomings are?

Currently the Houdini rendering implementations allow renders to split the export and submit a standalone render job after. For my USD Render ROP implementation through Husk I'm supporting this as well, however I'd like to implement a toggle to allow the Export Job to be exported locally instead of on the farm - and then only submit the render job.

This is because the USD exports from a lighting scene tend to contain mostly references only - the exports, even for large scenes in my test runs are rather quick (at most!). Many cases even run in just a few seconds. When going through the farm then suddenly I need farm machines checking out Houdini licenses for what then becomes a 30 second job due to Houdini starting up, the LOPs graph not being pre-loaded, etc.

So basically benefit is:

Not taking the render farm machines to do slowly what can be done instantly locally
Not taking Houdini licenses

Especially because the standalone render job doesn't need a Houdini license that means that after the submission with a local export the farm machines don't need the Houdini licenses at all. ❤️

How would you imagine the implementation of the feature?

I think it'd be good to refactor the Deadline submissions in a way so that the Abstract Submit Deadline plug-in just takes a 'sequence of jobs' which it can make dependent on each other. The default implementation just making the sequence of a single render job as is used across most DCCs.

Then inherited classes could just add into that sequence. By making the sequence of jobs to be stored in instance.data it also means that other plug-ins could adjust that order as well, to e.g. skip a certain job if run locally or to add a certain job (e.g. a thumbnail generation for something, after the render job). It could even be that the "publish job" often used for renders is just one of those steps then submitted and processed by one plug-in to Deadline.

Are there any labels you wish to add?

[X] I have added the relevant labels to the enhancement request.

Describe alternatives you've considered:

No response

Additional context:

No response

[cuID:OP-8057]

BigRoy commented 4 months ago

This is partially implemented because it currently allows rendering locally as well - but I don't believe it's possible yet to only do the exporting local yet submit the rendering to the farm. @MustafaJafar correct?

MustafaJafar commented 4 months ago

I don't believe it's possible yet to only do the exporting local yet submit the rendering to the farm. @MustafaJafar correct?

Correct, render target described in the issue is not implemented yet. I think it can be added easily.

Add new render target e.g. in mantra (I think we should move it to libs. it's reused many times) from https://github.com/ynput/ayon-houdini/blob/b2ceb5efdc93aed78b105a4f706d678a2e124a28/client/ayon_houdini/plugins/create/create_mantra_rop.py#L91-L96 to

        render_target_items = {
            "local": "Local machine rendering",
            "local_no_render": "Use existing frames (local)",
            "farm": "Farm Rendering",
            "farm_split": "Farm Rendering - Split export & render jobs",
            "farm_local_export": "Farm Rendering - Local export",
        }

Modify from https://github.com/ynput/ayon-houdini/blob/b2ceb5efdc93aed78b105a4f706d678a2e124a28/client/ayon_houdini/plugins/publish/collect_farm_instances.py#L23-L36 to

        # Collect Render Target
        if creator_attribute.get("render_target") not in {
            "farm_split", "farm", "farm_local_export"
        }:
            instance.data["farm"] = False
            instance.data["splitRender"] = False
            self.log.debug("Render on farm is disabled. "
                           "Skipping farm collecting.")
            return

        instance.data["farm"] = True
        instance.data["splitRender"] = (
            creator_attribute.get("render_target") == "farm_split"
        )

Modify ExtractRender from https://github.com/ynput/ayon-houdini/blob/b2ceb5efdc93aed78b105a4f706d678a2e124a28/client/ayon_houdini/plugins/publish/extract_render.py#L33 to

        if instance.data["splitRender"] or creator_attribute.get("render_target") == "farm_local_export" :

and from https://github.com/ynput/ayon-houdini/blob/b2ceb5efdc93aed78b105a4f706d678a2e124a28/client/ayon_houdini/plugins/publish/extract_render.py#L56-L62 to

        if creator_attribute.get("render_target") in {"local", "farm_local_export"}:
            ropnode = hou.node(instance.data.get("instance_node"))
            render_rop(ropnode)

        if instance.data.get("farm"):
            self.log.debug("Render should be processed on farm, skipping local render.")
            return

ynput / ayon-houdini