Houdini: Publish any ROP node (Generic Creator)

[BigRoy]

Changelog Description

This PR implements an idea for "lower level publishing" in Houdini. This implement Generic ROP publishing. Just create any Houdini ROP node (or custom Rop node HDA) and publish your product types from it!

This PR adds a Generic Creator: Currently, It injects the ayon meta data onto ROP nodes to behave as though they were produced by various creators from Ayon. This ensures that validators and collectors are retained without loss. e.g. using Generic creator while setting product type to pointcache should yield similar/same results as using pointcache creator Here is an example using labs karma node and make it behave as though it was made by karma creator. https://github.com/ynput/ayon-core/pull/542#issuecomment-2179333623

This PR originally also contained a Dynamic Runtime Instance creator. That is now separated to another PR here: https://github.com/ynput/ayon-core/pull/691

Additional info

As part of the Ynput Houdini Workgroup sessions I developed this quick prototype to expose a way to batch publish and ingest files. Consider it more of an exploration of what's possible then a "drop it in production now" ready-to-go solution.

Explainer

Yes, this requires some explanation. Here you go.

https://github.com/ynput/ayon-core/assets/2439881/5f767493-10bd-41f6-b54f-89313a96da33

What I forgot to add is that it currently still relies on detecting what the output files are for a ROP node based on a "file" parm that is often unique in Houdini per node. If anyone knows a way to just query the expected output files for a ROP node (similar to what PDG/TOPs seems to do I'd love to know!) but otherwise we'll just need to expand that list.

However, I also played with the idea of having "custom file list" attributes on the Node that when enabled could override the "Collector" logic and would instead use that list of files as the publish instance's files. So that e.g. one instance could also publish multiple representations. For that, @MustafaJafar did this lovely non-functional 'prototype' but it does get the idea across:

TODO

• [x] Use $OS as default variant name.
• [x] Add "OnCreated" callback to allow a studio to configure which nodes to automatically imprint as 'publishable'. See Fabia's first video here at 08:10 and code here
  • [ ] This is currently a first iteration - should still be exposed to project settings and allow user configuration.
• [x] The product types being an enum with predefined standards per node type.
• [ ] Implement good support for multiple representations, e.g. like this and Fabia's second video near the end.
• [ ] The comment field being available on the ROP nodes
  • [x] This does actually already work if you'd add the product type to ayon+settings://core/publish/collect_comment_per_instance (although not houdini-specific nor as nice?)
• [ ] Add "Publish" button directly on the node similar to "self publish" or whatever we had for others
• [ ] Allow exposing to settings which node types may need to be "automatically imprinted" on create for a studio to customize, with also settings for allowed product types (the default type, default variant name, etc.)
• [ ] Make sure this API approach remains a goal of the PR.
• [ ] Keep in mind Fabia's usage where their AX Publish Submitter can submit just the publish of existing files to the farm and create e.g. a nuke dependency job that generates custom previews of the intermediate media, etc.

Demo scene file

The demo scene file:

ynts_char_hero_pdg_v012.zip

Testing notes:

1. Check out the branch
2. Check out the explainer video
3. Test the demo scene file
4. Comment with great ideas on how to improve

[BigRoy]

Instances are not unique error in demo file

So in my recording I hit this issue with the low-level PDG example:

instance product names ['/HoudiniTesting > pointcacheMain'] are not unique. Please remove or rename duplicates

With that demo scene you'll hit the same error because those PDG nodes cook in-process so the logic runs in the current houdini process. Since it triggers a regular CreateContext logic it also collects whatever I happen to have in my Houdini scene as instances, like the pointcacheMain ROP I generated during the demo itself - oops.

So basically even without the dynamic instances the CreateContext already initializes to:

Collected create context with 3 instances
- pointcacheMain (pointcache)
- modelTess (pointcache)
- workfilePdg (workfile)

Deleting those nodes for model and pointcache from the workfile will fix that and allow you to test it further. Note that the workfile will always be collected (but the instance could technically be deactivated through the instance, etc.) Anyway, something to be aware of. (You could technically also open the publisher UI, toggle off all instances, click save and close it again so that they are off by default when the CreateContext finds them)

The better solution would be to run the cook for those PDG/TOPs nodes Cook (Out-of-process) but you'll need to make sure that those run in the same outer process so they have access to the exact python objects that is the CreateContext instance itself.

---

TODO

• [ ] Also noticed the dynamic logic doesn't like it yet if the ROP fetch uses a ROP that generates a single file - so that should be "todo" to fix as well, shouldn't be too hard.

[MustafaJafar]

Hey, I was testing Generic Creator. It looks cool. I made few modifications let me share them. I'll create two PRs one for each modification. First PR: https://github.com/BigRoy/ayon-core/pull/4 Houdini: Add 'Make AYON Publishable' to OPMenu Second PR: https://github.com/BigRoy/ayon-core/pull/5 Houdini: add 'generic' family to collect farm instances and skip render if local publishing.

Also, few notes/questions:

1. CreateHoudiniGeneric.create simply converts the publisher attr defs to native Houdini parameters and CreateHoudiniGeneric.collect_instances simply collects the values of these Houdini parameters in the same structure mimicking the regular instances.
2. family/product_type of the collected generic instances is forced to CreateHoudiniGeneric.product_type attribute which is always generic
3. generic instances trigger this early collector CollectNoProductTypeFamilyGeneric that ensures that family is set to "generic" and families is ["generic", "rop"]
4. only publish plugins with families set to ["*"] are triggered for generic instances. and no product specific publish plugins atm.
5. specific publish plugins may cause issues with generic instances because these plugins were built expecting a ROP node in /out.
6. which family is favored in publish plugins rop or generic ?
7. product name templates are ignored, correct ? maybe we can add a toggle compute name from template name profile and when enabled it disables the product name parameter and use ayon_core.pipeline.create.get_product_name

I find it cool as low level publishing that by passes all validators to just publish the exported data. But, I think we should consider the product specific publish plugins because e.g.

1. publishing a static mesh that is not static won't make sense.
2. publishing render will actually skip the AOVs because collect render product won't work. not that collect render products is used with local render publishing. but, I think this issue has an alternative solution which is Output Files list.

https://github.com/ynput/ayon-core/assets/20871534/eec09ec7-56db-4b0d-b247-4a54be97314b

[MustafaJafar]

What about converting the product type Houdini parameter in generic instances to a drop down menu. I think it'd be more user friendly although it would need some maintenance when adding/modifying creators.

from ayon_core.pipeline import registered_host
from ayon_core.pipeline.create import CreateContext
import hou

host = registered_host()
assert host, "No registered host."
create_context = CreateContext(host)

product_types = set()
for _, creator in create_context.manual_creators.items():
    if creator.product_type == "generic":
        continue
    elif creator.product_type in {
        "karma_rop", "mantra_rop", "arnold_rop",
        "redshift_rop", "vray_rop", "usdrender"
    }:
        product_types.add("render")
    else:
        product_types.add(creator.product_type)

print(product_types)

[fabiaserra]

Here's my demo(s)!

https://github.com/ynput/ayon-core/assets/4348536/5b2bece1-8242-441e-9a4d-3010c10420b6

I had to re-record the end because my microphone did a prank on me too

https://github.com/ynput/ayon-core/assets/4348536/8841a171-a5dc-4d82-98ee-03108643e502

[fabiaserra]

Here's my demo(s)!

I'm now realizing that I forgot to touch on what the AX Publisher Submitter ROP node does. It's basically a very simple approach to traverse through the whole upstream node graph (taking into account bypassed nodes and switches) and run the publish submission on the nodes that have the publish folder.

And I'm not sure if I mentioned this point on my second recording but the main thing here is that this pipeline I showed is production ready, all the tools that artists need to render/cache in the farm and publish all types are there. This is what an MVP (minimum viable product) is, after having this, then you can start adding complexity, but we should never start from a complex system and work all the way down IMO

[MustafaJafar]

Hey, I was testing the batch creator and I find it very cool. I wrote my notes about it in this gist https://gist.github.com/MustafaJafar/bd2a388e4a6aa3613d64a186ebb6660c as well as I extended the batch creator in PR https://github.com/BigRoy/ayon-core/pull/6 to support multiple representation.

Here's screen shot after seeing "Publish succeeded." for the first time.

Some demo

https://github.com/ynput/ayon-core/assets/20871534/373236ad-987c-4194-8712-4364dc026199

[BigRoy]

Just want to say thanks @fabiaserra @MustafaJafar - those are great details!

I like:

• [x] The variant (or in your case direct product name) being $OS by default. - this PR currently defaults to just Main.
• The comment field being available on the ROP nodes - which this PR lacks. -> added as todo on PR description
• The product types being an enum with predefined standards per node type. -> added as todo on PR description
• The publish logic being available even if just creating a regular Houdini ROP node. -> added as todo on PR description
• The flipbook tool! -> likely separate PR down the line.

I dislike (some from the demos, some are comments on how things currently work in AYON):

• Them not being detected as regular publish instances (which could greatly borrow from this PRs logic)
• The need to go directly through completely separate publishing/ingesting logic. What I'd want to come out of this PR and/or concept is have a low-level API that would make that technically feasible (and easy!) out of the box. Whether that'd still go through pyblish or not is up for discussion, but just having something almost as simple as a publish(product_name, representations) functionality exposed would be great.
• The validators are, like @fabiaserra described, too strict (or just plain wrong) and block just having this be a generic "publish me anything" workflow. We should find a way to have those validators be very targeted to only a very specific workflow, instead of just "all pointcache product types" from Houdini. Maybe even link it to a particular creator (or its identifier). What are your thoughts @iLLiCiTiT @antirotor ?
• I dislike the flipbook itself seeming quite 'unlinked' to a particular publish of e.g. a pointcache (I'd preferably have that tightly coupled together so we can pipeline-wise make assumptions like "this review belongs to this published pointcache")
• I want a headless publish to preferably also be able to generate a json publish report that can be loaded in the publish report viewer. However, @iLLiCiTiT told me that logic is still tightly coupled to the UI currently but shouldn't need to be. Once that's lower-level available on any publish then we can have say a PDG publish, or whatever publish log, be shared for someone on another machine (like a dev) to debug what went wrong with that particular publish.

So as @MustafaJafar stated:

What about converting the product type Houdini parameter in generic instances to a drop down menu.

Yes! But preferably in a way that we don't need to go create complex creators all over the place, etc.

---

Also got some comments from Alexandru Preoteasa (Static VFX):

I just took a look at the Lower level publishing concepts you put up on github. Looks interesting and seems like the AYON api gives you quite a decent amout of control. One thing I'm really interested in TOPs rather than batch publish and ingest is actually being able to automate processes that have to run in a chain and have all the reviews published along the way.

Which made me wonder @fabiaserra did you end up doing that in your setups as well? I suppose just a regular OpenGL node could be used in the /out network with review product type and due to how dependencies work with the publishing with Deadline that would automatically become like a dependent job, etc.

---

Deadline submissions, and then publishing.

@fabiaserra What's a bit unclear from your video is that you're using the regular deadline submission. Does that automatically also publish the output in a separate deadline job that gets created by the deadline submitter because you "hacked that into it"? Or how does that work? It seems you're basically submitting it to Deadline on its own - basically close to no AYON related logic to that, but then when/how does that publish through AYON?

---

Flipbook tool

Also really like the Flipbook tool you showed in the Part2 video at 03:30 with how easy it is to create the preview, check it and then publish it directly there.

[fabiaserra]

Thank you @BigRoy for summarizing your highlights and trying to transform these into action items! Let me try respond to some of your points.

• Them not being detected as regular publish instances (which could greatly borrow from this PRs logic)

I agree that would be nice to close the bridge with making use of the existing publish dialog and plugin system (although quite a lot of Houdini TDs would still prefer to submit their publishes through the node graph like we do with the AX Publisher Submitter and not a separate dialog) but the main point IMO is whether the AYON publish framework is fit for accommodating this workflow or we are trying to use a screwdriver as a hammer (as brought up a bit ago by Max on this post https://community.ynput.io/t/to-pyblish-or-not-to-pyblish/932).

• The need to go directly through completely separate publishing/ingesting logic. What I'd want to come out of this PR and/or concept is have a low-level API that would make that technically feasible (and easy!) out of the box. Whether that'd still go through pyblish or not is up for discussion, but just having something almost as simple as a publish(product_name, representations) functionality exposed would be great.

I agree and I guess this touches my prior point. We created our own separate API that abstracts the process of creating the .json file to run headless publishes and I keep using on all the tools that I have been building for our needs and I find it extremely useful. This stems from the fact that when I needed to create these tools I found it very hard to dissect the publish framework to run it in a simple manner from other places. If I had understood then how to do it through the system, I wouldn't have created this separate abstraction. I think the batch publishing introduced here kind of goes into this direction but I still find it too verbose and rigid, like mentioned by @BigRoy here https://gist.github.com/MustafaJafar/bd2a388e4a6aa3613d64a186ebb6660c?permalink_comment_id=5067094#gistcomment-5067094

• The validators are, like @fabiaserra described, too strict (or just plain wrong) and block just having this be a generic "publish me anything" workflow. We should find a way to have those validators be very targeted to only a very specific workflow, instead of just "all pointcache product types" from Houdini. Maybe even link it to a particular creator (or its identifier). What are your thoughts @iLLiCiTiT @antirotor ?

Yeah, unless the validators are 100% reliable and cover all use cases (quite impossible in reality, specially when AYON tries to fit a lot of workflows from different studios), it adds extra blockers to using the system. All validators should be optional and the publish dialog shouldn't show you ALL the plugins that exist in the system because it's very hard to know which plugins are actually required for that publish instance and they add too much noise to make the dialog actually useful to read.

• I dislike the flipbook itself seeming quite 'unlinked' to a particular publish of e.g. a pointcache (I'd preferably have that tightly coupled together so we can pipeline-wise make assumptions like "this review belongs to this published pointcache")

Yeah definitely, although that would be a very easy addition to the abstracted publish API to collect inputs on submission (reusing the same logic of the existing AYON plugin for doing that).

So as @MustafaJafar stated:

What about converting the product type Houdini parameter in generic instances to a drop down menu.

Yes! But preferably in a way that we don't need to go create complex creators all over the place, etc.

Please!

Which made me wonder @fabiaserra did you end up doing that in your setups as well? I suppose just a regular OpenGL node could be used in the /out network with review product type and due to how dependencies work with the publishing with Deadline that would automatically become like a dependent job, etc.

In my publish abstraction module, for image type sequences (i.e. render, review...) I spin up a Deadline task as a pre-dependency to the AYON/OP publish that runs a Deadline Nuke plugin that creates a .mov from the image sequence through a Nuke template script that we define (globally or per show if we want to set any overrides). That generated video is what gets uploaded into Shotgrid as the video representation of the publish version (by adding it into the expected representations of the .json that we use to do the headless publish) so we would very rarely need a separate review instance. Using OpenGL nodes as part of the publish process would be possible to add on the TOP dependency graph but that would just be only for flipbook type of reviews and it would have some limitations -> running the OpenGL ROP in the farm requires a GPU on the worker.

Deadline submissions, and then publishing.

@fabiaserra What's a bit unclear from your video is that you're using the regular deadline submission. Does that automatically also publish the output in a separate deadline job that gets created by the deadline submitter because you "hacked that into it"? Or how does that work? It seems you're basically submitting it to Deadline on its own - basically close to no AYON related logic to that, but then when/how does that publish through AYON?

Correct, we have two different Deadline submission entry points. I haven't hacked the vanilla Houdini Deadline submission yet to make it understand AYON and automatically create publish tasks as dependency tasks, although it would be certainly possible and not too hard to do. However, we currently are on the philosophy that artists need to always validate the things that get published so we wouldn't want to run the publish tasks on the same render submission. The two entry points are:

1. Deadline ROP submitter to render and orchestrate dependencies of the node graph in the farm (with only a few modifications to support USD Render and inject the OP/AYON env vars to extract the AYON environment on runtime) with very little saying from AYON.
2. AX Publisher Submitter to ONLY submit the publish of the previously generated outputs (it runs some simple pre-validations to make sure the outputs exist on disk) as separate Deadline tasks running the AYON plugin. For each node that's publishable we get a task such as this one (on this case it also automatically contains a Nuke task as a dependency that generates the .mov to upload as the review representation):

Flipbook tool

Also really like the Flipbook tool you showed in the Part2 video at 03:30 with how easy it is to create the preview, check it and then publish it directly there.

The tool is basically this right now, the code is not the cleanest as it was mostly just a copy paste from an old tool we had in our legacy pipeline to have an MVP they can start using already:

import os
import logging

from qtpy import QtWidgets, QtGui
import hou

from ayon_core.lib import path_tools
from ayon_core.modules.deadline.lib import publish

log = logging.getLogger(__name__)

class FlipbookDialog(QtWidgets.QDialog):
    def __init__(self, parent=None):
        QtWidgets.QDialog.__init__(self, parent)

        self.scene_viewer = hou.ui.paneTabOfType(hou.paneTabType.SceneViewer)

        # other properties
        self.setWindowTitle("Flipbook")

        # define general layout
        layout = QtWidgets.QVBoxLayout()
        groupLayout = QtWidgets.QVBoxLayout()

        # output toggles
        self.outputToMplay = QtWidgets.QCheckBox("MPlay Output", self)
        self.outputToMplay.setChecked(True)

        self.beautyPassOnly = QtWidgets.QCheckBox("Beauty Pass", self)
        self.useMotionblur = QtWidgets.QCheckBox("Motion Blur", self)

        # description widget
        self.descriptionLabel = QtWidgets.QLabel("Description")
        self.description = QtWidgets.QLineEdit()

        resolution = self.get_default_resolution()

        # resolution sub-widgets x
        self.resolutionX = QtWidgets.QWidget()
        resolutionXLayout = QtWidgets.QVBoxLayout()
        self.resolutionXLabel = QtWidgets.QLabel("Width")
        self.resolutionXLine = QtWidgets.QLineEdit(resolution[0])
        resolutionXLayout.addWidget(self.resolutionXLabel)
        resolutionXLayout.addWidget(self.resolutionXLine)
        self.resolutionX.setLayout(resolutionXLayout)

        # resolution sub-widgets y
        self.resolutionY = QtWidgets.QWidget()
        resolutionYLayout = QtWidgets.QVBoxLayout()
        self.resolutionYLabel = QtWidgets.QLabel("Height")
        self.resolutionYLine = QtWidgets.QLineEdit(resolution[1])
        resolutionYLayout.addWidget(self.resolutionYLabel)
        resolutionYLayout.addWidget(self.resolutionYLine)
        self.resolutionY.setLayout(resolutionYLayout)

        output_path = self.get_output_path()
        self.outputLabel = QtWidgets.QLabel(
            f"Flipbooking to: {output_path}"
        )

        # resolution group
        self.resolutionGroup = QtWidgets.QGroupBox("Resolution")
        resolutionGroupLayout = QtWidgets.QHBoxLayout()
        resolutionGroupLayout.addWidget(self.resolutionX)
        resolutionGroupLayout.addWidget(self.resolutionY)
        self.resolutionGroup.setLayout(resolutionGroupLayout)

        # frame range widget
        self.frameRange = QtWidgets.QGroupBox("Frame range")
        frameRangeGroupLayout = QtWidgets.QHBoxLayout()

        # frame range start sub-widget
        self.frameRangeStart = QtWidgets.QWidget()
        frameRangeStartLayout = QtWidgets.QVBoxLayout()
        self.frameRangeStartLabel = QtWidgets.QLabel("Start")
        self.frameRangeStartLine = QtWidgets.QLineEdit("$RFSTART")

        frameRangeStartLayout.addWidget(self.frameRangeStartLabel)
        frameRangeStartLayout.addWidget(self.frameRangeStartLine)
        self.frameRangeStart.setLayout(frameRangeStartLayout)
        frameRangeGroupLayout.addWidget(self.frameRangeStart)

        # frame range end sub-widget
        self.frameRangeEnd = QtWidgets.QWidget()
        frameRangeEndLayout = QtWidgets.QVBoxLayout()
        self.frameRangeEndLabel = QtWidgets.QLabel("End")
        self.frameRangeEndLine = QtWidgets.QLineEdit("$RFEND")

        frameRangeEndLayout.addWidget(self.frameRangeEndLabel)
        frameRangeEndLayout.addWidget(self.frameRangeEndLine)
        self.frameRangeEnd.setLayout(frameRangeEndLayout)
        frameRangeGroupLayout.addWidget(self.frameRangeEnd)

        self.frameRange.setLayout(frameRangeGroupLayout)

        # copy to path widget
        self.copyPathButton = QtWidgets.QPushButton("Copy Path to Clipboard")

        # options group
        self.optionsGroup = QtWidgets.QGroupBox("Flipbook options")
        groupLayout.addWidget(self.outputToMplay)
        groupLayout.addWidget(self.beautyPassOnly)
        groupLayout.addWidget(self.useMotionblur)
        groupLayout.addWidget(self.copyPathButton)
        self.optionsGroup.setLayout(groupLayout)

        # button box buttons
        self.cancelButton = QtWidgets.QPushButton("Cancel")
        self.startButton = QtWidgets.QPushButton("Start Flipbook")
        self.publishButton = QtWidgets.QPushButton("Submit to Publish")
        self.publishButton.setEnabled(
            os.path.exists(hou.expandString(output_path))
        )

        # lower right button box
        buttonBox = QtWidgets.QDialogButtonBox()
        buttonBox.addButton(self.startButton, QtWidgets.QDialogButtonBox.ActionRole)
        buttonBox.addButton(self.cancelButton, QtWidgets.QDialogButtonBox.ActionRole)
        buttonBox.addButton(self.publishButton, QtWidgets.QDialogButtonBox.ActionRole)

        # widgets additions
        layout.addWidget(self.outputLabel)
        layout.addWidget(self.descriptionLabel)
        layout.addWidget(self.description)
        layout.addWidget(self.frameRange)
        layout.addWidget(self.resolutionGroup)
        layout.addWidget(self.optionsGroup)
        layout.addWidget(buttonBox)

        # connect button functionality
        self.cancelButton.clicked.connect(self.close_window)
        self.startButton.clicked.connect(self.start_flipbook)
        self.publishButton.clicked.connect(self.submit_to_publish)
        self.copyPathButton.clicked.connect(self.copy_path_to_clipboard)
        self.description.textChanged.connect(self.update_output_path)

        # finally, set layout
        self.setLayout(layout)

    def close_window(self):
        self.close()

    def update_output_path(self):
        output_path = self.get_output_path()
        self.outputLabel.setText(f"Flipbooking to: {output_path}")
        self.publishButton.setEnabled(
            os.path.exists(hou.expandString(output_path))
        )

    # get a flipbook settings object and return with given inputs
    def get_flipbook_settings(self, input_settings):
        settings = self.scene_viewer.flipbookSettings().stash()
        log.info("Using '%s' object", settings)

        # standard settings
        settings.outputToMPlay(input_settings["mplay"])
        settings.output(input_settings["output"])
        settings.useResolution(True)
        settings.resolution(input_settings["resolution"])
        settings.cropOutMaskOverlay(True)
        settings.frameRange(input_settings["frameRange"])
        settings.beautyPassOnly(input_settings["beautyPass"])
        settings.antialias(hou.flipbookAntialias.HighQuality)
        settings.sessionLabel(input_settings["sessionLabel"])
        settings.useMotionBlur(input_settings["motionBlur"])

        return settings

    def get_output_path(self, expand=False):
        description = self.description.text().replace(" ", "_")
        path = "$HIP/flipbook/$HIPNAME/flipbook{}.$F4.jpg".format(
            f"_{description}" if description else ""
        )
        if expand:
            path = hou.expandString(path)

        return path

    def get_default_resolution(self):
        cam = self.scene_viewer.curViewport().camera()
        if not cam:  # Use the main render_cam if no viewport cam
            cam = hou.node("/obj/render_cam")

        if cam:
            x = cam.parm("resx").eval()
            y = float(cam.parm("resy").eval())
            pixel_ratio = cam.parm("aspect").eval()
            return (x, int(y / pixel_ratio))

        return ("$RESX", "$RESY")

    def start_flipbook(self):

        inputSettings = {}

        # validation of inputs
        inputSettings["frameRange"] = self.get_frame_range()
        inputSettings["resolution"] = self.get_resolution()
        inputSettings["mplay"] = self.outputToMplay.isChecked()
        inputSettings["beautyPass"] = self.beautyPassOnly.isChecked()
        inputSettings["motionBlur"] = self.useMotionblur.isChecked()

        outputPath = self.get_output_path()
        inputSettings["output"] = outputPath
        inputSettings["sessionLabel"] = outputPath

        log.info("Using the following settings, %s", inputSettings)

        base_dir = os.path.dirname(hou.expandString(outputPath))
        if not os.path.exists(base_dir):
            os.makedirs(base_dir)

        # retrieve full settings object
        settings = self.get_flipbook_settings(inputSettings)

        # run the actual flipbook
        try:
            with hou.InterruptableOperation(
                "Flipbooking",
                long_operation_name="Creating a flipbook",
                open_interrupt_dialog=True,
            ) as operation:
                operation.updateLongProgress(0.25, "Starting Flipbook")
                hou.SceneViewer.flipbook(self.scene_viewer, settings=settings)
                operation.updateLongProgress(1, "Flipbook successful")
                # self.close_window()

        except Exception as e:
            log.error("Oops, something went wrong!")
            log.error(e)
            return

        self.publishButton.setEnabled(
            os.path.exists(hou.expandString(outputPath))
        )

    def submit_to_publish(self):
        output_path = self.get_output_path(expand=True)
        product_name = os.path.basename(output_path).split(".")[0]
        # Add task name suffix to product name
        product_name = f"{product_name}_{os.getenv('AYON_TASK_NAME')}"

        if not os.path.exists(output_path):
            hou.ui.displayMessage(
                f"Flipbook path {output_path} does not exist, generate it first.",
                title="Path does not exist",
                severity=hou.severityType.Error,
            )

        button_idx, values = hou.ui.readMultiInput(
            "Publish Input",
            input_labels=("Comment", "Version (optional)"),
            buttons=("Submit", "Cancel"),
            default_choice=0,
            close_choice=1,
            initial_contents=(
                "",
                path_tools.get_version_from_path(hou.hipFile.basename())
            )
        )

        if button_idx:
            return

        comment, version = values

        publish_data = {"out_colorspace": "rec709"}

        if comment:
            publish_data["comment"] = comment

        if version:
            publish_data["version"] = int(version)

        message, success = publish.publish_version(
            os.getenv("AYON_PROJECT_NAME"),
            os.getenv("AYON_FOLDER_PATH"),
            os.getenv("AYON_TASK_NAME"),
            "review",
            product_name,
            {"jpg": output_path},
            publish_data=publish_data,
            overwrite_version=True if values[1] else False,
        )
        if success:
            hou.ui.displayMessage(
                message,
                title="Submission successful",
                severity=hou.severityType.Message,
            )
        else:
            hou.ui.displayMessage(
                message,
                title="Submission error",
                severity=hou.severityType.Error,
            )

    # copyPathButton callback
    # copy the output path to the clipboard
    def copy_path_to_clipboard(self):
        path = self.get_output_path(expand=True)
        log.info("Copying path to clipboard: %s", path)
        QtGui.QGuiApplication.clipboard().setText(path)

    def get_frame_range(self):
        return (
            int(hou.expandString(self.frameRangeStartLine.text())),
            int(hou.expandString(self.frameRangeEndLine.text())),
        )

    def get_resolution(self):
        return (
            int(hou.expandString(self.resolutionXLine.text())),
            int(hou.expandString(self.resolutionYLine.text())),
        )

def show_dialog(parent):
    dialog = FlipbookDialog(parent)
    dialog.show()

[krishnaavril]

Hey!

We have redesigned the LabsKarma node for rendering according to our studio requirements. it's really a good one!

If the ayon publishing part was added to it. The renders can be published without any hiccups. the current ayon creating karma node maynot be a best solution as its creating in "out" it doesn't make much sense TBH when artist have to set karma properties in LOP. I know lot of studios have there own file cache and render HDAs. I love the concept of adding AYON parameters into the node and publish through that node. Lot of studios are migrating to solaris Karma mostly due to materialx support. I think it will be beneficial. Do checkout the LabsKarma ayon can add parameters and publish through it, it would be awesome!

[krishnaavril]

Hi @MustafaJafar @BigRoy @fabiaserra !

I was just exploring this PR and I have some feedback on this: adding a fetch is really a cool idea, maybe we should whitelist it for the publishing Also product type of the fetch can be anything eg:render/pointcache, instead string parameter, we can add menu?

[BigRoy]

Also product type of the fetch can be anything eg:render/pointcache, instead string parameter, we can add menu?

Yes. That'd be similar to what @fabiaserra showed in his demo I suppose.

I was just exploring this PR and I have some feedback on this: adding a fetch is really a cool idea, maybe we should whitelist it for the publishing

Fetch is a tricky beast to tackle - this is also visible in e.g. deadline's source code. Technically it's not the fetch generating anything. Also fetch the way you seem to want to use brings up the debate whether that single fetch node would be one instance, or multiples with different product. Since technically you could fetch any files, any types and any upstream hierarchy which may generate many files?

Could you describe your use case - how do you want to use fetch and why?

[krishnaavril]

I wanted to put "Fetch" in a best use case as below:

We render the scenegraph using the Labskarma node Using the above node, we get the renders using deadline, which is straight forward. (Also I'm requesting that it would be easier if AyonPublish is directly available for this OTL too.)

When we wanted to publish these renders, I'm trying to fetch the usd render rop which is inside LabsKarma and publish the available frames. so the nuke guys can load from Ayon. Caches I'm not worried as OBJ/SOP caches publish is working great.

[MustafaJafar]

@krishna8008 I was able to make LabsKarma publishable using this PR and adding few more tweaks.

Steps:

1. Use the generic creator while setting product type to karma_rop.
2. Change the file method to explicit
3. add the following lines just below this

   elif node_type == "labs::karma::2.0":
       return node.parm("file")

4. render the frames
5. set render target to use existing frames (local)
6. publish

I admit it looks weird in the loader in Houdini. but hey, the files are where they should be and they are tracked by AYON.

Also, this doesn't support separated AOVs because the current karma_rop product was built based on the karma rop which doesn't support AOVs.

---

Well, your example exposed many things in the Houdini addon like

1. It's rigid.
2. The generic creator was designed to enable ROP nodes to behave as though they were produced by various creators from Ayon. This ensures that validators and collectors are retained without loss.

tbh, sometimes I imagine generic creator as fabia mentioned in his demo and as the following mockup screenshot. where you specify the product type and list of files to include but this leads to losing the validators and other plugins. I believe many studios can't abandon pyblish plugins because these plugins combined forms their pipeline.

let me know what do you think and what were your expectations.

[krishnaavril]

Thank You @MustafaJafar, its working great for LabsKarma, we can publish renders through it now!

But I was just wondering Its not working for the custom LabsKarma node! Please have a look at the below image

We saved as a new copy and added deadline and other features inside. I also matched the type of the HDA in lib.py. still its not working.

Every other studio/artist will have there own HDA. It would be great, if Ayon tools can be designed in a way that it can fit.

[MustafaJafar]

Thank You @MustafaJafar, its working great for LabsKarma, we can publish renders through it now!

I'm glad it worked

We saved as a new copy and added deadline and other features inside. I also matched the type of the HDA in lib.py. still its not working.

changes in lib.py requires restarting the Houdini app.

[fabiaserra]

Every other studio/artist will have there own HDA. It would be great, if Ayon tools can be designed in a way that it can fit.

That's been one of me points since I started proposing a refactor of the Houdini OP/AYON integration. As it is it's very opinionated on the workflow the Houdini artist should work and Houdini is the most customized and flexible DCC there is, like @krishna8008 is saying every studio/TD will have their own ways of creating outputs (even their own farm integrations at times) and so AYON should be just a very thin layer on top of that in order to be able to integrate any output into the pipeline. If someone wants to work in a very simple blackbox workflow what it is right now works good... but that's not true for most and that's why I ended up creating a different integration.

[fabiaserra]

tbh, sometimes I imagine generic creator as fabia mentioned in his demo and as the following mockup screenshot. where you specify the product type and list of files to include but this leads to losing the validators and other plugins.

This is not necessarily true. We should be able to run any validations without the current creator workflow. In fact, we should be able to implicitly create creators on vanilla nodes and close the bridge with those workflows

[krishnaavril]

Hey! @BigRoy @MustafaJafar @fabiaserra :

Like @fabiaserra and I are saying about the custom HDAs for any studio and individuals,. I was just wondering, instead hardcoding the type of the node in houdini, this elif node_type == "labs::karma::2.0": return node.parm("file")

Is there any better way to get the code working for custom node types, Hardcoding here in git will only work for Labs nodes; a dynamic change of script is probably for the custom HDAs support, or choosing any other method other than type?! So that the code works universally for everyone. A proper documentation will help the creators to guide them on how to do it in the right way, which is IMO a good way to go. I'm excited to see what @BigRoy was working on. Thanks Guys!

[BigRoy]

Thanks Krishna, the idea that I have is to try and expose some of these things to studio/project settings. Likely I will have that prototyped next week.

Note that I also added a list of Todo to the PR description.

[MustafaJafar]

[ ] Make sure this API approach remains a goal of the PR.

Will context.publish publish using the publisher like here or the pyblish api like here ?

[ ] Add "Publish" button directly on the node similar to "self publish" or whatever we had for others

Will it make use of context.publish ? I imagine it as context.publish(my_instance)

[MustafaJafar]

omg 😅 is this expected/intended ?

[BigRoy]

Will context.publish publish using the publisher like here or the pyblish api like here ?

This point actually relates more now to the separated PR #691 - but I'd say if we can go through CreateContext as that exposes more of an API by the way its designed and we have more control over designing it than the pyblish API.

Will it make use of context.publish ? I imagine it as context.publish(my_instance)

Not necessarily - this PR now focuses mostly on the 'generalization' of ROP nodes in Houdini to detect them as publishable. CreateContext.publish() isn't currently an available API function - however, we could create a lib.publish(create_context) method for the time being - since functionally there isn't much to that but in essence that is 99% already what self-publish code is doing. It's just exposing that as a proposal API method then.

[MustafaJafar]

onCreate event is cool it made me wonder, Should the generic creator be visible in the Creator UI ? Also, The default $OS is not favored by the creator UI.

[BigRoy]

omg 😅 is this expected/intended ?

Nope - pushed a hotfix for now. Had it on my radar and wanted to do it "the right way" but went on to other stuff.

[BigRoy]

Should the generic creator be visible in the Creator UI ?

I've had my doubts about that - what do you think?

Also, The default $OS is not favored by the creator UI.

Ah yes - that doesn't seem great. I'm starting to hate the publisher UI ;) Anyway, hacked around it with https://github.com/ynput/ayon-core/pull/542/commits/939ee37f69825eb3a563bf9ca3d8c7438757653b

[MustafaJafar]

For information. These don't match. it happens because mantra creator have more options than the generic creator. refreshing the publisher is enough although it will remove some item.

[BigRoy]

For information. These don't match.

Publisher attributes specific to the instance instead of the creator

Perfect - thanks. This will be hard to fix with the design of the publisher's logic unfortunately. A single Creator can't define its properties per instance, but only for the creator. That, by design, then limits how we can generalize these for a single Creator unfortunately since it can only have the state of one. Where in this case for render instances we should be displaying different properties.

---

There are some other major issues that I was hoping to write up - because @dee-ynput also mentioned in essence that's also a goal of this, basically figuring out what the needs are for an API and the UI to follow suit.

Other things to keep in mind are:

Single output path versus a ROP that may generate any amount of "linked files"

• The generalized logic now where "Collect Frames" detects the output files from a ROP node based on an output parameter is a nice generalization but also fails for a lot of special cases. For example:
  • .gltf file path may be set to $HIP/my.gltf but actually exports more files that should also be integrated (and actually have their filenames preserved on the publish as well since the gltf links relatively to the other files!)
  • a USD rop may export more than one .usd file in any structure necessary. It may also save sublayers, etc. (which also would have to be relinked to any published structure or has the same issue as gltf publishes that the filepaths inside the files have a hard link)
  • a USD rop may spawn other products like asset USDs, layer USD files which could each also need relinking. This is basically somewhat what I had to face in PR #295 - which made it solvable by focusing purely on the USD aspect of things without thinking about generalization of logic beyond that point to e.g. gltf or other file formats. (and also relying specifically on the USD API, which of course does not translate to other file formats)
  • This would also go for "look" publishing with textures. We may need to relink where the texture files are based on where we're publishing but it may also become ROP node specific what textures to collect and find and how to go about relinking, etc. Add into the mix that we may want to process the textures on publishing, e.g. generating .tx files and the API now still needs to be able to do 'runtime things' during publishing.
  • Then add that there may be the request that for render publishing we want to off-load the "review" generation to a separate process so that we can push that to a separate farm job to avoid it influencing main render publishing times (and potentially simplify being able to 'retrigger' that step)

This may be somewhat resolved by exposing attributes to the user where they can configure additional files to ingest, but for complex cases the only real sensible way to do this - is to do it programmatically, because we don't want to continuously have artists need to manually set tons of files a particular ROP may output (if those are dynamic) which again may make it hard to abstract/generalize.

Although for a particular ROP node we could have a get_expected_outputs implementation that does that for GLTF, for USD, etc. but still it'd be specific to the ROPs; what then about HDAs? and where do we maintain those custom implementations?

---

A lot of this boils down to finding what is the easiest open-accessible, easy to code yet also easy to maintain API structure we can design for creators and publishing - where we hopefully can avoid as much as possible hardcoding 'specificness' in design. I had the same discussion with @fabiaserra recently where he said "we should rely less on Creators for publishing in Houdini" but then at the same time his codebase has 'wrappers' for Houdini node types that in a way are trying to define what something should publish as, what the default parm values are, etc. In essence what he's done is also creating an API for what we need to prepare something for publishing, basically designing his own "Creator" API.

In his implementation it could work to e.g. have (oversimplified pseudocode):

class UsdRopNode(ayon_houdini.NodeWrapper):
    def on_created(self, node):
        self.make_publishable(node)
        self.set_parm_defaults(node)

    def get_expected_files(self):
        return [file1, file2]

    def set_parm_defaults(self, node):
        # update the parm defaults for this node
        pass

We're just shifting where we're putting the logic. However, being able to access some of these bits outside of pyblish where we don't need "collect" and validations, etc. does make it trivial to abstract other bits.

Yet, as soon as that wrapper also need to expose what settings are user configurable, etc. we're basically expanding the Creator class design to be somewhat what they are now yet also include what @fabiaserra does. Meaning we actually have less generalizing of Creator logic but start getting more specific per node in Houdini which I feel is funnily enough the exact opposite Fabia has been arguing for a lot, and me as well.

TL;DR - finding the simplest API that can do all we want is hard, but should be the goal

[MustafaJafar]

I was trying submitting to farm and it may need additional work because:

1. "generic" is not added to the families of some deadline and houdini plugins. or Should we include the original product type e.g. mantra_rop as well as render ?
2. The logic of deadline plugins depends on product type.

[MustafaJafar]

TL;DR - finding the simplest API that can do all we want is hard, but should be the goal

Thanks for the explanation.

[BigRoy]

I'll close this PR and reopen it on separated ayon-houdini repo once the addon separation has finished.