Houdini: Publish any ROP node (Generic Creator)

BigRoy commented 5 months ago

Implement Generic ROP publish from https://github.com/ynput/ayon-core/pull/542

Changelog Description

This PR implements an idea for "lower level publishing" in Houdini. This implement Generic ROP publishing. Just create any Houdini ROP node (or custom Rop node HDA) and publish your product types from it!

This PR adds a Generic Creator: Currently, It injects the ayon meta data onto ROP nodes to behave as though they were produced by various creators from Ayon. This ensures that validators and collectors are retained without loss. e.g. using Generic creator while setting product type to pointcache should yield similar/same results as using pointcache creator Here is an example using labs karma node and make it behave as though it was made by karma creator. https://github.com/ynput/ayon-core/pull/542#issuecomment-2179333623

Additional info

As part of the Ynput Houdini Workgroup sessions I developed this quick prototype to expose a way to batch publish and ingest files. Consider it more of an exploration of what's possible then a "drop it in production now" ready-to-go solution.

Explainer

Yes, this requires some explanation. Here you go.

https://github.com/ynput/ayon-core/assets/2439881/5f767493-10bd-41f6-b54f-89313a96da33

What I forgot to add is that it currently still relies on detecting what the output files are for a ROP node based on a "file" parm that is often unique in Houdini per node. If anyone knows a way to just query the expected output files for a ROP node (similar to what PDG/TOPs seems to do I'd love to know!) but otherwise we'll just need to expand that list.

However, I also played with the idea of having "custom file list" attributes on the Node that when enabled could override the "Collector" logic and would instead use that list of files as the publish instance's files. So that e.g. one instance could also publish multiple representations. For that, @MustafaJafar did this lovely non-functional 'prototype' but it does get the idea across:

mustafa_collect_multiple_representations_to_one_instance

Demo scene file

The demo scene file:

ynts_char_hero_pdg_v012.zip

TODO 📔 ✏️

[x] Use $OS as default variant name.
[x] Add "OnCreated" callback to allow a studio to configure which nodes to automatically imprint as 'publishable'. See Fabia's first video here at 08:10 and code here
- [ ] This is currently a first iteration - should still be exposed to project settings and allow user configuration.
[x] The product types being an enum with predefined standards per node type.
[x] The comment field being available on the ROP nodes
- [x] This does actually already work if you'd add the product type to ayon+settings://core/publish/collect_comment_per_instance ✅ - I guess this is fine for now.
[x] Add "Publish" button directly on the node similar to "self publish" or whatever we had for others
[ ] Allow exposing to settings which node types may need to be "automatically imprinted" on create for a studio to customize, with also settings for allowed product types (the default type, default variant name, etc.)
[ ] Implement good support for multiple representations, e.g. like this and Fabia's second video near the end.
[ ] Make sure this API approach remains a goal of the PR.
[ ] Keep in mind Fabia's usage where their AX Publish Submitter can submit just the publish of existing files to the farm and create e.g. a nuke dependency job that generates custom previews of the intermediate media, etc.
[ ] The product name is currently an 'exposed field' but should actually be locked off to the user for manual editing and should always adhere to the get_product_name logic of the Creator and the product types so that they adhere to project templates correctly.
- [ ] We need to make sure this isn't an expression that is heavy and re-evaluates the whole time. May need to be based on callback on "product type" and "variant" changes.

Known issues / Blockers 📔 🔥

The publisher, creator and create context can't differentiate attributes per instance, as such it can't e.g. differentiate whether "split export and render job" makes sense or not for a particular instance's attribute definitions. See comment
It will only detect the single file defined by the ROP node's output path and does not recognize whether e.g. a USD ROP or GLTF rop may generate other sidecar files (which often also if moved along should also remap paths, etc. so it might be non-trivial to support those out of the box)
- This basically makes it useless for GLTF exports, and in many cases useless for USD (if it were to write multiple files which also need remapping) and rendering (which may generate export files, need dedicated Deadline plug-ins to maybe render certain outputs (e.g. Husk) or may generate multiple sequences of files (like AOVs to separate files).
We do not know or detect the colorspace of the output files (there is no 'generic' way for ANY ROP to return what colorspace it may result in.
If certain ROPs may override the output FPS other than the workfile, we do not know it.
Due to Creators enforcing a single product type for which the attribute definitions are populated/created based on the Creator's product type we run into issues with a generic Creator that basically switches product type on the instances dynamically. See comments here.

Testing notes:

Check out the branch
Check out the explainer video
Test the demo scene file
Comment with great ideas on how to improve

Resolves #16

krishnaavril commented 4 months ago

Also, it would be great, if the node changed color or added a sticker on top when published, especially since this helps to identify the publish node easily in the scene. Not a very important one, but helps little.

BigRoy commented 4 months ago

Also, it would be great, if the node changed color or added a sticker on top when published, especially since this helps to identify the publish node easily in the scene. Not a very important one, but helps little.

Interesting request - when would you like the 'clear' this color or sticker? I can imagine it otherwise always having the sticker after the first publish you've done from the workfile.

krishnaavril commented 4 months ago

Also, it would be great, if the node changed color or added a sticker on top when published, especially since this helps to identify the publish node easily in the scene. Not a very important one, but helps little.

Interesting request - when would you like the 'clear' this color or sticker? I can imagine it otherwise always having the sticker after the first publish you've done from the workfile.

Yes, when its done fitst publish sounds good.

Another instance I can think is It helps a lot when the filecache node publishes, coz in sops its easy to mess around to find the publish node.

BigRoy commented 4 months ago

Adding screenshot of what it looks like now:

krishnaavril commented 3 months ago

Hey!

Just testing this branch, we cannot publish anything from houdini(workfiles/caches etc), somehow its getting conflict, I've attatched the error when we tried to publish workfiles with this branch


Error: The attempted operation failed.
Parameter name 'AYON_self_publish' is invalid or already exists

Traceback (most recent call last):
  File "C:\Users\avril3975\AppData\Local\Ynput\AYON\addons\core_0.4.4-dev.1\ayon_core\pipeline\create\context.py", line 2484, in _save_instance_changes
    creator.update_instances(update_list)
  File "C:\Users\avril3975\AppData\Local\Ynput\AYON\addons\houdini_0.3.12-dev.1\ayon_houdini\plugins\create\create_generic.py", line 393, in update_instances
    self.imprint(
  File "C:\Users\avril3975\AppData\Local\Ynput\AYON\addons\houdini_0.3.12-dev.1\ayon_houdini\plugins\create\create_generic.py", line 641, in imprint
    self.create_attribute_def_parms(instance_node, created_instance)
  File "C:\Users\avril3975\AppData\Local\Ynput\AYON\addons\houdini_0.3.12-dev.1\ayon_houdini\plugins\create\create_generic.py", line 624, in create_attribute_def_parms
    node.setParmTemplateGroup(parm_group)
  File "C:\PROGRA~1/SIDEEF~1/HOUDIN~1.547/houdini/python3.9libs\hou.py", line 16733, in setParmTemplateGroup
    return _hou.OpNode_setParmTemplateGroup(self, parm_template_group, rename_conflicting_parms)
hou.OperationFailed: The attempted operation failed.
Parameter name 'AYON_self_publish' is invalid or already exists

BigRoy commented 2 months ago

Hey!

Just testing this branch, we cannot publish anything from houdini(workfiles/caches etc), somehow its getting conflict, I've attatched the error when we tried to publish workfiles with this branch


Error: The attempted operation failed.
Parameter name 'AYON_self_publish' is invalid or already exists

Traceback (most recent call last):
  File "C:\Users\avril3975\AppData\Local\Ynput\AYON\addons\core_0.4.4-dev.1\ayon_core\pipeline\create\context.py", line 2484, in _save_instance_changes
    creator.update_instances(update_list)
  File "C:\Users\avril3975\AppData\Local\Ynput\AYON\addons\houdini_0.3.12-dev.1\ayon_houdini\plugins\create\create_generic.py", line 393, in update_instances
    self.imprint(
  File "C:\Users\avril3975\AppData\Local\Ynput\AYON\addons\houdini_0.3.12-dev.1\ayon_houdini\plugins\create\create_generic.py", line 641, in imprint
    self.create_attribute_def_parms(instance_node, created_instance)
  File "C:\Users\avril3975\AppData\Local\Ynput\AYON\addons\houdini_0.3.12-dev.1\ayon_houdini\plugins\create\create_generic.py", line 624, in create_attribute_def_parms
    node.setParmTemplateGroup(parm_group)
  File "C:\PROGRA~1/SIDEEF~1/HOUDIN~1.547/houdini/python3.9libs\hou.py", line 16733, in setParmTemplateGroup
    return _hou.OpNode_setParmTemplateGroup(self, parm_template_group, rename_conflicting_parms)
hou.OperationFailed: The attempted operation failed.
Parameter name 'AYON_self_publish' is invalid or already exists

Thanks @krishnaavril - should be solved now. It will now log a warning on creation. It's because it tries to create it twice somehow? you have the creation of self publish button enabled in settings.

But that would only have been the case if you triggered the generic creation on a node that was already a publishable instance. I've changed the code, but would still be lovely if you could share a screen capture of how you get it to the state that it breaks.

krishnaavril commented 2 months ago

Thanks, Yes I do have the self-publish button enabled in Ayon settings.

But that would only have been the case if you triggered the generic creation on a node that was already a publishable instance. No, I tried this with our inhouse otl, adding the parameter and otl name hardcoded in the ayon code which allows publishing our own cache and render nodes.

Will test these pushed changes today, thanks

krishnaavril commented 2 months ago

I've tested the code with the labs karma node, I think the issue is still there


Error: The attempted operation failed.
Parameter name 'AYON_self_publish' is invalid or already exists

Traceback (most recent call last):
  File "C:\Users\avril3975\AppData\Local\Ynput\AYON\addons\core_0.4.5-dev.3\ayon_core\pipeline\create\context.py", line 1278, in _save_instance_changes
    creator.update_instances(update_list)
  File "C:\Users\avril3975\AppData\Local\Ynput\AYON\addons\houdini_0.3.14-dev\ayon_houdini\plugins\create\create_generic.py", line 392, in update_instances
    self.imprint(
  File "C:\Users\avril3975\AppData\Local\Ynput\AYON\addons\houdini_0.3.14-dev\ayon_houdini\plugins\create\create_generic.py", line 640, in imprint
    self.create_attribute_def_parms(instance_node, created_instance)
  File "C:\Users\avril3975\AppData\Local\Ynput\AYON\addons\houdini_0.3.14-dev\ayon_houdini\plugins\create\create_generic.py", line 623, in create_attribute_def_parms
    node.setParmTemplateGroup(parm_group)
  File "C:\PROGRA~1/SIDEEF~1/HOUDIN~1.547/houdini/python3.10libs\hou.py", line 16733, in setParmTemplateGroup
    return _hou.OpNode_setParmTemplateGroup(self, parm_template_group, rename_conflicting_parms)
hou.OperationFailed: The attempted operation failed.
Parameter name 'AYON_self_publish' is invalid or already exists

after create was done and when I clicked on publish this error popped up!

ynput / ayon-houdini