search

AcademySoftwareFoundation / OpenCue

A render management system you can deploy for visual effects and animation productions.
https://www.opencue.io
Apache License 2.0
829 stars 199 forks source link

Integrate with Houdini PDG #361

Open gregdenton opened 5 years ago

gregdenton commented 5 years ago

Create a custom scheduler integration with Houdini's PDG dependency graphs. Talk with SideFX to get support out of the box with the default TOP node.

gregdenton commented 5 years ago

SideFX's PDG scheduler doc: http://www.sidefx.com/docs/houdini/tops/schedulers.html

manuelkoester commented 4 years ago

I have zero experience with OpenCue, but I have been keeping an eye on it. I have been working on different PDG schedulers and know what makes an integration with the RMS easier. Therefore couple of questions.

If I understood the glossary correctly the hierarchy of jobs in OpenCue works like this:

On what layers are dependencies possible? Can frames of different layers (or even jobs) be dependent on another? On what layer are the execution commands stored? Are they stored on layer level and the frame number is getting replaced on a frame by frame basis? Is it possible to add layers to an already existing OpenCue job? Is it possible to add frames to an already existing OpenCue layer?

suekieza commented 4 years ago

@fatboYY which RMS have you played around with before with PDG?

manuelkoester commented 4 years ago

@suekieza I have done some stuff with Deadline and a proprietary RMS in PDG.

suekieza commented 4 years ago

Ah cool, I'm on the hunt for someone who has done some testing with Royal Render. See what they have to say about the process.

gregdenton commented 4 years ago

To help answer some of the above questions:

On what layers are dependencies possible? Dependency relationships are pretty flexible and available to/from all objects. Here's a list of the various dependency types: JOB_ON_JOB JOB_ON_LAYER JOB_ON_FRAME LAYER_ON_JOB LAYER_ON_LAYER LAYER_ON_FRAME FRAME_ON_JOB FRAME_ON_LAYER FRAME_ON_FRAME FRAME_BY_FRAME PREVIOUS_FRAME LAYER_ON_SIM_FRAME

Can frames of different layers (or even jobs) be dependent on another? Yes.

On what layer are the execution commands stored? Are they stored on layer level and the frame number is getting replaced on a frame by frame basis? The command to run is stored on the layer along with the frame range. Like you mention, a frame token is replaced with the executing frame on a frame by frame basis.

Is it possible to add layers to an already existing OpenCue job? This is not currently available in the API, there are some "hacky" ways to to do it that we could productionalize if needed.

Is it possible to add frames to an already existing OpenCue layer? This is not currently available in the API. Adding frames to a layer may be easy to add but removing frames may cause some problems for the historical tables that keep track of resources used.

Schizo commented 3 years ago

Was there any movement on this? It would be great to have basic version of this running.

Eiken commented 3 years ago

Wondering the same. Otherwise we are thinking about giving it a go as we need it.

queglay commented 3 years ago

I would love to do this but I have too much on. I can help share what I've learnt though. I have written other schedulers for PDG and I'd be happy to do a recorded session to show anyone taking this on what I've learnt.

I would highly suggest forking the tractor PDG scheduler as it is easy to understand and it has some excellent new abilities like heartbeat messages to alleviate DB queries on task state (push messages instead of pull) - this greatly simplifies the work required.

Another very helpful module that can be ported easily is the tractorSpooler. This defers the submission delay of each item to a queue on the PDGMQ process running on the farm which frees up the main UI from having to wait for requests / job IDs. It is also possible to get the remote tractorSpooler to update attributes on the work item (like job IDs).

You can email me at andrew@firehawkvfx.com Id love to help by sharing some of what Ive gone through to get things running efficiently at scale.

nyqvistatwork commented 1 year ago

I have OpenCue as my main focus at work at the moment and since I'm getting Houdini installed anyways (sister company reasons) I would love to see if I could help out in any way. I might be new to PDG and I just got OpenCue to work but hey.. I'm a master at reading docs. And this project has roots in Cloudy With a Chance of Meatballs. Me and my kids loved that movie 🥰

bcipriano commented 1 year ago

@nyqvistatwork Sure thing, please give it a go! I don't know of any other progress on this.