AcademySoftwareFoundation / OpenCue

A render management system you can deploy for visual effects and animation productions.
https://www.opencue.io
Apache License 2.0
830 stars 202 forks source link

Expanded cloud functionality #1097

Open bcipriano opened 2 years ago

bcipriano commented 2 years ago

At the moment OpenCue is agnostic as to where resources run -- if RQD can run on a machine and that machine is networked to be able to reach the Cuebot, OpenCue doesn't really care where that machine is running. However, the TSC is currently discussing ways that cloud functionality can be expanded to be easier/more helpful to cloud users.

Major use cases

  1. Easily deploy OpenCue on the cloud.

    • Enable a "cloud-first" type of deployment, where the database, Cuebot, and RQD can all be easily deployed to the cloud.
    • The solution here should be cloud-agnostic -- ideally a single solution with flavors for each major cloud provider, and easy ways to extend the solution to other clouds.
    • The solution could provide a good alternative to users just getting started with OpenCue, similar to the sandbox.
  2. Easily scale RQD workers in the cloud. Help users make informed decisions about how many workers they need to complete their current OpenCue workload.

    • This is distinct from the first case -- users may have their RQD pools configured in many different ways, and our solution here should not assume they are using any specific tech.

Easily deploy OpenCue in the cloud

Ideas:

Easily scale RQD workers in the cloud

Ideas:

DiegoTavares commented 2 years ago

For logging, it would be interesting to add an HTTP endpoint to enable decoupling the GUI and the filesystem used by RQD.

bcipriano commented 2 years ago

For logging, it would be interesting to add an HTTP endpoint to enable decoupling the GUI and the filesystem used by RQD.

Great idea -- I've incorporated this into the ideas list.

malkia commented 1 year ago

We have a possible use case. We have cuebot on GCP and few render Windows workers there too. Initial tests indicate that this woks fine, scaling to more should not be an issue. The machines and cuebot can see and connect to each other without a problem.

The issue comes when a local worker (right now for development purposes) comes. It's able to talk to the cuebot, registers itself, but cuebot can't talk back (due to our global firewall settings, and we don't want to change that).

I was wondering how people solved this? Proxy maybe? It also made me think for this special case, whether it won't be worth creating an additional grpc bidi stream (both ways) that can both talk to the cuebot, and cuebot can talk back and serve jobs. Understandigly this would means more pressure on the cuebot to keep that connection, so it'll be only useful if there a handful of such machines.

Anyone else run into something like this?