search

pyblish / pyblish-deadline

Pyblish for Deadline
GNU Lesser General Public License v3.0
7 stars 4 forks source link

Multiple jobs per instance + job for context #38

Open asztalosdani opened 5 years ago

asztalosdani commented 5 years ago

Hi,

  1. I want to submit multiple jobs per Instance. As the instance.data("deadlineData")["job"] suggests, it is not possible at the moment.
  2. I want to submit jobs for the Context.

Both seem doable, I am committed to implementing it, and submit a PR when ready. Do you have any pre-implementation notes?

asztalosdani commented 5 years ago

Oh, I just found this: #20 Which is ok, but: for 1. I want the artist to see only one renderlayer instance, and this would submit an .ass export and an arnold render job. And for 2. I need context job, so after every instance is exported I can do a bit of a house cleaning. So I still think #20 is needed.

tokejepsen commented 5 years ago

Hey @asztalosdani :)

Good to see this repository being used.

I was wondering whether you have looked into having either the ass export or the Arnold render job as a non-optional instance? I certainly see why you wouldn't want too many instances exposed to the artist, but it could also be good to have two instances if you only wanted the ass export and not the Arnold render. For making them dependent on each other you could use the "order" in the deadline data.

As for a context job, am I right in guessing it's a job that runs last of all the jobs? So a job that is dependent on all the other jobs?

tokejepsen commented 5 years ago

Sorry I'm I have already covered what you already known, but gotta exasperate the options before moving forward :)

asztalosdani commented 5 years ago

Hi Toke,

Good to see this repository being used.

Well, it is in the pyblish universe, so I thought why not use it! :)

I was wondering whether you have looked into having either the ass export or the Arnold render job as a non-optional instance? I certainly see why you wouldn't want too many instances exposed to the artist, but it could also be good to have two instances if you only wanted the ass export and not the Arnold render.

I don't think it is the pyblish way. I think of the whole scene (more precisely a renderLayer) as an instance, which is being exported, and the ass export and the Arnold render are two Extractors (where the arnold could be optional), because they are outputting the same frames of the same instance, just two different representations.

As for a context job, am I right in guessing it's a job that runs last of all the jobs? So a job that is dependent on all the other jobs?

Yes, but the whole image looks like this, this time with an animation export, as it is a better example for the context-job-submission matter: The animators do their job, when done, they hit pyblish and see the two characters of this shot as instances in pyblish, and send them to the farm. I would like to see these jobs submitted with these orders as dependencies: -1 Custom maintenance job, which prepares the export process. It has to be done only once per scene (so per context), but it can be a long running job, so that's why we want to send it to farm and not run locally. -2 Maya job, exporting Character1 -2 Maya job, exporting Character2 -3 Custom maintenance job, which cleans up the stuff made in step 1. Again this has to be a context based submission.

asztalosdani commented 5 years ago

Oh, and I just found a bug: If you have instances with and without order, the ones without order won't get submitted, because the instances_no_order is only used if the instances_order is empty.

tokejepsen commented 5 years ago

I don't think it is the pyblish way.

The idea of an instance was to be the inverse of a file, but you can use it however you want. I agree it makes more sense for the users that it's something in the scene they are familiar with.

How do you see this implemented? Maybe have a instance.data["deadlineData"]["jobs"] entry and for the context submissions context.data["deadlineData"]["jobs"]?

I won't be able to help with any but suggesting changes, and I don't have access to computer for a while.

Oh, and I just found a bug

Nice! Thanks for reporting this. If you don't intend on fixing it yourself, then please make another issue for other people to potentially fix it in the future.

asztalosdani commented 5 years ago

All of them (multiple + context + bug) are implemented, will submit a PR, but in the meantime, you can check them here: submit multiple jobs per instance submit jobs from context I implemented it as #20 suggests, instance.data["deadlineData"] can be a list to maintain bacward compatibility, and context.data["deadlineData"] likewise. If you don't mind a backward incompatible change, we can come up with a better method to organize the submission data, instance.data["deadlineData"]["jobs"] as you suggested, instance.data["deadlineSubmissions"] or instance.data["deadlineJobs"].

tokejepsen commented 5 years ago

Cool, if you could send a PR with updates to the Readme, then I can merge.

tokejepsen commented 5 years ago

I've had a bit more of a think about this.

Since you probably would like to use separate Deadline plugins, in your case Maya ass export and Arnold standalone render, you would want to configure the plugin data as well. How about having a instance.data["deadlineData"]["submissions"] which will be a list of dictionaries containing job and plugin entries?