Shot publishing

[mottosso]

While writing a shot-publishing test, something occurred to me. We might either have to start making assumptions or providing configurable defaults for determining how to determine the output from various instances within a scene.

In this shot, the animation curves of Peter is expected to be published under the given frame range. But how does Publish find the actual curves, without making assumptions about the instance?

Relying on conventions of TDs

One suggestion could be to enforce that instances contain a particular set under a given name or containing a certain attribute. Publish would look up the set and publish what it finds. But this is touching on management and workflow, which I'd rather steer clear from.

Relying on conventions of Artists

Another suggestion may be to not include instances in the publishable set, but instead put the animation controls there directly.

However this would mean exposing the user, who would have to make this selection in the first place, to internal details of rigs and what not. Simple for Publish, not so simple for artists.

For in-house publishing tools, this is less of an issue, as they could rightfully hard-code certain conventions that work for them.

Thoughts?

[mottosso]

This is an example of how configurable defaults might work:

Here, parse_instance() takes as an argument the class at which it will use to parse an instance. (See Classes. Given a class such as Animation, it would parse the instance for predicates, such as by looking for an objectSet by a particular name. The method could be overriden by others to provide for custom predicates; e.g. some might instead use a transform group, whereas others may have in-house nodes in which outgoing data resides.

To keep things simple, the method outputs a plain Python-list of absolute paths to each associated node; in the case of animation, this is just a list of controllers. The controllers would then be subjected to the standard extraction process of grabbing the animation curves under the pre-defined frame-ranges.

Models, rigs and other non shot-based resources could skip this step and simply extract the entire contents of a publishable objectSet as usual.

This would move the responsibility of selection onto in-house developers, as opposed to artists or us.

Any suggestions?

[tokejepsen]

I would probably lean more towards a tagging of instances, instead of collecting them under a set. Maybe use the sets workflow for overrides. This way you can for example tag the original geo in the geo file, and which then gets progress down the line. Then it is less setup per shot.

[mottosso]

Interesting idea. By tagging, are you referring to, if we use Maya as an example, attaching attributes onto nodes to be published?

cmds.addAttr(publishable_node, 'publishable', dt='bool', default=True)

[tokejepsen]

yup, thats it:)

[mottosso]

Ah, do you mean that each controller would then be tagged? How would you separate between controls from different characters? And wouldn't you end up with quite a few attributes, how about modifying the attribute(s)?

One benefit of using sets is that you can store attributes onto the set itself; like frame-ranges and such. If related attributes are distributed across multiple objects, where would you store these settings?

As an example:

# Two assets with similar tags

# my_character.mb
for control in ['ctr1', 'ctr2', 'ctr3', 'ctr4']:
   cmds.addAttr(control, 'publishable', dt='bool', default=True)

# my_vehicle.mb
for control in ['ctr1', 'ctr2', 'ctr3', 'ctr4']:
   cmds.addAttr(control, 'publishable', dt='bool', default=True)

I think the problem remains, that there needs to be some way of distinguishing between which tags represents "meshes" and which represents "controls". The problem is how hard-coded we want to make this.

[tokejepsen]

Yeah, each controller would be tagged. As for separate characters, I would assume that the rig is referenced in which means you have a namespace that'll differentiate. Although that might need some more thinking to make it less Maya specific.

I would definitely have the sets option as well, for multiple customization.

[mottosso]

Okay, I think we're thinking of the same thing.

I tried highlighting two options above, conventions on part of the TD, and those on part of the Artist. Tagging each control would be a convention to be followed by TDs.

The issue is whether hard-coding such a conventions is a good idea, and if providing configurable defaults for it might get too complicated; both from a development point of view and for those configuring it.

For clarity, these are some use-cases I can think of:

1.1. TD prepares his assets in accordance with Publish conventions; tagging/collecting in object set. 1.2. Artist imports assets 1.3. Artist publishes assets 1.4. Publish reads from tags/objectset as prepared by TD

In this case, we provide instructions for how TDs prepare their assets. This may not fly if Publish is being integrated into a running project and might cause friction for those who aren't comfortable with our defaults.

Alternatively:

2.1 TD creates assets in any manner 2.2. Artist imports assets 2.3. Artists selects what to publish; i.e. including controllers, meshes 2.4. Publish reads from tags/objectset as prepared by Artist.

In this case, mistakes are quite likely to happen, as artists can't be trusted (tm)

Configuration

Configuration could happen on two fronts; either at the TDs or Artists.

3.1 Configuration is supplied to TDs 3.2 TDs prepare assets in accordance to configuration (schema, essentially)

Or:

4.1. Configuration is provided by the UI to help artists select large amounts of controllers/meshes. 4.2. Artist selects instances with help of UI

Lastly, one option may be to simply stick with publishing assets and to allow shot-publishing to be a separate concern/project.

[tokejepsen]

Does it have to be either/or? Would it not be possible to have TDs prep assets, then the Artist picks up any custom publishes?

If you have tagging of instances and sets, you could easily build a tool that collects tagged instances into sets so the Artist can see what they are expected to publish.

[mottosso]

No, you're right, there could be multiple options. The issue is; can we reasonably expect TDs or Artists to work in a certain way in order for them to publish their material?

At the moment, the only customisation is in the scene being published. With customised instances, Publish would affect the entire asset library before it could work properly. Do you get what I mean?

It may not be avoidable, but if there is a way, I'd rather not make too big of a splash when it comes to peoples working habits.

[tokejepsen]

I do see what you mean:)

[mottosso]

Cool. I like your suggestion though, about tagging with attributes. I'd encourage you to implement this behaviour once we get into code, would you be up for that?

[tokejepsen]

I'll definitely be up for helping out, now that I know where it could potentially sit within a pipeline.

[mottosso]

Awesome! I'm testing out your suggestion about adding attributes now; it might be the only way go. I ran into issues using only sets.

The process I was imagining was this:

1. Each instance, e.g. a character rig, has sets with geometry and controls. The sets then carry the "publishable" attribute.
2. In a shot, each instance is added to a "publish" set
3. Publish (the app) looks at each instance within the set and gets each contained set per instance.

But how can it derive which sets belongs to which instance? I thought about looking at namespaces, but that would make it necessary for TDs to work with namespaces, which I know can be quite a pain, and also not as portable to other apps, as you mentioned. The other might be to loop through each node within the hierarchy and look for connected sets, but that seemed rather brute.

Another way might be to loop through the hierarchy and look for "tags", or user-defined attributes on a per-node basis. Then you've got.

1. Same
2. Same
3. Publish looks within each instance for tagged nodes relevant to its class (e.g. controls or meshes)

Lastly, another way may be for Publish to record which sets are present upon publish as metadata. The sets could then be looked up from there.

[ljkart]

I have a few concern about using sets,

1. Is it would be very messy if we use sets for publish and artist uses their own multiple sets for different selection, it will make a scene with chunks of selection sets in outliner(for huge scene)?
2. what are the advantages we get by using sets over groups - a top group with all necessary data tagged on to it and all the other groups are underneath it , which is used to publish, the top group named according to the publish type like model, rig, texture etc. eg : /model    /asset_name     /geometry
3. Anyway sets means the reference to the groups already created, don't you feel this as duplication?

[mottosso]

Valid points.

1. I think the only way to be sure of this is to see it in action. Could you prepare a test for it? At the very least, I could imagine limiting sets to 1-2 per instance, and have them nest their sets in case they need more than that.

2. The main advantage is that we won't have to enforce convention on hierarchies.

   As an example, imagine someone wanted to use Publish on their already finished assets. If a specific hierarchy was required, there would be a need to rearrange it, potentially causing other issues.

   In some cases, our required hierarchy may not be possible/preferable; for example in cases where a rig may consist of multiple rigs (body, face, fingers) but are each separate and referenced. Sets would facilitate for this as hierarchies aren't fixed or enforced.

   These are potentially edge cases, but at the end of the day, I would suggest we steer clear from enforcing convention as much as we can, and I think sets could help with that.

3. Could you rephrase that? Or show an example? Not sure I follow.

[tokejepsen]

I think tagging nodes is most basic way of handling the meta data. I would then create tools around that workflow, rather than using sets, to ease the tagging process.

[mottosso]

Mm, I was thinking the same thing. This whole manual thing is mostly for starters during development. Pretty GUIs should be capable of handling all of these things down the line.

[mottosso]

@tokejepsen would it be possible to ask you for help on something? I'm really interested in your suggestion on adding attributes onto controllers as a means of distinguishing them from other things.

Do you think you could put together a little demoscene with an asset or two demonstrating your thoughts? I think I've got the idea, but I'm sure we'd come up with different outcomes, both of which potentially contributing something unique!

[tokejepsen]

sure, where should I be posting the files?

[mottosso]

Thanks. Are you familiar with the pull-request workflow? In a nutshell you:

1. Fork the repo
2. Clone it locally
3. Add the files under /tests/selection (see Writing tests in the wiki)
4. Push it to your repo
5. Submit a pull request.

From there, a thread will be started and we could talk about the additions.

Are you on hangouts? I'm at konstruktion@gmail.com if you want to chat.

[mottosso]

Anyway sets means the reference to the groups already created, don't you feel this as duplication?

It just hit me what you meant with this!

If a scene looks like this:

/
  Peter_AST
  publish_SEL/
      Peter_AST

Then I can understand why it looks duplicated, and why not just use Peter_AST directly? The reason is that not all objects in a scene may be up for being published. What I'd like is for a scene to be allowed to be messy, to be a work in progress, but still be publishable. So in order to distinguish between what is output, and what is to stay put, a set can be used. Picture a scene like this:

/
    Peter_AST/
    measurements_GRP/
        locator1
        locator2
    debug_locators_GRP/
    pCube1
    publish_SEL/
        Peter_AST

Here, there is no question about what is selected for publishing, and what is to remain behind. I would think that this behaviour is important, but if sets isn't the way to go, then do provide alternatives.

I made a note of it here:

• Partial Publishing

[BigRoy]

We currently use the described "tagging" workflow within Maya for our publishing system. For example we've chosen to "by default" exclude all nurbsCurves from being published from Animation to the Lighting scenes. Because of the nature of tagging with attributes we can decide at any point whether we want to force publish. We use a 'forcePublish' attribute that would be added to curves that we do want to be published in the end. Note that if the same attribute (if created as 'boolean' on a node) could also be used to distinguish nodes that should not be included in the publish. If a node doesn't have the attribute it uses the pipeline's default behaviour based on node type, if it has custom attributes it will use those to define how it should be published.

By default: We don't make any use of selection sets but include everything in the scene (that is visible) in the Publish. Or at least as much as possible (e.g. Alembic doesn't support everything). This way any 'messy' stuff we added for workarounds will likely transfer. We would rather 'overpublish' than 'underpublish'. With tight deadlines it's sometimes easier to just hide the stuff that was added by mistake than to go back, change the scene and republish.

The tricky thing with attributes is that they are bound to shapes. (Because hey, what happens if you have two nurbsCurve shapes under one transform?) If a shape gets deformed Maya sometimes happens to create a new shape that doesn't have the attribute - aaaaargh! - so it doesn't have the custom publishing information. We have created a custom 'copy attributes to deformed shapes' function to fix those cases.

[tokejepsen]

Thats cool. I like the notion of being able to publish everything in view of the camera.

How come you are tagging the shape nodes specificly? Couldn't you tag the transform, and filter down?

[BigRoy]

We once had the very specific use case where we required multiple shapes under one transform behaving differently. I think it was something with face based shader assignments at the time. In retrospect I think it might be way more stable 'tagging' the transform and assigning shape related stuff 'downwards' because Maya does a lot of mumbo-jumbo with Shapes (think deformers and such) that you don't wont. On our next revision ("Publish?") we might just go for tagging the IDs to transforms only. We'll have to go over the pros and cons before with the team on that though.

Another benefit of tagging to the transform (for shaders) would be that you can instance the shapes under a new transform and apply (pipeline-wise) another shader to the instance automatically. This isn't possible when tagging shapes, because instanced shapes means there's only one node.

[mottosso]

I'm glad you guys are here to talk about the shading-front of publishing, my experience lie mostly in deformation/simulation)

As there is a fountain of knowledge here, I'll try and paraphrase the topics, do correct me if I'm getting anything wrong!

Tagging shapes is good

This is bound to bite us in the butt one day, I'm sure of it. Intuitively, it would make sense to select a mesh and tag it, but as Maya separates between shapes and transforms intuition isn't always the best guide.

I was thinking that it may help to consider whether or not we could stick with a "cascading" behaviour on everything by default. In that, adding an attribute to a parent would infer the children are included; attributes such as whether or not it is to be published.

It'll have drawbacks too, but it does align rather well with how Maya treats hierarchies in everyday life; e.g. altering visibility on a parent implies that children are invisible too.

For anything that falls outside of this rule, we'll still have sets.

Overpublishing is good

Can't say much about it, other that it sounds dangerous, considering the additional amount of offline history gathered per reference, that would then be floating around once the publish has been slimmed down. Smells like rats-nest for bugs.

I'll reserve my opinion of it however until we can see it in action.

For example we've chosen to "by default" exclude all nurbsCurves from being published from Animation to the Lighting scenes.

Could you expand on this? Do you mean that lighting scenes are more than just pointcached meshes? Where does nurbsCurves come in?

I like the notion of being able to publish everything in view of the camera.

I think what @BigRoy was referring to wasn't about objects visible in the camera, but objects with their visibility attribute set to False.

[tokejepsen]

I was thinking about the tagging workflow, and couldn't we go with a hierarchical filtering system like:

Sets > parent > children

Where if something needs to be published, it'll get the "publish" tag. Other parameters gets overridden as you go down the chain. If a required tag like the class of a node, isn't present Publish looks upwards for the parameter. This way you could have tagging of transforms, with potential overrides on the shapes.

[BigRoy]

I think all of the methods described above can be accomplished with the method I've written out in the Architecture->Selection wiki page. By making it easy to create your own 'Selector' component one can even define more obscure or dynamic components that we might not think of. Even better is that the ordered queue of the selectors could allow to mix and match the different Selectors.

[mottosso]

Closing this, feel free to re-open if you have anything more to add.