Master Version Workflow

[mkolar]

Problem:

A few of our clients are requesting the option to use "master versions" or "push" workflow for certain projects. The reasons are simple efficiency with scene updates in projects that are unlikely to break with asset updates. Currently, the whole version workflow is very safe, however, when change is published, it requires a tremendous amount of time to update all it's dependent scenes, which might have a lot of nesting.

Example: Tree_model_v01.abc -> myForest_setdress_v01.ma -> sh020_layout_v01.ma -> sh020_lighting_v01 ...

When the tree is updated to tree_model_v02.abc, we need to update the myForest setdress, sh020_layout and sh020_lighting to v02 to have this change reflected all the way through the pipeline. If master version was possible, we would simply open the lighting scene and re-render. Now imagine that this happens on a tv show episode with 200 shots and all of them have to be re-done for one tree update, truly a PITA situation, and unfortunately quite common.

Of course, we can talk about better and more robust approaches to automatic updates through the files. Also if done properly the lighter can update the tree in the lighting scene without caring what happens underneath in layout and the setdress for example. But that's not ideal, considering the problem will still be there, next time he opens another shot.

The truth is that using a master version workflow is a very effective and viable, albeit somewhat dangerous, approach for working in many situations.

Solution

Write it to DB: We'd need to update version schema to allow the identification of this master version. I think that the best would be allowing using an empty version name that would automatically mean it's the master. We must be able to tell from the master version data which explicit version it currently corresponds to. Hence if we used an empty version name, we still need another data to specify what version number it really is (or the id of that version of course).

Another option would be leaving the version name the same as the one it corresponds to, but adding another data master simply set to true or false.

The question is whether it shouldn't be a separate type with its own schema. Which might simplify a lot of things, however, it would complicate others.

Master version should also have its own representations that would essentially be duplicates of the originals, however with different context and template.

Write it to Disk: This would be up to the config to sort out. It could simply fill the version key in the template with something else (master for instance), but we'd most likely use different template altogether and the file itself would live directly in the subset folder. That's however quite irrelevant for avalon-core itself.

The important thing is that these master publishes should always be re-written ( or hard-linked) with the latest/approved published version.

Load it via Loader

This should be a fairly simple update to the loader app, that would default to master version if it finds it, but otherwise function as it does currently

Manage it in the Scene There are a few caveats when it comes to managing scene versions.

1. we must be able to always convert the scene from master to locked versions. To achieve this we could store extra fields on containers that would specify whether we have master version loaded and what is the published version equivalent. We would then have to provide a script that crawls the containers and converts them from master to locked based on this information.
2. Storing information about current version equivalents for all containers should be happening fairly invisibly but in a safe manner. That we do it when the artist is certain the scene is "correct", personally I think that and integrator during publishing would be a good candidate.
3. Inventory should treat master version as always up to date. But of course at any time changeable to a locked version.

Alternatives We considered at length the option of just using version 0 or a very high number like 999. However for some of the studios we work with, this isn't an option. With certain jobs there are built-in client requirements for versions starting from ridiculous number ranges, say v2000 being the start (yep, it happens) for various reasons, while v000, is often used for first editorial sanity checks (making sure that plate going through comp and back, stays identical), or other purposes. So we'll need to implement this properly.

Anyone with some thoughts on the matter?

[BigRoy]

I think the issue is less about the versioning, but about a single path you're referencing to change in place. Meaning, that whatever you do you will need to swap that file out, either you're changing symlinks or actually overwriting a file.

For your specific use case, what if you turn the problem around. It seems what you're looking to produce is not even new versions at all, you're looking to "overwrite" whatever is there.

What if your instance has a versionUp boolean and when set to False the integrator (and pipeline) does not increment the version? Basically meaning that whatever the current version is will act as a master that will continuously override. This way, also whenever you do explicitly want a new version, you can do so by setting it to True again?

The downside of course is, that you are actually overwriting the same version. However, listening to your use case that's exactly what you'd want.

So basically you'd have v001 most of the time and it would never increase from there unless decided upon by the one who publishes? Would that be sufficient? And if not, why not?

Again, at best the workflow with a master version can be very dangerous and when things go wrong it can be non-trivial to recover from. It's not something I'd recommend very quickly but I can see the production use case.

[mkolar]

So basically you'd have v001 most of the time and it would never increase from there unless decided upon by the one who publishes? Would that be sufficient? And if not, why not?

this is exactly the approach that would make it dangerous. We're talking about keeping the standard versioning, but also having a single non-versioned file on top of that. So you get the best of both worlds. But most importantly at any point you can safely convert the scene to a locked version that will never change.

Yes it is primarily about overwriting a particular dedicated version file with each publish, but not about ditching versioning altogether.

t's not something I'd recommend very quickly but I can see the production use case.

I agree 100%, and it is our standpoint as well, but at the end of the day it's the client's decision about how safe vs how fast they want to be.

[vonbehr]

Maybe the way openPipeline handles this would be a good approach. In openPipeline the name of the master version never changes. Each time a new master version is published, 3 things happen:

1. A new WIP version is saved
2. A versioned copy of the master is saved in a subfolder.
3. The versionless masterfile is overwritten. So you end up with a versioned history of all master files and can go back easily if needed.

[mkolar]

that is precisely the workflow we are after. The key questions we're solving are, how to handle this in the DB and of course writing a bunch of tools to allow converting master to explicit and vice versa within the scene

[tokejepsen]

This may be too complicated but how about publishing two versions and link them as dependent?

The master version will always get overwritten when publishing, but you would update which version its dependent on.

[mkolar]

This may be too complicated but how about publishing two versions and link them as dependent?

we've decided to do something similar.

We'll create a new type called masterversion for now, which will be under at the same level in the db as standard versions. However instead of having it's own data it will only point to another version that it currently corresponds to. This master version must have it's own representations to be able to resolve paths using context and template, which could be different to standard versions.

To make everything backwards compatible, all the tools will simply look if masterversion exists and if it does, it will be used (in loader for example). We'll also store the number and id of the corresponding locked version on the component so we can switch from master to locked version at any time if needed.

Visually we were thinking of simply showing version number with some symbol to tell user he will be importing master rather than locked one. For example if the latest version publish is v003, then in the loader you'll see <v003> but when clicking on the version it will of course expand all available normal version as well, so user is not forced to use the master.

As mentioned before, the master will be overwriten with every publish to match the latest (or approved) version.

The plan is to make this completely transparent and backward compatible of course. If studio doesn't want to use masters, they simply won't publish them and everything works identically as it does now. So implementing it into a config should be as simple as writing a new integrator.

[mkolar]

We've implemented this in the pypeclub fork of core. From testing so far it looks very useable, albeit we'll still be recommending our clients not to use it unless the project absolutely requires it.

There are plenty of caveats hidden in this. For instance what happens if the file that we're overwriting is being used and locked, like arnold textures when publishing, but other artist rendering at that time.

We'll need to write some extra tools to lock the versions in the scene during publishing for example.