Enhancement: Expose saving artist notes or workfile document data in an API

[BigRoy]

Is there an existing issue for this?

• [X] I have searched the existing issues.

Please describe the feature you have in mind and explain what the current shortcomings are?

Through the workfiles tool the user can get and/or save an "Artist note" per workfile and potentially store other data into the workfile documents. This functionality seems deeply embedded into the UIs but there seems to be no simple API to set these through python code.

Why?

Exposing this through the API could allow it to be used for:

1. A post-integration plug-in that stores e.g. the publish date/time of a workfile (when that file was involved in a publish) into its artist note in the form of: Published {datetime} to subsets: {",".join(subsets)} so that it could e.g. show:

   Published 07-08-2023 12:31 to subsets: modelMain v001, reviewMain v002 Disabled instances: pointcacheEnvironment, character_hero_01

2. When a workfile is downloaded on launch via e.g. this PR https://github.com/ynput/OpenPype/pull/5660 then the Artist note could set for downloaded files, e.g.:

   Download from published workfileAnimation > v091

3. It could also be used by custom ingesting tools someone might use, e.g. we have some for quickly ingesting rigs - the note could then contain more details about what the source file of the ingestion was to our workfiles.
4. The Workfiles Tool browsing "published files" using Copy & Open could then also store an artist note saying what published workfile it was copied from, similar to idea 2) listed here.
5. The artist notes for a workfile could then also be shown elsewhere then only in the workfiles tool - so that it might get more usage overall.

How would you imagine the implementation of the feature?

I'd expect a simple set_workfile_data and/or set_workfile_note function that'd work for both OpenPype and AYON.

Here's working code that could be the basics for the OpenPype API but it'll likely not match for AYON:

import os
import copy

from openpype.pipeline import Anatomy
from openpype.client import get_workfile_info
from openpype.client.operations import (
    OperationsSession,
    new_workfile_info_doc,
    prepare_workfile_info_update_data,
)

def get_workfile_doc(project_name,
                     asset_id,
                     task_name,
                     filepath):
    """Return workfile document from database, if it exists"""
    filename = os.path.basename(filepath)
    return get_workfile_info(
        project_name, asset_id, task_name, filename
    )

def create_workfile_doc(project_name,
                        asset_id,
                        task_name,
                        filepath):
    """Create workfile document in database.

    If it already exists, the existing document is returned.

    """

    workdir, filename = os.path.split(filepath)
    workfile_doc = get_workfile_info(
        project_name, asset_id, task_name, filename
    )
    if workfile_doc:
        return workfile_doc

    # Create a new document
    anatomy = Anatomy(project_name)
    success, rootless_dir = anatomy.find_root_template_from_path(workdir)
    filepath = "/".join([
        os.path.normpath(rootless_dir).replace("\\", "/"),
        filename
    ])

    workfile_doc = new_workfile_info_doc(
        filename, asset_id, task_name, [filepath]
    )

    session = OperationsSession()
    session.create_entity(project_name, "workfile", workfile_doc)
    session.commit()

    return workfile_doc

def set_workfile_note(
    project_name,
    asset_id,
    task_name,
    filepath,
    note
):
    """Update the artist note for a workfile.

    If the workfile document does not exist in the database yet
    it will be created.
    """
    # This does not create the document if it already exists.
    workfile_doc = create_workfile_doc(
        project_name, asset_id, task_name, filepath
    )
    if workfile_doc.get("data", {}).get("note") == note:
        # Nothing to update
        return

    new_workfile_doc = copy.deepcopy(workfile_doc)
    new_workfile_doc.setdefault("data", {})["note"] = note
    update_data = prepare_workfile_info_update_data(
        workfile_doc, new_workfile_doc
    )
    if not update_data:
        return

    session = OperationsSession()
    session.update_entity(
        project_name, "workfile", workfile_doc["_id"], update_data
    )
    session.commit()

# Example to setting current workfile
from openpype.pipeline import registered_host
from openpype.client import get_asset_by_name
host = registered_host()
filepath = host.get_current_workfile()
if not filepath:
    raise RuntimeError("Current file not saved.")

context = host.get_current_context()
project_name = context["project_name"]
asset_name = context["asset_name"]
task_name = context["task_name"]
asset_doc = get_asset_by_name(project_name, asset_name, fields=["_id"])
asset_id = asset_doc["_id"]
set_workfile_note(project_name, asset_id, task_name, filepath, note="hello monkeytown")

Are there any labels you wish to add?

• [X] I have added the relevant labels to the enhancement request.

Describe alternatives you've considered:

No response

Additional context:

@iLLiCiTiT mentioned that for AYON workfiles implementation a separate controller implementation of saving workfile notes, etc. has been implemented. As such it's likely that both the old tool and new tool should be refactored to use a singular API (which in the API might need an OpenPype way and an AYON way - with the OpenPype code being described in the issue ☝️ )

---

If the API is written, I'm personally not sure where this would be stored within OpenPype or Ayon? Is it openpype.pipeline.workfile.{???}?

[cuID:OP-7102]

[fabiaserra]

Thank you @BigRoy for starting this. I like where this is going but could we rethink this proposal so it's not only for workfiles and we can use the same API to write "comment"/"description" to any product entity? The only difference here would be that the workfile UI shows that "comment" for the workfile entities

[BigRoy]

I like where this is going but could we rethink this proposal so it's not only for workfiles and we can use the same API to write "comment"/"description" to any product entity? The only difference here would be that the workfile UI shows that "comment" for the workfile entities

Could be nice - I'm not sure how that models to the different entities. I know e.g. assets have data.label (and data.description maybe?) but no data.comment and version entities have data.comment. Workfiles apparently have data.note - haha.

I'm not sure how the data model in AYON would be streamlined to maybe ease these efforts. With those differing so much (in OpenPype at least) I'm not sure what type of consistently exposed functions make sense here. Maybe @mkolar can offer more insight how this might change with AYON.

By the way, looking at the basics of the logic above, updating any data of an entity seems to follow this API:

Please consider these pseudocode examples - since these operations are intended to 'update database' entries proceed with care when running them as tests. ;)

doc = {...} # project, asset, subset, version or workfile doc?
new_doc = copy.deepcopy(doc)

# Make changes to document here

update_data = prepare_workfile_info_update_data(
    doc, new_doc
)
if update_data:
    session = OperationsSession()
    session.update_entity(
        project_name, doc["type"], doc["_id"], update_data
    )
    session.commit()

Which means that updating/setting comment, note or label would be as simple as doing that with either data.note or data.label, etc. Taking that a bit further, some pseudocode like this could be used:

import copy
from openpype.client.operations import (
    OperationsSession,
    # Note that operations actually has
    # prepare update data functions for the
    # different types - to ensure some keys
    # are never updated
    _prepare_update_data
)

def update_entity(doc, changes):
    new_doc = copy.deepcopy(doc)

    for key, value in changes.items():
        if "." in key:
            # Allow specifying e.g. key data.note
            # to only adjust data.note
            parent_keys = key.split(".")
            key = parents.pop()
            target = new_doc
            for parent_key in parents:
                # TODO: What to do if an existing parent key
                #   isn't a dict itself. Do we want to replace it?
                target = target.setdefault(parent_keys, {})
            target[key] = value
        else:
            new_doc[key] = value

    update_data = _prepare_update_data(doc, new_doc)
    if not update_data:
        return

    session = OperationsSession()
    session.update_entity(
        project_name, doc["type"], doc["_id"], update_data
    )
    session.commit()

# Some examples    
update_entity(asset_doc, {"data.label": "new label"})
update_entity(version_doc, {"data.comment": "new group"})
update_entity(version_doc, {"data.subsetGroup": "my new subset group"})
update_entity(workfile_doc, {"data.note": "new note"})

Note that I have not tested these pseudocode examples. Since they "update database entries" please proceed with caution when testing. But as the comment there shows the openpype.client.server.operations has dedicated update functions for the different entity types which should be used instead to have "some safeguards"

[fabiaserra]

Coincidentally, today I had one of these issues were I had to update some published mongo data so thank you @BigRoy for that code snippet! Here's the updated working version I used for updating a few versions:

from openpype.client.operations import (
    OperationsSession,
    prepare_subset_update_data
)

def update_entity(project_name, doc, changes):
    new_doc = copy.deepcopy(doc)

    for key, value in changes.items():
        if "." in key:
            # Allow specifying e.g. key data.note
            # to only adjust data.note
            parent_keys = key.split(".")
            key = parent_keys.pop()
            target = new_doc
            for parent_key in parent_keys:
                # TODO: What to do if an existing parent key
                #   isn't a dict itself. Do we want to replace it?
                target = target.setdefault(parent_key, {})
            target[key] = value
        else:
            new_doc[key] = value

    update_data = prepare_subset_update_data(doc, new_doc, replace=False)
    if not update_data:
        return

    session = OperationsSession()
    session.update_entity(
        project_name, doc["type"], doc["_id"], update_data
    )
    session.commit()

And we should make sure to validate that the doc given to update_entity has all the "data" queried... otherwise you end up removing the data fields that weren't queried (I might or might have not done that mistake and ended up having to write extra logic to rewrite it)!

[fabiaserra]

is there any difference really on prepare_subset_update_data or prepare_version_update_data?

[BigRoy]

And we should make sure to validate that the doc given to update_entity has all the "data" queried... otherwise you end up removing the data fields that weren't queried (I might or might have not done that mistake and ended up having to write extra logic to rewrite it)!

Note: Example below is not entirely correct due to top level dicts basically being applied as full replacement

By the way, I believe if you only want to update values and never remove values then I think you can maybe do without prepare_update_data and directly pass only the to change data to:

session.update_entity(
        project_name, doc["type"], doc["_id"], update_data
)

Since the preparing function seems to only record the 'changes' between the documents and has a special value for removals. _N

Again, test this assumption ;) do not run it live, etc. <- this assumption was wrong.

For example:

update_data = {"data": {'hello": "world"}}

This would replace the full data key with the new dict - it doesn't just update "data.hello".

EDIT: Looking at this code I'm pretty sure nested dicts will be completely replaced, not updated partially.

Updating only sub keys of dicts

You might "beat that" by turning nested dicts into flattened keys like: {"data.note": "value"} in MongoDB operations But I'm not sure if those will eventually also work with the AYON API. Maybe @iLLiCiTiT knows. (referencing discord question here)

---

is there any difference really on prepare_subset_update_data or prepare_version_update_data?

As I see it, currently not but of course it might be in the future.

[fabiaserra]

Yeah that was my issue that prepare_update_data value is directly the nested dictionary of data... so it was replacing it completely with my small subset of data...

[fabiaserra]

Is there any easy way to retrieve the mongo _id in any of the OP uis? I was thinking that it would have been useful if I could just see it in the Loader "Version info" tab

[BigRoy]

Is there any easy way to retrieve the mongo _id in any of the OP uis? I was thinking that it would have been useful if I could just see it in the Loader "Version info" tab

Right click in the version tab, copy raw data:

But I believe that only provides the representation document.

Otherwise you might want something like this Debug Loader (note that I believe this one only enables when running from a build or OPENPYPE_DEBUG env var is 1 - it doesn't come natively with OpenPype however)

[fabiaserra]

nice, that's useful, I will add it to my fork, thank you!!

[Sharkitty]

One problem I see here is that artist notes as they are now only seem to affect local workfiles, which in OpenPype aren't tied to the db. So applying artist notes to products instead (it's equivalent to subsets right?), would be a completely different thing. In the context of workfiles, it would mean that the notes would be written to published workfiles (which are subsets, therefore they should be able to use the same data as other subsets), but that would not cover cases in which we would want to add notes to local workfiles. Shouldn't local workfiles and products have their own separate implementation?

[BigRoy]

One problem I see here

Where is that exactly? Are you responding to something specific by Fabia? I don't see him requesting to set artist notes on published workfiles. The way I interpreted it is: "Can we make the API more generalized so that it becomes trivial to update the comment or note or anything else on published data.

I agree that the request differs from the current PRs direct needs - and regarding comment, notes, descriptions, etc. as I remarked here there's also not one general name for it.

I'd say having a simple "update any data" API would be nice but might be best left for a separate issue referencing this PR as a prototype for one aspect.

[Sharkitty]

I'm just saying local workfiles and products are two different things. I'm not sure how this API change could work out for both, unless it's separated, with one feature for workfile note, and another for products comment. What do you think about it?

[BigRoy]

Agreed. I think this PR focused on just workfiles is fine. @fabiaserra how would you conclude after above discussion?

[fabiaserra]

yeah definitely seems like it would have to be two different APIs for local workfiles and products, how are the artist notes for local workfiles stored?

[mkolar]

@iLLiCiTiT if I'm not mistaken, this is not really relevant in AYON, right? or rather fixed by the new API

[Sharkitty]

@iLLiCiTiT if I'm not mistaken, this is not really relevant in AYON, right? or rather fixed by the new API

I believe it's been said that the workfile API is different in AYON, therefore I don't think this can be used as is in AYON. Also the initial goal with this feature was to be used in #5660 which has been closed. Its implementation in #5808 has been closed as well. I think it would be fine to close this issue.