search

AcademySoftwareFoundation / openfx

OpenFX effects API
Other
393 stars 119 forks source link

Clip and Image Metadata #142

Open fxtech opened 4 months ago

fxtech commented 4 months ago

Open Effects Proposal for Standard Change

Please read the contribution guidelines first.

Standard Change Workflow

Requirements for accepting a standard change:

Summary

This proposal aims to add metadata read/write capability to plugins.

Motivation

Some plugins want to be able to get the file path and frame number of the media that sourced the current clip and/or frame. Other plugins may want to query information about the lens used, or the stack of previous effects applied. The ability to add metadata to the output may also be useful.

Problem

Currently there is no way to query anything about input images to an effect came from, such as the name or path of the source clip.

Impact

This is a new feature, embodied as a new set of properties. It dos not affect any existing suites or properties.

This feature is entirely optional. A host that does not implement it will simply appear to have no meta data to an effect. There is no requirement for an effect to make use of it at all.

Documentation Impact

The documentation for this feature could be embodied entirely in the header file additions.

Stakeholders

Plugin writers should benefit the most from this change.

Discussion

fxtech commented 4 months ago

Some thoughts on this:

  1. This API should be contained entirely in a new file: ofxMetadata.h
  2. It would be useful for metadata to be available at the Clip and Image levels. For example, it would be meaningful to query a clip to see what the file path is, but you'd need to query the input image to learn the frame number or timecode it came from.
  3. It would be useful to be able to add or modify input metadata and pass it on to the output.
  4. Metadata should be encapsulated in PropertySet, since it already acts like a dictionary.
  5. There needs to be a mechanism to enumerate all of the metadata available in the PropertySet.
  6. There should be a well-defined set of standard metadata keys and value types, possibly derived from OTIO conventions.
  7. Hosts should be free to implement the metadata API with lazy evaluation. There should not be a performance penalty if a plugin never uses it.

In order to best handle point 2 above, I propose the API is handled with a new OfxMetadataSuiteV1.

The workflow would be to first get a property set from either a Clip or an Image using the existing suites for doing so, then obtain the metadata property set from that, using metadataGet(). This is likely to be handle around some possibly-expensive implementation details, so must be released with metadataRelease(). Property keys can be enumerated with metadataEnumerate(). Note that using the OfxPropertySuiteV1 to modify metadata properties can change them immediately. It should not be necessary to copy metadata from input clips or images to the output clip or images - the host will do that on its own. But it should be possible to add to or override the metadata in the output clip and/or image.

typedef OfxStatus (OfxMetadataEnumationFuncV1)(OfxPropertySetHandle metadataPropSet, const char *property, void *userData);

typedef struct OfxMetadataSuiteV1 {  
  OfxStatus (*metadataGet)(OfxPropertySetHandle clipOrImagePropertySetHandle, OfxPropertySetHandle *metadataPropHandle);
  OfxStatus (*metadataRelease)(OfxPropertySetHandle metadataPropHandle);
  OfxStatus (*metadataEnumerate)(OfxPropertySetHandle metadataPropHandle, OfxMetadataEnumationFuncV1 callback, void *userData);
};
fxtech commented 4 months ago

Some standard metadata:

Vendor-specific metadata keys should be prefixed with the vendor ID (ie. com.borisfx) File-format-specific metadata should be prefixed with the file format ID if there is one. (ie. openexr)

AdrianEddy commented 4 months ago

Some additional useful metadata:

I'm strongly in favor of this feature, the media file path and accurate timestamp are essential for my use case and OpenFX currenty doesn't provide them

barretpj commented 4 months ago

On Sat, 10 Feb 2024 at 22:01, Paul Miller @.***> wrote:

Some standard metadata:

  • media file path (as a URL)

Should we define how to represent a per-clip path for file-per-frame (sequence) media? printf-style ("%07d" with %% escaping)? ####### style?

  • source frame number
  • source timecode (OTIO uses HH:MM:SS;FRAME or HH:MM:SS.MS)

Where ";FRAME" implies drop-frame, presumably, and ":FRAME" non-drop?

Vendor-specific metadata keys should be prefixed with the vendor ID (ie. com.borisfx)

Where "vendor-specific" means everything that's not defined in the suite header file?

File-format-specific metadata should be prefixed with the file format ID if there is one. (ie. openexr)

File-format prefixes are unlikely to be possible in my host I fear. Do we need to define how OpenEXR metadata types with non-obvious mapping to OFX string values are to be handled (e.g. chromaticities, box2i, m44f etc)? Do we need to define what the prefixes are for some well-known file formats? Or is it just "copy Nuke"? There's ambiguity between what's vendor-specific and what's file-format-specific too, unless we do say "if it's at all different from Nuke then it's vendor-specific"?

revisionfx commented 4 months ago

prefixes are unlikely to be possible in my host ETC

I was thinking for frame meta-data we would pack chunks of meta-data inside a named group of data - then EXR or CameraX "values" can just be inside that without prefixes...

group "EXR" { list... } - easier to search if you are looking just for something in particular then scanning all

garyo commented 4 months ago

@revisionfx I'm not sure what you're getting at with your grouping idea. Are you thinking the actual metadata values would be a JSON string or something, represented as a single OpenFX string property? OpenFX doesn't have hierarchy/groups for properties otherwise.

revisionfx commented 4 months ago

Phil used the example of OpenEXR Notice the new attributes (long list): https://openexr.com/en/latest/StandardAttributes.html I am suggesting that we could use a generic meta-data suite and "invent" such grouping and call it maybe "schema" ? What would be in the "EXR" schema is a list of Properties (the attributes above). The Meta-Data suite would allow one to request the "EXR" meta-data handle... as opposed to try to parse a long list of stuff I might not know anything about.

revisionfx commented 4 months ago

DO you need a PropReason for change so frame meta-data change upstream if a node is a client get rerender? Making sure caching based on parameter changes are not skipping effect then

fxtech commented 4 months ago

We agreed that metadata property keys should be prefixed with 'where it came from', and define a common set with the ofx prefix.

fxtech commented 4 months ago

In order to support "unknown/binary blob" types of metadata we should add a try "raw data" type to PropertSet and while we're at it, parameter support. I'll work up a separate issue for that.

MrKepzie commented 4 months ago

Just as a side-note, I had implemented this in openfx host support 3 years back, with also support for time varying metadata.
I also had to add Bytes support to property types spec looked like this:

typedef struct OfxMetaDataSuiteV1
{
  /*  @brief Get the time-invariant metadata property set corresponding to the given key.
   @param inArgs The current action inArgs that were passed by the host to avoid lost
   of context by the host.
   @param clip The clip on which to get the metadata from
   @param desiredKeys Optional array of metadata key that are needed. Each metadata desired key is an array
   representing the hierarchy of metadata containers leading to the desired metadata name, e.g: ["temp",project1","scene1","metadata"]
   If NULL, the host must return a property set containing all metadata available on the clip.
   @param nGrouping For each metadata in desiredKeys, the number of items in the grouping
   @param nDesiredKeys The number of keys in desiredKeys
   @returns kOfxStatOK - The metadata property set was successfully fetched and returned
   in the handle
   kOfxStatFailed - The metadata property set could not be fetched because
   no metadata matches the corresponding key
   kOfxStatErrBadHandle - The inArgs handle or clip handle was invalid
   kOfxStatErrMemory - The host had not enough memory to complete the operation
   plug-in should abort whatever it was doing
   */
  OfxStatus (*clipGetMetadata) (OfxPropertySetHandle inArgs,
                                OfxImageClipHandle clip,
                                const char*** desiredKeys,
                                int* nGrouping,
                                int nDesiredKeys,
                                OfxPropertySetHandle* propertySetHandle);

  /*  @brief Get the time-variant metadata property set corresponding to the given key.
   @param inArgs The current action inArgs that were passed by the host to avoid lost
   of context by the host.
   @param clip The clip on which to get the metadata from
   @param time The time at which to sample the metadata
   @param view The view on which to sample the metadata (to support the multi-view suite)
   @param desiredKeys Optional array of metadata key that are needed. Each metadata desired key is an array
   representing the hierarchy of metadata containers leading to the desired metadata name, e.g: ["temp",project1","scene1","metadata"]
   If NULL, the host must return a property set containing all metadata available on the clip.
   @param nGrouping For each metadata in desiredKeys, the number of items in the grouping
   @param nDesiredKeys The number of keys in desiredKeys
   @returns kOfxStatOK - The metadata property set was successfully fetched and returned
   in the handle
   kOfxStatFailed - The metadata property set could not be fetched because
   no metadata matches the corresponding key
   kOfxStatErrBadHandle - The inArgs handle or clip handle was invalid
   kOfxStatErrMemory - The host had not enough memory to complete the operation
   plug-in should abort whatever it was doing
   */
  OfxStatus (*clipGetMetadataAtTime) (OfxPropertySetHandle inArgs,
                                      OfxImageClipHandle clip,
                                      OfxTime time,
                                      int view,
                                      const char*** desiredKeys,
                                      int* nGrouping,
                                      int nDesiredKeys,
                                      OfxPropertySetHandle* propertySetHandle);

} OfxMetaDataSuiteV1;

/*
Action to fetch time invariant metadata on an image effect instance.
This action may only be called from the main thread. This action generalizes and replaces
the old kOfxImageEffectActionGetClipPreferences action, and provides a similar behavior.

inArgs:

outArgs:

A property set where the plug-in can create properties of type int, double or string:
these properties are the metadatas encoded by the plug-in.

A string property may contain binary data, so it should be handled correctly by the host and the plug-in.
The property suite has been modified accordingly.

The effect may only return time invariant metadata from this action.

- For each clip, a property whose name is the clip name appended with kOfxImageEffectPropPassThroughMetadata (see declaration below)

- kOfxImageEffectPropMetadataPassThroughClip (see declaration below)
 */
#define kOfxImageEffectActionGetTimeInvariantMetadata "OfxImageEffectActionGetTimeInvariantMetadata"

/*
 Action to fetch time/view variant metadata on an image effect instance. May be called
 on a render thread.

 The image effect may only return time/view variant  metadatas from this action, such as a timecode.

 inArgs:

 — kOfxPropTime double x1, The time at which to sample the metadata
 - kFnOfxImageEffectPropView, int x1 (only if view aware)

 outArgs:

 A property set where the plug-in can create properties of type int, double or string:
 these properties are the metadatas encoded by the plug-in.

 A string property may contain binary data, so it should be handled correctly by the host and the plug-in.
 The property suite has been modified accordingly.

 The effect may only return time invariant metadata from this action.

 - kOfxImag- For each clip, a property whose name is the clip name appended with kOfxImageEffectPropPassThroughMetadata (see declaration below)
eEffectPropPassThroughMetadata (see below)

 - kOfxImageEffectPropMetadataPassThroughClip (see below)
 */
#define kOfxImageEffectActionGetTimeVariantMetadata "OfxImageEffectActionGetTimeVariantMetadata"

/* @brief string x N property
 Property Set: outArgs of the kOfxImageEffectActionGetTimeInvariantMetadata or kOfxImageEffectActionGetTimeVariantMetadata action

 The plug-in may indicate into it the metadata key(s) of all metadata inherited from a given clip (whose name
 must be prepended to this string) that it does not create.
 Omitted metadata are considered deleted and should not be made available on downstream image effects by the host.
 */
#define kOfxImageEffectPropPassThroughMetadata "_PassThroughMetadata"

/* @brief int x1
 Property Set: Parameter descriptor (read/write), parameter instance (read only)
 Valid values - 0 or 1
 Default value - 0
 Indicates whether a change to this parameter may involve a change to the metadata.

 If set to 1, the host should not only call the kOfxActionInstanceChanged action on the corresponding image effect,
 but also call this action with a type kOfxTypeClip with argument kOfxImageEffectOutputClipName recursively downstream on all image effects.

 This enables a plug-in to update its parameters when a metadata is changed upstream.
 */
#define kOfxParamPropMetadataSlave "OfxParamPropMetadataSlave"

typedef struct OfxPropertySuiteV2 {

  /// Inherited from OfxPropertySuiteV1
  OfxPropertySuiteV1 suiteV1;
  /// End Inherited from OfxPropertySuiteV1

  /* @brief Creates an integer property with the given name and the given
    @param property The name of the property
    @param dimension The number of dimensions
    @param defaultValue An array of size dimension containing the default values
    @returns - kOfxStatFailed: The host declines creation of this property
             - kOfxStatErrExists: A property with the same name already exists in the property set
             - kOfxStatErrUnsupported:  The host does not support properties creation
   */
  OfxStatus (*propCreateInt) (OfxPropertySetHandle properties, const char* property, int dimension, int defaultValue);

  /* @brief Same as other propCreateInt, except for double properties
   */
  OfxStatus (*propCreateDouble) (OfxPropertySetHandle properties, const char* property, int dimension, double defaultValue);

  /* @brief Same as other propCreateInt, except for c-string properties.
   The host should copy the default string values so that their lifetime outlive
   the action within which the plug-in is currently calling this function.
   */
  OfxStatus (*propCreateString) (OfxPropertySetHandle properties, const char* property, int dimension, const char* defaultValue);

  /* @brief Same as other propCreateInt, except for pointer properties.
   The pointers given in defaultValue should outlive the action from which the plug-in is currently calling
   this function until the next action is called on the corresponding image effect.
   */
  OfxStatus (*propCreatePointer) (OfxPropertySetHandle properties, const char* property, int dimension, void* defaultValue);

  /* @brief Same as other propCreateString, except that it also takes a nBytes array
   of size 'dimension' indicating for each c-string its size in bytes.
   */
  OfxStatus (*propCreateBytes) (OfxPropertySetHandle properties, const char* property, int dimension, int nBytes, const unsigned char* defaultValue);

  /* @brief Creates a new property that acts as a container for other properties.
   This allows to create recursive property sets.
   @param property The name of the property
   @returns
     - \ref kOfxStatFailed: The host declines creation of this property
     - \ref kOfxStatErrExists: A property with the same name already exists in the property set
     - \ref kOfxStatErrUnsupported:  The host does not support properties creation
   */
  OfxStatus (*propCreatePropertySet) (OfxPropertySetHandle properties, const char* property, OfxPropertySetHandle* createdSetHandle);

  /* @brief Get a handle to a property of type property set.
   @param properties is the handle of the thing holding the property
   @param property is the string labelling the property
   @param setHandle is the property set handle
   @returns
   - ::kOfxStatOK
   - ::kOfxStatErrBadHandle
   - ::kOfxStatErrUnknown
   */
  OfxStatus (*propGetPropertySet) (OfxPropertySetHandle properties, const char* property, OfxPropertySetHandle* setHandle);

  /* @brief Set a single value in a string property being plain binary data.
   @param properties is the handle of the thing holding the property
   @param property is the string labelling the property
   @param index is for multidimenstional properties and is dimension of the one we are setting
   @param nBytes The number of bytes to copy from the "value" buffer.
   @param value is the buffer we are setting. The host should make a copy of this buffer.
   @returns
   - ::kOfxStatOK
   - ::kOfxStatErrBadHandle
   - ::kOfxStatErrUnknown
   - ::kOfxStatErrBadIndex
   - ::kOfxStatErrValue
   */
  OfxStatus (*propSetBytes) (OfxPropertySetHandle properties, const char* property, int index, int nBytes, const unsigned char* value);

  /* @brief Set a single value in a string property being plain binary data.
   @param properties is the handle of the thing holding the property
   @param property is the string labelling the property
   @param index is for multidimenstional properties and is dimension of the one we are setting
   @param nBytes The number of bytes to copy onto the "value" buffer.
   @param value is the buffer we are getting. The buffer should already be allocated of nBytes.
   The host should make a copy to this buffer.
   @returns
   - ::kOfxStatOK
   - ::kOfxStatErrBadHandle
   - ::kOfxStatErrUnknown
   - ::kOfxStatErrBadIndex
   */
  OfxStatus (*propGetBytes) (OfxPropertySetHandle properties, const char* property, int index, int *nBytes, unsigned char** value);

  /* @brief Set a single value in a string property being plain binary data.
   @param properties is the handle of the thing holding the property
   @param property is the string labelling the property
   @param index is for multidimenstional properties and is dimension of the one we are setting
   @param nBytes The number of bytes to copy onto the "value" buffer.
   @param value is the buffer we are getting. The buffer should already be allocated of nBytes.
   The host should make a copy to this buffer.
   @returns
   - ::kOfxStatOK
   - ::kOfxStatErrBadHandle
   - ::kOfxStatErrUnknown
   - ::kOfxStatErrBadIndex
   */
  OfxStatus (*propGetNumBytes) (OfxPropertySetHandle properties, const char* property, int index, int *nBytes);

  /* @brief Set all values at once in a string property being plain binary data.
   @param properties is the handle of the thing holding the property
   @param property is the string labelling the property
   @param count is for multidimensional properties and is number of dimensions of the property
   @param nBytes The number of bytes to copy from the "value" buffer. This is an array of size "count" containing
   for each dimension the size in bytes of the corresponding buffer in "value".
   @param value is an array of size "count" containing for each dimension
   the buffer we are setting. The host should make a copy of this buffer.
   @returns
   - ::kOfxStatOK
   - ::kOfxStatErrBadHandle
   - ::kOfxStatErrUnknown
   - ::kOfxStatErrBadIndex
   - ::kOfxStatErrValue
   */
  OfxStatus (*propSetBytesN) (OfxPropertySetHandle properties, const char* property, int count, const int *nBytes, const unsigned char** value);

  /* @brief Set all values at once in a string property being plain binary data.
   @param properties is the handle of the thing holding the property
   @param property is the string labelling the property
   @param count is for multidimensional properties and is number of dimensions of the property
   @param nBytes The number of bytes to copy to the "value" buffer. This is an array of size "count" containing
   for each dimension the size in bytes of the corresponding buffer in "value".
   @param value is an array of size "count" containing for each dimension
   the buffer we are setting. Each buffer is already allocated of at least nBytes.
   The host should make a copy to this buffer.
   @returns
   - ::kOfxStatOK
   - ::kOfxStatErrBadHandle
   - ::kOfxStatErrUnknown
   - ::kOfxStatErrBadIndex
   - ::kOfxStatErrValue
   */
  OfxStatus (*propGetBytesN) (OfxPropertySetHandle properties, const char* property, int count, int *nBytes, unsigned char** value);

  /** @brief Get the number of properties in a property set.
   @param numProperties In output, this is the number of properties in the set.
   @returns - kOfxStatFailed  The host did not manage to figure out the number of properties
   in the set or does not want the plug-in to be able to iterate over the properties.
   - kOfxStatOk The host successfully returned the number of properties in the property set.
   - kOfxStatErrBadHandle The properties handle is invalid
   */
  OfxStatus (*propGetNumProperties) (OfxPropertySetHandle properties, int* numProperties);

  /** @brief Get the name and type of a property in a property set
   @param index The index of the property to fetch in the set. This must be a number ranging
   from 0 to the number of properties returned by the propGetNumProperties function.
   @param property In output, a UTF-8 encoded c-string containing the name of the property.
   This name can be used to call any other function of the property suite. The string is
   valid until the action ends.
   @param typeName In output, a UTF-8 encoded c-string containing the name of the type of the property.
   This can either be "int", "double", "pointer", "string", "bytes", "set".
   @returns - kOfxStatFailed  The host did not manage to figure out the property name
   in the set or does not want the plug-in to be able to iterate over the properties.
   - kOfxStatOk The host successfully returned the property name
   - kOfxStatErrBadIndex The index is invalid
   - kOfxStatErrBadHandle The properties handle is invalid
   */
  OfxStatus (*propGetPropertyNameAndType) (OfxPropertySetHandle properties, int index, char** property, char** typeName);

} OfxPropertySuiteV2;
fxtech commented 4 months ago

Well, this is interesting! It pretty much handles everything we have been discussing recently regarding Metadata as well. Anyone else want to take a look and comment?

garyo commented 4 months ago

Nice! Thanks for this, @MrKepzie ! @fxtech how does this compare with your version? (We don't have the multi view suite, so would removing that be a problem, Alexandre?)

fxtech commented 4 months ago

One key design difference I see is this API requires loaders to "create" properties, where the new proposal is inteded to be merely a wrapper around potentially-lazily-evaluated metadata. It would be difficult for Silhouette to have arbitrary plugins "create" their own properties, as internally these properties don't really "exist" in a list with a length, per-se.

MrKepzie commented 4 months ago

My use case was:

Temporal/View variant was added too, but since OpenFX doesn't have the concept of view out of the box, this should be stripped off. The system was really to mock up what you can do in Nuke and pass metadata through the graph

MrKepzie commented 4 months ago

@fxtech This is lazily evaluated as long as the host doesn't call the action or just calls it with the desired metadata keys

fxtech commented 4 months ago

OK I didn't see you had added an Action. That's an interesting approach. I encourage others to chime in.

garyo commented 4 months ago

The action (especially the recursive signaling of downstream effects) may be a bit of work for hosts to implement. Can we get to a "MVP" version of this without that feature, i.e. where only the host may specify metadata and it's read-only from the plugin side? Then we can have more interesting discussions about plugin-created metadata and how it propagates, gets signaled, etc.

fxtech commented 4 months ago

Alexandre's implementation seems pretty robust, but for what we're trying to achieve in terms of simplicity I propose we go forward with my proposal for now. Would like to hear from others though because I am definitely not one to prefer reinventing the wheel.

revisionfx commented 1 month ago

@fxtech - just refreshing my memory on this.

revisionfx commented 1 month ago

Just to be clear following today's conversation. I am not quite clear what meta-data types are: simple raw types (e.g. float,double, int, strings...) or support for more dimensional constructs, e.g. box, color...