Add property to support analytics across apps

There's a growing interest in tracking workflow and data usages, with tools like DataDog, OpenTelemetry, Sumo, or Zipkin. While Desktop Agents already have a lot of the information they need to provide these integrations, there's no reliable way for a DA to identify when multiple API calls are related. Proposed below is an optional addition to context payloads, which will provide hooks for telemetry tracking tools.

This proposal follows the same approach that HTTP used for their traceparent headers, as well as the approach used in the open source OpenTelemetry standard.

Supercedes the previously submitted issue https://github.com/finos/FDC3/issues/1142

Proposal

While a full integration with a telemetry tool would need to include information about which app shared which context when, none of those features would require a change to the Desktop Agent standard. The DA already knows all of those pieces. The only thing the DA doesn't already know, per the standard, is if an FDC3 context payload is shared in response to another FDC3 message.

This proposal is for a single, optional property change that enables the Desktop Agent and individual Apps to communicate this narrow detail.

Desktop Agent

Desktop Agents SHOULD include a traceId (a properly random UUID). If an intent/context is raised/broadcast with a traceId, the desktop agent should do nothing. If there is no traceId, the deskotp agent SHOULD add one.

Individual App

If an app receives a context payload, and wants to do something in response, the app should forward the received traceId. If the app is not responding to anything, it can safely ignore the traceId.

Example

Price App broadcasts this context.

{
  "type": "fdc3.instrument",
  "id": {
    "ticker": "MSFT"
  }
}

Since there's no traceId, the desktop agent will inject one. Order App receives this context payload:

{
  "type": "fdc3.instrument",
  "id": {
    "ticker": "MSFT"
  },
  "traceId": "699381f3-284f-4014-a476-e114ef951cef"
}

The Order App will then send off price information about MSFT. Since it's responding to a context, it will include the traceId of the context it's responding to.

{
  "type": "fdc3.valuation",
  "price": 20,
  "value": 100,
  "CURRENCY_ISOCODE": "USD",
  "traceId": "699381f3-284f-4014-a476-e114ef951cef"
}

Now, the desktop agent can see that Order App has responded to Price App. Without the traceId, the desktop agent would have to guess if the contexts were related.

Other considerations

This feature could also provide desktop agents the necessary data to help identify infinite loops. For example, you could have Price App send a message, which causes Order App to respond, which causes Price App to respond, and then they ping pong off of each other forever. In networking, this is called a "broadcast storm." There is theoretically no way to determine the difference between legitimate high traffic versus illegitimate infinte loops - at least, without this kind of tracing information.

Conclusion

This proposal makes it possible for apps or desktop agents to implement telemetry, while not being perscriptive about any particular integration or tool.

Hey to follow back on this from our discussion, I feel that some form of correlation mechanism is a welcomed addition to the spec.

Looking at the types in browser types I do feel that context is not a good place for this however and instead metadata is a more suitable home. I see we already have a requestUUID in metadata. I wonder if this correlation across a workflow could be achieved with a a new metadata property of originatingRequestId and would achieve the same as a traceId but would be more focused. It would then be clear to copy this value in agents and make it immutable. In the end though that can be considered semantics.

Hi @Davidhanson90, the metadata elements in the message schemas are not somewhere that we can put this as they are an artefact of the communication protocol and not the FDC3 API. Any solution we propose MUST work via the API interface: https://fdc3.finos.org/docs/api/ref/DesktopAgent, as thats what apps actually interact with - they do not touch the messages directly, as that will be handled by the DesktopAgentProxy returned by the getAgent implementation and in browser cases (containers implement the API interface themselves independently making the comms protocol messages irrelevant to them).

That means that we either have to pass this metadata in the context object or add optional arguments to multiple API calls (most notably fdc3.broadcast, channel.broadcast, fdc3.raiseIntent and fdc3.raiseIntentForContext). Adding multiple arguments will be problematic - the only viable suggestion I have there is to add a single metadata object argument to the aforementioned functions and then have desktop agents copy its context into the ContextMetadata object passed to context handlers and intent handlers alongside the originatingAppMetadata that Desktop Agents add to it.

As I mentioned, ContextMetadata/originatingAppMetadata is currently an optional feature that is flagged in the ImplementaitonMetadata for a desktop agent - as there were demands to see it in use before it becomes a requirement in a subsequent version - for that to happen we'll need feedback on it's use or new use cases that the community considers required (tracing, signing or app-specific metadata could be such).

So here's a concrete alternative proposal (to adding a metadata element to the context schema - to support this and other proposals):

We would update the following API calls:

// Desktop Agent API
broadcast(context: Context): Promise<void>;
raiseIntent(intent: string, context: Context, app?: AppIdentifier): Promise<IntentResolution>;
raiseIntentForContext(context: Context, app?: AppIdentifier): Promise<IntentResolution>;

to have a new, optional metadata parameter

broadcast(context: Context, metadata?: MetadataType): Promise<void>;
raiseIntent(intent: string, context: Context, app?: AppIdentifier, metadata?: MetadataType): Promise<IntentResolution>;
raiseIntentForContext(context: Context, app?: AppIdentifier, metadata?: MetadataType): Promise<IntentResolution>;

Noting that if you didn't want to target your intent you would need to pass null as the second argument if using the third, e.g.:

const resolution = await fdc3.raiseIntent("ViewChart", null, { traceId: "some-uuid", signature: "some-signature-token@ });

Metadata passed would then be copied (by the Desktop Agent) into the ContextMetadata object (already) passed to the Context and Intent listeners:

type ContextHandler = (context: Context, metadata?: ContextMetadata) => void;
type IntentHandler = (context: Context, metadata?: ContextMetadata) => Promise<IntentResult> | void;

I would think that we can use ContextMetadata for the typing for both input and output metadata, with the one caveat that the DA should always overwrite or remove the existing source property to avoid spoofing (one app pretending to be another):

interface ContextMetadata {
  /** Identifier for the app instance that sent the context and/or intent. 
   *  @experimental 
   */
  readonly source: AppIdentifier;
}

This keeps the separation between the metadata and context itself. However, it will require work in DA's to implement support and, hence, may be harder to agree - the existing originating appId metadata (source field) is still optional in FDC3 - which I personally consider to be a failure to standardize...

@Davidhanson90 do you prefer this proposal? Do others object to it more than the alternative (tucking metadata into Context - there was at least one other objection to that in the past).

finos / FDC3