[Custom threshold] Add span label for async operations during rule execution

Summary

In order to be able to have a better overview of how long different async operations take, we want to add better span labeling around async operations during the custom threshold rule execution. As an example, you can check how the security team does this using withSecuritySpan.

const timestampFieldCaps = await withSecuritySpan('fieldCaps', () =>
  services.scopedClusterClient.asCurrentUser.fieldCaps(
    {
      index: inputIndex,
      fields: secondaryTimestamp
        ? [primaryTimestamp, secondaryTimestamp]
        : [primaryTimestamp],
      include_unmapped: true,
      runtime_mappings: runtimeMappings,
      ignore_unavailable: true,
    },
    { meta: true }
  )
);

There is an ongoing effort to label spans in the alerting framework as well.

Note

We only consider adding span for async operations, although we can do that for sync operations that we think might take long, but keep in mind that adding many spans will have a negative impact on performance and possibly debugging.

Implementation related info

Add the following to your Kibana config:

elastic.apm.active: true
elastic.apm.transactionSampleRate: 1.0
elastic.apm.environment: YOUR_USERNAME

Going to https://kibana-cloud-apm.elastic.dev/ -> APM page, you should be able to filter the environment using your YOUR_USERNAME.

Acceptance criteria

Identify async operations during the custom threshold rule execution and add a proper name.

elastic / kibana