[Unified observability][Proposal] Data sharing between observability applications

afgomez commented 2 years ago

Historically the applications of the observability solution have only accessed their own data.

Our mission in unified observability is thinking about the overall experience for users. To fulfill this mission, different applications need to have easy access to the data of other applications. We need to create an approach to data sharing between the apps.

This issue describes the requirements for such a data sharing approach.

Terminology

Data refers to any of the observability related data. This can be low level signals (logs, metrics and traces), or high level entities (hosts, containers, services, alerts, etc.)
A data provider is the plugin that owns the data. For example, for services will be apm. For hosts will be infra.
A fetcher is a function that returns data. The requirements and conventions for fetcher functions will be specified below.
A consumer is a piece of code that uses a fetcher function to access some data.
...

Requirements

Fetchers are plugin-independent to prevent circular dependencies. They will live either in the observability plugin or in a separate package (to be determined).

Fetchers are statically typed. Consumers will have type safety. This allows easier refactors when the fetcher changes its signature.

Fetchers should behave the same in the server and in the client. For this they must the data service. Using the data service will also enable fetchers to support partial queries, cancellable queries, etc. To preserve this functionality fetchers need to return an RxJS Observable (Observables can be trivially treated as a Promise with the lasValueFrom function).

The return value should be something that the consumer can easily make sense of (so, avoid raw responses from Elasticsearch).

Fetchers know where their data is. Consumers don't need to specify the indices and/or data views to query.

Anatomy of a fetcher

function fetch[EntityOrSignal]({ ...options }): Observable;

Fetchers follow a simple naming convention:

The prefix fetch
The name of the data they are fetching.

If a fetcher returns a list of entities or signals, the name will be plural.

fetchLogs(); // Observable<LogEntry[]>
fetchServices(); // Observable<Service[]>

If a fetcher returns details for a single entity or signal, the name will be singular. The first argument must always be the identifier of the entity or signal. What is considered an identifier is up to the data provider.

fetchLogEntry('some_ES_id'); // Observable<LogEntry>
fetchService('service_id'); // Observable<Host>

Conventions and recommendations

Common parameters will be standarized. For example, date ranges will always be named from and to. Query fields will always be named query, etc.

[ ] Look for existing names in Kibana and use those.

Date ranges need to be resolved by the consumer to prevent time shifts when calling multiple fetchers.

// Bad. Each fetcher will resolve the range to different timestamps.
fetchLogs({ from: 'now-15m', to: 'now' });
fetchServices({ from: 'now-15', to: 'now' });

// Good.
const from = datemath.parse('now-15m').valueOf();
const to = datemath.parse('now').valueOf();

fetchLogs({ from, to });
fetchServices({ from, to });

Implementation challenges

There are two main challenges to be solved:

How to inject the data plugin, ideally not through an argument.
How fetchers determine where to get their data. We can use some sort of registry that the data providers need to fill at runtime.

Example usage

import { fetchLogs } from '@kbn/oblt-data';
import { useObservable } from 'react-use';

interface LogStreamProps {
  from: string;
  to: string;
  query?: string;
}

function LogStream(props: LogStreamProps) {
  const logs = useObservable(fetchLogs({ from: props.from, to: props.to, query: props.query }));

  if (!logs) {
    return <div>Loading...</div>
  }

  if (logs.length === 0) {
    return <div>No logs</div>
  }

  return (
    <>
      {logs.map(log => <div key={log.id}>log.message</div>)}
    </>
  );
}

elasticmachine commented 2 years ago

Pinging @elastic/unified-observability (Team:Unified observability)

jasonrhodes commented 2 years ago

Linked issues and discussions:

elastic / kibana