Add GetPipelines() to Host interface so that pipelines are available to extensions

[abeach-nr]

Is your feature request related to a problem? Please describe. I am trying to build an extension to provide debugging information about pipelines.

I believe it is only possible to debug pipelines via zpagesexention as an http endpoint /debug/pipelinez https://github.com/open-telemetry/opentelemetry-collector/blob/e0eea95c1a6913741fbfe622ea41dfa33ecf8df2/service/zpages.go#L36

Describe the solution you'd like I would like to add a GetPipelines method to the Host interface:

https://github.com/open-telemetry/opentelemetry-collector/blob/e0eea95c1a6913741fbfe622ea41dfa33ecf8df2/component/host.go#L23

type Host interface {
    // ReportFatalError is used to report to the host that the component
    // encountered a fatal error (i.e.: an error that the instance can't recover
    // from) after its start function had already returned.
    //
    // ReportFatalError should be called by the component anytime after Component.Start() ends and
    // before Component.Shutdown() begins.
    ReportFatalError(err error)

    // GetFactory of the specified kind. Returns the factory for a component type.
    // This allows components to create other components. For example:
    //   func (r MyReceiver) Start(host component.Host) error {
    //     apacheFactory := host.GetFactory(KindReceiver,"apache").(component.ReceiverFactory)
    //     receiver, err := apacheFactory.CreateMetricsReceiver(...)
    //     ...
    //   }
    //
    // GetFactory can be called by the component anytime after Component.Start() begins and
    // until Component.Shutdown() ends. Note that the component is responsible for destroying
    // other components that it creates.
    GetFactory(kind Kind, componentType config.Type) Factory

    // GetExtensions returns the map of extensions. Only enabled and created extensions will be returned.
    // Typically is used to find an extension by type or by full config name. Both cases
    // can be done by iterating the returned map. There are typically very few extensions,
    // so there are no performance implications due to iteration.
    //
    // GetExtensions can be called by the component anytime after Component.Start() begins and
    // until Component.Shutdown() ends.
    GetExtensions() map[config.ComponentID]Extension

    // GetExporters returns the map of exporters. Only enabled and created exporters will be returned.
    // Typically is used to find exporters by type or by full config name. Both cases
    // can be done by iterating the returned map. There are typically very few exporters,
    // so there are no performance implications due to iteration.
    // This returns a map by DataType of maps by exporter configs to the exporter instance.
    // Note that an exporter with the same name may be attached to multiple pipelines and
    // thus we may have an instance of the exporter for multiple data types.
    // This is an experimental function that may change or even be removed completely.
    //
    // GetExporters can be called by the component anytime after Component.Start() begins and
    // until Component.Shutdown() ends.
    GetExporters() map[config.DataType]map[config.ComponentID]Exporter

        GetPipelines()pipelines.Pipelines

}

With that interface, i can build an exention with the following:

type MyExtension struct {
}
func (m *MyExtension) Start(ctx context.Context, host component.Host) error {
    pipelines := host.GetPipelines() 
    // do stuff with pipelines
}

Describe alternatives you've considered another option is modifying zpages to include json version of the /pipelinez and other endpoints

[Aneurysm9]

another option is modifying zpages to include json version of the /pipelinez and other endpoints

This sounds like the way to go if it would meet your needs. zpages is already intended to provide debugging information and it could certainly be useful to provide it in a more easily machine readable format. Would love to see Accept-based content negotiation on those resources.

[abeach-nr]

@Aneurysm9 extensions would be preferable, as I wouldn't need another process to then consume that endpoint. An extension could say, log the DAG representation of the pipelines, or the extension could instrument that DAG information and push to to another server.

[abeach-nr]

There is another option without changing the Host interface.

currently, the Host interface does not implement the RegisterZPages method.

the 'zpagesexention' gets around this because extensions use a host wrapper

https://github.com/open-telemetry/opentelemetry-collector/blob/e0eea95c1a6913741fbfe622ea41dfa33ecf8df2/service/internal/components/host_wrapper.go#L48

that adds the RegisterZPages

and then there is a type cast. https://github.com/open-telemetry/opentelemetry-collector/blob/e0eea95c1a6913741fbfe622ea41dfa33ecf8df2/extension/zpagesextension/zpagesextension.go#L70

I could add GetPipelines to the host wrapper and do the same interface type casting approach the zpagesextension does.

However, this approach seems like it should be avoided: https://github.com/open-telemetry/opentelemetry-collector/blob/e0eea95c1a6913741fbfe622ea41dfa33ecf8df2/service/internal/components/host_wrapper.go#L47

[djaglowski]

What specific information are you trying to discover about the pipelines?

This capability would directly expose a substantial portion of the collector's internals. I think we should avoid this if at all possible and instead provide targeted access for specific use cases.

[abeach-nr]

@djaglowski ultimately I would like to aggregate collector metrics (Prometheus metrics from the collector) on a pipeline level for the purposes of monitoring pipelines within a collector. I want to be able to look at individual component metrics but in the larger context of a pipeline. With the current collector metrics, there is no way to associate a components metrics to its pipeline. Zpages only really provides the structure of the pipeline, which is a start.

[Aneurysm9]

I think that points to a need to augment the metrics (and preferably also log) data with this information. I tried doing this earlier with logs, but because receivers and exporters can be shared across many pipelines I was only able to do so for processors.