RemoteTap Extension: a new start

wildum commented 1 month ago

Component(s)

extension/remotetap

Is your feature request related to a problem? Please describe.

The RemoteTap extension was abandoned because the first implementation was not merged. The extension has been left in a skeleton state since its beginning 9 months ago.

The big advantage of this component compared to the remotetap processor is that the user does not need to modify the pipeline to see the data at any stages.

In Alloy (Grafana's Opentelemetry collector distribution), we implemented a similar feature called live debugging. Observability is not an easy topic and we believe this feature is a big step in making it more accessible.

Describe the solution you'd like

The extension could maintain a registry where components can register themselves. Additionally, it could expose an endpoint that allows users to select from these registered components.

Processors would register on start to the extension and publish data to it after every processing (some could publish the data also before if relevant). Receivers could also register to it and publish data before sending it to the next consumer.

Components would only publish data when a remotetap stream is open to prevent any unnecessary computing

Describe alternatives you've considered

No response

Additional context

I would be happy to use my experience with live debugging in Alloy to contribute to the implementation of this feature.

@atoulme

github-actions[bot] commented 1 month ago

Pinging code owners:

extension/remotetap: @atoulme

See Adding Labels via Comments if you do not have permissions to add labels yourself.

atoulme commented 1 month ago

The extension actually requires the remotetap processor to be deployed, and lists them in its configuration.

Whatever you can bring forward that helps is most welcome. Please feel free to expand on your proposal, given the component model of the collector. In particular, can you please describe the config yaml for this development?

wildum commented 1 month ago

I would like the extension not to depend on the remotetap processor with the idea to deprecate the remotetap processor once the extension is mature enough.

The config for the extension would only contain "confighttp.ServerConfig `mapstructure:",squash"" and would look like this in yaml:

extensions:
  remotetap:
    endpoint: localhost:12001

If you need to modify the config of a faulty collector to debug it, you run the risk to break it completely or to make the problem go away by reloading the config. Having the possibility to remote tap any processor/receiver of a running collector without any disruptions is a massive +.

I believe that with the following design, users might be happy to keep the extension even in prod environments because it's not directly part of their pipelines and it does not impact performances:

On start:

remote tap extension starts the server
processors/receivers that support the feature register on start to the extensions via host.GetExtensions()["remotetap"].Register(componentID)

Let's say that the user wants to remote tap the component "metricsgeneration" processor:

the user sends the request localhost:12001/metricsgeneration via the UI
the remote tap extension receives the request, keeps the connection open and tracks it in a map
the metricsgeneration processor checks after every processing if a connection is open via host.GetExtensions()["remotetap"].IsActive(componentID)
if a connection is active, the processor will publish the metrics to the extension via host.GetExtensions()["remotetap"].PublishMetrics(componentID, md Metrics)
the extension will marshall the metrics into text and send the data over the open connection to the front-end

The interface for the components to interact with the extension would be the following:

type RemoteTapPublisher interface {
        // Register the component to the RemoteTap extension
        Register(componentID)
    // IsActive returns true when at least one connection is open for the given componentID.
    IsActive(componentID ComponentID) bool
    // PublishMetrics sends metrics for a given componentID to the RemoteTap extension.
    PublishMetrics(componentID ComponentID, md pmetric.Metrics)
    // PublishTraces sends traces for a given componentID to the RemoteTap extension.
    PublishTraces(componentID ComponentID, td ptrace.Traces)
    // PublishLogs sends logs for a given componentID to the RemoteTap extension.
    PublishLogs(componentID ComponentID, ld plog.Logs)
    // PublishData sends data for a given componentID to the RemoteTap extension.
    PublishData(componentID ComponentID, data string)
}

The UI should contain some basic controls to make debugging easier:

sample rate to handle heavy loads
filter
pause/resume the stream
clear the page

What do you think? I could try a POC with the support for one processor and a very basic UI. If people are happy with it we could gradually extend it to more components and improve the UI.

jaronoff97 commented 4 weeks ago

@wildum I'd be happy to assist in reviewing and working on this. I am currently using the remotetap processor for exactly this:

sample rate to handle heavy loads
filter
pause/resume the stream
clear the page

In my tails project

djaglowski commented 4 weeks ago

In my opinion the ideal solution for this would be more deeply integrated into the collector so that individual component developers do not need to be concerned with managing it, and so that performance and correctness concerns are handled in a uniform way.

Roughly the following:

When the user has not enabled data tapping, there is no additional work done within the collector vs today.
When data is tapped, it respects consumer.Capabilities to make copies as appropriate.
Receiver outputs, processor inputs and outputs, and exporter outputs may be tapped.
Tapped data passes through an interface which may eventually have multiple implementations (e.g. as local web service, OpAMP, any exporter)

I would also point out that if we ever land https://github.com/open-telemetry/opentelemetry-collector/issues/9077, then this becomes a trivial problem where the solution is just adding one more exporter that subscribes to all data producers.

wildum commented 3 days ago

Hey @atoulme @djaglowski @jaronoff97, following the SIG meeting (and a week off), I worked on a 2nd POC. As discussed, this time I implemented this concept in the core repository using the processorhelper pkg. You can find the new POC here: https://github.com/open-telemetry/opentelemetry-collector/pull/10962 And a clean lightweight version of the concept here: https://github.com/open-telemetry/opentelemetry-collector/pull/10963

Please have a look at the concept branch when you have time and let me know what you think :)

wildum commented 2 days ago

Following @atoulme's comment, I moved the component back to contrib and kept only the changes related to the processor helper in core. @djaglowski @jaronoff97 Here are the new relevant links:

core: processor helper changes
contrib: remotetap extension
POC in core: https://github.com/open-telemetry/opentelemetry-collector/pull/10962

open-telemetry / opentelemetry-collector-contrib