New component: RabbitMQ (message) Receiver

frzifus commented 3 months ago

The purpose and use-cases of the new component

Since version v0.104.0 the opentelemetry collector includes a RabbitMQ exporter. While a rabbitmq receiver exists, it is only used to retrieve node metadata.

With the enhanced ability to read from the queue with an OpenTelemetry Collector, it can be used to reroute, manipulate, or further process traffic. https://github.com/open-telemetry/opentelemetry-collector-contrib/issues/28891#issuecomment-2103770417

Example configuration for the component

Minimal receiver options required to ensure feature parity with exporter

receivers:
  rabbitmq:
    connection:
      endpoint: amqp://localhost:5672 # (required) The RabbitMQ endpoint to connect to.
      vhost # (optional) The RabbitMQ virtual host to connect to.
      tls: # (optional): TLS configuration.
        insecure: true
        ...
      auth:
        plain: # (optional) Configuration if using SASL PLAIN authentication 
          username: user
          password: pass
    queue:
      name: otlp_spans
    encoding_extension: otlp_encoding/rabbitmq #  (required) ID of the encoding/decoding extension used to unmarshal data.

extensions:
  otlp_encoding/rabbitmq:
    protocol: otlp_json

Telemetry data types supported

Logs, metrics, and traces

Is this a vendor-specific component?

[ ] This is a vendor-specific component
[ ] If this is a vendor-specific component, I am a member of the OpenTelemetry organization.
[ ] If this is a vendor-specific component, I am proposing to contribute and support it as a representative of the vendor.

Code Owner(s)

@frzifus

Sponsor (optional)

No response

Additional context

Since an exporter and receiver already exist, does it make sense to add the desired functionality to the receiver?

Perhaps someone has an alternative name suggestion which is better then RabbitMQDataReceiver ? :sweat_smile:

cc @atoulme @swar8080 @djaglowski @cpheps

frzifus commented 3 months ago

Yesterday I got a poc up and running. It uses the existing retry logic of the rabbitmqexporter.

To avoid hard to maintain copy pasta we may want to move this logic into e.g. internal/rabbitmq so that it can be reused by a potential receiver.

Config

receivers:
  rabbitmqdata:
    encoding_extension: otlp_encoding/rabbitmq
    connection:
      vhost: "/"
      endpoint: amqp://localhost:5672
      auth:
        plain:
          username: user
          password: password
      queue:
        - name: otlp_metrics
          consumer: col1
          exclusive: true

exporters:
  debug:

extensions:
  otlp_encoding/rabbitmq:
    protocol: otlp_json

service:
  extensions: [otlp_encoding/rabbitmq]
  pipelines:
    metrics:
      receivers: [rabbitmqdata]
      exporters: [debug]

Output

swar8080 commented 3 months ago

Kind of a mouthful but rabbitmqmessagereciever is another option for the component id to help distinguish it from rabbitmqreceiver

andrzej-stencel commented 1 month ago

Kind of a mouthful but rabbitmqmessagereciever is another option for the component id to help distinguish it from rabbitmqreceiver

Good point about the naming, we need to help users distinguish collecting messages from RabbitMQ queues from collecting metrics about RabbitMQ itself.

Another option: rename the current RabbitMQ receiver to RabbitMQ Metrics receiver rabbitmq_metrics. This is what Kafka receiver and Kafka Metrics receiver do.

I agree that a RabbitMQ (messages) receiver makes total sense, especially that Contrib already has the exporter. Unfortunately I'm not able to sponsor this component formally as I'm worried I wouldn't have the capacity for code ownership that comes with sponsoring.

open-telemetry / opentelemetry-collector-contrib