search

open-telemetry / opentelemetry-collector-contrib

Contrib repository for the OpenTelemetry Collector
https://opentelemetry.io
Apache License 2.0
2.75k stars 2.19k forks source link

[receiver/sqlquery] `initial_delay` setting is not supported for logs #29671

Open andrzej-stencel opened 7 months ago

andrzej-stencel commented 7 months ago

Component(s)

receiver/sqlquery

What happened?

Description

The SQL Query receiver supports the initial_delay property (it's not documented in the README), but it only works for metrics and not for logs.

Steps to Reproduce

Run the collector with the below config and observe collector's logs.

Expected Result

Metrics and logs are first scraped after 5 seconds, then at 10 second intervals.

Actual Result

Metrics collection behavior is correct: first scraped after 5 seconds, then at 10 second intervals.

Logs collection behavior is incorrect: the initial_delay of 5s is ignored. Logs are first scraped after 10 seconds and then at 10 second intervals.

Collector version

0.90.1

Environment information

Any

OpenTelemetry Collector configuration

exporters:
  debug:
    verbosity: basic
receivers:
  sqlquery/logs:
    collection_interval: 10s
    datasource: "postgresql://postgres:postgres@localhost:5432/postgres?sslmode=disable"
    driver: postgres
    initial_delay: 5s
    queries:
      - sql: select id, body from simple_logs where id > $$1
        tracking_column: id
        tracking_start_value: 0
        logs:
          - body_column: body
  sqlquery/metrics:
    collection_interval: 10s
    datasource: "postgresql://postgres:postgres@localhost/postgres?sslmode=disable"
    driver: postgres
    initial_delay: 5s
    queries:
      - sql: |-
          select * from my_metrics
        metrics:
          - metric_name: somemetric
            value_column: value
service:
  pipelines:
    logs:
      exporters:
        - debug
      receivers:
        - sqlquery/logs
    metrics:
      exporters:
        - debug
      receivers:
        - sqlquery/metrics

Log output

2023-12-06T12:53:10.396+0100    info    service@v0.90.1/telemetry.go:86 Setting up own telemetry...
2023-12-06T12:53:10.397+0100    info    service@v0.90.1/telemetry.go:203        Serving Prometheus metrics      {"address": ":8888", "level": "Basic"}
2023-12-06T12:53:10.397+0100    info    exporter@v0.90.1/exporter.go:275        Development component. May change in the future.        {"kind": "exporter", "data_type": "logs", "name": "debug"}
2023-12-06T12:53:10.397+0100    info    exporter@v0.90.1/exporter.go:275        Development component. May change in the future.        {"kind": "exporter", "data_type": "metrics", "name": "debug"}
2023-12-06T12:53:10.397+0100    info    receiver@v0.90.1/receiver.go:296        Development component. May change in the future.        {"kind": "receiver", "name": "sqlquery/logs", "data_type": "logs"}
2023-12-06T12:53:10.398+0100    info    service@v0.90.1/service.go:148  Starting otelcol-contrib...     {"Version": "0.90.1", "NumCPU": 16}
2023-12-06T12:53:10.398+0100    info    extensions/extensions.go:34     Starting extensions...
2023-12-06T12:53:10.398+0100    info    service@v0.90.1/service.go:174  Everything is ready. Begin running and processing data.
2023-12-06T12:53:15.410+0100    info    MetricsExporter {"kind": "exporter", "data_type": "metrics", "name": "debug", "resource metrics": 1, "metrics": 2, "data points": 2}
2023-12-06T12:53:20.410+0100    info    LogsExporter    {"kind": "exporter", "data_type": "logs", "name": "debug", "resource logs": 1, "log records": 6}
2023-12-06T12:53:25.400+0100    info    MetricsExporter {"kind": "exporter", "data_type": "metrics", "name": "debug", "resource metrics": 1, "metrics": 2, "data points": 2}
^C2023-12-06T12:53:28.121+0100  info    otelcol@v0.90.1/collector.go:258        Received signal from OS {"signal": "interrupt"}
2023-12-06T12:53:28.121+0100    info    service@v0.90.1/service.go:188  Starting shutdown...
2023-12-06T12:53:28.121+0100    info    extensions/extensions.go:52     Stopping extensions...
2023-12-06T12:53:28.121+0100    info    service@v0.90.1/service.go:202  Shutdown complete.

Additional context

This is due to the initial_delay being a part of the https://github.com/open-telemetry/opentelemetry-collector/tree/main/receiver/scraperhelper that the metrics implementation uses internally. To fix this, The logs receiver implementation would need to explicitly add support for initial_delay, just as it currently supports collection_interval.

github-actions[bot] commented 7 months ago

Pinging code owners:

See Adding Labels via Comments if you do not have permissions to add labels yourself.

crobert-1 commented 7 months ago

I think since logs are considered Development stability and this config option is undocumented, I'm going to label as an enhancement, but I agree this is a good idea to support.

As @astencel-sumo pointed out, the initial_delay option is coming from the scraper helper. The reason this option is available for metrics but not logs is shown when creating the receivers. When creating the metric receiver we're using the scraper helper's receiver, but the logs receiver uses its own internal model of a receiver.

I think the best option is to rely on the existing scraper helper functionality, but code owners (or anyone interested on resolving this) will have to confirm this is possible for logs in this receiver. Adding the initial_delay configuration option to the README should also be done as a part of this change.

andrzej-stencel commented 7 months ago

I think the best option is to rely on the existing scraper helper functionality

The scraperhelper package currently only supports metrics, but perhaps it could be extended to also support logs? 🤔

Adding the initial_delay configuration option to the README should also be done as a part of this change.

💯 agree 👍

crobert-1 commented 7 months ago

The scraperhelper package currently only supports metrics, but perhaps it could be extended to also support logs? 🤔

My bad, I didn't realize it only supports metrics. In that case, since it's a single (and pretty simple) option, it's not a big deal to implement it in this receiver. If there's some other existing library that we can use it'd be great, but by no means required. 👍

github-actions[bot] commented 5 months ago

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

Pinging code owners:

See Adding Labels via Comments if you do not have permissions to add labels yourself.

github-actions[bot] commented 3 months ago

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

Pinging code owners:

See Adding Labels via Comments if you do not have permissions to add labels yourself.

github-actions[bot] commented 1 month ago

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

Pinging code owners:

See Adding Labels via Comments if you do not have permissions to add labels yourself.