search

open-telemetry / opentelemetry-collector-contrib

Contrib repository for the OpenTelemetry Collector
https://opentelemetry.io
Apache License 2.0
3.02k stars 2.33k forks source link

Snowflake receiver doesn't seem to be collecting storage data #29750

Closed luthes closed 10 months ago

luthes commented 10 months ago

Component(s)

receiver/snowflake

What happened?

Description

When using the Snowflake receiver, I noticed that we weren't getting the storage metrics I'd expect: snowflake.storage.stage_bytes.total, snowflake.storage.storage_bytes.total or snowflake.storage.failsafe_bytes.total. It does appear that the query for this is running, as I can see the query in the query history. I don't, however, see this getting sent to my OTLP endpoint (New Relic, in this case), or even logged out using the debug exporter.

All other metrics are received, including optional metrics.

Steps to Reproduce

Docker container with config for Snowflake connector and

Expected Result

I should see logs in the debug exporter that it's finding metrics that it's querying data, or at least

Actual Result

No logs, no errors, no data in otlp endpoint.

Collector version

0.90.1

Environment information

Official Docker Container

OpenTelemetry Collector configuration

receivers:
  snowflake:
    username: ${SNOWFLAKE_USERNAME}
    password: ${SNOWFLAKE_PASSWORD}
    account: my-snowflake-account
    warehouse: my-warehouse
    collection_interval: 1m
    metrics:
      [...]
      # All other metrics are explicitly disabled, including default metrics.
      snowflake.storage.stage_bytes.total:
        enabled: true
      snowflake.storage.storage_bytes.total:
        enabled: true
      snowflake.storage.failsafe_bytes.total:
        enabled: true
     [...]
processors:
  batch:
exporters:
  debug:
    verbosity: detailed
    sampling_initial: 5
    sampling_thereafter: 200
  otlp:
    # New Relic Config
    [...]

service:
  telemetry:
    logs:
      level: "debug"
  pipelines:
    metrics:
      receivers:
        - snowflake
      processors: [batch]
      exporters: [otlp, debug]

Log output

snowflake-otel-collector  | 2023-12-12T00:08:07.346Z    info    service@v0.90.1/telemetry.go:86 Setting up own telemetry...           
snowflake-otel-collector  | 2023-12-12T00:08:07.346Z    info    service@v0.90.1/telemetry.go:203        Serving Prometheus metrics   {"address": ":8888", "level": "Basic"}                                                                                                 
snowflake-otel-collector  | 2023-12-12T00:08:07.346Z    debug   extension@v0.90.1/extension.go:162      Beta component. May change in the future.     {"kind": "extension", "name": "zpages"}                                                                               
snowflake-otel-collector  | 2023-12-12T00:08:07.346Z    debug   extension@v0.90.1/extension.go:162      Beta component. May change in the future.     {"kind": "extension", "name": "health_check"}                                                                         
snowflake-otel-collector  | 2023-12-12T00:08:07.346Z    debug   exporter@v0.90.1/exporter.go:273        Stable component.       {"kind": "exporter", "data_type": "metrics", "name": "otlp"}                                                                                
snowflake-otel-collector  | 2023-12-12T00:08:07.346Z    info    exporter@v0.90.1/exporter.go:275        Development component. May change in the future.      {"kind": "exporter", "data_type": "metrics", "name": "debug"}                                                 
snowflake-otel-collector  | 2023-12-12T00:08:07.346Z    debug   processor@v0.90.1/processor.go:287      Stable component.       {"kind ": "processor", "name": "batch", "pipeline": "metrics"}                                                                               
snowflake-otel-collector  | 2023-12-12T00:08:07.346Z    debug   receiver@v0.90.1/receiver.go:294        Alpha component. May change in the future.    {"kind": "receiver", "name": "snowflake", "data_type": "metrics"}                              snowflake-otel-collector  | 2023-12-12T00:08:07.346Z    debug   receiver@v0.90.1/receiver.go:294        Alpha component. May change in the future.    {"kind": "receiver", "name": "snowflake", "data_type": "metrics"}                                snowflake-otel-collector  | 2023-12-12T00:08:07.347Z    info    service@v0.90.1/service.go:148  Starting otelcol-contrib...     {"Version": "0.90.1", "NumCPU": 20}                                                                                                         snowflake-otel-collector  | 2023-12-12T00:08:07.347Z    info    extensions/extensions.go:34     Starting extensions...
snowflake-otel-collector  | 2023-12-12T00:08:07.347Z    info    extensions/extensions.go:37     Extension is starting...        {"kind": "extension", "name": "health_check"}                                                                                               
snowflake-otel-collector  | 2023-12-12T00:08:07.347Z    info    healthcheckextension@v0.90.1/healthcheckextension.go:35 Starting health_check extension       {"kind": "extension", "name": "health_check", "config": {"Endpoint":"0.0.0.0:13133","TLSSetting":null,"CORS":null,"Auth":null,"MaxRequestBodySize":0,"IncludeMetadata":false,"ResponseHeaders":null,"Path":"/status/check","ResponseBody":null,"CheckCollectorPipeline":{"Enabled":false,"Interval":"5m","ExporterFailureThreshold":5}}}
snowflake-otel-collector  | 2023-12-12T00:08:07.347Z    warn    internal@v0.90.1/warning.go:40  Using the 0.0.0.0 address exposes this server to every network interface, which may facilitate Denial of Service attacks      {"kind": "extension", "name": "health_check", "documentation": "https://github.com/open-telemetry/opentelemetry-collector/blob/main/docs/security-best-practices.md#safeguards-against-denial-of-service-attacks"}
snowflake-otel-collector  | 2023-12-12T00:08:07.347Z    info    extensions/extensions.go:45     Extension started.      {"kind": "extension", "name": "health_check"}
snowflake-otel-collector  | 2023-12-12T00:08:07.347Z    info    extensions/extensions.go:37     Extension is starting...        {"kind": "extension", "name": "zpages"} 
snowflake-otel-collector  | 2023-12-12T00:08:07.347Z    info    zpagesextension@v0.90.1/zpagesextension.go:53   Registered zPages span processor on tracer provider   {"kind": "extension", "name": "zpages"}
snowflake-otel-collector  | 2023-12-12T00:08:07.347Z    info    zpagesextension@v0.90.1/zpagesextension.go:63   Registered Host's zPages      {"kind": "extension", "name": "zpages"}
snowflake-otel-collector  | 2023-12-12T00:08:07.347Z    info    zpagesextension@v0.90.1/zpagesextension.go:75   Starting zPages extension     {"kind": "extension", "name": "zpages", "config": {"TCPAddr":{"Endpoint":":55679"}}}
snowflake-otel-collector  | 2023-12-12T00:08:07.347Z    info    extensions/extensions.go:45     Extension started.      {"kind": "extension", "name": "zpages"}
snowflake-otel-collector  | 2023-12-12T00:08:07.347Z    info    zapgrpc/zapgrpc.go:178  [core] [Channel #1] Channel created     {"grpc_log": true}
snowflake-otel-collector  | 2023-12-12T00:08:07.347Z    info    zapgrpc/zapgrpc.go:178  [core] [Channel #1] original dial target is: "otlp.nr-data.net:4318"  {"grpc_log": true}
snowflake-otel-collector  | 2023-12-12T00:08:07.347Z    info    zapgrpc/zapgrpc.go:178  [core] [Channel #1] parsed dial target is: {URL:{Scheme:otlp.nr-data.net Opaque:4318 User: Host: Path: RawPath: OmitHost:false ForceQuery:false RawQuery: Fragment: RawFragment:}}  {"grpc_log": true}
snowflake-otel-collector  | 2023-12-12T00:08:07.347Z    info    zapgrpc/zapgrpc.go:178  [core] [Channel #1] fallback to scheme "passthrough"  {"grpc_log": true}
snowflake-otel-collector  | 2023-12-12T00:08:07.347Z    info    zapgrpc/zapgrpc.go:178  [core] [Channel #1] parsed dial target is: {URL:{Scheme:passthrough Opaque: User: Host: Path:/otlp.nr-data.net:4318 RawPath: OmitHost:false ForceQuery:false RawQuery: Fragment: RawFragment:}}     {"grpc_log": true}
snowflake-otel-collector  | 2023-12-12T00:08:07.347Z    info    zapgrpc/zapgrpc.go:178  [core] [Channel #1] Channel authority set to "otlp.nr-data.net:4318"  {"grpc_log": true}
snowflake-otel-collector  | 2023-12-12T00:08:07.347Z    info    zapgrpc/zapgrpc.go:178  [core] [Channel #1] Resolver state updated: {
snowflake-otel-collector  |   "Addresses": [
snowflake-otel-collector  |     {
snowflake-otel-collector  |       "Addr": "otlp.nr-data.net:4318",
snowflake-otel-collector  |       "ServerName": "",
snowflake-otel-collector  |       "Attributes": null,
snowflake-otel-collector  |       "BalancerAttributes": null,
snowflake-otel-collector  |       "Metadata": null
snowflake-otel-collector  |     }
snowflake-otel-collector  |   ],
snowflake-otel-collector  |   "Endpoints": [
snowflake-otel-collector  |     {
snowflake-otel-collector  |       "Addresses": [
snowflake-otel-collector  |         {
snowflake-otel-collector  |           "Addr": "otlp.nr-data.net:4318",
snowflake-otel-collector  |           "ServerName": "",
snowflake-otel-collector  |           "Attributes": null,
snowflake-otel-collector  |           "BalancerAttributes": null,
snowflake-otel-collector  |           "Metadata": null
snowflake-otel-collector  |         }
snowflake-otel-collector  |       ],
snowflake-otel-collector  |       "Attributes": null
snowflake-otel-collector  |     }
snowflake-otel-collector  |   ],
snowflake-otel-collector  |   "ServiceConfig": null,
snowflake-otel-collector  |   "Attributes": null
snowflake-otel-collector  | } (resolver returned new addresses) {"grpc_log": true}
snowflake-otel-collector  | 2023-12-12T00:08:07.347Z    info    zapgrpc/zapgrpc.go:178  [core] [Channel #1] Channel switches to new LB policy "pick_first"    {"grpc_log": true}
snowflake-otel-collector  | 2023-12-12T00:08:07.347Z    info    zapgrpc/zapgrpc.go:178  [core] [pick-first-lb 0xc002d81a10] Received new config {
snowflake-otel-collector  |   "shuffleAddressList": false
snowflake-otel-collector  | }, resolver state {
snowflake-otel-collector  |   "Addresses": [
snowflake-otel-collector  |     {
snowflake-otel-collector  |       "Addr": "otlp.nr-data.net:4318",
snowflake-otel-collector  |       "ServerName": "",
snowflake-otel-collector  |       "Attributes": null,
snowflake-otel-collector  |       "BalancerAttributes": null,
snowflake-otel-collector  |       "Metadata": null
snowflake-otel-collector  |     }
snowflake-otel-collector  |   ],
snowflake-otel-collector  |   "Endpoints": [
snowflake-otel-collector  |     {
snowflake-otel-collector  |       "Addresses": [
snowflake-otel-collector  |         {
snowflake-otel-collector  |           "Addr": "otlp.nr-data.net:4318",
snowflake-otel-collector  |           "ServerName": "",
snowflake-otel-collector  |           "Attributes": null,
snowflake-otel-collector  |           "BalancerAttributes": null,
snowflake-otel-collector  |           "Metadata": null
snowflake-otel-collector  |         }
snowflake-otel-collector  |       ],
snowflake-otel-collector  |       "Attributes": null
snowflake-otel-collector  |     }
snowflake-otel-collector  |   ],
snowflake-otel-collector  |   "ServiceConfig": null,
snowflake-otel-collector  |   "Attributes": null
snowflake-otel-collector  | }   {"grpc_log": true}
snowflake-otel-collector  | 2023-12-12T00:08:07.347Z    info    zapgrpc/zapgrpc.go:178  [core] [Channel #1 SubChannel #2] Subchannel created  {"grpc_log": true}
snowflake-otel-collector  | 2023-12-12T00:08:07.347Z    info    zapgrpc/zapgrpc.go:178  [core] [Channel #1] Channel Connectivity change to CONNECTING {"grpc_log": true}
snowflake-otel-collector  | 2023-12-12T00:08:07.347Z    info    zapgrpc/zapgrpc.go:178  [core] [Channel #1 SubChannel #2] Subchannel Connectivity change to CONNECTING        {"grpc_log": true}
snowflake-otel-collector  | 2023-12-12T00:08:07.347Z    info    zapgrpc/zapgrpc.go:178  [core] [Channel #1 SubChannel #2] Subchannel picks a new address "otlp.nr-data.net:4318" to connect   {"grpc_log": true}
snowflake-otel-collector  | 2023-12-12T00:08:07.347Z    info    zapgrpc/zapgrpc.go:178  [core] [pick-first-lb 0xc002d81a10] Received SubConn state update: 0xc002d81bc0, {ConnectivityState:CONNECTING ConnectionError:<nil>} {"grpc_log": true}
snowflake-otel-collector  | 2023-12-12T00:08:07.347Z    info    healthcheck/handler.go:132      Health Check state change       {"kind": "extension", "name": "health_check", "status": "ready"}
snowflake-otel-collector  | 2023-12-12T00:08:07.347Z    info    service@v0.90.1/service.go:174  Everything is ready. Begin running and processing data.
snowflake-otel-collector  | 2023-12-12T00:08:07.507Z    info    zapgrpc/zapgrpc.go:178  [core] [Channel #1 SubChannel #2] Subchannel Connectivity change to READY     {"grpc_log": true}
snowflake-otel-collector  | 2023-12-12T00:08:07.508Z    info    zapgrpc/zapgrpc.go:178  [core] [pick-first-lb 0xc002d81a10] Received SubConn state update: 0xc002d81bc0, {ConnectivityState:READY ConnectionError:<nil>}      {"grpc_log": true}
snowflake-otel-collector  | 2023-12-12T00:08:07.508Z    info    zapgrpc/zapgrpc.go:178  [core] [Channel #1] Channel Connectivity change to READY      {"grpc_log": true}

Additional context

I'm not sure if it matters, but the type in the client.go is of type int64, but it looks like the type returned from Snowflake might be of type float64 like some of the other types.

For example, this query (from the client.go):

storageMetricsQuery          = "select STORAGE_BYTES, STAGE_BYTES, FAILSAFE_BYTES from STORAGE_USAGE ORDER BY USAGE_DATE DESC LIMIT 1;"

returns the following:

STORAGE_BYTES                   STAGE_BYTES         FAILSAFE_BYTES
123412341234123.000000          123412341234.000000 12341234123412.000000

But frankly, I'm not sure if there is some type conversion happening somewhere that would impact this, or if it matters at all.

Just to be clear, these are the only three metrics that don't work. If I explicity enable all other metrics, or allow defaults with empty metric:, I see all of the other data points reflected in the OTLP endpoint.

github-actions[bot] commented 10 months ago

Pinging code owners:

See Adding Labels via Comments if you do not have permissions to add labels yourself.

shalper2 commented 10 months ago

@luthes hey! Thanks for opening the issue, I was able to reproduce using your config. Your intuition was right, there is a typing issue with the rows.Scan() (from database/sql) used in client.go.