search

GoogleCloudPlatform / ops-agent

Apache License 2.0
139 stars 67 forks source link

Ops Agent does not support traceSampled field mapping #1700

Open PawelKGalka opened 3 months ago

PawelKGalka commented 3 months ago

Describe the bug When trying to set jsonPayload."logging.googleapis.com/trace_sampled" it is not displayed as mapped in Logs Explorer and remains in jsonPayload

To Reproduce Steps to reproduce the behavior:

  1. Start a GCE VM with image RHEL8
  2. Install Ops Agent version 2.46.1
  3. Customize the agent with configuration 
    logging:
receivers:
messages:
  type: files
  include_paths:
  - /var/log/messages
secure:
  type: files
  include_paths:
  - /var/log/secure
applogs_tcp:
  type: tcp
  format: json
  listen_port: 5160
processors:
parse:
  type: parse_json
  time_key: jsonPayload.timestamp
  time_format: "%Y-%m-%dT%H:%M:%S.%L%Z"
adjust_fields_to_stackdriver:
  type: modify_fields
  fields:
    jsonPayload."logging.googleapis.com/trace_sampled":
      default_value: false
service:
pipelines:
  default_pipeline:
    receivers: [messages, secure]
  applogs_pipeline:
    receivers: [applogs_tcp]
    processors: [parse, adjust_fields_to_stackdriver]

    and restart the agent

  4. See log entry does not have traceSampled property and it remains in jsonPayload as jsonPayload."logging.googleapis.com/trace_sampled": "false"

Expected behavior Value mapped in logs explorer

Environment (please complete the following information):

braydonk commented 3 months ago

Hi @PawelKGalka,

Can you try doing logging.googleapis.com/traceSampled (instead of trace_sampled) and see if that works?

PawelKGalka commented 3 months ago

hi @braydonk No luck with above, I was referring to Google documentation on structured logging: https://cloud.google.com/logging/docs/structured-logging

We were facing similar issue with logging.googleapis.com/spanId field in same manner but I got resolved I think with some release of ops agent so potentially this needs similar development. Also we are unable to send it as boolean in jsonPayload to map so would it work if we use map_values mapping?

map_values:
  "true" : true
  "false": false
jefferbrecht commented 3 months ago

Hi @PawelKGalka, just to learn a bit more about your use case, would it be acceptable if the Ops Agent were to set traceSampled automatically based on whether there's a spanId and trace present?

We've had a few other folks ask for ability to set traceSampled manually as well, and it seems like all use cases so far would benefit from something automatic instead so that you don't need to configure anything.

PawelKGalka commented 3 months ago

Hi @jefferbrecht,

Our application is utilizing tracing in 3 modes based on particular criterias - DROP,RECORD,RECORD_AND_SAMPLE (from OTEL lib). This means there are cases where we trace to correlate transaction in the logs but we do not sample it to Cloud Trace. So for our use case it would be best to have this configurable in the config. Or it can take precedence over default setting when spanId/traceId is set