trace_id field extraction from logs

ghost commented 1 year ago

Dear Team, We have installed Signoz using helm charts by supplying override-values.yaml in our EKS cluster. After installation -we configured OTEL URL in our apps.

In the Frontend dashboard, we were able to see logs coming in but we were not getting value for trace_id field. I have attached the screenshot where traced_id is showing body{} but not coming in the interested fields.

Can someone please help here with what could be the issue? I have attached my override-values.yaml and screenshot missing trace_id from logs

override-values.yaml :

global: storageClass: gp2 cloud: gcp

clickhouse: installCustomStorageClass: true otel: receiver: otlp: protocols: grpc: endpoint: otel-collector.default.svc.cluster.local:4317 exporter: jaeger: endpoint: http://jaeger-query:16686 logging: loglevel: debug logformat: json processor: batch: timeout: 5s attribute: actions:

key: trace_id action: upsert from_attribute: log value_type: string otlp: protocols: grpc: parser: json: timestamp: field: "time" format: "RFC3339Nano" attributes:
- key: log field: "message"
- key: trace_id field: "trace_id"

signoz

nityanandagohain commented 1 year ago

You will have to use the trace parser for this https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/pkg/stanza/docs/operators/trace_parser.md

Here is an example https://github.com/SigNoz/logs-benchmark/blob/0b2451e6108d8fa5fdd5808c4e174bd52b9d55d3/signoz/signoz-client/otel-collector-config.yaml#L22

charanparvathaneni commented 1 year ago

@nityanandagohain We are using below override-values.yaml but still not seeing the trace_id value in the trace_id field. Please help

global: storageClass: gp2 cloud: gcp

clickhouse: installCustomStorageClass: true

receivers: tcplog: listen_address: "0.0.0.0:2255" attributes: {} resource: host: "mys" add_attributes: false operators:

type: regex_parser regex: '^.(?P{.})' output: parsedata
type: json_parser id: parsedata parse_from: attributes.body
type: move from: attributes["dummyBytes"] to: body
type: time_parser parse_from: attributes.datetime layout_type: strptime layout: '%d/%b/%Y:%H:%M:%S %z'
type: trace_parser trace_id: parse_from: attributes.traceId span_id: parse_from: attributes.spanId
type: remove field: attributes.body
type: remove field: attributes.datetime
type: remove field: attributes.traceId
type: remove field: attributes.spanId
type: move from: attributes["user-identifier"] to: attributes.user_identifier

processors: batch: send_batch_size: 30000 send_batch_max_size: 31000 timeout: 10s

extensions: health_check: endpoint: 0.0.0.0:13133 zpages: endpoint: 0.0.0.0:55679 pprof: endpoint: 0.0.0.0:1777

exporters: logging: loglevel: debug log_attributes:

trace_id
span_id
body otlp/signoz: endpoint: 172.31.1.23:4317 tls: insecure: true

service: telemetry: metrics: address: 0.0.0.0:8888 logs: level: "debug" extensions:

health_check
zpages
pprof pipelines: logs: receivers: [tcplog] processors: [batch] exporters: [otlp/signoz]

nityanandagohain commented 1 year ago

@charanparvathaneni what is the format of logs you are sending, the above configuration is for parsing logs of this format.

{
  "host": "6.127.39.239",
  "user-identifier": "wisoky5015",
  "datetime": "09/Nov/2022:04:48:14 +0000",
  "method": "HEAD",
  "request": "/revolutionary/roi",
  "protocol": "HTTP/2.0",
  "status": 501,
  "bytes": 11466,
  "referer": "http://www.seniorrevolutionize.name/intuitive/matrix/24/7",
  "traceId": "3aee4c94836ffe064b37f054dc0e1c50",
  "spanId": "4b37f054dc0e1c50",
  "dummyBytes": "kingship grumble swearer hording wiggling dipper execution's crock's clobbers Spaniard's priestess's organises Harrods's Josef's Wilma meditating movable's photographers polestars pectorals Coventries rehearsal Arkhangelsk's Kasai barometer junkier Osgood's impassively illogical cardsharp's underbrush shorter patronymics plagiarises gargle's Chandra intransigent deathtrap extraneously hairless inferred glads frail polka jeez ohm bigoted sari's cannonades vestibule's pursuer vanity ceremony Kodaly's swaths disturbance's belt Samoan's invulnerability dumping Alfreda padded corrosive universal dishwasher's multiplier decisive selloff eastbound moods Konrad's appositive's reset's ingenuously checkup's overselling evens Darrin's supernumerary liberalism productivity's legrooms Lorenzo including Robbin savourier foxglove's buckshot businesswomen abalones tare Chaitin Stephenson's inpatients distinction cryings aspic empire's healed perspiring"
}

charanparvathaneni commented 1 year ago

@nityanandagohain our application-sample log looks like below.Please help .

######################## { "hostname": "account-api-c695c7cd7-pvk9n", "level": "info", "message": "Response /api/account/users/me", "method": "GET", "responseBody": { "_id": "64636f00ab5974d2945c7c4b", "authUserId": "56e989ef-96e0-409f-a167-5a3fb1ac1362", "createdAt": "Tue May 16 2023 11:54:40 GMT+0000 (Coordinated Universal Time)", "createdBy": "9570e51d-720b-4195-af89-be611aca2c9d", "email": "doctor104@mailinator.com", "firstName": "Sudheer", "lastName": "Sudheer", "phoneNumber": { "dialCode": "+91", "value": 999999999 }, "status": "Invited", "tenant": [ { "status": "InActive", "tenantId": "6463609cf72b9f2b0b54cabc" } ], "updatedAt": "Tue May 16 2023 11:54:40 GMT+0000 (Coordinated Universal Time)" }, "span_id": "46a992e6f848706f", "timestamp": "2023-05-18T05:57:20.791Z", "trace_flags": "01", "trace_id": "8c67114211b24caf694625e933175897", "type": "RESPONSE", "url": "/api/account/users/me" }

SigNoz / charts

trace_id field extraction from logs #223

override-values.yaml :