Exporter does not recognize log levels or any extra attributes

[kushal-ti]

I'm using the open telemetry filelog receiver to collect logs (since using the otlp log receiver is failing in the current release due to the issue mentioned here https://github.com/GoogleCloudPlatform/opentelemetry-operations-go/issues/435.) The output of the filelog receiver logs, after being passed through the processor in the otel collector, looks like this

{
    "resourceLogs": [
        {
            "resource": {},
            "scopeLogs": [
                {
                    "scope": {},
                    "logRecords": [
                        {
                            "observedTimeUnixNano": "1655957310265130687",
                            "severityText": "INFO",
                            "body": {
                                "stringValue": "Entering healthCheckPost with list of 2 arguments"
                            },
                            "attributes": [
                                {
                                    "key": "gcp.log_name",
                                    "value": {
                                        "stringValue": "otel-collector-pipeline"
                                    }
                                },
                                {
                                    "key": "gcp.arguments",
                                    "value": {
                                        "stringValue": "[{\"type\": \"String\", \"value\": \"kushal\"}, {\"type\": \"CustomBody\", \"value\": {\"member1\":10,\"member2\":100,\"member3\":[10,100]}}]"
                                    }
                                } ... (truncated for brevity)
                            ],
                            "traceId": "e8f84199ef8f4e1779cd21577aaacbc1",
                            "spanId": "ee43d418bc51ca1f"
                        }
                    ]
                }
            ]
        }
    ]
}

However, this same log on the console looks like so

According to the mapping in the logs data model mentioned here https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/logs/data-model.md#google-cloud-logging any extra attributes should be mentioned as gcp. , but gcp.arguments attributes in the above log were dropped. https://github.com/GoogleCloudPlatform/opentelemetry-operations-go/pull/424 talks about non gcp. attributes so i was under the implication that gcp.* attributes should have been retained, in the current release itself.

Additionally, it also does not retain the log severity, (INFO in this case), instead, the log severity I see on gcp console is default Do i set some other property so that log severity is picked up by stackdriver?. Also, do gcp.* attributes only cover specific list of attributes that stackdriver recognises ? and if so, is there any documentation of this list?

For reference, this is my otel config file

extensions:
  health_check:
  pprof:
    endpoint: 0.0.0.0:1777
  zpages:
    endpoint: 0.0.0.0:55679

receivers:
  filelog:
    include: [input_logs.json]
    start_at: end
    operators:
      - type: json_parser
        parse_from: body

  fluentforward:
    endpoint: 0.0.0.0:8000
  otlp:
    protocols:
      grpc:
      http:    
processors:
  batch:
  transform:
    logs:
      queries:
        - set(trace_id, attributes["trace_id"])
        - set(span_id, attributes["span_id"])
        - set(severity_text, attributes["severity"])
        - set(body, attributes["message"])

  attributes:
    # Override the default log name.  `gcp.log_name` takes precedence
    # over the `default_log_name` specified in the exporter.
    actions:
      - key: gcp.log_name
        action: insert
        value: otel-collector-pipeline
      - key: gcp.arguments
        from_attribute: arguments
        action: upsert

exporters:
  logging:
    loglevel: debug
  file:
    path: output_logs.json
  googlecloud:
    retry_on_failure:
      enabled: true
    project: codenation-186008
    log:
      default_log_name: app

service:
  telemetry:
    logs:
      level: debug
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [googlecloud]

    logs:
      receivers: [filelog]
      processors: [batch, transform, attributes]
      exporters: [file, googlecloud]

  extensions: [health_check, pprof, zpages]

[damemi]

Thanks for reporting this @kushal-ti. We did test the exporter using the filelog receiver's severity parser (before the transform processor was available, which I see you are using). Maybe because of that, we worked with some assumptions about the severity that were specific to the parser. I'll look into those and make sure we can parse severityText properly.

In the mean time, would you be able to try using the severity parsing operator to see if that works?

For the attributes, the intent of that wildcard gcp.* for "any additional fields" was to refer to any LogEntry or GCP-specific fields not explicitly called out in the doc. The exporter actually skips over attributes with gcp. when parsing them to labels, because we assume those have been parsed to LogEntry fields elsewhere in the exporter code. We should make this clearer in the docs, and may want to revisit that behavior if it's too aggressive.

The tl;dr is if you have your own attributes, you can just use your own naming style. All OTel attributes that can't be mapped to a GCP field get converted to labels on the entry.

[kushal-ti]

@damemi just an update, apparently just adding severity text isn't enough, you also need to add a severity number, After i modified my otel config file to look like this, i could see the logs turning up in gcp console with the proper severity level

transform:
    logs:
      queries:
        - set(trace_id, attributes["trace_id"])
        - set(span_id, attributes["span_id"])
        - set(severity_text, attributes["severity"])
        - set(severity_number, 9) where attributes["severity"] == "INFO"
        - set(severity_number, 13) where atributes["severity"] == "WARN"
        - set(severity_number, 17) where attributes["severity"] == "ERROR"
        - set(body, attributes["message"])

i will also try using severity parser operator if that leads to a cleaner result and update the outcome in this thread

[kushal-ti]

The tl;dr is if you have your own attributes, you can just use your own naming style. All OTel attributes that can't be mapped to a GCP field get converted to labels on the entry.

I assume this will be the case after the patch made in https://github.com/GoogleCloudPlatform/opentelemetry-operations-go/pull/424. Thanks, this makes things clearer.

I have just one question to ask, according to the patch reference above, all attributes will be inserted as string labels. Normally, when i'm logging directly to console and leaving it up to gcp agent to pick up the logs, fields which are json are parsed as json

for example, if i have a log with the following json structure

{
     "message": "my log message",
     "attributes": {
         "key1": "value1",  
         "key2":"value2"
      }
}

the attributes key is represented as a JSON structure (inside the json payload object in gcp logs console) in GCP logs, which allows me to search and filter based on the values of the inner keys I tried to add my attributes using gcp.* in the hope of replicating this behaviour If my custom attributes are all set as labels, as far as i understand, they will only be set as a map of <string, string>, which would mean the filtering wouldn't work quite as well. for instance the above json would be represented as

   "message": "my log message"
   "label": {
         "attributes": "{\"key1\": \"value1\", \"key2\":\"value2\"}"
    }

Is there any way i can export a particular key of the log as a json rather than a string value

For what it's worth, according to the mapping for the log data model, the json_payload attribute of gcp logs is mapped to the body attribute for otlp logs, but even when i try to encode my entire log message as a JSON string and send it in the body field of the otlp logs, gcp still interprets it as a text_payload. Is there any configuration i can modify so that the exporter considers the body as a json payload and interprets it accordingly

[kushal-ti]

going through the source code i find this line which indicates there is a way to export the log body as a Map type (or a JSON type ?) Any idea how i can set this configuration, so that the type of body is Map instead of string ?

[kushal-ti]

and may want to revisit that behavior if it's too aggressive.

if it counts, currently, i have no way of setting the traceSampled gcp attribute in the right place in logs. Even though trace sampled is on of the keys in LogEntry, setting gcp.trace_sampled in the collector does not translate to the field showing up in logs (exporting the trace_flags attribute in traces also does not do the trick)

[kushal-ti]

Even though trace sampled is on of the keys in LogEntry, setting gcp.trace_sampled in the collector does not translate to the field showing up in logs

A colleague made a pull request to address this issue, here https://github.com/GoogleCloudPlatform/opentelemetry-operations-go/pull/444 but it wasn't accepted, the details seem to indicate that he wasn't allowed to run the e2e tests and other checks.

It's a relatively minor change (only 4 lines of code change and one additional test) and we have verified that building the contrib collector with that change allows us to export trace sampled and have the trace details button operational on gcp console. Please do go through it and accept it if everything is in order

[damemi]

Thanks for the added information, and the PR to address the traceSampled field.

In order for us to accept that PR, your colleague will need to sign the CLA (more info and a link to the CLA is available here, from the cla-bot's comment https://github.com/GoogleCloudPlatform/opentelemetry-operations-go/pull/444#issuecomment-1164841594).

The changes look good to me, so as long as they are able to sign the CLA then I don't see any problem merging it.

[davidgargti20]

I have signed the CLA could you review and accept it now

[damemi]

@kushal-ti sorry for the delayed response, but some updates on the other questions in this thread:

It looks like the severity parsing operator does some translation to automatically set the SeverityNumber if an equivalent SeverityText is parsed. I'm assuming that setting SeverityText directly with the transform processor doesn't do this (which is why you had to add the alias values yourself to get it to work). I opened https://github.com/GoogleCloudPlatform/opentelemetry-operations-go/pull/446 to add our own attempt at translating these generic text values from the data model in cases where SeverityNumber isn't set to match them.

Re: https://github.com/GoogleCloudPlatform/opentelemetry-operations-go/issues/442#issuecomment-1164380818, there isn't anything you can set in the config to force the body type to be interpreted a certain way. This is done by the OTel libraries, and the GCP exporter just passes it to the matching payload field based on how OTel interprets the type. You would likely have to use another operator, or processor, to parse specific keys out of a JSON-encoded string to an actual key in the log body for OTel to determine it is an object and not just a string (otherwise it has no way of knowing). The JSON parser is probably what you want.

To keep this issue from getting too crowded I'm going to close it when #446 merges, but if you would like to continue discussion on the JSON trouble (or find anything new), feel free to open a new issue so we can track it there. Thanks!