Supported json encoded messages in googlecloudpubsub exporter

[brandonbirdj]

Component(s)

exporter/googlecloudpubsub

Is your feature request related to a problem? Please describe.

Many of the out of the box ways to get logs into pubsub from google cloud utilize jsonPayload, and json encoding. One example is when configuring a log sink from Google Cloud Logging.

In order to make it easier to have a drop in replacement it would be very useful to also support json encoded messages. Putting the message body inside a field called jsonPayload could simple be done in the log pipeline of the collector, no exporter changes needed.

Additionally I think there is perhaps an opportunity to make the attributes set a little more dynamic as opposed to the current cloud events ones.

Describe the solution you'd like

It would be useful to perhaps allow a user to supply additonal attributes. I do not presently have a usecase for this, and would also be happy to just not use the cloud events ones if json encoding is provided.

exporters:
  googlecloudpubsub:
    project: abc
    endpoint: mytestendpoint
    user_agent: projects/abc/topics/requests
    additional_attributes:
    - version
    encoding: json

Describe alternatives you've considered

Forking this provider, or converting all other dependent logic presently relying on json encoding are the only alternatives I have considered.

Additional context

I would be happy to help write this, but would like direction on how to handle the additional attributes and what to do with the current ce ones.

[github-actions[bot]]

Pinging code owners:

• exporter/googlecloudpubsub: @alexvanboxel

  See Adding Labels via Comments if you do not have permissions to add labels yourself.

[alexvanboxel]

Please extract the attribute issue in another issue, so the discussion can be decoppeled.

I'm willing to accept contributions for JSON encoding only if its the standard encoded pdata in JSON format provided by the marshallers in the collector itself (this means it MUST NOT be wrapped in an jsonPayload and the encoding needs to be added to the ce-content-encoding field. Ideally, the JSON encoding SHOULD be added to the receiver. The receiver and exporter are a duo.

[brandonbirdj]

Please extract the attribute issue in another issue, so the discussion can be decoppeled.

I'm willing to accept contributions for JSON encoding only if its the standard encoded pdata in JSON format provided by the marshallers in the collector itself (this means it MUST NOT be wrapped in an jsonPayload and the encoding needs to be added to the ce-content-encoding field. Ideally, the JSON encoding SHOULD be added to the receiver. The receiver and exporter are a duo.

@alexvanboxel to clarify what you said you are saying the receiver should not use "raw_json" encoding, but rather detect from the payload? Similarly the exporter will auto detect, so neither will have a config option making it required to set?

Specifically I'm confused by the following comment since the receiver can support raw_json

the JSON encoding SHOULD be added to the receiver. The receiver and exporter are a duo.

Since cloud events supports json perhaps simply allowing auto detection and use of"content-type": "application/cloudevents+json; charset=UTF-8" would be simpler? https://github.com/google/knative-gcp/blob/main/docs/spec/pubsub-protocol-binding.md#22-data-attribute

[github-actions[bot]]

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

Pinging code owners:

• exporter/googlecloudpubsub: @alexvanboxel

See Adding Labels via Comments if you do not have permissions to add labels yourself.