New component: Cloudflare Logs Receiver

[dehaansa]

The purpose and use-cases of the new component

This Cloudflare receiver will receive logs sent from the LogPush API to collect logs from the Cloudflare logs aggregation system. A use case of this receiver is to ingest Cloudflare logs into OTLP format or other desired log platform.

Users will be required to set up their LogPush job external to the receiver, and the receiver is really only specialized to LogPush in the format of the data expected.

The LogPull API ended up not matching our needs (https://github.com/open-telemetry/opentelemetry-collector-contrib/issues/17376) and it was decided that we would write a receiver that supported data being pushed from the LogPush API in a manner similar to the API supported for MongoDB Atlas Alerts

Example configuration for the component

cloudflare:
  tls: # TLS is required for the LogPush API. Self-signed certificates are not supported.
    key_file: some_key_file
    cert_file: some_cert_file
  endpoint: 0.0.0.0:12345 # http server endpoint, no default value
  secret: 1234567890abcdef1234567890abcdef # Secret is optional, but is a way to ensure that the requests contain the specified value in a header to ensure requests are valid
  timestamp_field: EdgeStartTimestamp # The default intended support of this receiver is the http_requests dataset, but if a user wishes to send a different dataset they will want to specify a different field to look at for the record's timestamp

Telemetry data types supported

Logs

Is this a vendor-specific component?

• [ ] This is a vendor-specific component
• [ ] If this is a vendor-specific component, I am proposing to contribute this as a representative of the vendor.

Sponsor (optional)

No response

Additional context

No response

[djaglowski]

Can you explain more about the format in which the logs are sent? Will it be possible to save test inputs in json files or other human readable format?

The API says it can deliver payloads of up to 100,000 logs. Is there a configurable maximum?

[dehaansa]

Can you explain more about the format in which the logs are sent? Will it be possible to save test inputs in json files or other human readable format?

The logs come in a gzipped "file" containing JSON logs separated by newline characters. Definitely can be saved & tested.

The API says it can deliver payloads of up to 100,000 logs. Is there a configurable maximum?

Individual LogPush jobs can have their payload maximums defined as documented here. 1,000 to 100,000 appears to be the supported range of maximum payload size. These variables will be configured by users when they configure their job, outside of the receiver itself.

[djaglowski]

Thanks @dehaansa, I'll sponsor this.