search

open-telemetry / opentelemetry-collector-releases

OpenTelemetry Collector Official Releases
https://opentelemetry.io
Apache License 2.0
252 stars 162 forks source link

Create a PaaS distribution #459

Open atoulme opened 10 months ago

atoulme commented 10 months ago

This is a proposal for a use case related to the collector, alongside a select configuration and selection of components.

Heroku and other similar Platform-as-a-Service solutions allow to run the collector as a sidecar.

They have significant limitations in the size of the binary allowed to run, and limited use cases around data collection.

We currently maintain a Heroku buildpack project that could use this distribution moving forward: https://github.com/signalfx/splunk-otel-collector-heroku

As you can see in the config file used, the distribution uses a few select components.

Here is a sample manifest for it:

dist:
  module: github.com/open-telemetry/opentelemetry-collector-releases/paas
  name: otelcol-paas
  description: OpenTelemetry Collector PaaS
  version: 0.92.0
  output_path: ./_build
  otelcol_version: 0.92.0

receivers:
  - gomod: go.opentelemetry.io/collector/receiver/otlpreceiver v0.92.0
  - gomod: github.com/open-telemetry/opentelemetry-collector-contrib/receiver/prometheusreceiver v0.92.0
  - gomod: github.com/open-telemetry/opentelemetry-collector-contrib/receiver/zipkinreceiver v0.92.0

exporters:
  - gomod: go.opentelemetry.io/collector/exporter/debugexporter v0.92.0
  - gomod: go.opentelemetry.io/collector/exporter/otlpexporter v0.92.0
  - gomod: go.opentelemetry.io/collector/exporter/otlphttpexporter v0.92.0

extensions:
  - gomod: go.opentelemetry.io/collector/extension/zpagesextension v0.92.0
  - gomod: github.com/open-telemetry/opentelemetry-collector-contrib/extension/healthcheckextension v0.92.0
  - gomod: github.com/open-telemetry/opentelemetry-collector-contrib/extension/pprofextension v0.92.0

processors:
  - gomod: go.opentelemetry.io/collector/processor/batchprocessor v0.92.0
  - gomod: github.com/open-telemetry/opentelemetry-collector-contrib/processor/transformprocessor v0.92.0
SaschaBrechmannVHV commented 10 months ago

Hi @atoulme , nice proposal . i like this simple way. Some thing to think about: We use the attributes extension to add some extra Key to the SPANs:

  attributes/insert_otel_receiver:
    actions:
      - key: 'otel_receiver'
        value: '${ env:HOSTNAME }'
        action: insert

We do not use K8s or container. But we have many collectors in many different Security-Zones and this helps a lot for Error-Debugging/Searching. So one could see the collector, with is used.

Regards, Sascha

Update

As i see in the link for example config file, you all ready using this.. The Attributes-Prozessor was not in the List. Is it included with the transform-processor or is it missing?

atoulme commented 10 months ago

Yes, I swapped it for the transformprocessor, which I think might be more versatile in the future. We can add attributes, resource, span processors if we want, though transformprocessor will do the job.

SaschaBrechmannVHV commented 10 months ago

If i use the transformprocessor a config would look like this?

transform:
  error_mode: ignore
  trace_statements:
    - context: span
      statements:
        - set(attributes["otel_receiver"], "${ env:HOSTNAME }")
jpkrohling commented 10 months ago

I like this idea, but I wonder if we couldn't/shouldn't reduce the number of components even further: Jaeger can send OTLP for quite some time now, so, I'd start the distribution without Jaeger. Logging should also be replaced by Debug, and the we could be having OTLP-only exporters (in addition to debug, that is).

atoulme commented 10 months ago

I have reflected your recommendations in the initial comment.

mx-psi commented 10 months ago

@atoulme What's the diff with the core distro?

atoulme commented 10 months ago

Here is the diff:

2,4c2,4
<   module: github.com/open-telemetry/opentelemetry-collector-releases/core
<   name: otelcol
<   description: OpenTelemetry Collector
---
>   module: github.com/open-telemetry/opentelemetry-collector-releases/paas
>   name: otelcol-paas
>   description: OpenTelemetry Collector PaaS
11,14d10
<   - gomod: github.com/open-telemetry/opentelemetry-collector-contrib/receiver/hostmetricsreceiver v0.92.0
<   - gomod: github.com/open-telemetry/opentelemetry-collector-contrib/receiver/jaegerreceiver v0.92.0
<   - gomod: github.com/open-telemetry/opentelemetry-collector-contrib/receiver/kafkareceiver v0.92.0
<   - gomod: github.com/open-telemetry/opentelemetry-collector-contrib/receiver/opencensusreceiver v0.92.0
20d15
<   - gomod: go.opentelemetry.io/collector/exporter/loggingexporter v0.92.0
23,28d17
<   - gomod: github.com/open-telemetry/opentelemetry-collector-contrib/exporter/fileexporter v0.92.0
<   - gomod: github.com/open-telemetry/opentelemetry-collector-contrib/exporter/kafkaexporter v0.92.0
<   - gomod: github.com/open-telemetry/opentelemetry-collector-contrib/exporter/opencensusexporter v0.92.0
<   - gomod: github.com/open-telemetry/opentelemetry-collector-contrib/exporter/prometheusexporter v0.92.0
<   - gomod: github.com/open-telemetry/opentelemetry-collector-contrib/exporter/prometheusremotewriteexporter v0.92.0
<   - gomod: github.com/open-telemetry/opentelemetry-collector-contrib/exporter/zipkinexporter v0.92.0
32d20
<   - gomod: go.opentelemetry.io/collector/extension/ballastextension v0.92.0
38,46c26
<   - gomod: go.opentelemetry.io/collector/processor/memorylimiterprocessor v0.92.0
<   - gomod: github.com/open-telemetry/opentelemetry-collector-contrib/processor/attributesprocessor v0.92.0
<   - gomod: github.com/open-telemetry/opentelemetry-collector-contrib/processor/resourceprocessor v0.92.0
<   - gomod: github.com/open-telemetry/opentelemetry-collector-contrib/processor/spanprocessor v0.92.0
<   - gomod: github.com/open-telemetry/opentelemetry-collector-contrib/processor/probabilisticsamplerprocessor v0.92.0
<   - gomod: github.com/open-telemetry/opentelemetry-collector-contrib/processor/filterprocessor v0.92.0
< 
< connectors:
<   - gomod: go.opentelemetry.io/collector/connector/forwardconnector v0.92.0
---
>   - gomod: github.com/open-telemetry/opentelemetry-collector-contrib/processor/transformprocessor v0.92.0

This distro has markedly less components than core, and adds the transformprocessor.

mx-psi commented 10 months ago

Thanks!

TylerHelmuth commented 10 months ago

Similar to https://github.com/open-telemetry/opentelemetry-collector-releases/issues/357 we show hold off until we have split up our CI. I am very close to a solution, right now I'm waiting on goreleaser-pro 1.24 which has a bug fix we need.

atoulme commented 3 months ago

OK, I think we're ready to put this back on. I would like to have this being examined as a potential solution for PaaS environments.