search

open-telemetry / opentelemetry-collector

OpenTelemetry Collector
https://opentelemetry.io
Apache License 2.0
4.36k stars 1.44k forks source link

Collector with custom plugins compilation broken #11485

Open lgtti opened 22 hours ago

lgtti commented 22 hours ago

Hi,

we are using our collector with contrib plugins using ocb.

The config file is the following:

dist:
  name: edge
  description: open telemetry collector with additional filelog receiver
  output_path: .
  otelcol_version: 0.99.0

receivers:
  - gomod:
      github.com/open-telemetry/opentelemetry-collector-contrib/receiver/filelogreceiver v0.99.0
  - gomod:
      github.com/open-telemetry/opentelemetry-collector-contrib/receiver/hostmetricsreceiver v0.99.0
  - gomod:
      go.opentelemetry.io/collector/receiver/otlpreceiver v0.99.0
processors:
  - gomod:
      go.opentelemetry.io/collector/processor/batchprocessor v0.99.0
  - gomod:
      github.com/open-telemetry/opentelemetry-collector-contrib/processor/resourceprocessor v0.99.0
  - gomod:
      go.opentelemetry.io/collector/processor/memorylimiterprocessor v0.99.0
exporters:
  - gomod:
      go.opentelemetry.io/collector/exporter/debugexporter v0.99.0
  - gomod:
      go.opentelemetry.io/collector/exporter/otlphttpexporter v0.99.0
  - gomod:
      github.com/open-telemetry/opentelemetry-collector-contrib/exporter/lokiexporter v0.99.0

These are the steps:

curl --proto '=https' --tlsv1.2 -fL -o src/ocb \
    https://github.com/open-telemetry/opentelemetry-collector/releases/download/cmd%2Fbuilder%2Fv0.99.0/ocb_0.99.0_linux_amd64

chmod +x src/ocb

GOFLAGS=-buildvcs=false ./ocb --config builder_config.yaml

We have successfully compiled it at 2024-07-30 in the morning and we have tried a new compilation yesterday (2024-10-17) with this error:

+ GOFLAGS=-buildvcs=false ./ocb --config builder_config.yaml

2024-10-17T14:46:22.551Z    INFO    internal/command.go:125 OpenTelemetry Collector Builder {"version": "0.99.0", "date": "2024-04-22T20:31:41Z"}

2024-10-17T14:46:22.552Z    INFO    internal/command.go:161 Using config file   {"path": "builder_config.yaml"}

2024-10-17T14:46:22.552Z    INFO    builder/config.go:118   Using go    {"go-executable": "/usr/local/go/bin/go"}

2024-10-17T14:46:22.553Z    INFO    builder/main.go:101 Sources created {"path": "."}

2024-10-17T14:46:39.085Z    INFO    builder/main.go:197 Getting go modules

2024-10-17T14:46:42.544Z    INFO    builder/main.go:112 Compiling

Error: failed to compile the OpenTelemetry Collector distribution: go subcommand failed with args '[build -trimpath -o edge -ldflags=-s -w]': exit status 1, error message: # go.opentelemetry.io/otel/sdk/metric

/go/pkg/mod/go.opentelemetry.io/otel/sdk/metric@v1.25.0/meter.go:65:22: cannot use (*meter)(nil) (value of type *meter) as "go.opentelemetry.io/otel/metric".Meter value in variable declaration: *meter does not implement "go.opentelemetry.io/otel/metric".Meter (missing method Float64Gauge)

/go/pkg/mod/go.opentelemetry.io/otel/sdk/metric@v1.25.0/provider.go:90:9: cannot use mp.meters.Lookup(s, func() *meter {…}) (value of type *meter) as "go.opentelemetry.io/otel/metric".Meter value in return statement: *meter does not implement "go.opentelemetry.io/otel/metric".Meter (missing method Float64Gauge)

How is possible? The plugins, the ocb and base otel collector versions are the same but the compilation fails. This is a huge stability problems if the build version is not reproducible.

Is there a workaround or a bufix for this problem?

mx-psi commented 21 hours ago

Hey, to further help you with this, could you pass --skip-compilation and share the go.mod of the resulting code?

lgtti commented 20 hours ago

Sure!

// Code generated by "go.opentelemetry.io/collector/cmd/builder". DO NOT EDIT.

module go.opentelemetry.io/collector/cmd/builder

go 1.21.0

toolchain go1.22.1

require (
    github.com/open-telemetry/opentelemetry-collector-contrib/exporter/lokiexporter v0.99.0
    github.com/open-telemetry/opentelemetry-collector-contrib/processor/resourceprocessor v0.99.0
    github.com/open-telemetry/opentelemetry-collector-contrib/receiver/filelogreceiver v0.99.0
    github.com/open-telemetry/opentelemetry-collector-contrib/receiver/hostmetricsreceiver v0.99.0
    go.opentelemetry.io/collector/component v0.99.0
    go.opentelemetry.io/collector/confmap v0.99.0
    go.opentelemetry.io/collector/confmap/converter/expandconverter v0.99.0
    go.opentelemetry.io/collector/confmap/provider/envprovider v0.99.0
    go.opentelemetry.io/collector/confmap/provider/fileprovider v0.99.0
    go.opentelemetry.io/collector/confmap/provider/httpprovider v0.99.0
    go.opentelemetry.io/collector/confmap/provider/httpsprovider v0.99.0
    go.opentelemetry.io/collector/confmap/provider/yamlprovider v0.99.0
    go.opentelemetry.io/collector/connector v0.99.0
    go.opentelemetry.io/collector/exporter v0.99.0
    go.opentelemetry.io/collector/exporter/debugexporter v0.99.0
    go.opentelemetry.io/collector/exporter/otlphttpexporter v0.99.0
    go.opentelemetry.io/collector/extension v0.99.0
    go.opentelemetry.io/collector/otelcol v0.99.0
    go.opentelemetry.io/collector/processor v0.99.0
    go.opentelemetry.io/collector/processor/batchprocessor v0.99.0
    go.opentelemetry.io/collector/processor/memorylimiterprocessor v0.99.0
    go.opentelemetry.io/collector/receiver v0.99.0
    go.opentelemetry.io/collector/receiver/otlpreceiver v0.99.0
    golang.org/x/sys v0.25.0
)

require (
    github.com/beorn7/perks v1.0.1 // indirect
    github.com/bmatcuk/doublestar/v4 v4.6.1 // indirect
    github.com/cenkalti/backoff/v4 v4.3.0 // indirect
    github.com/cespare/xxhash/v2 v2.3.0 // indirect
    github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc // indirect
    github.com/dennwc/varint v1.0.0 // indirect
    github.com/expr-lang/expr v1.16.5 // indirect
    github.com/felixge/httpsnoop v1.0.4 // indirect
    github.com/fsnotify/fsnotify v1.7.0 // indirect
    github.com/go-kit/log v0.2.1 // indirect
    github.com/go-logfmt/logfmt v0.6.0 // indirect
    github.com/go-logr/logr v1.4.2 // indirect
    github.com/go-logr/stdr v1.2.2 // indirect
    github.com/go-ole/go-ole v1.2.6 // indirect
    github.com/go-viper/mapstructure/v2 v2.0.0-alpha.1 // indirect
    github.com/gogo/protobuf v1.3.2 // indirect
    github.com/golang/groupcache v0.0.0-20210331224755-41bb18bfe9da // indirect
    github.com/golang/snappy v0.0.4 // indirect
    github.com/google/uuid v1.6.0 // indirect
    github.com/grafana/loki/pkg/push v0.0.0-20231127162423-bd505f8e2d37 // indirect
    github.com/grafana/regexp v0.0.0-20221122212121-6b5c0a4cb7fd // indirect
    github.com/grpc-ecosystem/grpc-gateway/v2 v2.19.1 // indirect
    github.com/hashicorp/go-version v1.6.0 // indirect
    github.com/hashicorp/golang-lru/v2 v2.0.7 // indirect
    github.com/inconshreveable/mousetrap v1.1.0 // indirect
    github.com/influxdata/go-syslog/v3 v3.0.1-0.20230911200830-875f5bc594a4 // indirect
    github.com/json-iterator/go v1.1.12 // indirect
    github.com/klauspost/compress v1.17.8 // indirect
    github.com/knadh/koanf/maps v0.1.1 // indirect
    github.com/knadh/koanf/providers/confmap v0.1.0 // indirect
    github.com/knadh/koanf/v2 v2.1.1 // indirect
    github.com/leodido/ragel-machinery v0.0.0-20181214104525-299bdde78165 // indirect
    github.com/leoluk/perflib_exporter v0.2.1 // indirect
    github.com/lufia/plan9stats v0.0.0-20211012122336-39d0f177ccd0 // indirect
    github.com/mitchellh/copystructure v1.2.0 // indirect
    github.com/mitchellh/reflectwalk v1.0.2 // indirect
    github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd // indirect
    github.com/modern-go/reflect2 v1.0.2 // indirect
    github.com/mostynb/go-grpc-compression v1.2.2 // indirect
    github.com/open-telemetry/opentelemetry-collector-contrib/internal/coreinternal v0.99.0 // indirect
    github.com/open-telemetry/opentelemetry-collector-contrib/internal/filter v0.99.0 // indirect
    github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza v0.99.0 // indirect
    github.com/open-telemetry/opentelemetry-collector-contrib/pkg/translator/loki v0.99.0 // indirect
    github.com/open-telemetry/opentelemetry-collector-contrib/pkg/translator/prometheus v0.99.0 // indirect
    github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2 // indirect
    github.com/power-devops/perfstat v0.0.0-20210106213030-5aafc221ea8c // indirect
    github.com/prometheus/client_golang v1.19.0 // indirect
    github.com/prometheus/client_model v0.6.1 // indirect
    github.com/prometheus/common v0.52.3 // indirect
    github.com/prometheus/procfs v0.13.0 // indirect
    github.com/prometheus/prometheus v0.51.2-0.20240405174432-b4a973753c6e // indirect
    github.com/rs/cors v1.10.1 // indirect
    github.com/shirou/gopsutil/v3 v3.24.3 // indirect
    github.com/shoenig/go-m1cpu v0.1.6 // indirect
    github.com/spf13/cobra v1.8.0 // indirect
    github.com/spf13/pflag v1.0.5 // indirect
    github.com/stretchr/testify v1.9.0 // indirect
    github.com/tklauser/go-sysconf v0.3.12 // indirect
    github.com/tklauser/numcpus v0.6.1 // indirect
    github.com/valyala/fastjson v1.6.4 // indirect
    github.com/yusufpapurcu/wmi v1.2.4 // indirect
    go.opencensus.io v0.24.0 // indirect
    go.opentelemetry.io/collector v0.99.0 // indirect
    go.opentelemetry.io/collector/config/configauth v0.99.0 // indirect
    go.opentelemetry.io/collector/config/configcompression v1.6.0 // indirect
    go.opentelemetry.io/collector/config/configgrpc v0.99.0 // indirect
    go.opentelemetry.io/collector/config/confighttp v0.99.0 // indirect
    go.opentelemetry.io/collector/config/confignet v0.99.0 // indirect
    go.opentelemetry.io/collector/config/configopaque v1.6.0 // indirect
    go.opentelemetry.io/collector/config/configretry v0.99.0 // indirect
    go.opentelemetry.io/collector/config/configtelemetry v0.99.0 // indirect
    go.opentelemetry.io/collector/config/configtls v0.99.0 // indirect
    go.opentelemetry.io/collector/config/internal v0.99.0 // indirect
    go.opentelemetry.io/collector/consumer v0.99.0 // indirect
    go.opentelemetry.io/collector/extension/auth v0.99.0 // indirect
    go.opentelemetry.io/collector/featuregate v1.6.0 // indirect
    go.opentelemetry.io/collector/filter v0.99.0 // indirect
    go.opentelemetry.io/collector/pdata v1.6.0 // indirect
    go.opentelemetry.io/collector/semconv v0.99.0 // indirect
    go.opentelemetry.io/collector/service v0.99.0 // indirect
    go.opentelemetry.io/contrib/config v0.5.0 // indirect
    go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc v0.54.0 // indirect
    go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.54.0 // indirect
    go.opentelemetry.io/contrib/propagators/b3 v1.25.0 // indirect
    go.opentelemetry.io/otel v1.29.0 // indirect
    go.opentelemetry.io/otel/bridge/opencensus v1.25.0 // indirect
    go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetricgrpc v1.25.0 // indirect
    go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetrichttp v1.25.0 // indirect
    go.opentelemetry.io/otel/exporters/otlp/otlptrace v1.25.0 // indirect
    go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc v1.25.0 // indirect
    go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp v1.25.0 // indirect
    go.opentelemetry.io/otel/exporters/prometheus v0.47.0 // indirect
    go.opentelemetry.io/otel/exporters/stdout/stdoutmetric v1.25.0 // indirect
    go.opentelemetry.io/otel/exporters/stdout/stdouttrace v1.25.0 // indirect
    go.opentelemetry.io/otel/metric v1.29.0 // indirect
    go.opentelemetry.io/otel/sdk v1.29.0 // indirect
    go.opentelemetry.io/otel/sdk/metric v1.25.0 // indirect
    go.opentelemetry.io/otel/trace v1.29.0 // indirect
    go.opentelemetry.io/proto/otlp v1.2.0 // indirect
    go.uber.org/atomic v1.11.0 // indirect
    go.uber.org/multierr v1.11.0 // indirect
    go.uber.org/zap v1.27.0 // indirect
    golang.org/x/exp v0.0.0-20240119083558-1b970713d09a // indirect
    golang.org/x/net v0.29.0 // indirect
    golang.org/x/oauth2 v0.23.0 // indirect
    golang.org/x/text v0.18.0 // indirect
    golang.org/x/time v0.6.0 // indirect
    gonum.org/v1/gonum v0.15.0 // indirect
    google.golang.org/genproto/googleapis/api v0.0.0-20240903143218-8af14fe29dc1 // indirect
    google.golang.org/genproto/googleapis/rpc v0.0.0-20240903143218-8af14fe29dc1 // indirect
    google.golang.org/grpc v1.66.2 // indirect
    google.golang.org/protobuf v1.34.2 // indirect
    gopkg.in/yaml.v3 v3.0.1 // indirect
)
mx-psi commented 18 hours ago

Thanks, I believe the issue is because of the mismatch between these two dependencies:

I am going to ask the Go SIG for advice here.

In terms of reproducibility: if you want your builds to be reproducible, I would recommend using --skip-compilation and run the go build command itself. This ensures that all dependencies are pinned. A Collector builder manifest does not necessarily pin all the indirect dependencies.