ocb doesn't build and complains about go.opentelemetry.io/collector/config

[utezduyar]

I am following the README on the repository but it complains about not contain package go.opentelemetry.io/collector/config.

I also played around with the yaml file and seems like the build works if zpagesextension is in the yaml file.

The failure is at cmd := exec.Command(cfg.Distribution.Go, "mod", "tidy", "-compat=1.18") in GetModules.

umuttl@MAC umut % cat umut.yaml 
receivers:
  - gomod: go.opentelemetry.io/collector/receiver/otlpreceiver v0.69.0
exporters:
  - gomod: go.opentelemetry.io/collector/exporter/loggingexporter v0.69.0
extensions:
#  - gomod: go.opentelemetry.io/collector/extension/zpagesextension v0.69.0

umuttl@MAC umut % /Users/umuttl/go/bin/builder --config=./umut.yaml --output-path=./output
Flag --output-path has been deprecated, use config distribution::output_path
2023-01-11T14:52:44.402+0100    INFO    internal/command.go:125 OpenTelemetry Collector Builder {"version": "dev", "date": "unknown"}
2023-01-11T14:52:44.403+0100    INFO    internal/command.go:158 Using config file   {"path": "./umut.yaml"}
2023-01-11T14:52:44.403+0100    INFO    builder/config.go:107   Using go    {"go-executable": "/opt/homebrew/bin/go"}
2023-01-11T14:52:44.404+0100    INFO    builder/main.go:76  Sources created {"path": "./output"}
Error: failed to update go.mod: exit status 1. Output:
go: finding module for package go.opentelemetry.io/collector/config
go.opentelemetry.io/collector/cmd/builder imports
    go.opentelemetry.io/collector/otelcol imports
    go.opentelemetry.io/collector/service tested by
    go.opentelemetry.io/collector/service.test imports
    go.opentelemetry.io/collector/extension/zpagesextension imports
    go.opentelemetry.io/collector/config: module go.opentelemetry.io/collector@latest found (v0.69.0), but does not contain package go.opentelemetry.io/collector/config

umuttl@MAC umut % go version
go version go1.19.4 darwin/arm64

umuttl@MAC umut % cat ./output/go.mod 
// Code generated by "go.opentelemetry.io/collector/cmd/builder". DO NOT EDIT.

module go.opentelemetry.io/collector/cmd/builder

go 1.18

require (
    go.opentelemetry.io/collector/receiver/otlpreceiver v0.69.0
    go.opentelemetry.io/collector/exporter/loggingexporter v0.69.0
    go.opentelemetry.io/collector v0.69.0
)

It works if I enable zpagesextension

umuttl@MAC umut % cat umut.yaml 
receivers:
  - gomod: go.opentelemetry.io/collector/receiver/otlpreceiver v0.69.0
exporters:
  - gomod: go.opentelemetry.io/collector/exporter/loggingexporter v0.69.0
extensions:
  - gomod: go.opentelemetry.io/collector/extension/zpagesextension v0.69.0

umuttl@MAC umut % /Users/umuttl/go/bin/builder --config=./umut.yaml --output-path=./output
Flag --output-path has been deprecated, use config distribution::output_path
2023-01-11T14:57:23.069+0100    INFO    internal/command.go:125 OpenTelemetry Collector Builder {"version": "dev", "date": "unknown"}
2023-01-11T14:57:23.070+0100    INFO    internal/command.go:158 Using config file   {"path": "./umut.yaml"}
2023-01-11T14:57:23.070+0100    INFO    builder/config.go:107   Using go    {"go-executable": "/opt/homebrew/bin/go"}
2023-01-11T14:57:23.071+0100    INFO    builder/main.go:76  Sources created {"path": "./output"}
2023-01-11T14:57:23.313+0100    INFO    builder/main.go:118 Getting go modules
2023-01-11T14:57:23.364+0100    INFO    builder/main.go:87  Compiling
2023-01-11T14:57:25.595+0100    INFO    builder/main.go:99  Compiled    {"binary": "./output/otelcol-custom"}

umuttl@MAC umut % ./output/otelcol-custom components
buildinfo:
    command: otelcol-custom
    description: Custom OpenTelemetry Collector distribution
    version: 1.0.0
receivers:
    - otlp
processors: []
exporters:
    - logging
extensions:
    - zpages

[MovieStoreGuy]

Hey @jpkrohling,

Looking at https://github.com/open-telemetry/opentelemetry-collector/blob/v0.69.0/go.mod#L25

It looks like it depends on the previous version of pdata which is trying to resolve the packages for v0.68.0 and causes issues since the go code in config/ was removed causing it to fail.

I suspect this didn't fail the builds in the collector repos since it uses replace statements.

[mx-psi]

It seems like adding some contrib components to the list fixes the issue. For example, adding the hostmetrics receiver fixes this (the following manifest builds correctly):

# This one builds successfully

receivers:
  - gomod: go.opentelemetry.io/collector/receiver/otlpreceiver v0.69.0
  - gomod: github.com/open-telemetry/opentelemetry-collector-contrib/receiver/hostmetricsreceiver v0.69.0
exporters:
  - gomod: go.opentelemetry.io/collector/exporter/loggingexporter v0.69.0

But adding some other contrib components does not fix the build; e.g. only adding the file exporter does not (the following manifest fails to build with the same error as the above):

# This one fails to build with the same error

receivers:
  - gomod: go.opentelemetry.io/collector/receiver/otlpreceiver v0.69.0
exporters:
  - gomod: go.opentelemetry.io/collector/exporter/loggingexporter v0.69.0
  - gomod: github.com/open-telemetry/opentelemetry-collector-contrib/exporter/fileexporter v0.69.0

I have not been able to tell what the difference is between the two that makes one fail and the other succeed.

The fact that adding the host metrics receiver fixes the build explains why the official binaries were built without any problems, since both distros include this component. This also means that the official binaries have the correct version of every module (I haven't checked this explicitly, but am reasonably sure about it).

I think we need a core v0.69.1 tag that fixes the issue @MovieStoreGuy mentioned.

[jpkrohling]

How about adding a replace statement for go.opentelemetry.io/collector/pdata, pointing to the expected version? Would it make the build work? I suspect this has to do with the module splits that happened in the core, not caused by the builder itself.

[mx-psi]

@jpkrohling see https://github.com/open-telemetry/opentelemetry-collector/issues/6908#issuecomment-1380135220, it looks like the require statements on the main core go.mod file for v0.69.0 are incorrectly pointing to v0.68.0/v1.0.0-rc2

[jpkrohling]

Sorry, I meant to add a replace statement to the manifest used with the builder.

[mx-psi]

Ah, my bad, thanks for clarifying :)

[MovieStoreGuy]

How about adding a replace statement for go.opentelemetry.io/collector/pdata, pointing to the expected version? Would it make the build work? I suspect this has to do with the module splits that happened in the core, not caused by the builder itself.

I had tried doing that in my internal project, and it has no affect unfortunately.

[mx-psi]

I am not able to reproduce this with v0.69.1, but there is a bug report of the issue on v0.69.1 over at #6942.

@utezduyar, does v0.69.1 fix the issue for you?

[utezduyar]

0.69.1 works for me. To follow up I have created https://github.com/open-telemetry/opentelemetry-collector/pull/6944