Builder support to create a module instead of main binary

[mauri870]

Is your feature request related to a problem? Please describe. Currently, the OTel Collector Builder produces a main binary. While it is possible to skip compilation with --skip-compilation, it still generates main files, and the package name for the generated files is main.

In certain scenarios, it would be helpful if the builder supported a mode where it only generates a Go module instead of a main binary. This would allow the collector to be more easily integrated into an existing Go program. For example, at Elastic, we embed our Collector Distribution into Elastic Agent as a subcommand elastic-agent otel.

Describe the solution you'd like I suggest adding a new flag or config.yml property to instruct the Builder to create a module instead of a main binary. When this option is provided, the Builder would skip generating the main files and instead export an entry point that can be used as a library.

[mx-psi]

We have discussed this in the past, to me the biggest open questions are whether this is actually useful and what stability guarantees we would provide (#7464 discusses the latter question).

Why is using the builder useful when compared to just using the Collector as a library?

[mauri870]

We have discussed this in the past, to me the biggest open questions are whether this is actually useful and what stability guarantees we would provide (#7464 discusses the latter question).

Why is using the builder useful when compared to just using the Collector as a library?

Thanks for the added context. I think the main advantage of using the builder is code generation. Having a single yml file with all the exports, receivers, etc is very handy to produce the final code, as well as having an easier time bumping the versions of the dependencies and a go.mod generated for you.

The same advantages that makes the builder a great tool to produce main collector binaries are also useful here, the only change is the entrypoint to the code (main vs exported function). It is possible to achieve something like this today with the builder by leveraging --skip-compilation plus some lines of scripting and a shim export layer, I wish it was easier/supported.

[mx-psi]

Having a single yml file with all the exports, receivers, etc is very handy to produce the final code

I can understand this, but I don't see the point after the first time. Why is it better to keep using ocb after the initial step? Is it because of lack of stability of the underlying APIs?

having an easier time bumping the versions of the dependencies and a go.mod generated for you.

What's the advantage of it vs something like dependabot or renovatebot?

[mauri870]

I understand your point, but I don't see the need to continue using OCB after the initial step. Is it due to the instability of the underlying APIs?

Good point. Part of the reason is the lack of stability, and part is that we occasionally add new modules, which would need to be manually integrated into components. I'd prefer to rely on OCB to generate these files. We also plan to allow our customers to create their own collectors based on the Elastic Collector Distribution. They can start with our config.yml and generate their own customized collector.

What's the advantage of using this over tools like Dependabot or RenovateBot?

I think we can use Dependabot or a similar tool to keep versions up to date, but new modules would still have to be added manuall to the source and go.mod as mentioned earlier.

[mx-psi]

Thanks for the added context. I am also interested in why the Elastic Collector Distribution cannot be built with ocb. What's missing there?

I don't feel like getting into the business of library code generation is good for ocb maintainability, so I would like to explore all the alternatives

[mauri870]

Thanks for the added context. I am also interested in why the Elastic Collector Distribution cannot be built with ocb. What's missing there?

I don't feel like getting into the business of library code generation is good for ocb maintainability, so I would like to explore all the alternatives

For context, it can be built into a binary using ocb. The problem is that we embed the collector into another Go program, so this program must instantiate and run the collector in a separate goroutine, as opposed to compiling the main generated by ocb. Why we do this is to reduce the size of the production bundle, as we already ship bigger binaries (multiple beats, elastic agent) and otelcol would be another one. In fact, that is exactly what we are doing now, writing the collector code and a shim export layer on top.

I understand the hesitation to introduce and support library code generation along with maintainance. Perhaps it is wise to keep this issue open and see if there is more interest in this feature. I would love to hear from the community possible use cases for this.

[mx-psi]

Perhaps it is wise to keep this issue open and see if there is more interest in this feature. I would love to hear from the community possible use cases for this.

Okay, that sounds reasonable. I would also like to revisit this once we mark otelcol as 1.0 to see if the situation is better