My main concern with the pipes library is that it is full of "magic" and for someone not knowing it internally (someone that it's not me 😅) it's very difficult to reason about how it works. It also had:
Provided multiple ways to do the same task, which might be confusing.
The annotation-based connection of nodes was not typesafe, and a source of runtime errors with error messages that might not be clarifying.
All the magic was based on an intensive use of the reflection API, and the library reached a point of complexity where it was pretty difficult to extend and maintain even for the creator of the library. This was a long-term risk for the team.
I've been lately working on a complete rewrite of the library, which:
Provides a unified API for creating graphs.
Every configuration is done explicitly via code in a typesafe manner. No more annotations. No more altering the behavior of the nodes via implicit interfaces.
The actual code API will be descriptive enough to be able to understand what's happenning in the documentation popups of each function/type, without having to continuously refer to the library tutorials.
Reduced from 3500 to 1200 the total lines of source code. Part of the reduction is becase the new version removes a lot of functionalities that remained unused in Beyla.
Replacing the annotations with explicit Go language constructs might add few extra boilerplate code and externally could seem as it does not change much the results, but with this new version, any new contributor has better API guidance to add or change elements in the pipeline.
My main concern with the pipes library is that it is full of "magic" and for someone not knowing it internally (someone that it's not me 😅) it's very difficult to reason about how it works. It also had:
I've been lately working on a complete rewrite of the library, which:
Replacing the annotations with explicit Go language constructs might add few extra boilerplate code and externally could seem as it does not change much the results, but with this new version, any new contributor has better API guidance to add or change elements in the pipeline.