jjbustamante
commented
1 year ago Hi @shresthaujjwal
Thanks for creating this issue! I think we will need to work on a sub-team RFC to get more details about which opentelemetry standards we want to support. I am glad to help you with it.
Description
Need a way to stream pack log data and buildpack logs to analytical systems to better understand the build pipeline data and also get a better understanding of how the buildpack are used or where the errors are happening etc
Proposed solution
Proposal is to not just stream the log but also make it configurable or use opentelemetry standards like embed https://github.com/open-telemetry/opentelemetry-go sdk in all go app so that we have standards. Also ability to control log level and where we stream the logs as we could stream this to our internal OTLP endpoints
Describe alternatives you've considered
Alternative is to scrape the console log of pack cli and write your own processor and stream the logs ourself and also override any opensource packeto buildpack and add our logs and logics that would stream logs to our OTLP endpoint
Additional context
This should be specs for buildpack as well so that all the buildpack maintainer and creators who had logs enabled has listeners enabled that would automatically stream the logs to OTLP endpoint that configured
This logs is very important for large organization who support and run 1000s of jobs across organization and wants to get holistics view of how pipeline is behaving, where the failure is happening, which buildpack is very popular or at what version or which stack or builder version or who is running rebase etc.