Add observability with opentelemetry

[ericrlessa]

To improve the observability of our project, we propose integrating OpenTelemetry for logging and metrics. Initially, our focus will be on implementing logging and metrics functionalities. As our project currently operates as a monolith, tracing can be considered for future iterations.

For logging and metrics, we intend to use CloudWatch as the provider. However, it's crucial for us to maintain platform independence and isolate specific API usage. OpenTelemetry provides a solution that allows us to achieve this goal effectively.

Here are some suggested metrics to monitor:

Notifications: Track various notification events, such as emails sent, errors encountered, messages bounced, or those treated as spam.

Baskets: Monitor the lifecycle of shopping baskets, including creation, reservation, and delivery statuses.

Stocks: Keep track of product availability metrics, such as products out of stock, sold-out items, and high-demand products.

General Errors: Capture and analyze general errors occurring within the system to identify and address issues promptly.

[faermanj]

I believe the next step here would be to experiment and record a demo. What do you think @IamNatri @idmcarvalho

[StuxxNet]

Hey guys!

Is there a branch where you are currently working in this topic? If so, what's that? I would like to participate on this topic and for that would be cool to understand what's the current status of the task and what are we striving for, although I could get some context based on the description of this issue.

[idmcarvalho]

Hello @StuxxNet I'm trying to implement some configurations to handle the Jaeger exporter and provide some traces. Your help would be much appreciated. Have you joined our Discord channel? So we can discuss a little more https://discord.gg/NJ99Qat2

at last i didnt' commited anything, my changes stayed local but we can create an feat-opentelemtry branch to start this issue

[idmcarvalho]

@faermanj @StuxxNet Let me know when we could meet to discuss about this issue

[IamNatri]

The otel instrumentation is already up to date, check the branch

Now we are capturing metrics and trances with OTELP (OpenTelemetry protocol). For logs we need to export for a specific vendor as quarkus don't have a native export component for logs in OTLP.

API test metric endpoint (api/product) the status of the issue and futher steps are:

• [x] Otel instrumentation
• [x] prometheus ready to use
• [x] zipikin ready to date
• [x] grafana ready to use with prometheus and zipikin
• [ ] logs export in quarkus api and Instrument grafana with the log vendor config
• [ ] create custom metrics