Better observability from "business" perspective

[MingZhang-PS]

Description Considering I am an extension application developer and my extension app is deployed on Kyma. My extension is simple, it reacts event from SAP field service management, and update record of SAP field service management back according to some business logic defined inside the extension. Someday, I found the record is not updated. I am not very sure

1. If the event is sent from SAP FSM
2. If the mTLS connection between SAP FSM and Kyma is broken
3. If the event is lost in Kyma eventing mesh due to unknown reason
4. If the event is not dispatched due to incorrect change of my extension
5. If the oauth client id and secret cached in Kyma application gateway goes invalidated

As you can see, my extension is designed in typical Kyma pattern. However, in order to check different issue possibilities, I have to use different tool/skill, like: a. For 1) and 2), SAP FSM kibana log => However, as an extension developer, I am most likely from a partner company, how can I check SAP FSM kibana log? b. For 3) and 4), Components log of Kyma eventing mesh => However, Kyma eventing mesh contains lots of internal components, it is hard for an extension developer to sort out c. For 5), again, I have to check SAP FSM kibana log or Kyma application gateway log

Reasons There are some logging, tracing tool provided by Kyma, like Loki and Jaegar. However, they are more generic and techincal, not customized for typical Kyma use case. As an extension developer, I hope there is some dedicated and convenience UI tool, helps to visualization the event dispatching and callback workflow. I hope it tells me where the event is lost, Kyma itself, my extension, or somewhere else? why extension can't callback SAP FSM, oauth issue or SAP FSM down, or something else?

[a-thaler]

Hi @MingZhang-PS

Kyma collects the observability data (traces, metrics, logs) coming from the cluster itself, and provides some tooling out of the box to visualize that data. The out-of-the-box tooling can be improved, for example by naming technical components much better and making the pre-configured integration into grafana even better. However, I guess it will not meet your requirements as it is based on the local cluster context only.

Instead, Kyma needs to open-up in regards to integrations for external observability backends, so that you could easily plug Kyma to a product like datadog or splunk. Here, you can integrate all your systems together, assuring easy aggregation and correlation of metrics, logs, and traces in a convenient accessible way, providing also the required business context.

In regards to SAP, probably there should be a solution in place as part of the SAP Cloud Platform fulfilling that needs, where kyma will be just one system contributing it's data to.

From a roadmap perspective for kyma, we will work on making the integration of the observability backend of your choice possible. Furthermore, we will see that traces are improved in regards of naming and also completeness.

[a-thaler]

Will close it, feel free to re-open if you disagree or you want to focus on other aspects

[MingZhang-PS]

Hi Andreas Thanks for the reply! Customer nowadays use Kyma in a way that hard to image. Let me take 2 examples.

• Customer may not use SAP cloud platform for cost reason or to simplify the complexity of their infrastructure.
• Customer may not use SAP managed Kyma instance since they already have their own K8S cluster and basic operation capability.

However, they still want to leverage Kyma as a free open source project to extend their SAP software. So it is worth to give them "business oriented" diagnostic tool under these backgrounds. I get the point that in order to maximum the openness and flexibility, it is not suitable to be put such tool as standard delivery of Kyma. Indeed it is true, so I am thinking if such tool can be delivered as a Kyma "addon" (Addon is a great feature in Kyma I would say)? Customer then can choose to install this addon when he wants to solve problem quickly. Or he can choose to build with bigger solution

[a-thaler]

Good to hear that Kyma gets used Standalone, it was the intention from the beginning that users are free to choose how to operate it. And the same should go with observability. There is the free "batteries included" stack which is not HA and is based on pre-configured 3party software. And you can integrate your own solution and you also can buy the enterprise solution from a vendor like SAP.

The question is: what must be improved on the free stack, so that it can bring more value also in the extensibility scenario without writing a lot of custom code (it is not the focus of Kyma to provide an own project in this area). As I mentioned, I see a need to:

• name technical components better
• exclude irrelevant spans like the istio-mixer
• more easy correlation between traces, logs and metrics
• assuring that for all scenarios there is a end-2-end trace available, so no loss of the traceID in the eventing for example
• components of the extensibility scenario must provide good enough metrics and traces so that we can have focused grafana dashboards. An expired secret in the application-gateway should trigger an alert with a related dashboard visualizing the statistics of the gateway.

So if you see concrete things that we can improve in the eventing and application connectivity (especially in regards to metrics and visualizing them in grafana) we are happy to hear about it. In best case in combination with a Pull Request :)

[MingZhang-PS]

Hi Andreas Besides the need you mentioned above, I would like to add one more ^-^:

• View the events received by specific Kyma application, includes its receiving time, event type, source application, payload (nice to have)