kyma-project / kyma

Kyma is an opinionated set of Kubernetes-based modular building blocks, including all necessary capabilities to develop and run enterprise-grade cloud-native applications.
https://kyma-project.io
Apache License 2.0
1.52k stars 405 forks source link

Configurable logging #11236

Closed a-thaler closed 2 years ago

a-thaler commented 3 years ago

Description

The current logging setup is only configurable via installation overrides and restricts the user of changing the configuration dynamically at runtime without triggering the kyma upgrade procedure. Furthermore it is tightly coupled/packaged with the Loki integration and is dependent on istio as a service mesh.

Goal is to decouple the log collection and shipment layer from the actual logging backend and UI. The kyma core should not even bring any backend but simply enabled the user to collect and ship logs via a simple configuration change at runtime. There should be no dependency to istio to still ship logs if there are disruptions with the service-mesh itself. There should be an optional in-cluster backend and UI deployable by the user.

logging

Preparation:

Actions:

Topics which are not required for a first initial scope are covered in https://github.com/kyma-project/kyma/issues/13142


:warning: Funnel process - do not remove!
Name|Value :--- | :--- theme | Modularization business value | 10 technical value | 10 user value | 10 open-source value | 10 effort | requested by |
ghost commented 3 years ago

This issue has been automatically marked as stale due to the lack of recent activity. It will soon be closed if no further activity occurs. Thank you for your contributions.

ghost commented 2 years ago

This issue has been automatically marked as stale due to the lack of recent activity. It will soon be closed if no further activity occurs. Thank you for your contributions.

gowrisankar22 commented 2 years ago

@a-thaler Is there a planned release date for this feature?

a-thaler commented 2 years ago

@gowrisankar22 we are heading towards a first beta release maybe already with Kyma 2.3. We are still in performance analysis having some findings which require to adjust the concept

gowrisankar22 commented 2 years ago

@a-thaler That's fantastic news. A long-awaited feature that will allow us to remove the fluentbit component dependency from our list.

a-thaler commented 2 years ago

Another update: while implementing the feature and doing more and more tests with fluent-bit helped us to understand it more and more. The results had indeed bigger impact to the design and the sum of modifications revealed, that the initial design is not fitting well anymore. We propose an adjusted design which is fully built on top of the already implemented one and will make the planned "LogPreset" abstraction obsolete, so there will be zero waste out of that. We will implement the few new API elements required to directly support a very simple integration with a HTTP based OpenSearch system.

Feel free to give feedback on the new idea here: https://github.com/kyma-project/community/pull/675 Please be aware of that the proposal discusses a future API, not the minimal one. For the minimal one please focus on the example provided here: https://github.com/kyma-project/community/pull/675/files#diff-a3e0d5b98feddb0d93dffda5a6dc176f78648e578b86ba8a7cf9daba08b92addR170

a-thaler commented 2 years ago

The documentation is making good progress as well. If you are interested in a pre-read, look here: https://deploy-preview-14063--kyma-project-docs-preview.netlify.app/docs/kyma/preview/01-overview/main-areas/observability/obsv-04-telemetry-in-kyma/ That is not covering the discussed API additions of above comment

a-thaler commented 2 years ago

Based on the new proposal we re-vamped the backlog. It might look impressive from the amount of tasks. Indeed the tasks are all well-defined with not much uncertainty left so that they can be implemented with a faster progress. We see so much value in the adjusted API that we want to include it before the release.

a-thaler commented 2 years ago

MVP is done and will bundled with 2.6