NeonGeckoCom / neon-diana-utils

Scripts and Automation for running Neon DIANA
Other
5 stars 6 forks source link

[FEAT] Log Aggregation #68

Open NeonDaniel opened 5 months ago

NeonDaniel commented 5 months ago

Objective

Currently, many services interact with each other, so viewing synchronized logs is sometimes necessary. Additionally, we have no documented tools for viewing container logs in k8s (k9s is not really a log viewer).

Initial Implementation Requirements

Other Considerations

NeonDaniel commented 5 months ago

The license for Sentry appears to be modified Apache, so would have to evaluate if that is compatible and acceptable for our use.

Open Telemetry is Apache 2.0 licensed and may also be applicable for metrics reporting in addition to logging

Graylog uses the same license as MongoDb apparently which I believe we have already evaluated as acceptable for our use cases.

NeonDaniel commented 5 months ago

Allure was also mentioned but I have not researched that implementation

NeonDaniel commented 5 months ago

Official docs for Kubernetes outline some different architecture options. It seems a node logging agent might be the easiest to implement as it requires no changes to containers and only adds one container per node vs a sidecar container per pod.

I believe all of the logging we care about is already using ovos-utils.log so formatting is uniform and log names can be parsed in the logging backend

NeonDaniel commented 5 months ago

From @mikejgray this article outlines some good background information on logging. For the moment, kubetail or one of the other utilities they mention may be sufficient to debug issues with the default logging we have.

Longer-term we may want to use sidecar containers to build structured logs and implement proper aggregation on a dedicated pod

NeonDaniel commented 4 months ago

As discussed, we will get an account opened with Sentry for @NeonKirill to integrate with Klat to start. If this works well, we can extend other services to support this and work it into the diana setup CLI as an optional component