search

oracle-quickstart / oci-kubernetes-monitoring

Kubernetes Monitoring with OCI Observability & Management Platform
https://www.oracle.com/manageability/logging-analytics/
Universal Permissive License v1.0
42 stars 32 forks source link

Fluentd suddenly stopped sending logs to LA #84

Closed sourabhgupta385 closed 1 month ago

sourabhgupta385 commented 2 months ago

Hi Team,

We are running https://github.com/oracle-quickstart/oci-kubernetes-monitoring in our kubernetes cluster. We are noticing a very strange thing that some nodes stopped sending logs to LA. oci-onm-logan pods are generating normal logs that they are tailing the log files but oci-logging-analytics.log inside the pod /var/log log file stopped generating any logs and since then there are no logs uploaded to LA.

We have enabled DEBUG logs also but there is nothing useful. Can you please help us with this situation? This is happening very randomly. The solution for now is to restart the pod and then it again starts sending logs normally but that is manual. We would like to know what will be permanent fix for it or how to debug this?

Regards, Sourabh

santhoshkvuda commented 1 month ago

Resolved this internally. It's due to a bad source sending single log record exceeding max chunk limit.