Describe what happened:
There is a discrepancy between count of files actually being tailed by Datadog and what is being reported.
First log of each container is 0.log in /var/log/pods//. When kubelet is terminated in an unclean way, e.g. by hard-rebooting the worker node, this container termination is handled like restart (especially in case of statefulsets) and new log is created in the same folder. Due to unclean kubelet/node termination, 0.log is not deleted and new container is logging to 1.log. So far so good, this is standard kubelet behavior.
After hard restarting the node 4 times, we end up with 5 log files named 0.log to 4.log in the same folder. Now I would expect Datadog to tail only one file -> 4.log, since it is the only file that is actually in use by live container, since previous 4 containers no longer exist.
Example container:
dev01-fcmi/cmi-repo-host-0/pgbackrest
-------------------------------------
- Type: file
Identifier: ce025fb0a1db8c11c30a41838f2e531cd850a2d06446219b597d2be99f74bd21
Path: /var/log/pods/dev01-fcmi_cmi-repo-host-0_502543d6-bf0a-4743-b2fc-bb68610c3a0f/pgbackrest/*.log
Service: crunchy-pgbackrest
Source: crunchy-pgbackrest
Status: OK
5 files tailed out of 5 files matching
Inputs:
/var/log/pods/dev01-fcmi_cmi-repo-host-0_502543d6-bf0a-4743-b2fc-bb68610c3a0f/pgbackrest/4.log
BytesRead: 0
Average Latency (ms): 0
24h Average Latency (ms): 0
Peak Latency (ms): 0
24h Peak Latency (ms): 0
When we check file descriptors, we see that only one file, the correct one, is actually opened/presumably tailed:
root@ip-10-223-124-76:/proc/5424/fd# ls -l | grep dev01-fcmi_cmi-repo-host-0_502543d6-bf0a-4743-b2fc-bb68610c3a0f/pgbackrest/
lr-x------. 1 root root 64 Jan 17 12:57 41 -> /var/log/pods/dev01-fcmi_cmi-repo-host-0_502543d6-bf0a-4743-b2fc-bb68610c3a0f/pgbackrest/4.log
However the "agent status" output is reporting that 5 files out of 5 are being tailed, which is not true. This has one big implication -> sum of these values is compared against open_files_limit and it doesn't take me much to reach limit of 200 with just couple hard restarts if I run enough pods . This is mainly causing issues on development clusters which are shut-down (gracefuly) for weekends, but after some timeout node is shut down regardless.
Deployment pod names get changed on every upgrade, so even if there are some leftover logs, Datadog ignores them eventually. But with statefulsets, these leftover logs can accumulate in the same directory and with the behavior described above, we end up with Datadog reporting "taling" 30 log files, from which some are 6 months old, while in reality it's only tailing one file.
Once we hit open_file_limit, with each restart less and less files are actually being tailed. Only solution is manual cleanup of /var/log/pods.
Example (/var/log/pods is cleaned beforehand):
Startup:
Reported as tailed by Datadog: 56
Open FDs: 56
Actual files on filesystem: 101
Restart 1:
Reported as tailed by Datadog: 113
Open FDs: 56
Actual files on filesystem: 203
Restart 2:
Reported as tailed by Datadog: 170
Open FDs: 56
Actual files on filesystem: 305
Restart 3:
Reported as tailed by Datadog: 200
Open FDs: 50
Actual files on filesystem: 407
Restart 4:
Reported as tailed by Datadog: 200
Open FDs: 39
Actual files on filesystem: 510
Restart 5:
Reported as tailed by Datadog: 200
Open FDs: 34
Actual files on filesystem: 612
Complete status after Restart 5:
root@ip-10-223-124-76:/proc/5380/fd# agent status
Getting the status from the agent.
2023-01-17 14:57:25 UTC | CORE | WARN | (pkg/util/log/log.go:618 in func1) | Deactivating Autoconfig will disable most components. It's recommended to use autoconfig_exclude_features and autoconfig_include_features to activate/deactivate features selectively
===============
Agent (v7.41.1)
===============
Status date: 2023-01-17 14:57:25.785 UTC (1673967445785)
Agent start: 2023-01-17 14:42:14.652 UTC (1673966534652)
Pid: 5380
Go Version: go1.18.9
Python Version: 3.8.14
Build arch: amd64
Agent flavor: agent
Check Runners: 4
Log Level: WARN
Paths
=====
Config File: /etc/datadog-agent/datadog.yaml
conf.d: /etc/datadog-agent/conf.d
checks.d: /etc/datadog-agent/checks.d
Clocks
======
NTP offset: -190µs
System time: 2023-01-17 14:57:25.785 UTC (1673967445785)
Host Info
=========
bootTime: 2023-01-17 14:41:07 UTC (1673966467000)
hostId: ec28db03-5c7a-ef19-1335-d09190515d6e
kernelArch: x86_64
kernelVersion: 6.0.10-200.fc36.x86_64
os: linux
platform: fedora
platformFamily: fedora
platformVersion: 36 (CoreOS)
procs: 383
uptime: 1m16s
Hostnames
=========
cluster-name: cortex-mgmt01
ec2-hostname: ip-10-223-124-76.eu-central-1.compute.internal
host_aliases: [i-03dcdc76f8df26cd5 ip-10-223-124-76.eu-central-1.compute.internal-cortex-mgmt01]
hostname: i-03dcdc76f8df26cd5
instance-id: i-03dcdc76f8df26cd5
socket-fqdn: 10-223-124-76.ovn-kubernetes-node.openshift-ovn-kubernetes.svc.cluster.local.
socket-hostname: ip-10-223-124-76
host tags:
cluster_name:cortex-mgmt01
kube_cluster_name:cortex-mgmt01
kube_node_role:worker
hostname provider: aws
unused hostname providers:
'hostname' configuration/environment: hostname is empty
'hostname_file' configuration/environment: 'hostname_file' configuration is not enabled
azure: azure_hostname_style is set to 'os'
fargate: agent is not runnning on Fargate
fqdn: FQDN hostname is not usable
gce: unable to retrieve hostname from GCE: cloud provider is disabled by configuration
os: OS hostname is not usable
Metadata
========
agent_version: 7.41.1
cloud_provider: AWS
config_apm_dd_url:
config_dd_url:
config_logs_dd_url:
config_logs_socks5_proxy_address:
config_no_proxy: []
config_process_dd_url:
config_proxy_http:
config_proxy_https:
config_site:
feature_apm_enabled: false
feature_cspm_enabled: false
feature_cws_enabled: false
feature_logs_enabled: true
feature_networks_enabled: false
feature_networks_http_enabled: false
feature_networks_https_enabled: false
feature_otlp_enabled: false
feature_process_enabled: false
feature_processes_container_enabled: true
flavor: agent
hostname_source: aws
install_method_installer_version: datadog-3.7.2
install_method_tool: helm
install_method_tool_version: Helm
logs_transport: HTTP
========
JMXFetch
========
Information
==================
Initialized checks
==================
no checks
Failed checks
=============
no checks
=========
Forwarder
=========
Transactions
============
Cluster: 0
ClusterRole: 0
ClusterRoleBinding: 0
CronJob: 0
DaemonSet: 0
Deployment: 0
Dropped: 0
HighPriorityQueueFull: 0
Ingress: 0
Job: 0
Namespace: 0
Node: 0
PersistentVolume: 0
PersistentVolumeClaim: 0
Pod: 0
ReplicaSet: 0
Requeued: 0
Retried: 0
RetryQueueSize: 0
Role: 0
RoleBinding: 0
Service: 0
ServiceAccount: 0
StatefulSet: 0
Transaction Successes
=====================
Total number: 187
Successes By Endpoint:
check_run_v1: 60
intake: 6
metadata_v1: 2
series_v2: 119
On-disk storage
===============
On-disk storage is disabled. Configure `forwarder_storage_max_size_in_bytes` to enable it.
API Keys status
===============
API key ending with f7da4: API Key valid
==========
Endpoints
==========
https://app.datadoghq.eu - API Key ending with:
- f7da4
==========
Logs Agent
==========
Reliable: Sending compressed logs in HTTPS to agent-http-intake.logs.datadoghq.eu on port 443
BytesSent: 3.6119746e+07
EncodedBytesSent: 971926
LogsProcessed: 24905
LogsSent: 24738
Warnings
========
The limit on the maximum number of files in use (200) has been reached. Increase this limit (thanks to the attribute logs_config.open_files_limit in datadog.yaml) or decrease the number of tailed file.
dev01-cashpresso/cashpresso-f-prod-risk-function-0/pulsar-function
------------------------------------------------------------------
- Type: file
Identifier: 229a3b552c988142c7614f4ad2d9b559646f6614b8350a284ccd6a6eeab9c32b
Path: /var/log/pods/dev01-cashpresso_cashpresso-f-prod-risk-function-0_dc1b2366-57cb-4c4f-81eb-3db403192466/pulsar-function/*.log
Service: neuron-functions-python-runner
Source: neuron-functions-python-runner
Status: Pending
6 files tailed out of 6 files matching
BytesRead: 915
Average Latency (ms): 0
24h Average Latency (ms): 0
Peak Latency (ms): 0
24h Peak Latency (ms): 0
Lines Combined: 0
MultiLine matches: 1
dev01-argos/argos-neuron-connector-78cd5574c-86lqj/argos-neuron-connector
-------------------------------------------------------------------------
- Type: file
Identifier: d9fcadfd38756300c9fd609ef76c0865393b0950dfd1ce527255b5026a044e44
Path: /var/log/pods/dev01-argos_argos-neuron-connector-78cd5574c-86lqj_6e552843-16bf-4a78-b294-0689e8c1e086/argos-neuron-connector/*.log
Service: adc-argos-neuron-pulse-application
Source: adc-argos-neuron-pulse-application
Status: Pending
0 files tailed out of 6 files matching
BytesRead: 0
Average Latency (ms): 0
24h Average Latency (ms): 0
Peak Latency (ms): 0
24h Peak Latency (ms): 0
dev01-neuron-pulsar/dev01-neuron-pulsar-bookie-1/pulsar-bookkeeper-verify-clusterid
-----------------------------------------------------------------------------------
- Type: file
Identifier: da2956ccf4114f095796cc21b51fcc672101457191aae21e308578e4fc841817
Path: /var/log/pods/dev01-neuron-pulsar_dev01-neuron-pulsar-bookie-1_124fc7cf-e13f-4f25-989a-866b7c653e97/pulsar-bookkeeper-verify-clusterid/*.log
Service: pulsar-bookie
Source: neuron-pulsar-all
Status: OK
0 files tailed out of 6 files matching
BytesRead: 0
Average Latency (ms): 0
24h Average Latency (ms): 0
Peak Latency (ms): 0
24h Peak Latency (ms): 0
dev01-capture/pgo-56d4d654c9-p4zg2/operator
-------------------------------------------
- Type: file
Identifier: 00d3483c950786221278629cf411018141903315ec3c91d7a85860f686aea6ef
Path: /var/log/pods/dev01-capture_pgo-56d4d654c9-p4zg2_b2787835-4d06-46f3-8b6a-9c225e7d77d0/operator/*.log
Service: postgres-operator
Source: postgres-operator
Status: OK
6 files tailed out of 6 files matching
Inputs:
/var/log/pods/dev01-capture_pgo-56d4d654c9-p4zg2_b2787835-4d06-46f3-8b6a-9c225e7d77d0/operator/5.log
BytesRead: 10637
Average Latency (ms): 0
24h Average Latency (ms): 0
Peak Latency (ms): 0
24h Peak Latency (ms): 0
cortex-argo/controller-manager-867fbd77b4-8j846/controller-manager
------------------------------------------------------------------
- Type: file
Identifier: 059db3f6754548d50c9c2776b140c18bfab3ecf9d32a81e54e77f3f665e7ca92
Path: /var/log/pods/cortex-argo_controller-manager-867fbd77b4-8j846_5f99c975-edff-4a9b-8cb1-4289bc8053fd/controller-manager/*.log
Service: argo-events
Source: argo-events
Status: Pending
7 files tailed out of 7 files matching
BytesRead: 0
Average Latency (ms): 0
24h Average Latency (ms): 0
Peak Latency (ms): 0
24h Peak Latency (ms): 0
ci-idc-24412-ftr11-improv-fairo/abc-calcproductrisk-function-0/pulsar-function
------------------------------------------------------------------------------
- Type: file
Identifier: 2c7ae22a37785e18e76a8648da684569fda2ef13d412d9ee038bd25fd8d04698
Path: /var/log/pods/ci-idc-24412-ftr11-improv-fairo_abc-calcproductrisk-function-0_8b25b43f-371c-4e02-b466-6109ad5d5544/pulsar-function/*.log
Service: abc-calcproductrisk
Source: pulsar
Status: Pending
0 files tailed out of 6 files matching
BytesRead: 0
Average Latency (ms): 0
24h Average Latency (ms): 0
Peak Latency (ms): 0
24h Peak Latency (ms): 0
dev01-fairo/fairo-cust-risk-2-function-0/pulsar-function
--------------------------------------------------------
- Type: file
Identifier: 22fbb756bb1910364e04d4d92ce0a6dea2c6d3842271f358496b62876721afdd
Path: /var/log/pods/dev01-fairo_fairo-cust-risk-2-function-0_a8d55a36-d2cc-45a1-bdab-9c4a6b98b2a6/pulsar-function/*.log
Service: fairo-cust-risk-2
Source: pulsar
Status: Pending
0 files tailed out of 6 files matching
BytesRead: 0
Average Latency (ms): 0
24h Average Latency (ms): 0
Peak Latency (ms): 0
24h Peak Latency (ms): 0
dev01-fcmi/cmi-repo-host-0/pgbackrest-log-dir
---------------------------------------------
- Type: file
Identifier: 433215f5ac4d60988fd30cd359dc5f654eae4bf6ef77c7adf7a81ad63d85442f
Path: /var/log/pods/dev01-fcmi_cmi-repo-host-0_502543d6-bf0a-4743-b2fc-bb68610c3a0f/pgbackrest-log-dir/*.log
Service: crunchy-pgbackrest
Source: crunchy-pgbackrest
Status: OK
6 files tailed out of 6 files matching
Inputs:
/var/log/pods/dev01-fcmi_cmi-repo-host-0_502543d6-bf0a-4743-b2fc-bb68610c3a0f/pgbackrest-log-dir/5.log
BytesRead: 0
Average Latency (ms): 0
24h Average Latency (ms): 0
Peak Latency (ms): 0
24h Peak Latency (ms): 0
ci-idc-24412-int-scenario-fairo/abc-calccountryrisk-function-0/pulsar-function
------------------------------------------------------------------------------
- Type: file
Identifier: 2936c6e060781c4d270bc5c72aa2b5677dab9b0c37626e813e26cfd8c4ac3cc9
Path: /var/log/pods/ci-idc-24412-int-scenario-fairo_abc-calccountryrisk-function-0_b0d148fb-ee9e-4c2a-967b-fd8cf8cec552/pulsar-function/*.log
Service: abc-calccountryrisk
Source: neuron-functions-python-runner
Status: Pending
6 files tailed out of 6 files matching
BytesRead: 2052
Average Latency (ms): 0
24h Average Latency (ms): 0
Peak Latency (ms): 0
24h Peak Latency (ms): 0
Lines Combined: 0
MultiLine matches: 4
ci-idc-24412-rmu-scenario-fairo/fairo-f-wlm-split-function-0/pulsar-function
----------------------------------------------------------------------------
- Type: file
Identifier: c6531f9c387ff4ad3e88549a8db029f923752a98390a82080ffa4680e3fc0e06
Path: /var/log/pods/ci-idc-24412-rmu-scenario-fairo_fairo-f-wlm-split-function-0_da876b26-1fd0-47f4-bb4d-c1afd2f77bb1/pulsar-function/*.log
Service: fairo-f-wlm-split
Source: pulsar
Status: Pending
6 files tailed out of 6 files matching
BytesRead: 887
Average Latency (ms): 0
24h Average Latency (ms): 0
Peak Latency (ms): 0
24h Peak Latency (ms): 0
Lines Combined: 0
MultiLine matches: 1
dev01-fairo/abc-minerva-com-function-0/pulsar-function
------------------------------------------------------
- Type: file
Identifier: c92842c4fd9d9bf24874870dad22c9d4926fe7cd91a90fae638fc3d40f4e26fc
Path: /var/log/pods/dev01-fairo_abc-minerva-com-function-0_ff637a00-6418-4da9-ab8b-15124d4ccf9c/pulsar-function/*.log
Service: abc-minerva-com
Source: pulsar
Status: Pending
0 files tailed out of 6 files matching
BytesRead: 0
Average Latency (ms): 0
24h Average Latency (ms): 0
Peak Latency (ms): 0
24h Peak Latency (ms): 0
cortex-pipeline/cortex-release-pipeline-sensor-dfs7q-6f9867b5d-gkfhv/main
-------------------------------------------------------------------------
- Type: file
Identifier: 22f6deeda22a5af9234eb39786754b664d92691253c56ba2e4879042aba36f93
Path: /var/log/pods/cortex-pipeline_cortex-release-pipeline-sensor-dfs7q-6f9867b5d-gkfhv_9bab9b60-1340-453d-98de-4770746e85d8/main/*.log
Service: argo-events
Source: argo-events
Status: OK
6 files tailed out of 6 files matching
Inputs:
/var/log/pods/cortex-pipeline_cortex-release-pipeline-sensor-dfs7q-6f9867b5d-gkfhv_9bab9b60-1340-453d-98de-4770746e85d8/main/5.log
BytesRead: 8144
Average Latency (ms): 0
24h Average Latency (ms): 0
Peak Latency (ms): 0
24h Peak Latency (ms): 0
dev02-capture/capture-db-capturedb-7266-0/nss-wrapper-init
----------------------------------------------------------
- Type: file
Identifier: 77b68b85ba0a224520b012bce4ea401eeb64c93242d8a692718c7daacd81cb13
Path: /var/log/pods/dev02-capture_capture-db-capturedb-7266-0_6bbdaf48-7991-41ad-818e-e3c26dc8aada/nss-wrapper-init/*.log
Service: crunchy-postgres
Source: crunchy-postgres
Status: OK
6 files tailed out of 6 files matching
Inputs:
/var/log/pods/dev02-capture_capture-db-capturedb-7266-0_6bbdaf48-7991-41ad-818e-e3c26dc8aada/nss-wrapper-init/5.log
BytesRead: 0
Average Latency (ms): 0
24h Average Latency (ms): 0
Peak Latency (ms): 0
24h Peak Latency (ms): 0
dev01-neuron-pulsar/dev01-neuron-pulsar-proxy-1/dev01-neuron-pulsar-proxy
-------------------------------------------------------------------------
- Type: file
Identifier: fb7e4f0a10b276f54dcdf1c5e28bc950f183c0192ada8f652834573f86a84bfe
Path: /var/log/pods/dev01-neuron-pulsar_dev01-neuron-pulsar-proxy-1_d330a865-5000-4978-9515-af4fbe4e737e/dev01-neuron-pulsar-proxy/*.log
Service: pulsar-proxy
Source: pulsar
Status: OK
6 files tailed out of 6 files matching
Inputs:
/var/log/pods/dev01-neuron-pulsar_dev01-neuron-pulsar-proxy-1_d330a865-5000-4978-9515-af4fbe4e737e/dev01-neuron-pulsar-proxy/5.log
BytesRead: 187348
Average Latency (ms): 0
24h Average Latency (ms): 0
Peak Latency (ms): 25
24h Peak Latency (ms): 25
Lines Combined: 0
MultiLine matches: 893
dev02-capture/capture-db-capturedb-7266-0/pgbackrest
----------------------------------------------------
- Type: file
Identifier: ebe46ff18c20aa03e445caeb6d1e3ae226f69c714197ea895d899d37fe50553c
Path: /var/log/pods/dev02-capture_capture-db-capturedb-7266-0_6bbdaf48-7991-41ad-818e-e3c26dc8aada/pgbackrest/*.log
Service: crunchy-pgbackrest
Source: crunchy-pgbackrest
Status: OK
0 files tailed out of 6 files matching
BytesRead: 0
Average Latency (ms): 0
24h Average Latency (ms): 0
Peak Latency (ms): 0
24h Peak Latency (ms): 0
dev01-neuron-pulsar/dev01-neuron-pulsar-proxy-1/wait-zookeeper-ready
--------------------------------------------------------------------
- Type: file
Identifier: 1a31db4fc7e4f9d26e43b5c9756aece1d94e315837fdc969d6ce732ae815365a
Path: /var/log/pods/dev01-neuron-pulsar_dev01-neuron-pulsar-proxy-1_d330a865-5000-4978-9515-af4fbe4e737e/wait-zookeeper-ready/*.log
Service: pulsar-proxy
Source: neuron-pulsar-all
Status: OK
0 files tailed out of 6 files matching
BytesRead: 0
Average Latency (ms): 0
24h Average Latency (ms): 0
Peak Latency (ms): 0
24h Peak Latency (ms): 0
ci-idc-24412-rmu-scenario-fairo/abc-calcproductrisk-function-0/pulsar-function
------------------------------------------------------------------------------
- Type: file
Identifier: 7a19e1f0439b0099cd79a8ee684b6903ff917b14c24820890a2a9906df8ee209
Path: /var/log/pods/ci-idc-24412-rmu-scenario-fairo_abc-calcproductrisk-function-0_6bd79182-91a9-4baf-9ce9-8e97d19b2bbe/pulsar-function/*.log
Service: abc-calcproductrisk
Source: pulsar
Status: Pending
6 files tailed out of 6 files matching
BytesRead: 2059
Average Latency (ms): 0
24h Average Latency (ms): 0
Peak Latency (ms): 0
24h Peak Latency (ms): 0
Lines Combined: 0
MultiLine matches: 4
ci-idc-24412-rmu-scenario-fairo/fairo-f-lookup-split-function-0/pulsar-function
-------------------------------------------------------------------------------
- Type: file
Identifier: b70bda1bd670d113fbacb352108d48386b773c253f5bbe5732bdf9b792e1418a
Path: /var/log/pods/ci-idc-24412-rmu-scenario-fairo_fairo-f-lookup-split-function-0_4be1f4c1-e08f-4540-a23b-71c20945c822/pulsar-function/*.log
Service: fairo-f-lookup-split
Source: pulsar
Status: Pending
0 files tailed out of 6 files matching
BytesRead: 0
Average Latency (ms): 0
24h Average Latency (ms): 0
Peak Latency (ms): 0
24h Peak Latency (ms): 0
cortex-argo/cortex-argoworkflows-argo-workflows-server-75749b6ffd-22t55/argo-server
-----------------------------------------------------------------------------------
- Type: file
Identifier: 233b4641ff588003836772c6f13fbeec621caa924da143555549f9b34c0fe9c6
Path: /var/log/pods/cortex-argo_cortex-argoworkflows-argo-workflows-server-75749b6ffd-22t55_f9b96ce3-f82e-4dce-a42a-d9e2c367a7c7/argo-server/*.log
Service: argocli
Source: argocli
Status: Pending
6 files tailed out of 6 files matching
BytesRead: 61061
Average Latency (ms): 0
24h Average Latency (ms): 0
Peak Latency (ms): 18
24h Peak Latency (ms): 18
dev01-neuron-filestreamer/eventbus-default-stan-0/metrics
---------------------------------------------------------
- Type: file
Identifier: 0221587ad576ab62616960b96bda470dba57399a54ab3ff5bf8a61c7afc94d58
Path: /var/log/pods/dev01-neuron-filestreamer_eventbus-default-stan-0_2e497708-ee8a-43b3-aaaf-dfbe98565003/metrics/*.log
Service: prometheus-nats-exporter
Source: prometheus-nats-exporter
Status: Pending
6 files tailed out of 6 files matching
BytesRead: 0
Average Latency (ms): 0
24h Average Latency (ms): 0
Peak Latency (ms): 0
24h Peak Latency (ms): 0
ci-idc-24412-rmu-scenario-fairo/fairo-aml-rmu-60-function-0/pulsar-function
---------------------------------------------------------------------------
- Type: file
Identifier: e45bc3d59c367abd9fb868a8315f485ca91552b1a2b27173845a3a5baf4cf225
Path: /var/log/pods/ci-idc-24412-rmu-scenario-fairo_fairo-aml-rmu-60-function-0_fc4b29fa-46f5-4b9d-889e-d3cc79961df2/pulsar-function/*.log
Service: neuron-functions-python-runner
Source: neuron-functions-python-runner
Status: Pending
0 files tailed out of 6 files matching
BytesRead: 0
Average Latency (ms): 0
24h Average Latency (ms): 0
Peak Latency (ms): 0
24h Peak Latency (ms): 0
ci-idc-24412-int-scenario-fairo/fairo-aml-int-06-2-function-0/pulsar-function
-----------------------------------------------------------------------------
- Type: file
Identifier: 81172d21d9138f9c8cf330036915dc3e1e488ad32740c7095c014a33972d26fb
Path: /var/log/pods/ci-idc-24412-int-scenario-fairo_fairo-aml-int-06-2-function-0_372c6c60-7d2b-457d-ad16-e55906c6d633/pulsar-function/*.log
Service: neuron-functions-python-runner
Source: neuron-functions-python-runner
Status: Pending
6 files tailed out of 6 files matching
BytesRead: 394
Average Latency (ms): 0
24h Average Latency (ms): 0
Peak Latency (ms): 0
24h Peak Latency (ms): 0
dev01-cashpresso/apollo-f-aml-hr-trx-function-0/pulsar-function
---------------------------------------------------------------
- Type: file
Identifier: 5c095af42760bb45b3a97490080dd335b978a50d105e2ca354c70d70b3d9534f
Path: /var/log/pods/dev01-cashpresso_apollo-f-aml-hr-trx-function-0_ff6bb273-fb99-4af0-a5d3-4b7d45947824/pulsar-function/*.log
Service: neuron-functions-python-runner
Source: neuron-functions-python-runner
Status: Pending
6 files tailed out of 6 files matching
BytesRead: 1681
Average Latency (ms): 0
24h Average Latency (ms): 0
Peak Latency (ms): 0
24h Peak Latency (ms): 0
Lines Combined: 0
MultiLine matches: 3
dev01-fairo/fairo-prod-risk-2-function-0/pulsar-function
--------------------------------------------------------
- Type: file
Identifier: 20e41896ebcf64142848219a70fb3b7f6b9c8a2f1bc7bc855cb6cca502133de1
Path: /var/log/pods/dev01-fairo_fairo-prod-risk-2-function-0_c6d6cd96-bbf8-4bc7-8497-4df395f8671f/pulsar-function/*.log
Service: fairo-prod-risk-2
Source: pulsar
Status: Pending
0 files tailed out of 6 files matching
BytesRead: 0
Average Latency (ms): 0
24h Average Latency (ms): 0
Peak Latency (ms): 0
24h Peak Latency (ms): 0
cortex-servicemesh/istiod-mesh-57f55dfd57-d4tkr/discovery
---------------------------------------------------------
- Type: file
Identifier: 40576d949a15880d5d84b952b98fb0677870bb5c2b3c5dfb02eeb41a4b8eba82
Path: /var/log/pods/cortex-servicemesh_istiod-mesh-57f55dfd57-d4tkr_280ced8d-9c7a-434c-9b3a-b23c80ee3c91/discovery/*.log
Service: pilot-rhel8
Source: pilot-rhel8
Status: Pending
0 files tailed out of 6 files matching
BytesRead: 449
Average Latency (ms): 0
24h Average Latency (ms): 0
Peak Latency (ms): 0
24h Peak Latency (ms): 0
Lines Combined: 0
MultiLine matches: 1
ci-idc-24412-int-scenario-fairo/fairo-txn-risk-2-function-0/pulsar-function
---------------------------------------------------------------------------
- Type: file
Identifier: 00ec799baf7bbaeeaa6ac8d3004686e025e80e91ef70b0bf3807316925e3145a
Path: /var/log/pods/ci-idc-24412-int-scenario-fairo_fairo-txn-risk-2-function-0_2e109941-b16a-44a6-a9c2-185df91c3124/pulsar-function/*.log
Service: fairo-txn-risk-2
Source: neuron-functions-python-runner
Status: Pending
0 files tailed out of 6 files matching
BytesRead: 0
Average Latency (ms): 0
24h Average Latency (ms): 0
Peak Latency (ms): 0
24h Peak Latency (ms): 0
dev01-neuron-pulsar/dev01-neuron-pulsar-proxy-1/wait-broker-ready
-----------------------------------------------------------------
- Type: file
Identifier: 90d6a94d2dc51b22abe20316b03dc1f661573e7670484f7beeb0f9a03bb1cb66
Path: /var/log/pods/dev01-neuron-pulsar_dev01-neuron-pulsar-proxy-1_d330a865-5000-4978-9515-af4fbe4e737e/wait-broker-ready/*.log
Service: pulsar-proxy
Source: neuron-pulsar-all
Status: OK
0 files tailed out of 6 files matching
BytesRead: 0
Average Latency (ms): 0
24h Average Latency (ms): 0
Peak Latency (ms): 0
24h Peak Latency (ms): 0
cortex-cgroups-gc/k8s-gc-frlcz/garbage-collector
------------------------------------------------
- Type: file
Identifier: f591ee05e1a571c9cf78d0e6ca26de596164849ee0979e3fde1bad6e54eef63f
Path: /var/log/pods/cortex-cgroups-gc_k8s-gc-frlcz_2cba6d51-848c-42b1-9916-d74a5a322fec/garbage-collector/*.log
Service: support-tools
Source: support-tools
Status: OK
6 files tailed out of 6 files matching
Inputs:
/var/log/pods/cortex-cgroups-gc_k8s-gc-frlcz_2cba6d51-848c-42b1-9916-d74a5a322fec/garbage-collector/5.log
BytesRead: 416
Average Latency (ms): 0
24h Average Latency (ms): 0
Peak Latency (ms): 0
24h Peak Latency (ms): 0
Lines Combined: 0
MultiLine matches: 1
dev01-fcmi/cmi-repo-host-0/nss-wrapper-init
-------------------------------------------
- Type: file
Identifier: 671ddd494bb0d65c9be19124a9d31b8976f58e61d3cfbc406c0ce6c12e5bbe48
Path: /var/log/pods/dev01-fcmi_cmi-repo-host-0_502543d6-bf0a-4743-b2fc-bb68610c3a0f/nss-wrapper-init/*.log
Service: crunchy-pgbackrest
Source: crunchy-pgbackrest
Status: OK
6 files tailed out of 6 files matching
Inputs:
/var/log/pods/dev01-fcmi_cmi-repo-host-0_502543d6-bf0a-4743-b2fc-bb68610c3a0f/nss-wrapper-init/5.log
BytesRead: 0
Average Latency (ms): 0
24h Average Latency (ms): 0
Peak Latency (ms): 0
24h Peak Latency (ms): 0
dev02-capture/capture-db-capturedb-7266-0/pgbackrest-config
-----------------------------------------------------------
- Type: file
Identifier: d95c29b993e02617e1bdd5d8722b37f5d8f3edd14dad23a4437e0f98f6c7605e
Path: /var/log/pods/dev02-capture_capture-db-capturedb-7266-0_6bbdaf48-7991-41ad-818e-e3c26dc8aada/pgbackrest-config/*.log
Service: crunchy-pgbackrest
Source: crunchy-pgbackrest
Status: OK
6 files tailed out of 6 files matching
Inputs:
/var/log/pods/dev02-capture_capture-db-capturedb-7266-0_6bbdaf48-7991-41ad-818e-e3c26dc8aada/pgbackrest-config/5.log
BytesRead: 0
Average Latency (ms): 0
24h Average Latency (ms): 0
Peak Latency (ms): 0
24h Peak Latency (ms): 0
dev01-fairo/fairo-fairo-com-neu-function-0/pulsar-function
----------------------------------------------------------
- Type: file
Identifier: f51e9ac4dddb376b8ecfbd12713183e4a63a98f1cb8485093f5a2e1757128acf
Path: /var/log/pods/dev01-fairo_fairo-fairo-com-neu-function-0_39c49626-4cd5-4fa4-8061-6021d2cf2569/pulsar-function/*.log
Service: fairo-fairo-com-neu
Source: pulsar
Status: Pending
0 files tailed out of 6 files matching
BytesRead: 0
Average Latency (ms): 0
24h Average Latency (ms): 0
Peak Latency (ms): 0
24h Peak Latency (ms): 0
dev01-cashpresso/apollo-wlm-rescreening-function-0/pulsar-function
------------------------------------------------------------------
- Type: file
Identifier: 02c13c5230b1ad45a296d946d07bcb5c1a51f38fc0a53c4de7c3b934c92d2dfd
Path: /var/log/pods/dev01-cashpresso_apollo-wlm-rescreening-function-0_7182eafe-ce7f-4c93-a38b-53fba58bdd06/pulsar-function/*.log
Service: neuron-functions-python-runner
Source: neuron-functions-python-runner
Status: Pending
0 files tailed out of 6 files matching
BytesRead: 0
Average Latency (ms): 0
24h Average Latency (ms): 0
Peak Latency (ms): 0
24h Peak Latency (ms): 0
dev01-fairo/fairo-txn-risk-2-function-0/pulsar-function
-------------------------------------------------------
- Type: file
Identifier: 6ba8cfc29a07468c9629f1725ade747198dfd46f946cf8d6d4fe1e61c999f3f2
Path: /var/log/pods/dev01-fairo_fairo-txn-risk-2-function-0_c3abbe43-46d0-49ae-b43c-52e5d468ae98/pulsar-function/*.log
Service: fairo-txn-risk-2
Source: pulsar
Status: OK
6 files tailed out of 6 files matching
Inputs:
/var/log/pods/dev01-fairo_fairo-txn-risk-2-function-0_c3abbe43-46d0-49ae-b43c-52e5d468ae98/pulsar-function/5.log
BytesRead: 1668
Average Latency (ms): 0
24h Average Latency (ms): 0
Peak Latency (ms): 0
24h Peak Latency (ms): 0
dev01-fcmi/cmi-repo-host-0/pgbackrest-config
--------------------------------------------
- Type: file
Identifier: 08c0db18142ff8107a54097592fbc581e24ec7d93ae752c8d43c20c4a64b1856
Path: /var/log/pods/dev01-fcmi_cmi-repo-host-0_502543d6-bf0a-4743-b2fc-bb68610c3a0f/pgbackrest-config/*.log
Service: crunchy-pgbackrest
Source: crunchy-pgbackrest
Status: OK
6 files tailed out of 6 files matching
Inputs:
/var/log/pods/dev01-fcmi_cmi-repo-host-0_502543d6-bf0a-4743-b2fc-bb68610c3a0f/pgbackrest-config/5.log
BytesRead: 0
Average Latency (ms): 0
24h Average Latency (ms): 0
Peak Latency (ms): 0
24h Peak Latency (ms): 0
cortex-certmanager/cortex-certmanager-webhook-5dd6c5d77c-2j2cb/cert-manager
---------------------------------------------------------------------------
- Type: file
Identifier: 806ee255ccf09580aceaa4faa91e11eafffa8e3c0ba045484cbf8ec80b87de7e
Path: /var/log/pods/cortex-certmanager_cortex-certmanager-webhook-5dd6c5d77c-2j2cb_4f97561b-3080-48c1-b72d-dacfc5a874c7/cert-manager/*.log
Service: cert-manager-webhook
Source: cert-manager-webhook
Status: OK
6 files tailed out of 6 files matching
Inputs:
/var/log/pods/cortex-certmanager_cortex-certmanager-webhook-5dd6c5d77c-2j2cb_4f97561b-3080-48c1-b72d-dacfc5a874c7/cert-manager/5.log
BytesRead: 0
Average Latency (ms): 0
24h Average Latency (ms): 0
Peak Latency (ms): 0
24h Peak Latency (ms): 0
dev01-cashpresso/apollo-ctry-risk-2-function-0/pulsar-function
--------------------------------------------------------------
- Type: file
Identifier: ec143bd8b9488ce8854e93ed30c4b33237a7ddd9f4a4ff69d31764a04e7982cd
Path: /var/log/pods/dev01-cashpresso_apollo-ctry-risk-2-function-0_8bb0d034-f5d5-4c61-8d7b-4c89ed77931a/pulsar-function/*.log
Service: neuron-functions-python-runner
Source: neuron-functions-python-runner
Status: Pending
6 files tailed out of 6 files matching
BytesRead: 1281
Average Latency (ms): 0
24h Average Latency (ms): 0
Peak Latency (ms): 0
24h Peak Latency (ms): 0
Lines Combined: 0
MultiLine matches: 2
dev01-neuron-pulsar/dev01-neuron-pulsar-zookeeper-2/dev01-neuron-pulsar-zookeeper
---------------------------------------------------------------------------------
- Type: file
Identifier: 3f1256b907de21e9cbc3f849a246958834ca1a35c7dd24105faa1977a7146d67
Path: /var/log/pods/dev01-neuron-pulsar_dev01-neuron-pulsar-zookeeper-2_0f042e11-6194-471d-bcbd-1875560d4e50/dev01-neuron-pulsar-zookeeper/*.log
Service: pulsar-zookeeper
Source: pulsar
Status: OK
6 files tailed out of 6 files matching
Inputs:
/var/log/pods/dev01-neuron-pulsar_dev01-neuron-pulsar-zookeeper-2_0f042e11-6194-471d-bcbd-1875560d4e50/dev01-neuron-pulsar-zookeeper/5.log
BytesRead: 10640
Average Latency (ms): 0
24h Average Latency (ms): 0
Peak Latency (ms): 0
24h Peak Latency (ms): 0
Lines Combined: 0
MultiLine matches: 56
dev01-fcmi/cmi-repo-host-0/pgbackrest
-------------------------------------
- Type: file
Identifier: c28880cea1b45bc5f27c91f93d1fa8445437871be6cd7fe83c28b501bcbf04f8
Path: /var/log/pods/dev01-fcmi_cmi-repo-host-0_502543d6-bf0a-4743-b2fc-bb68610c3a0f/pgbackrest/*.log
Service: crunchy-pgbackrest
Source: crunchy-pgbackrest
Status: OK
6 files tailed out of 6 files matching
Inputs:
/var/log/pods/dev01-fcmi_cmi-repo-host-0_502543d6-bf0a-4743-b2fc-bb68610c3a0f/pgbackrest/5.log
BytesRead: 0
Average Latency (ms): 0
24h Average Latency (ms): 0
Peak Latency (ms): 0
24h Peak Latency (ms): 0
testkube/testkube-nats-box-6c6c5f8cc9-2sxhq/nats-box
----------------------------------------------------
- Type: file
Identifier: efd0e62297cb8dc5f34be9d29cdde561fe407b178d5c3511420016a3d249af1d
Path: /var/log/pods/testkube_testkube-nats-box-6c6c5f8cc9-2sxhq_69269f6c-785e-42fa-8bf5-0ddc48cee483/nats-box/*.log
Service: nats-box
Source: nats-box
Status: OK
6 files tailed out of 6 files matching
Inputs:
/var/log/pods/testkube_testkube-nats-box-6c6c5f8cc9-2sxhq_69269f6c-785e-42fa-8bf5-0ddc48cee483/nats-box/5.log
BytesRead: 0
Average Latency (ms): 0
24h Average Latency (ms): 0
Peak Latency (ms): 0
24h Peak Latency (ms): 0
ci-idc-24412-rmu-scenario-fairo/fairo-ctry-risk-2-function-0/pulsar-function
----------------------------------------------------------------------------
- Type: file
Identifier: 5bfc9e3ded90e02bd0c769e413faa3d610ff9e6bd56b37a2195bc0d3ff14c174
Path: /var/log/pods/ci-idc-24412-rmu-scenario-fairo_fairo-ctry-risk-2-function-0_218a91de-52ac-4504-9214-091b128a2be9/pulsar-function/*.log
Service: fairo-ctry-risk-2
Source: pulsar
Status: OK
6 files tailed out of 6 files matching
Inputs:
/var/log/pods/ci-idc-24412-rmu-scenario-fairo_fairo-ctry-risk-2-function-0_218a91de-52ac-4504-9214-091b128a2be9/pulsar-function/5.log
BytesRead: 1267
Average Latency (ms): 0
24h Average Latency (ms): 0
Peak Latency (ms): 0
24h Peak Latency (ms): 0
ci-idc-24412-int-scenario-fairo/fairo-cust-risk-2-function-0/pulsar-function
----------------------------------------------------------------------------
- Type: file
Identifier: 005e8a1469e81cfba3639234355e1171f50e04305c4c66b0505146f6434da3c4
Path: /var/log/pods/ci-idc-24412-int-scenario-fairo_fairo-cust-risk-2-function-0_71ee342c-241a-4a7a-bf1d-bd15dd9f6f82/pulsar-function/*.log
Service: fairo-cust-risk-2
Source: neuron-functions-python-runner
Status: Pending
0 files tailed out of 6 files matching
BytesRead: 0
Average Latency (ms): 0
24h Average Latency (ms): 0
Peak Latency (ms): 0
24h Peak Latency (ms): 0
ci-idc-24412-int-scenario-fairo/fairo-f-over-risk2-function-0/pulsar-function
-----------------------------------------------------------------------------
- Type: file
Identifier: 7409ec8c0d194ffe766e09bd65456af7dcfb8c71559eb5f62f7e05ef9b19a1b6
Path: /var/log/pods/ci-idc-24412-int-scenario-fairo_fairo-f-over-risk2-function-0_317282dc-cca4-41c5-bf1d-4e779eb4a05c/pulsar-function/*.log
Service: fairo-f-over-risk2
Source: neuron-functions-python-runner
Status: Pending
1 files tailed out of 6 files matching
BytesRead: 2056
Average Latency (ms): 0
24h Average Latency (ms): 0
Peak Latency (ms): 0
24h Peak Latency (ms): 0
Lines Combined: 0
MultiLine matches: 4
dev02-capture/capture-db-capturedb-7266-0/exporter
--------------------------------------------------
- Type: file
Identifier: b90463b05a10d1805982642100a16e994a8c6924db0b0b0dbe7dff9434ec4056
Path: /var/log/pods/dev02-capture_capture-db-capturedb-7266-0_6bbdaf48-7991-41ad-818e-e3c26dc8aada/exporter/*.log
Service: crunchy-postgres-exporter
Source: crunchy-postgres-exporter
Status: OK
0 files tailed out of 6 files matching
BytesRead: 2982
Average Latency (ms): 0
24h Average Latency (ms): 0
Peak Latency (ms): 0
24h Peak Latency (ms): 0
dev02-capture/capture-db-capturedb-7266-0/replication-cert-copy
---------------------------------------------------------------
- Type: file
Identifier: c2c5a270eca0b07c01b95eabcb4d34574e39558a2828a2e64ec4a957c284d21a
Path: /var/log/pods/dev02-capture_capture-db-capturedb-7266-0_6bbdaf48-7991-41ad-818e-e3c26dc8aada/replication-cert-copy/*.log
Service: crunchy-postgres
Source: crunchy-postgres
Status: OK
0 files tailed out of 6 files matching
BytesRead: 0
Average Latency (ms): 0
24h Average Latency (ms): 0
Peak Latency (ms): 0
24h Peak Latency (ms): 0
dev02-capture/capture-db-capturedb-7266-0/postgres-startup
----------------------------------------------------------
- Type: file
Identifier: bf656a14cf0e69f5d6603faf7616f2788002d029ab6836b3e74d45328c5ea6bc
Path: /var/log/pods/dev02-capture_capture-db-capturedb-7266-0_6bbdaf48-7991-41ad-818e-e3c26dc8aada/postgres-startup/*.log
Service: crunchy-postgres
Source: crunchy-postgres
Status: OK
0 files tailed out of 6 files matching
BytesRead: 0
Average Latency (ms): 0
24h Average Latency (ms): 0
Peak Latency (ms): 0
24h Peak Latency (ms): 0
dev02-capture/capture-db-capturedb-7266-0/database
--------------------------------------------------
- Type: file
Identifier: 895a03ff8586779652da58f9869b2422d5faf4607a03c1877dc2501fe8e5bc7b
Path: /var/log/pods/dev02-capture_capture-db-capturedb-7266-0_6bbdaf48-7991-41ad-818e-e3c26dc8aada/database/*.log
Service: crunchy-postgres
Source: crunchy-postgres
Status: OK
0 files tailed out of 6 files matching
BytesRead: 940
Average Latency (ms): 0
24h Average Latency (ms): 0
Peak Latency (ms): 0
24h Peak Latency (ms): 0
ci-idc-24412-int-scenario-fairo/mq-source-connector-source-0/pulsar-source
--------------------------------------------------------------------------
- Type: file
Identifier: 072dc49c40ce9e98e0f49531611a215d6d7c020f47aa7c65f3741f1099d27496
Path: /var/log/pods/ci-idc-24412-int-scenario-fairo_mq-source-connector-source-0_6411ac40-12e9-4c6a-b264-2dbe5c5b0bb8/pulsar-source/*.log
Service: sectormq
Source: pulsar-all
Status: OK
0 files tailed out of 11 files matching
BytesRead: 0
Average Latency (ms): 0
24h Average Latency (ms): 0
Peak Latency (ms): 0
24h Peak Latency (ms): 0
cortex-pipeline/cortex-community-cd-test-promotion-sensor-7nvgh-7bb59f5458sq2dr/main
------------------------------------------------------------------------------------
- Type: file
Identifier: 9c92cc7417e391249ab84e9f8c74e9c08d2ce872ed572887364764e60d6e85dd
Path: /var/log/pods/cortex-pipeline_cortex-community-cd-test-promotion-sensor-7nvgh-7bb59f5458sq2dr_15c2f509-4e2e-4bf9-93a9-23d8c1bfaaca/main/*.log
Service: argo-events
Source: argo-events
Status: OK
6 files tailed out of 6 files matching
Inputs:
/var/log/pods/cortex-pipeline_cortex-community-cd-test-promotion-sensor-7nvgh-7bb59f5458sq2dr_15c2f509-4e2e-4bf9-93a9-23d8c1bfaaca/main/5.log
BytesRead: 19401
Average Latency (ms): 0
24h Average Latency (ms): 0
Peak Latency (ms): 0
24h Peak Latency (ms): 0
dev01-cashpresso/cashpresso-cre-2-out-function-0/pulsar-function
----------------------------------------------------------------
- Type: file
Identifier: d07114d4de8e69336e35e6fa43d76aeb87ef6b3eb81923bf2a6e09a969bdd751
Path: /var/log/pods/dev01-cashpresso_cashpresso-cre-2-out-function-0_74e6f3b0-d9fc-4941-92f0-3c452753889a/pulsar-function/*.log
Service: neuron-functions-python-runner
Source: neuron-functions-python-runner
Status: Pending
6 files tailed out of 6 files matching
BytesRead: 904
Average Latency (ms): 0
24h Average Latency (ms): 0
Peak Latency (ms): 0
24h Peak Latency (ms): 0
Lines Combined: 0
MultiLine matches: 1
dev01-fairo/fairo-wlm-rep-function-0/pulsar-function
----------------------------------------------------
- Type: file
Identifier: a625c3208d9ebbb73a57b81a36307ca3fcd548a029957465313cceb57cbae86f
Path: /var/log/pods/dev01-fairo_fairo-wlm-rep-function-0_d2df7c98-605e-4bb0-bc5f-244fc797b5d7/pulsar-function/*.log
Service: fairo-wlm-rep
Source: pulsar
Status: Pending
6 files tailed out of 6 files matching
BytesRead: 1651
Average Latency (ms): 0
24h Average Latency (ms): 0
Peak Latency (ms): 0
24h Peak Latency (ms): 0
Lines Combined: 0
MultiLine matches: 3
testkube/testkube-api-server-7fcbf975cf-ckwlz/testkube-api
----------------------------------------------------------
- Type: file
Identifier: 230536765c7025e303e7b856ba6e7d59c614de4e92b8474a757ca0b4bde3c3bd
Path: /var/log/pods/testkube_testkube-api-server-7fcbf975cf-ckwlz_3309bcc7-2698-4f86-a55b-5fa019512cea/testkube-api/*.log
Service: testkube-api-server
Source: testkube-api-server
Status: OK
0 files tailed out of 6 files matching
BytesRead: 0
Average Latency (ms): 0
24h Average Latency (ms): 0
Peak Latency (ms): 0
24h Peak Latency (ms): 0
dev01-neuron-pulsar/dev01-neuron-pulsar-bookie-1/dev01-neuron-pulsar-bookie
---------------------------------------------------------------------------
- Type: file
Identifier: 3e0a0dc809346c7093b431325e6f0406a21c859c76d5170cc777b7644c5c629a
Path: /var/log/pods/dev01-neuron-pulsar_dev01-neuron-pulsar-bookie-1_124fc7cf-e13f-4f25-989a-866b7c653e97/dev01-neuron-pulsar-bookie/*.log
Service: pulsar-bookie
Source: pulsar
Status: OK
6 files tailed out of 6 files matching
Inputs:
/var/log/pods/dev01-neuron-pulsar_dev01-neuron-pulsar-bookie-1_124fc7cf-e13f-4f25-989a-866b7c653e97/dev01-neuron-pulsar-bookie/5.log
BytesRead: 4.520116e+06
Average Latency (ms): 0
24h Average Latency (ms): 0
Peak Latency (ms): 19
24h Peak Latency (ms): 19
Lines Combined: 62
MultiLine matches: 21331
ci-idc-24412-int-scenario-fairo/bae-mx-transformer-function-0/pulsar-function
-----------------------------------------------------------------------------
- Type: file
Identifier: 235862f05abd4d497ad8c6925a256be04c68226ffb8223e97604a4134118b5ad
Path: /var/log/pods/ci-idc-24412-int-scenario-fairo_bae-mx-transformer-function-0_376dd1f2-5eea-4715-bb32-4eff4a0e25c2/pulsar-function/*.log
Service: bae-mx-transformer
Source: neuron-functions-python-runner
Status: Pending
6 files tailed out of 6 files matching
BytesRead: 1301
Average Latency (ms): 0
24h Average Latency (ms): 0
Peak Latency (ms): 0
24h Peak Latency (ms): 0
Lines Combined: 0
MultiLine matches: 2
dev01-neuron-filestreamer/eventbus-default-stan-0/stan
------------------------------------------------------
- Type: file
Identifier: d71f19f6ad8239f8b629f93cb58b2ac290b8c3fd073c17f2f63e931d9fcb177d
Path: /var/log/pods/dev01-neuron-filestreamer_eventbus-default-stan-0_2e497708-ee8a-43b3-aaaf-dfbe98565003/stan/*.log
Service: nats-streaming
Source: nats-streaming
Status: Pending
6 files tailed out of 6 files matching
BytesRead: 0
Average Latency (ms): 0
24h Average Latency (ms): 0
Peak Latency (ms): 0
24h Peak Latency (ms): 0
ci-idc-24412-ftr11-improv-fairo/fairo-f-frc-crs-function-0/pulsar-function
--------------------------------------------------------------------------
- Type: file
Identifier: 3cb86d02edca9d8ff0c8020866fbb9842885e32b86b38678b79c0338d3adab4c
Path: /var/log/pods/ci-idc-24412-ftr11-improv-fairo_fairo-f-frc-crs-function-0_28b80eb1-5222-47aa-a482-9da9ded42964/pulsar-function/*.log
Service: fairo-f-frc-crs
Source: pulsar
Status: OK
6 files tailed out of 6 files matching
Inputs:
/var/log/pods/ci-idc-24412-ftr11-improv-fairo_fairo-f-frc-crs-function-0_28b80eb1-5222-47aa-a482-9da9ded42964/pulsar-function/5.log
BytesRead: 1642
Average Latency (ms): 0
24h Average Latency (ms): 0
Peak Latency (ms): 0
24h Peak Latency (ms): 0
dev01-cashpresso/cashpresso-f-wlm-split-function-0/pulsar-function
------------------------------------------------------------------
- Type: file
Identifier: 9b919f96638a08c278f706978c3363d4df8febe80f6f1ecf542f882af4e0c526
Path: /var/log/pods/dev01-cashpresso_cashpresso-f-wlm-split-function-0_89f2f691-5fc4-41fa-be22-9af92295e2d9/pulsar-function/*.log
Service: neuron-functions-python-runner
Source: neuron-functions-python-runner
Status: OK
6 files tailed out of 6 files matching
Inputs:
/var/log/pods/dev01-cashpresso_cashpresso-f-wlm-split-function-0_89f2f691-5fc4-41fa-be22-9af92295e2d9/pulsar-function/5.log
BytesRead: 907
Average Latency (ms): 0
24h Average Latency (ms): 0
Peak Latency (ms): 0
24h Peak Latency (ms): 0
=============
Process Agent
=============
Version: 7.41.1
Status date: 2023-01-17 14:57:25.797 UTC (1673967445797)
Process Agent Start: 2023-01-17 14:42:15.458 UTC (1673966535458)
Pid: 5479
Go Version: go1.18.9
Build arch: amd64
Log Level: WARN
Enabled Checks: [container rtcontainer process_discovery pod]
Allocated Memory: 36,146,592 bytes
Hostname: i-03dcdc76f8df26cd5
=================
Process Endpoints
=================
https://process.datadoghq.eu - API Key ending with:
- f7da4
=========
Collector
=========
Last collection time: 2023-01-17 14:57:23
Docker socket:
Number of processes: 0
Number of containers: 83
Process Queue length: 0
RTProcess Queue length: 0
Connections Queue length: 0
Event Queue length: 0
Pod Queue length: 0
Process Bytes enqueued: 0
RTProcess Bytes enqueued: 0
Connections Bytes enqueued: 0
Event Bytes enqueued: 0
Pod Bytes enqueued: 0
Drop Check Payloads: []
=========
APM Agent
=========
Status: Not running or unreachable on localhost:8126.
Error: Get "http://localhost:8126/debug/vars": dial tcp [::1]:8126: connect: connection refused
==========
Aggregator
==========
Checks Metric Sample: 630,343
Dogstatsd Metric Sample: 2,402
Event: 1
Events Flushed: 1
Number Of Flushes: 60
Series Flushed: 594,750
Service Check: 2,602
Service Checks Flushed: 2,644
=========
DogStatsD
=========
Event Packets: 0
Event Parse Errors: 0
Metric Packets: 2,401
Metric Parse Errors: 0
Service Check Packets: 0
Service Check Parse Errors: 0
Udp Bytes: 257,200
Udp Packet Reading Errors: 0
Udp Packets: 1,240
Uds Bytes: 0
Uds Origin Detection Errors: 0
Uds Packet Reading Errors: 0
Uds Packets: 1
Unterminated Metric Errors: 0
=====================
Datadog Cluster Agent
=====================
- Datadog Cluster Agent endpoint detected: https://172.30.143.253:5005
Successfully connected to the Datadog Cluster Agent.
- Running: 7.41.1+commit.4f39b9e
=============
Autodiscovery
=============
Enabled Features
================
cri
kubernetes
Configuration Errors
====================
ci-idc-24412-int-scenario-fairo/mq-source-connector-source-0 (6411ac40-12e9-4c6a-b264-2dbe5c5b0bb8)
---------------------------------------------------------------------------------------------------
annotation ad.datadoghq.com/pulsarfunction.logs is invalid: pulsarfunction doesn't match a container identifier [pulsar-source]
====
OTLP
====
Status: Not enabled
Collector status: Not running
Agent Environment Agent 7.41.1 - Commit: 4f39b9e - Serialization version: v5.0.39 - Go version: go1.18.9
Describe what happened: There is a discrepancy between count of files actually being tailed by Datadog and what is being reported.
First log of each container is 0.log in /var/log/pods//. When kubelet is terminated in an unclean way, e.g. by hard-rebooting the worker node, this container termination is handled like restart (especially in case of statefulsets) and new log is created in the same folder. Due to unclean kubelet/node termination, 0.log is not deleted and new container is logging to 1.log. So far so good, this is standard kubelet behavior.
After hard restarting the node 4 times, we end up with 5 log files named 0.log to 4.log in the same folder. Now I would expect Datadog to tail only one file -> 4.log, since it is the only file that is actually in use by live container, since previous 4 containers no longer exist.
Example container:
When we check file descriptors, we see that only one file, the correct one, is actually opened/presumably tailed:
However the "agent status" output is reporting that 5 files out of 5 are being tailed, which is not true. This has one big implication -> sum of these values is compared against open_files_limit and it doesn't take me much to reach limit of 200 with just couple hard restarts if I run enough pods . This is mainly causing issues on development clusters which are shut-down (gracefuly) for weekends, but after some timeout node is shut down regardless.
Deployment pod names get changed on every upgrade, so even if there are some leftover logs, Datadog ignores them eventually. But with statefulsets, these leftover logs can accumulate in the same directory and with the behavior described above, we end up with Datadog reporting "taling" 30 log files, from which some are 6 months old, while in reality it's only tailing one file.
Once we hit open_file_limit, with each restart less and less files are actually being tailed. Only solution is manual cleanup of /var/log/pods.
Example (/var/log/pods is cleaned beforehand): Startup: Reported as tailed by Datadog: 56 Open FDs: 56 Actual files on filesystem: 101
Restart 1: Reported as tailed by Datadog: 113 Open FDs: 56 Actual files on filesystem: 203
Restart 2: Reported as tailed by Datadog: 170 Open FDs: 56 Actual files on filesystem: 305
Restart 3: Reported as tailed by Datadog: 200 Open FDs: 50 Actual files on filesystem: 407
Restart 4: Reported as tailed by Datadog: 200 Open FDs: 39 Actual files on filesystem: 510
Restart 5: Reported as tailed by Datadog: 200 Open FDs: 34 Actual files on filesystem: 612
Complete status after Restart 5:
File descriptors:
Describe what you expected: In this case I would expect "agent" status to report correct number of files tailed.
Steps to reproduce the issue: Hard restart k8s worker node
Additional environment details (Operating System, Cloud provider, etc): Running Fedora on Amazon EC2 Running OKD version 4.11.0-0.okd-2022-12-02-145640