search

DataDog / datadog-agent

Main repository for Datadog Agent
https://docs.datadoghq.com/
Apache License 2.0
2.87k stars 1.21k forks source link

[BUG] Tailed log files not counted correctly #15110

Closed davtex closed 1 year ago

davtex commented 1 year ago

Agent Environment Agent 7.41.1 - Commit: 4f39b9e - Serialization version: v5.0.39 - Go version: go1.18.9

Describe what happened: There is a discrepancy between count of files actually being tailed by Datadog and what is being reported.

First log of each container is 0.log in /var/log/pods//. When kubelet is terminated in an unclean way, e.g. by hard-rebooting the worker node, this container termination is handled like restart (especially in case of statefulsets) and new log is created in the same folder. Due to unclean kubelet/node termination, 0.log is not deleted and new container is logging to 1.log. So far so good, this is standard kubelet behavior.

After hard restarting the node 4 times, we end up with 5 log files named 0.log to 4.log in the same folder. Now I would expect Datadog to tail only one file -> 4.log, since it is the only file that is actually in use by live container, since previous 4 containers no longer exist.

Example container:

  dev01-fcmi/cmi-repo-host-0/pgbackrest
  -------------------------------------
    - Type: file
      Identifier: ce025fb0a1db8c11c30a41838f2e531cd850a2d06446219b597d2be99f74bd21
      Path: /var/log/pods/dev01-fcmi_cmi-repo-host-0_502543d6-bf0a-4743-b2fc-bb68610c3a0f/pgbackrest/*.log
      Service: crunchy-pgbackrest
      Source: crunchy-pgbackrest
      Status: OK
        5 files tailed out of 5 files matching
      Inputs:
        /var/log/pods/dev01-fcmi_cmi-repo-host-0_502543d6-bf0a-4743-b2fc-bb68610c3a0f/pgbackrest/4.log
      BytesRead: 0
      Average Latency (ms): 0
      24h Average Latency (ms): 0
      Peak Latency (ms): 0
      24h Peak Latency (ms): 0

When we check file descriptors, we see that only one file, the correct one, is actually opened/presumably tailed:

root@ip-10-223-124-76:/proc/5424/fd# ls -l | grep dev01-fcmi_cmi-repo-host-0_502543d6-bf0a-4743-b2fc-bb68610c3a0f/pgbackrest/
lr-x------. 1 root root 64 Jan 17 12:57 41 -> /var/log/pods/dev01-fcmi_cmi-repo-host-0_502543d6-bf0a-4743-b2fc-bb68610c3a0f/pgbackrest/4.log

However the "agent status" output is reporting that 5 files out of 5 are being tailed, which is not true. This has one big implication -> sum of these values is compared against open_files_limit and it doesn't take me much to reach limit of 200 with just couple hard restarts if I run enough pods . This is mainly causing issues on development clusters which are shut-down (gracefuly) for weekends, but after some timeout node is shut down regardless.

Deployment pod names get changed on every upgrade, so even if there are some leftover logs, Datadog ignores them eventually. But with statefulsets, these leftover logs can accumulate in the same directory and with the behavior described above, we end up with Datadog reporting "taling" 30 log files, from which some are 6 months old, while in reality it's only tailing one file.

Once we hit open_file_limit, with each restart less and less files are actually being tailed. Only solution is manual cleanup of /var/log/pods.

Example (/var/log/pods is cleaned beforehand): Startup: Reported as tailed by Datadog: 56 Open FDs: 56 Actual files on filesystem: 101

Restart 1: Reported as tailed by Datadog: 113 Open FDs: 56 Actual files on filesystem: 203

Restart 2: Reported as tailed by Datadog: 170 Open FDs: 56 Actual files on filesystem: 305

Restart 3: Reported as tailed by Datadog: 200 Open FDs: 50 Actual files on filesystem: 407

Restart 4: Reported as tailed by Datadog: 200 Open FDs: 39 Actual files on filesystem: 510

Restart 5: Reported as tailed by Datadog: 200 Open FDs: 34 Actual files on filesystem: 612

Complete status after Restart 5:

root@ip-10-223-124-76:/proc/5380/fd# agent status
Getting the status from the agent.

2023-01-17 14:57:25 UTC | CORE | WARN | (pkg/util/log/log.go:618 in func1) | Deactivating Autoconfig will disable most components. It's recommended to use autoconfig_exclude_features and autoconfig_include_features to activate/deactivate features selectively

===============
Agent (v7.41.1)
===============

  Status date: 2023-01-17 14:57:25.785 UTC (1673967445785)
  Agent start: 2023-01-17 14:42:14.652 UTC (1673966534652)
  Pid: 5380
  Go Version: go1.18.9
  Python Version: 3.8.14
  Build arch: amd64
  Agent flavor: agent
  Check Runners: 4
  Log Level: WARN

  Paths
  =====
    Config File: /etc/datadog-agent/datadog.yaml
    conf.d: /etc/datadog-agent/conf.d
    checks.d: /etc/datadog-agent/checks.d

  Clocks
  ======
    NTP offset: -190µs
    System time: 2023-01-17 14:57:25.785 UTC (1673967445785)

  Host Info
  =========
    bootTime: 2023-01-17 14:41:07 UTC (1673966467000)
    hostId: ec28db03-5c7a-ef19-1335-d09190515d6e
    kernelArch: x86_64
    kernelVersion: 6.0.10-200.fc36.x86_64
    os: linux
    platform: fedora
    platformFamily: fedora
    platformVersion: 36 (CoreOS)
    procs: 383
    uptime: 1m16s

  Hostnames
  =========
    cluster-name: cortex-mgmt01
    ec2-hostname: ip-10-223-124-76.eu-central-1.compute.internal
    host_aliases: [i-03dcdc76f8df26cd5 ip-10-223-124-76.eu-central-1.compute.internal-cortex-mgmt01]
    hostname: i-03dcdc76f8df26cd5
    instance-id: i-03dcdc76f8df26cd5
    socket-fqdn: 10-223-124-76.ovn-kubernetes-node.openshift-ovn-kubernetes.svc.cluster.local.
    socket-hostname: ip-10-223-124-76
    host tags:
      cluster_name:cortex-mgmt01
      kube_cluster_name:cortex-mgmt01
      kube_node_role:worker
    hostname provider: aws
    unused hostname providers:
      'hostname' configuration/environment: hostname is empty
      'hostname_file' configuration/environment: 'hostname_file' configuration is not enabled
      azure: azure_hostname_style is set to 'os'
      fargate: agent is not runnning on Fargate
      fqdn: FQDN hostname is not usable
      gce: unable to retrieve hostname from GCE: cloud provider is disabled by configuration
      os: OS hostname is not usable

  Metadata
  ========
    agent_version: 7.41.1
    cloud_provider: AWS
    config_apm_dd_url: 
    config_dd_url: 
    config_logs_dd_url: 
    config_logs_socks5_proxy_address: 
    config_no_proxy: []
    config_process_dd_url: 
    config_proxy_http: 
    config_proxy_https: 
    config_site: 
    feature_apm_enabled: false
    feature_cspm_enabled: false
    feature_cws_enabled: false
    feature_logs_enabled: true
    feature_networks_enabled: false
    feature_networks_http_enabled: false
    feature_networks_https_enabled: false
    feature_otlp_enabled: false
    feature_process_enabled: false
    feature_processes_container_enabled: true
    flavor: agent
    hostname_source: aws
    install_method_installer_version: datadog-3.7.2
    install_method_tool: helm
    install_method_tool_version: Helm
    logs_transport: HTTP

========
JMXFetch
========

  Information
  ==================
  Initialized checks
  ==================
    no checks

  Failed checks
  =============
    no checks

=========
Forwarder
=========

  Transactions
  ============
    Cluster: 0
    ClusterRole: 0
    ClusterRoleBinding: 0
    CronJob: 0
    DaemonSet: 0
    Deployment: 0
    Dropped: 0
    HighPriorityQueueFull: 0
    Ingress: 0
    Job: 0
    Namespace: 0
    Node: 0
    PersistentVolume: 0
    PersistentVolumeClaim: 0
    Pod: 0
    ReplicaSet: 0
    Requeued: 0
    Retried: 0
    RetryQueueSize: 0
    Role: 0
    RoleBinding: 0
    Service: 0
    ServiceAccount: 0
    StatefulSet: 0

  Transaction Successes
  =====================
    Total number: 187
    Successes By Endpoint:
      check_run_v1: 60
      intake: 6
      metadata_v1: 2
      series_v2: 119

  On-disk storage
  ===============
    On-disk storage is disabled. Configure `forwarder_storage_max_size_in_bytes` to enable it.

  API Keys status
  ===============
    API key ending with f7da4: API Key valid

==========
Endpoints
==========
  https://app.datadoghq.eu - API Key ending with:
      - f7da4

==========
Logs Agent
==========
    Reliable: Sending compressed logs in HTTPS to agent-http-intake.logs.datadoghq.eu on port 443
    BytesSent: 3.6119746e+07
    EncodedBytesSent: 971926
    LogsProcessed: 24905
    LogsSent: 24738

  Warnings
  ========
    The limit on the maximum number of files in use (200) has been reached. Increase this limit (thanks to the attribute logs_config.open_files_limit in datadog.yaml) or decrease the number of tailed file.

  dev01-cashpresso/cashpresso-f-prod-risk-function-0/pulsar-function
  ------------------------------------------------------------------
    - Type: file
      Identifier: 229a3b552c988142c7614f4ad2d9b559646f6614b8350a284ccd6a6eeab9c32b
      Path: /var/log/pods/dev01-cashpresso_cashpresso-f-prod-risk-function-0_dc1b2366-57cb-4c4f-81eb-3db403192466/pulsar-function/*.log
      Service: neuron-functions-python-runner
      Source: neuron-functions-python-runner
      Status: Pending
        6 files tailed out of 6 files matching
      BytesRead: 915
      Average Latency (ms): 0
      24h Average Latency (ms): 0
      Peak Latency (ms): 0
      24h Peak Latency (ms): 0  
      Lines Combined: 0   
      MultiLine matches: 1 

  dev01-argos/argos-neuron-connector-78cd5574c-86lqj/argos-neuron-connector
  -------------------------------------------------------------------------
    - Type: file
      Identifier: d9fcadfd38756300c9fd609ef76c0865393b0950dfd1ce527255b5026a044e44
      Path: /var/log/pods/dev01-argos_argos-neuron-connector-78cd5574c-86lqj_6e552843-16bf-4a78-b294-0689e8c1e086/argos-neuron-connector/*.log
      Service: adc-argos-neuron-pulse-application
      Source: adc-argos-neuron-pulse-application
      Status: Pending
        0 files tailed out of 6 files matching
      BytesRead: 0
      Average Latency (ms): 0
      24h Average Latency (ms): 0
      Peak Latency (ms): 0
      24h Peak Latency (ms): 0

  dev01-neuron-pulsar/dev01-neuron-pulsar-bookie-1/pulsar-bookkeeper-verify-clusterid
  -----------------------------------------------------------------------------------
    - Type: file
      Identifier: da2956ccf4114f095796cc21b51fcc672101457191aae21e308578e4fc841817
      Path: /var/log/pods/dev01-neuron-pulsar_dev01-neuron-pulsar-bookie-1_124fc7cf-e13f-4f25-989a-866b7c653e97/pulsar-bookkeeper-verify-clusterid/*.log
      Service: pulsar-bookie
      Source: neuron-pulsar-all
      Status: OK
        0 files tailed out of 6 files matching
      BytesRead: 0
      Average Latency (ms): 0
      24h Average Latency (ms): 0
      Peak Latency (ms): 0
      24h Peak Latency (ms): 0

  dev01-capture/pgo-56d4d654c9-p4zg2/operator
  -------------------------------------------
    - Type: file
      Identifier: 00d3483c950786221278629cf411018141903315ec3c91d7a85860f686aea6ef
      Path: /var/log/pods/dev01-capture_pgo-56d4d654c9-p4zg2_b2787835-4d06-46f3-8b6a-9c225e7d77d0/operator/*.log
      Service: postgres-operator
      Source: postgres-operator
      Status: OK
        6 files tailed out of 6 files matching
      Inputs:
        /var/log/pods/dev01-capture_pgo-56d4d654c9-p4zg2_b2787835-4d06-46f3-8b6a-9c225e7d77d0/operator/5.log
      BytesRead: 10637
      Average Latency (ms): 0
      24h Average Latency (ms): 0
      Peak Latency (ms): 0
      24h Peak Latency (ms): 0

  cortex-argo/controller-manager-867fbd77b4-8j846/controller-manager
  ------------------------------------------------------------------
    - Type: file
      Identifier: 059db3f6754548d50c9c2776b140c18bfab3ecf9d32a81e54e77f3f665e7ca92
      Path: /var/log/pods/cortex-argo_controller-manager-867fbd77b4-8j846_5f99c975-edff-4a9b-8cb1-4289bc8053fd/controller-manager/*.log
      Service: argo-events
      Source: argo-events
      Status: Pending
        7 files tailed out of 7 files matching
      BytesRead: 0
      Average Latency (ms): 0
      24h Average Latency (ms): 0
      Peak Latency (ms): 0
      24h Peak Latency (ms): 0

  ci-idc-24412-ftr11-improv-fairo/abc-calcproductrisk-function-0/pulsar-function
  ------------------------------------------------------------------------------
    - Type: file
      Identifier: 2c7ae22a37785e18e76a8648da684569fda2ef13d412d9ee038bd25fd8d04698
      Path: /var/log/pods/ci-idc-24412-ftr11-improv-fairo_abc-calcproductrisk-function-0_8b25b43f-371c-4e02-b466-6109ad5d5544/pulsar-function/*.log
      Service: abc-calcproductrisk
      Source: pulsar
      Status: Pending
        0 files tailed out of 6 files matching
      BytesRead: 0
      Average Latency (ms): 0
      24h Average Latency (ms): 0
      Peak Latency (ms): 0
      24h Peak Latency (ms): 0

  dev01-fairo/fairo-cust-risk-2-function-0/pulsar-function
  --------------------------------------------------------
    - Type: file
      Identifier: 22fbb756bb1910364e04d4d92ce0a6dea2c6d3842271f358496b62876721afdd
      Path: /var/log/pods/dev01-fairo_fairo-cust-risk-2-function-0_a8d55a36-d2cc-45a1-bdab-9c4a6b98b2a6/pulsar-function/*.log
      Service: fairo-cust-risk-2
      Source: pulsar
      Status: Pending
        0 files tailed out of 6 files matching
      BytesRead: 0
      Average Latency (ms): 0
      24h Average Latency (ms): 0
      Peak Latency (ms): 0
      24h Peak Latency (ms): 0

  dev01-fcmi/cmi-repo-host-0/pgbackrest-log-dir
  ---------------------------------------------
    - Type: file
      Identifier: 433215f5ac4d60988fd30cd359dc5f654eae4bf6ef77c7adf7a81ad63d85442f
      Path: /var/log/pods/dev01-fcmi_cmi-repo-host-0_502543d6-bf0a-4743-b2fc-bb68610c3a0f/pgbackrest-log-dir/*.log
      Service: crunchy-pgbackrest
      Source: crunchy-pgbackrest
      Status: OK
        6 files tailed out of 6 files matching
      Inputs:
        /var/log/pods/dev01-fcmi_cmi-repo-host-0_502543d6-bf0a-4743-b2fc-bb68610c3a0f/pgbackrest-log-dir/5.log
      BytesRead: 0
      Average Latency (ms): 0
      24h Average Latency (ms): 0
      Peak Latency (ms): 0
      24h Peak Latency (ms): 0

  ci-idc-24412-int-scenario-fairo/abc-calccountryrisk-function-0/pulsar-function
  ------------------------------------------------------------------------------
    - Type: file
      Identifier: 2936c6e060781c4d270bc5c72aa2b5677dab9b0c37626e813e26cfd8c4ac3cc9
      Path: /var/log/pods/ci-idc-24412-int-scenario-fairo_abc-calccountryrisk-function-0_b0d148fb-ee9e-4c2a-967b-fd8cf8cec552/pulsar-function/*.log
      Service: abc-calccountryrisk
      Source: neuron-functions-python-runner
      Status: Pending
        6 files tailed out of 6 files matching
      BytesRead: 2052
      Average Latency (ms): 0
      24h Average Latency (ms): 0
      Peak Latency (ms): 0
      24h Peak Latency (ms): 0  
      Lines Combined: 0   
      MultiLine matches: 4 

  ci-idc-24412-rmu-scenario-fairo/fairo-f-wlm-split-function-0/pulsar-function
  ----------------------------------------------------------------------------
    - Type: file
      Identifier: c6531f9c387ff4ad3e88549a8db029f923752a98390a82080ffa4680e3fc0e06
      Path: /var/log/pods/ci-idc-24412-rmu-scenario-fairo_fairo-f-wlm-split-function-0_da876b26-1fd0-47f4-bb4d-c1afd2f77bb1/pulsar-function/*.log
      Service: fairo-f-wlm-split
      Source: pulsar
      Status: Pending
        6 files tailed out of 6 files matching
      BytesRead: 887
      Average Latency (ms): 0
      24h Average Latency (ms): 0
      Peak Latency (ms): 0
      24h Peak Latency (ms): 0  
      Lines Combined: 0   
      MultiLine matches: 1 

  dev01-fairo/abc-minerva-com-function-0/pulsar-function
  ------------------------------------------------------
    - Type: file
      Identifier: c92842c4fd9d9bf24874870dad22c9d4926fe7cd91a90fae638fc3d40f4e26fc
      Path: /var/log/pods/dev01-fairo_abc-minerva-com-function-0_ff637a00-6418-4da9-ab8b-15124d4ccf9c/pulsar-function/*.log
      Service: abc-minerva-com
      Source: pulsar
      Status: Pending
        0 files tailed out of 6 files matching
      BytesRead: 0
      Average Latency (ms): 0
      24h Average Latency (ms): 0
      Peak Latency (ms): 0
      24h Peak Latency (ms): 0

  cortex-pipeline/cortex-release-pipeline-sensor-dfs7q-6f9867b5d-gkfhv/main
  -------------------------------------------------------------------------
    - Type: file
      Identifier: 22f6deeda22a5af9234eb39786754b664d92691253c56ba2e4879042aba36f93
      Path: /var/log/pods/cortex-pipeline_cortex-release-pipeline-sensor-dfs7q-6f9867b5d-gkfhv_9bab9b60-1340-453d-98de-4770746e85d8/main/*.log
      Service: argo-events
      Source: argo-events
      Status: OK
        6 files tailed out of 6 files matching
      Inputs:
        /var/log/pods/cortex-pipeline_cortex-release-pipeline-sensor-dfs7q-6f9867b5d-gkfhv_9bab9b60-1340-453d-98de-4770746e85d8/main/5.log
      BytesRead: 8144
      Average Latency (ms): 0
      24h Average Latency (ms): 0
      Peak Latency (ms): 0
      24h Peak Latency (ms): 0

  dev02-capture/capture-db-capturedb-7266-0/nss-wrapper-init
  ----------------------------------------------------------
    - Type: file
      Identifier: 77b68b85ba0a224520b012bce4ea401eeb64c93242d8a692718c7daacd81cb13
      Path: /var/log/pods/dev02-capture_capture-db-capturedb-7266-0_6bbdaf48-7991-41ad-818e-e3c26dc8aada/nss-wrapper-init/*.log
      Service: crunchy-postgres
      Source: crunchy-postgres
      Status: OK
        6 files tailed out of 6 files matching
      Inputs:
        /var/log/pods/dev02-capture_capture-db-capturedb-7266-0_6bbdaf48-7991-41ad-818e-e3c26dc8aada/nss-wrapper-init/5.log
      BytesRead: 0
      Average Latency (ms): 0
      24h Average Latency (ms): 0
      Peak Latency (ms): 0
      24h Peak Latency (ms): 0

  dev01-neuron-pulsar/dev01-neuron-pulsar-proxy-1/dev01-neuron-pulsar-proxy
  -------------------------------------------------------------------------
    - Type: file
      Identifier: fb7e4f0a10b276f54dcdf1c5e28bc950f183c0192ada8f652834573f86a84bfe
      Path: /var/log/pods/dev01-neuron-pulsar_dev01-neuron-pulsar-proxy-1_d330a865-5000-4978-9515-af4fbe4e737e/dev01-neuron-pulsar-proxy/*.log
      Service: pulsar-proxy
      Source: pulsar
      Status: OK
        6 files tailed out of 6 files matching
      Inputs:
        /var/log/pods/dev01-neuron-pulsar_dev01-neuron-pulsar-proxy-1_d330a865-5000-4978-9515-af4fbe4e737e/dev01-neuron-pulsar-proxy/5.log
      BytesRead: 187348
      Average Latency (ms): 0
      24h Average Latency (ms): 0
      Peak Latency (ms): 25
      24h Peak Latency (ms): 25  
      Lines Combined: 0   
      MultiLine matches: 893 

  dev02-capture/capture-db-capturedb-7266-0/pgbackrest
  ----------------------------------------------------
    - Type: file
      Identifier: ebe46ff18c20aa03e445caeb6d1e3ae226f69c714197ea895d899d37fe50553c
      Path: /var/log/pods/dev02-capture_capture-db-capturedb-7266-0_6bbdaf48-7991-41ad-818e-e3c26dc8aada/pgbackrest/*.log
      Service: crunchy-pgbackrest
      Source: crunchy-pgbackrest
      Status: OK
        0 files tailed out of 6 files matching
      BytesRead: 0
      Average Latency (ms): 0
      24h Average Latency (ms): 0
      Peak Latency (ms): 0
      24h Peak Latency (ms): 0

  dev01-neuron-pulsar/dev01-neuron-pulsar-proxy-1/wait-zookeeper-ready
  --------------------------------------------------------------------
    - Type: file
      Identifier: 1a31db4fc7e4f9d26e43b5c9756aece1d94e315837fdc969d6ce732ae815365a
      Path: /var/log/pods/dev01-neuron-pulsar_dev01-neuron-pulsar-proxy-1_d330a865-5000-4978-9515-af4fbe4e737e/wait-zookeeper-ready/*.log
      Service: pulsar-proxy
      Source: neuron-pulsar-all
      Status: OK
        0 files tailed out of 6 files matching
      BytesRead: 0
      Average Latency (ms): 0
      24h Average Latency (ms): 0
      Peak Latency (ms): 0
      24h Peak Latency (ms): 0

  ci-idc-24412-rmu-scenario-fairo/abc-calcproductrisk-function-0/pulsar-function
  ------------------------------------------------------------------------------
    - Type: file
      Identifier: 7a19e1f0439b0099cd79a8ee684b6903ff917b14c24820890a2a9906df8ee209
      Path: /var/log/pods/ci-idc-24412-rmu-scenario-fairo_abc-calcproductrisk-function-0_6bd79182-91a9-4baf-9ce9-8e97d19b2bbe/pulsar-function/*.log
      Service: abc-calcproductrisk
      Source: pulsar
      Status: Pending
        6 files tailed out of 6 files matching
      BytesRead: 2059
      Average Latency (ms): 0
      24h Average Latency (ms): 0
      Peak Latency (ms): 0
      24h Peak Latency (ms): 0  
      Lines Combined: 0   
      MultiLine matches: 4 

  ci-idc-24412-rmu-scenario-fairo/fairo-f-lookup-split-function-0/pulsar-function
  -------------------------------------------------------------------------------
    - Type: file
      Identifier: b70bda1bd670d113fbacb352108d48386b773c253f5bbe5732bdf9b792e1418a
      Path: /var/log/pods/ci-idc-24412-rmu-scenario-fairo_fairo-f-lookup-split-function-0_4be1f4c1-e08f-4540-a23b-71c20945c822/pulsar-function/*.log
      Service: fairo-f-lookup-split
      Source: pulsar
      Status: Pending
        0 files tailed out of 6 files matching
      BytesRead: 0
      Average Latency (ms): 0
      24h Average Latency (ms): 0
      Peak Latency (ms): 0
      24h Peak Latency (ms): 0

  cortex-argo/cortex-argoworkflows-argo-workflows-server-75749b6ffd-22t55/argo-server
  -----------------------------------------------------------------------------------
    - Type: file
      Identifier: 233b4641ff588003836772c6f13fbeec621caa924da143555549f9b34c0fe9c6
      Path: /var/log/pods/cortex-argo_cortex-argoworkflows-argo-workflows-server-75749b6ffd-22t55_f9b96ce3-f82e-4dce-a42a-d9e2c367a7c7/argo-server/*.log
      Service: argocli
      Source: argocli
      Status: Pending
        6 files tailed out of 6 files matching
      BytesRead: 61061
      Average Latency (ms): 0
      24h Average Latency (ms): 0
      Peak Latency (ms): 18
      24h Peak Latency (ms): 18

  dev01-neuron-filestreamer/eventbus-default-stan-0/metrics
  ---------------------------------------------------------
    - Type: file
      Identifier: 0221587ad576ab62616960b96bda470dba57399a54ab3ff5bf8a61c7afc94d58
      Path: /var/log/pods/dev01-neuron-filestreamer_eventbus-default-stan-0_2e497708-ee8a-43b3-aaaf-dfbe98565003/metrics/*.log
      Service: prometheus-nats-exporter
      Source: prometheus-nats-exporter
      Status: Pending
        6 files tailed out of 6 files matching
      BytesRead: 0
      Average Latency (ms): 0
      24h Average Latency (ms): 0
      Peak Latency (ms): 0
      24h Peak Latency (ms): 0

  ci-idc-24412-rmu-scenario-fairo/fairo-aml-rmu-60-function-0/pulsar-function
  ---------------------------------------------------------------------------
    - Type: file
      Identifier: e45bc3d59c367abd9fb868a8315f485ca91552b1a2b27173845a3a5baf4cf225
      Path: /var/log/pods/ci-idc-24412-rmu-scenario-fairo_fairo-aml-rmu-60-function-0_fc4b29fa-46f5-4b9d-889e-d3cc79961df2/pulsar-function/*.log
      Service: neuron-functions-python-runner
      Source: neuron-functions-python-runner
      Status: Pending
        0 files tailed out of 6 files matching
      BytesRead: 0
      Average Latency (ms): 0
      24h Average Latency (ms): 0
      Peak Latency (ms): 0
      24h Peak Latency (ms): 0

  ci-idc-24412-int-scenario-fairo/fairo-aml-int-06-2-function-0/pulsar-function
  -----------------------------------------------------------------------------
    - Type: file
      Identifier: 81172d21d9138f9c8cf330036915dc3e1e488ad32740c7095c014a33972d26fb
      Path: /var/log/pods/ci-idc-24412-int-scenario-fairo_fairo-aml-int-06-2-function-0_372c6c60-7d2b-457d-ad16-e55906c6d633/pulsar-function/*.log
      Service: neuron-functions-python-runner
      Source: neuron-functions-python-runner
      Status: Pending
        6 files tailed out of 6 files matching
      BytesRead: 394
      Average Latency (ms): 0
      24h Average Latency (ms): 0
      Peak Latency (ms): 0
      24h Peak Latency (ms): 0

  dev01-cashpresso/apollo-f-aml-hr-trx-function-0/pulsar-function
  ---------------------------------------------------------------
    - Type: file
      Identifier: 5c095af42760bb45b3a97490080dd335b978a50d105e2ca354c70d70b3d9534f
      Path: /var/log/pods/dev01-cashpresso_apollo-f-aml-hr-trx-function-0_ff6bb273-fb99-4af0-a5d3-4b7d45947824/pulsar-function/*.log
      Service: neuron-functions-python-runner
      Source: neuron-functions-python-runner
      Status: Pending
        6 files tailed out of 6 files matching
      BytesRead: 1681
      Average Latency (ms): 0
      24h Average Latency (ms): 0
      Peak Latency (ms): 0
      24h Peak Latency (ms): 0  
      Lines Combined: 0   
      MultiLine matches: 3 

  dev01-fairo/fairo-prod-risk-2-function-0/pulsar-function
  --------------------------------------------------------
    - Type: file
      Identifier: 20e41896ebcf64142848219a70fb3b7f6b9c8a2f1bc7bc855cb6cca502133de1
      Path: /var/log/pods/dev01-fairo_fairo-prod-risk-2-function-0_c6d6cd96-bbf8-4bc7-8497-4df395f8671f/pulsar-function/*.log
      Service: fairo-prod-risk-2
      Source: pulsar
      Status: Pending
        0 files tailed out of 6 files matching
      BytesRead: 0
      Average Latency (ms): 0
      24h Average Latency (ms): 0
      Peak Latency (ms): 0
      24h Peak Latency (ms): 0

  cortex-servicemesh/istiod-mesh-57f55dfd57-d4tkr/discovery
  ---------------------------------------------------------
    - Type: file
      Identifier: 40576d949a15880d5d84b952b98fb0677870bb5c2b3c5dfb02eeb41a4b8eba82
      Path: /var/log/pods/cortex-servicemesh_istiod-mesh-57f55dfd57-d4tkr_280ced8d-9c7a-434c-9b3a-b23c80ee3c91/discovery/*.log
      Service: pilot-rhel8
      Source: pilot-rhel8
      Status: Pending
        0 files tailed out of 6 files matching
      BytesRead: 449
      Average Latency (ms): 0
      24h Average Latency (ms): 0
      Peak Latency (ms): 0
      24h Peak Latency (ms): 0  
      Lines Combined: 0   
      MultiLine matches: 1 

  ci-idc-24412-int-scenario-fairo/fairo-txn-risk-2-function-0/pulsar-function
  ---------------------------------------------------------------------------
    - Type: file
      Identifier: 00ec799baf7bbaeeaa6ac8d3004686e025e80e91ef70b0bf3807316925e3145a
      Path: /var/log/pods/ci-idc-24412-int-scenario-fairo_fairo-txn-risk-2-function-0_2e109941-b16a-44a6-a9c2-185df91c3124/pulsar-function/*.log
      Service: fairo-txn-risk-2
      Source: neuron-functions-python-runner
      Status: Pending
        0 files tailed out of 6 files matching
      BytesRead: 0
      Average Latency (ms): 0
      24h Average Latency (ms): 0
      Peak Latency (ms): 0
      24h Peak Latency (ms): 0

  dev01-neuron-pulsar/dev01-neuron-pulsar-proxy-1/wait-broker-ready
  -----------------------------------------------------------------
    - Type: file
      Identifier: 90d6a94d2dc51b22abe20316b03dc1f661573e7670484f7beeb0f9a03bb1cb66
      Path: /var/log/pods/dev01-neuron-pulsar_dev01-neuron-pulsar-proxy-1_d330a865-5000-4978-9515-af4fbe4e737e/wait-broker-ready/*.log
      Service: pulsar-proxy
      Source: neuron-pulsar-all
      Status: OK
        0 files tailed out of 6 files matching
      BytesRead: 0
      Average Latency (ms): 0
      24h Average Latency (ms): 0
      Peak Latency (ms): 0
      24h Peak Latency (ms): 0

  cortex-cgroups-gc/k8s-gc-frlcz/garbage-collector
  ------------------------------------------------
    - Type: file
      Identifier: f591ee05e1a571c9cf78d0e6ca26de596164849ee0979e3fde1bad6e54eef63f
      Path: /var/log/pods/cortex-cgroups-gc_k8s-gc-frlcz_2cba6d51-848c-42b1-9916-d74a5a322fec/garbage-collector/*.log
      Service: support-tools
      Source: support-tools
      Status: OK
        6 files tailed out of 6 files matching
      Inputs:
        /var/log/pods/cortex-cgroups-gc_k8s-gc-frlcz_2cba6d51-848c-42b1-9916-d74a5a322fec/garbage-collector/5.log
      BytesRead: 416
      Average Latency (ms): 0
      24h Average Latency (ms): 0
      Peak Latency (ms): 0
      24h Peak Latency (ms): 0  
      Lines Combined: 0   
      MultiLine matches: 1 

  dev01-fcmi/cmi-repo-host-0/nss-wrapper-init
  -------------------------------------------
    - Type: file
      Identifier: 671ddd494bb0d65c9be19124a9d31b8976f58e61d3cfbc406c0ce6c12e5bbe48
      Path: /var/log/pods/dev01-fcmi_cmi-repo-host-0_502543d6-bf0a-4743-b2fc-bb68610c3a0f/nss-wrapper-init/*.log
      Service: crunchy-pgbackrest
      Source: crunchy-pgbackrest
      Status: OK
        6 files tailed out of 6 files matching
      Inputs:
        /var/log/pods/dev01-fcmi_cmi-repo-host-0_502543d6-bf0a-4743-b2fc-bb68610c3a0f/nss-wrapper-init/5.log
      BytesRead: 0
      Average Latency (ms): 0
      24h Average Latency (ms): 0
      Peak Latency (ms): 0
      24h Peak Latency (ms): 0

  dev02-capture/capture-db-capturedb-7266-0/pgbackrest-config
  -----------------------------------------------------------
    - Type: file
      Identifier: d95c29b993e02617e1bdd5d8722b37f5d8f3edd14dad23a4437e0f98f6c7605e
      Path: /var/log/pods/dev02-capture_capture-db-capturedb-7266-0_6bbdaf48-7991-41ad-818e-e3c26dc8aada/pgbackrest-config/*.log
      Service: crunchy-pgbackrest
      Source: crunchy-pgbackrest
      Status: OK
        6 files tailed out of 6 files matching
      Inputs:
        /var/log/pods/dev02-capture_capture-db-capturedb-7266-0_6bbdaf48-7991-41ad-818e-e3c26dc8aada/pgbackrest-config/5.log
      BytesRead: 0
      Average Latency (ms): 0
      24h Average Latency (ms): 0
      Peak Latency (ms): 0
      24h Peak Latency (ms): 0

  dev01-fairo/fairo-fairo-com-neu-function-0/pulsar-function
  ----------------------------------------------------------
    - Type: file
      Identifier: f51e9ac4dddb376b8ecfbd12713183e4a63a98f1cb8485093f5a2e1757128acf
      Path: /var/log/pods/dev01-fairo_fairo-fairo-com-neu-function-0_39c49626-4cd5-4fa4-8061-6021d2cf2569/pulsar-function/*.log
      Service: fairo-fairo-com-neu
      Source: pulsar
      Status: Pending
        0 files tailed out of 6 files matching
      BytesRead: 0
      Average Latency (ms): 0
      24h Average Latency (ms): 0
      Peak Latency (ms): 0
      24h Peak Latency (ms): 0

  dev01-cashpresso/apollo-wlm-rescreening-function-0/pulsar-function
  ------------------------------------------------------------------
    - Type: file
      Identifier: 02c13c5230b1ad45a296d946d07bcb5c1a51f38fc0a53c4de7c3b934c92d2dfd
      Path: /var/log/pods/dev01-cashpresso_apollo-wlm-rescreening-function-0_7182eafe-ce7f-4c93-a38b-53fba58bdd06/pulsar-function/*.log
      Service: neuron-functions-python-runner
      Source: neuron-functions-python-runner
      Status: Pending
        0 files tailed out of 6 files matching
      BytesRead: 0
      Average Latency (ms): 0
      24h Average Latency (ms): 0
      Peak Latency (ms): 0
      24h Peak Latency (ms): 0

  dev01-fairo/fairo-txn-risk-2-function-0/pulsar-function
  -------------------------------------------------------
    - Type: file
      Identifier: 6ba8cfc29a07468c9629f1725ade747198dfd46f946cf8d6d4fe1e61c999f3f2
      Path: /var/log/pods/dev01-fairo_fairo-txn-risk-2-function-0_c3abbe43-46d0-49ae-b43c-52e5d468ae98/pulsar-function/*.log
      Service: fairo-txn-risk-2
      Source: pulsar
      Status: OK
        6 files tailed out of 6 files matching
      Inputs:
        /var/log/pods/dev01-fairo_fairo-txn-risk-2-function-0_c3abbe43-46d0-49ae-b43c-52e5d468ae98/pulsar-function/5.log
      BytesRead: 1668
      Average Latency (ms): 0
      24h Average Latency (ms): 0
      Peak Latency (ms): 0
      24h Peak Latency (ms): 0

  dev01-fcmi/cmi-repo-host-0/pgbackrest-config
  --------------------------------------------
    - Type: file
      Identifier: 08c0db18142ff8107a54097592fbc581e24ec7d93ae752c8d43c20c4a64b1856
      Path: /var/log/pods/dev01-fcmi_cmi-repo-host-0_502543d6-bf0a-4743-b2fc-bb68610c3a0f/pgbackrest-config/*.log
      Service: crunchy-pgbackrest
      Source: crunchy-pgbackrest
      Status: OK
        6 files tailed out of 6 files matching
      Inputs:
        /var/log/pods/dev01-fcmi_cmi-repo-host-0_502543d6-bf0a-4743-b2fc-bb68610c3a0f/pgbackrest-config/5.log
      BytesRead: 0
      Average Latency (ms): 0
      24h Average Latency (ms): 0
      Peak Latency (ms): 0
      24h Peak Latency (ms): 0

  cortex-certmanager/cortex-certmanager-webhook-5dd6c5d77c-2j2cb/cert-manager
  ---------------------------------------------------------------------------
    - Type: file
      Identifier: 806ee255ccf09580aceaa4faa91e11eafffa8e3c0ba045484cbf8ec80b87de7e
      Path: /var/log/pods/cortex-certmanager_cortex-certmanager-webhook-5dd6c5d77c-2j2cb_4f97561b-3080-48c1-b72d-dacfc5a874c7/cert-manager/*.log
      Service: cert-manager-webhook
      Source: cert-manager-webhook
      Status: OK
        6 files tailed out of 6 files matching
      Inputs:
        /var/log/pods/cortex-certmanager_cortex-certmanager-webhook-5dd6c5d77c-2j2cb_4f97561b-3080-48c1-b72d-dacfc5a874c7/cert-manager/5.log
      BytesRead: 0
      Average Latency (ms): 0
      24h Average Latency (ms): 0
      Peak Latency (ms): 0
      24h Peak Latency (ms): 0

  dev01-cashpresso/apollo-ctry-risk-2-function-0/pulsar-function
  --------------------------------------------------------------
    - Type: file
      Identifier: ec143bd8b9488ce8854e93ed30c4b33237a7ddd9f4a4ff69d31764a04e7982cd
      Path: /var/log/pods/dev01-cashpresso_apollo-ctry-risk-2-function-0_8bb0d034-f5d5-4c61-8d7b-4c89ed77931a/pulsar-function/*.log
      Service: neuron-functions-python-runner
      Source: neuron-functions-python-runner
      Status: Pending
        6 files tailed out of 6 files matching
      BytesRead: 1281
      Average Latency (ms): 0
      24h Average Latency (ms): 0
      Peak Latency (ms): 0
      24h Peak Latency (ms): 0  
      Lines Combined: 0   
      MultiLine matches: 2 

  dev01-neuron-pulsar/dev01-neuron-pulsar-zookeeper-2/dev01-neuron-pulsar-zookeeper
  ---------------------------------------------------------------------------------
    - Type: file
      Identifier: 3f1256b907de21e9cbc3f849a246958834ca1a35c7dd24105faa1977a7146d67
      Path: /var/log/pods/dev01-neuron-pulsar_dev01-neuron-pulsar-zookeeper-2_0f042e11-6194-471d-bcbd-1875560d4e50/dev01-neuron-pulsar-zookeeper/*.log
      Service: pulsar-zookeeper
      Source: pulsar
      Status: OK
        6 files tailed out of 6 files matching
      Inputs:
        /var/log/pods/dev01-neuron-pulsar_dev01-neuron-pulsar-zookeeper-2_0f042e11-6194-471d-bcbd-1875560d4e50/dev01-neuron-pulsar-zookeeper/5.log
      BytesRead: 10640
      Average Latency (ms): 0
      24h Average Latency (ms): 0
      Peak Latency (ms): 0
      24h Peak Latency (ms): 0  
      Lines Combined: 0   
      MultiLine matches: 56 

  dev01-fcmi/cmi-repo-host-0/pgbackrest
  -------------------------------------
    - Type: file
      Identifier: c28880cea1b45bc5f27c91f93d1fa8445437871be6cd7fe83c28b501bcbf04f8
      Path: /var/log/pods/dev01-fcmi_cmi-repo-host-0_502543d6-bf0a-4743-b2fc-bb68610c3a0f/pgbackrest/*.log
      Service: crunchy-pgbackrest
      Source: crunchy-pgbackrest
      Status: OK
        6 files tailed out of 6 files matching
      Inputs:
        /var/log/pods/dev01-fcmi_cmi-repo-host-0_502543d6-bf0a-4743-b2fc-bb68610c3a0f/pgbackrest/5.log
      BytesRead: 0
      Average Latency (ms): 0
      24h Average Latency (ms): 0
      Peak Latency (ms): 0
      24h Peak Latency (ms): 0

  testkube/testkube-nats-box-6c6c5f8cc9-2sxhq/nats-box
  ----------------------------------------------------
    - Type: file
      Identifier: efd0e62297cb8dc5f34be9d29cdde561fe407b178d5c3511420016a3d249af1d
      Path: /var/log/pods/testkube_testkube-nats-box-6c6c5f8cc9-2sxhq_69269f6c-785e-42fa-8bf5-0ddc48cee483/nats-box/*.log
      Service: nats-box
      Source: nats-box
      Status: OK
        6 files tailed out of 6 files matching
      Inputs:
        /var/log/pods/testkube_testkube-nats-box-6c6c5f8cc9-2sxhq_69269f6c-785e-42fa-8bf5-0ddc48cee483/nats-box/5.log
      BytesRead: 0
      Average Latency (ms): 0
      24h Average Latency (ms): 0
      Peak Latency (ms): 0
      24h Peak Latency (ms): 0

  ci-idc-24412-rmu-scenario-fairo/fairo-ctry-risk-2-function-0/pulsar-function
  ----------------------------------------------------------------------------
    - Type: file
      Identifier: 5bfc9e3ded90e02bd0c769e413faa3d610ff9e6bd56b37a2195bc0d3ff14c174
      Path: /var/log/pods/ci-idc-24412-rmu-scenario-fairo_fairo-ctry-risk-2-function-0_218a91de-52ac-4504-9214-091b128a2be9/pulsar-function/*.log
      Service: fairo-ctry-risk-2
      Source: pulsar
      Status: OK
        6 files tailed out of 6 files matching
      Inputs:
        /var/log/pods/ci-idc-24412-rmu-scenario-fairo_fairo-ctry-risk-2-function-0_218a91de-52ac-4504-9214-091b128a2be9/pulsar-function/5.log
      BytesRead: 1267
      Average Latency (ms): 0
      24h Average Latency (ms): 0
      Peak Latency (ms): 0
      24h Peak Latency (ms): 0

  ci-idc-24412-int-scenario-fairo/fairo-cust-risk-2-function-0/pulsar-function
  ----------------------------------------------------------------------------
    - Type: file
      Identifier: 005e8a1469e81cfba3639234355e1171f50e04305c4c66b0505146f6434da3c4
      Path: /var/log/pods/ci-idc-24412-int-scenario-fairo_fairo-cust-risk-2-function-0_71ee342c-241a-4a7a-bf1d-bd15dd9f6f82/pulsar-function/*.log
      Service: fairo-cust-risk-2
      Source: neuron-functions-python-runner
      Status: Pending
        0 files tailed out of 6 files matching
      BytesRead: 0
      Average Latency (ms): 0
      24h Average Latency (ms): 0
      Peak Latency (ms): 0
      24h Peak Latency (ms): 0

  ci-idc-24412-int-scenario-fairo/fairo-f-over-risk2-function-0/pulsar-function
  -----------------------------------------------------------------------------
    - Type: file
      Identifier: 7409ec8c0d194ffe766e09bd65456af7dcfb8c71559eb5f62f7e05ef9b19a1b6
      Path: /var/log/pods/ci-idc-24412-int-scenario-fairo_fairo-f-over-risk2-function-0_317282dc-cca4-41c5-bf1d-4e779eb4a05c/pulsar-function/*.log
      Service: fairo-f-over-risk2
      Source: neuron-functions-python-runner
      Status: Pending
        1 files tailed out of 6 files matching
      BytesRead: 2056
      Average Latency (ms): 0
      24h Average Latency (ms): 0
      Peak Latency (ms): 0
      24h Peak Latency (ms): 0  
      Lines Combined: 0   
      MultiLine matches: 4 

  dev02-capture/capture-db-capturedb-7266-0/exporter
  --------------------------------------------------
    - Type: file
      Identifier: b90463b05a10d1805982642100a16e994a8c6924db0b0b0dbe7dff9434ec4056
      Path: /var/log/pods/dev02-capture_capture-db-capturedb-7266-0_6bbdaf48-7991-41ad-818e-e3c26dc8aada/exporter/*.log
      Service: crunchy-postgres-exporter
      Source: crunchy-postgres-exporter
      Status: OK
        0 files tailed out of 6 files matching
      BytesRead: 2982
      Average Latency (ms): 0
      24h Average Latency (ms): 0
      Peak Latency (ms): 0
      24h Peak Latency (ms): 0

  dev02-capture/capture-db-capturedb-7266-0/replication-cert-copy
  ---------------------------------------------------------------
    - Type: file
      Identifier: c2c5a270eca0b07c01b95eabcb4d34574e39558a2828a2e64ec4a957c284d21a
      Path: /var/log/pods/dev02-capture_capture-db-capturedb-7266-0_6bbdaf48-7991-41ad-818e-e3c26dc8aada/replication-cert-copy/*.log
      Service: crunchy-postgres
      Source: crunchy-postgres
      Status: OK
        0 files tailed out of 6 files matching
      BytesRead: 0
      Average Latency (ms): 0
      24h Average Latency (ms): 0
      Peak Latency (ms): 0
      24h Peak Latency (ms): 0

  dev02-capture/capture-db-capturedb-7266-0/postgres-startup
  ----------------------------------------------------------
    - Type: file
      Identifier: bf656a14cf0e69f5d6603faf7616f2788002d029ab6836b3e74d45328c5ea6bc
      Path: /var/log/pods/dev02-capture_capture-db-capturedb-7266-0_6bbdaf48-7991-41ad-818e-e3c26dc8aada/postgres-startup/*.log
      Service: crunchy-postgres
      Source: crunchy-postgres
      Status: OK
        0 files tailed out of 6 files matching
      BytesRead: 0
      Average Latency (ms): 0
      24h Average Latency (ms): 0
      Peak Latency (ms): 0
      24h Peak Latency (ms): 0

  dev02-capture/capture-db-capturedb-7266-0/database
  --------------------------------------------------
    - Type: file
      Identifier: 895a03ff8586779652da58f9869b2422d5faf4607a03c1877dc2501fe8e5bc7b
      Path: /var/log/pods/dev02-capture_capture-db-capturedb-7266-0_6bbdaf48-7991-41ad-818e-e3c26dc8aada/database/*.log
      Service: crunchy-postgres
      Source: crunchy-postgres
      Status: OK
        0 files tailed out of 6 files matching
      BytesRead: 940
      Average Latency (ms): 0
      24h Average Latency (ms): 0
      Peak Latency (ms): 0
      24h Peak Latency (ms): 0

  ci-idc-24412-int-scenario-fairo/mq-source-connector-source-0/pulsar-source
  --------------------------------------------------------------------------
    - Type: file
      Identifier: 072dc49c40ce9e98e0f49531611a215d6d7c020f47aa7c65f3741f1099d27496
      Path: /var/log/pods/ci-idc-24412-int-scenario-fairo_mq-source-connector-source-0_6411ac40-12e9-4c6a-b264-2dbe5c5b0bb8/pulsar-source/*.log
      Service: sectormq
      Source: pulsar-all
      Status: OK
        0 files tailed out of 11 files matching
      BytesRead: 0
      Average Latency (ms): 0
      24h Average Latency (ms): 0
      Peak Latency (ms): 0
      24h Peak Latency (ms): 0

  cortex-pipeline/cortex-community-cd-test-promotion-sensor-7nvgh-7bb59f5458sq2dr/main
  ------------------------------------------------------------------------------------
    - Type: file
      Identifier: 9c92cc7417e391249ab84e9f8c74e9c08d2ce872ed572887364764e60d6e85dd
      Path: /var/log/pods/cortex-pipeline_cortex-community-cd-test-promotion-sensor-7nvgh-7bb59f5458sq2dr_15c2f509-4e2e-4bf9-93a9-23d8c1bfaaca/main/*.log
      Service: argo-events
      Source: argo-events
      Status: OK
        6 files tailed out of 6 files matching
      Inputs:
        /var/log/pods/cortex-pipeline_cortex-community-cd-test-promotion-sensor-7nvgh-7bb59f5458sq2dr_15c2f509-4e2e-4bf9-93a9-23d8c1bfaaca/main/5.log
      BytesRead: 19401
      Average Latency (ms): 0
      24h Average Latency (ms): 0
      Peak Latency (ms): 0
      24h Peak Latency (ms): 0

  dev01-cashpresso/cashpresso-cre-2-out-function-0/pulsar-function
  ----------------------------------------------------------------
    - Type: file
      Identifier: d07114d4de8e69336e35e6fa43d76aeb87ef6b3eb81923bf2a6e09a969bdd751
      Path: /var/log/pods/dev01-cashpresso_cashpresso-cre-2-out-function-0_74e6f3b0-d9fc-4941-92f0-3c452753889a/pulsar-function/*.log
      Service: neuron-functions-python-runner
      Source: neuron-functions-python-runner
      Status: Pending
        6 files tailed out of 6 files matching
      BytesRead: 904
      Average Latency (ms): 0
      24h Average Latency (ms): 0
      Peak Latency (ms): 0
      24h Peak Latency (ms): 0  
      Lines Combined: 0   
      MultiLine matches: 1 

  dev01-fairo/fairo-wlm-rep-function-0/pulsar-function
  ----------------------------------------------------
    - Type: file
      Identifier: a625c3208d9ebbb73a57b81a36307ca3fcd548a029957465313cceb57cbae86f
      Path: /var/log/pods/dev01-fairo_fairo-wlm-rep-function-0_d2df7c98-605e-4bb0-bc5f-244fc797b5d7/pulsar-function/*.log
      Service: fairo-wlm-rep
      Source: pulsar
      Status: Pending
        6 files tailed out of 6 files matching
      BytesRead: 1651
      Average Latency (ms): 0
      24h Average Latency (ms): 0
      Peak Latency (ms): 0
      24h Peak Latency (ms): 0  
      Lines Combined: 0   
      MultiLine matches: 3 

  testkube/testkube-api-server-7fcbf975cf-ckwlz/testkube-api
  ----------------------------------------------------------
    - Type: file
      Identifier: 230536765c7025e303e7b856ba6e7d59c614de4e92b8474a757ca0b4bde3c3bd
      Path: /var/log/pods/testkube_testkube-api-server-7fcbf975cf-ckwlz_3309bcc7-2698-4f86-a55b-5fa019512cea/testkube-api/*.log
      Service: testkube-api-server
      Source: testkube-api-server
      Status: OK
        0 files tailed out of 6 files matching
      BytesRead: 0
      Average Latency (ms): 0
      24h Average Latency (ms): 0
      Peak Latency (ms): 0
      24h Peak Latency (ms): 0

  dev01-neuron-pulsar/dev01-neuron-pulsar-bookie-1/dev01-neuron-pulsar-bookie
  ---------------------------------------------------------------------------
    - Type: file
      Identifier: 3e0a0dc809346c7093b431325e6f0406a21c859c76d5170cc777b7644c5c629a
      Path: /var/log/pods/dev01-neuron-pulsar_dev01-neuron-pulsar-bookie-1_124fc7cf-e13f-4f25-989a-866b7c653e97/dev01-neuron-pulsar-bookie/*.log
      Service: pulsar-bookie
      Source: pulsar
      Status: OK
        6 files tailed out of 6 files matching
      Inputs:
        /var/log/pods/dev01-neuron-pulsar_dev01-neuron-pulsar-bookie-1_124fc7cf-e13f-4f25-989a-866b7c653e97/dev01-neuron-pulsar-bookie/5.log
      BytesRead: 4.520116e+06
      Average Latency (ms): 0
      24h Average Latency (ms): 0
      Peak Latency (ms): 19
      24h Peak Latency (ms): 19  
      Lines Combined: 62   
      MultiLine matches: 21331 

  ci-idc-24412-int-scenario-fairo/bae-mx-transformer-function-0/pulsar-function
  -----------------------------------------------------------------------------
    - Type: file
      Identifier: 235862f05abd4d497ad8c6925a256be04c68226ffb8223e97604a4134118b5ad
      Path: /var/log/pods/ci-idc-24412-int-scenario-fairo_bae-mx-transformer-function-0_376dd1f2-5eea-4715-bb32-4eff4a0e25c2/pulsar-function/*.log
      Service: bae-mx-transformer
      Source: neuron-functions-python-runner
      Status: Pending
        6 files tailed out of 6 files matching
      BytesRead: 1301
      Average Latency (ms): 0
      24h Average Latency (ms): 0
      Peak Latency (ms): 0
      24h Peak Latency (ms): 0  
      Lines Combined: 0   
      MultiLine matches: 2 

  dev01-neuron-filestreamer/eventbus-default-stan-0/stan
  ------------------------------------------------------
    - Type: file
      Identifier: d71f19f6ad8239f8b629f93cb58b2ac290b8c3fd073c17f2f63e931d9fcb177d
      Path: /var/log/pods/dev01-neuron-filestreamer_eventbus-default-stan-0_2e497708-ee8a-43b3-aaaf-dfbe98565003/stan/*.log
      Service: nats-streaming
      Source: nats-streaming
      Status: Pending
        6 files tailed out of 6 files matching
      BytesRead: 0
      Average Latency (ms): 0
      24h Average Latency (ms): 0
      Peak Latency (ms): 0
      24h Peak Latency (ms): 0

  ci-idc-24412-ftr11-improv-fairo/fairo-f-frc-crs-function-0/pulsar-function
  --------------------------------------------------------------------------
    - Type: file
      Identifier: 3cb86d02edca9d8ff0c8020866fbb9842885e32b86b38678b79c0338d3adab4c
      Path: /var/log/pods/ci-idc-24412-ftr11-improv-fairo_fairo-f-frc-crs-function-0_28b80eb1-5222-47aa-a482-9da9ded42964/pulsar-function/*.log
      Service: fairo-f-frc-crs
      Source: pulsar
      Status: OK
        6 files tailed out of 6 files matching
      Inputs:
        /var/log/pods/ci-idc-24412-ftr11-improv-fairo_fairo-f-frc-crs-function-0_28b80eb1-5222-47aa-a482-9da9ded42964/pulsar-function/5.log
      BytesRead: 1642
      Average Latency (ms): 0
      24h Average Latency (ms): 0
      Peak Latency (ms): 0
      24h Peak Latency (ms): 0

  dev01-cashpresso/cashpresso-f-wlm-split-function-0/pulsar-function
  ------------------------------------------------------------------
    - Type: file
      Identifier: 9b919f96638a08c278f706978c3363d4df8febe80f6f1ecf542f882af4e0c526
      Path: /var/log/pods/dev01-cashpresso_cashpresso-f-wlm-split-function-0_89f2f691-5fc4-41fa-be22-9af92295e2d9/pulsar-function/*.log
      Service: neuron-functions-python-runner
      Source: neuron-functions-python-runner
      Status: OK
        6 files tailed out of 6 files matching
      Inputs:
        /var/log/pods/dev01-cashpresso_cashpresso-f-wlm-split-function-0_89f2f691-5fc4-41fa-be22-9af92295e2d9/pulsar-function/5.log
      BytesRead: 907
      Average Latency (ms): 0
      24h Average Latency (ms): 0
      Peak Latency (ms): 0
      24h Peak Latency (ms): 0

=============
Process Agent
=============

  Version: 7.41.1
  Status date: 2023-01-17 14:57:25.797 UTC (1673967445797)
  Process Agent Start: 2023-01-17 14:42:15.458 UTC (1673966535458)
  Pid: 5479
  Go Version: go1.18.9
  Build arch: amd64
  Log Level: WARN
  Enabled Checks: [container rtcontainer process_discovery pod]
  Allocated Memory: 36,146,592 bytes
  Hostname: i-03dcdc76f8df26cd5

  =================
  Process Endpoints
  =================
    https://process.datadoghq.eu - API Key ending with:
        - f7da4

  =========
  Collector
  =========
    Last collection time: 2023-01-17 14:57:23
    Docker socket: 
    Number of processes: 0
    Number of containers: 83
    Process Queue length: 0
    RTProcess Queue length: 0
    Connections Queue length: 0
    Event Queue length: 0
    Pod Queue length: 0
    Process Bytes enqueued: 0
    RTProcess Bytes enqueued: 0
    Connections Bytes enqueued: 0
    Event Bytes enqueued: 0
    Pod Bytes enqueued: 0
    Drop Check Payloads: []

=========
APM Agent
=========

  Status: Not running or unreachable on localhost:8126.
  Error: Get "http://localhost:8126/debug/vars": dial tcp [::1]:8126: connect: connection refused

==========
Aggregator
==========
  Checks Metric Sample: 630,343
  Dogstatsd Metric Sample: 2,402
  Event: 1
  Events Flushed: 1
  Number Of Flushes: 60
  Series Flushed: 594,750
  Service Check: 2,602
  Service Checks Flushed: 2,644

=========
DogStatsD
=========
  Event Packets: 0
  Event Parse Errors: 0
  Metric Packets: 2,401
  Metric Parse Errors: 0
  Service Check Packets: 0
  Service Check Parse Errors: 0
  Udp Bytes: 257,200
  Udp Packet Reading Errors: 0
  Udp Packets: 1,240
  Uds Bytes: 0
  Uds Origin Detection Errors: 0
  Uds Packet Reading Errors: 0
  Uds Packets: 1
  Unterminated Metric Errors: 0

=====================
Datadog Cluster Agent
=====================

  - Datadog Cluster Agent endpoint detected: https://172.30.143.253:5005
  Successfully connected to the Datadog Cluster Agent.
  - Running: 7.41.1+commit.4f39b9e

=============
Autodiscovery
=============
  Enabled Features
  ================
    cri
    kubernetes

  Configuration Errors
  ====================
    ci-idc-24412-int-scenario-fairo/mq-source-connector-source-0 (6411ac40-12e9-4c6a-b264-2dbe5c5b0bb8)
    ---------------------------------------------------------------------------------------------------
        annotation ad.datadoghq.com/pulsarfunction.logs is invalid: pulsarfunction doesn't match a container identifier [pulsar-source]

====
OTLP
====

  Status: Not enabled
  Collector status: Not running

File descriptors:

root@ip-10-223-124-76:/proc/5380/fd# ls -l        
total 0
lrwx------. 1 root root 64 Jan 17 14:42 0 -> /dev/null
l-wx------. 1 root root 64 Jan 17 14:42 1 -> 'pipe:[42372]'
lrwx------. 1 root root 64 Jan 17 14:42 10 -> 'socket:[49212]'
lrwx------. 1 root root 64 Jan 17 14:42 12 -> 'socket:[55450]'
lrwx------. 1 root root 64 Jan 17 14:42 13 -> 'socket:[55459]'
lrwx------. 1 root root 64 Jan 17 14:42 14 -> 'socket:[70524]'
lrwx------. 1 root root 64 Jan 17 14:42 15 -> 'socket:[78643]'
lrwx------. 1 root root 64 Jan 17 14:42 16 -> 'socket:[78644]'
lrwx------. 1 root root 64 Jan 17 14:42 18 -> 'socket:[75281]'
lr-x------. 1 root root 64 Jan 17 14:42 19 -> /var/log/pods/dev01-cashpresso_apollo-ctry-risk-2-function-0_8bb0d034-f5d5-4c61-8d7b-4c89ed77931a/pulsar-function/5.log
l-wx------. 1 root root 64 Jan 17 14:42 2 -> 'pipe:[42373]'
lrwx------. 1 root root 64 Jan 17 14:42 20 -> 'socket:[80160]'
lrwx------. 1 root root 64 Jan 17 14:42 21 -> 'socket:[75284]'
lrwx------. 1 root root 64 Jan 17 14:42 22 -> 'socket:[75282]'
lrwx------. 1 root root 64 Jan 17 14:42 23 -> 'socket:[75285]'
lr-x------. 1 root root 64 Jan 17 14:42 24 -> /var/log/pods/cortex-cgroups-gc_k8s-gc-frlcz_2cba6d51-848c-42b1-9916-d74a5a322fec/garbage-collector/5.log
lr-x------. 1 root root 64 Jan 17 14:42 25 -> /var/log/pods/cortex-pipeline_cortex-community-cd-test-promotion-sensor-7nvgh-7bb59f5458sq2dr_15c2f509-4e2e-4bf9-93a9-23d8c1bfaaca/main/5.log
lr-x------. 1 root root 64 Jan 17 14:43 26 -> /var/log/pods/dev01-fairo_fairo-wlm-rep-function-0_d2df7c98-605e-4bb0-bc5f-244fc797b5d7/pulsar-function/5.log
lr-x------. 1 root root 64 Jan 17 14:42 27 -> /var/log/pods/dev02-capture_capture-db-capturedb-7266-0_6bbdaf48-7991-41ad-818e-e3c26dc8aada/pgbackrest-config/5.log
lrwx------. 1 root root 64 Jan 17 14:42 29 -> 'socket:[80297]'
lrwx------. 1 root root 64 Jan 17 14:42 3 -> 'anon_inode:[pidfd]'
lr-x------. 1 root root 64 Jan 17 14:44 30 -> /proc/cpuinfo
lr-x------. 1 root root 64 Jan 17 14:43 31 -> /proc/meminfo
lr-x------. 1 root root 64 Jan 17 14:42 32 -> /var/log/pods/dev02-capture_capture-db-capturedb-7266-0_6bbdaf48-7991-41ad-818e-e3c26dc8aada/nss-wrapper-init/5.log
lr-x------. 1 root root 64 Jan 17 14:42 33 -> /var/log/pods/dev01-fcmi_cmi-repo-host-0_502543d6-bf0a-4743-b2fc-bb68610c3a0f/pgbackrest-log-dir/5.log
lr-x------. 1 root root 64 Jan 17 14:42 34 -> /var/log/pods/cortex-argo_controller-manager-867fbd77b4-8j846_5f99c975-edff-4a9b-8cb1-4289bc8053fd/controller-manager/8.log
lrwx------. 1 root root 64 Jan 17 14:42 35 -> 'socket:[94409]'
lr-x------. 1 root root 64 Jan 17 14:42 36 -> /var/log/pods/dev01-neuron-pulsar_dev01-neuron-pulsar-proxy-1_d330a865-5000-4978-9515-af4fbe4e737e/dev01-neuron-pulsar-proxy/5.log
lr-x------. 1 root root 64 Jan 17 14:42 37 -> /var/log/pods/testkube_testkube-nats-box-6c6c5f8cc9-2sxhq_69269f6c-785e-42fa-8bf5-0ddc48cee483/nats-box/5.log
lr-x------. 1 root root 64 Jan 17 14:42 4 -> /dev/urandom
lr-x------. 1 root root 64 Jan 17 14:42 43 -> /var/log/pods/dev01-capture_pgo-56d4d654c9-p4zg2_b2787835-4d06-46f3-8b6a-9c225e7d77d0/operator/5.log
lr-x------. 1 root root 64 Jan 17 14:42 44 -> /var/log/pods/cortex-pipeline_cortex-release-pipeline-sensor-dfs7q-6f9867b5d-gkfhv_9bab9b60-1340-453d-98de-4770746e85d8/main/5.log
lr-x------. 1 root root 64 Jan 17 14:42 45 -> /var/log/pods/dev01-cashpresso_cashpresso-cre-2-out-function-0_74e6f3b0-d9fc-4941-92f0-3c452753889a/pulsar-function/5.log
lr-x------. 1 root root 64 Jan 17 14:42 46 -> /var/log/pods/dev01-cashpresso_cashpresso-f-prod-risk-function-0_dc1b2366-57cb-4c4f-81eb-3db403192466/pulsar-function/5.log
lrwx------. 1 root root 64 Jan 17 14:42 47 -> 'socket:[97534]'
lrwx------. 1 root root 64 Jan 17 14:42 48 -> 'socket:[97535]'
lrwx------. 1 root root 64 Jan 17 14:42 49 -> 'socket:[97536]'
lrwx------. 1 root root 64 Jan 17 14:42 5 -> 'anon_inode:[eventpoll]'
lr-x------. 1 root root 64 Jan 17 14:42 51 -> /var/log/pods/ci-idc-24412-int-scenario-fairo_abc-calccountryrisk-function-0_b0d148fb-ee9e-4c2a-967b-fd8cf8cec552/pulsar-function/5.log
lr-x------. 1 root root 64 Jan 17 14:42 52 -> /var/log/pods/dev01-fcmi_cmi-repo-host-0_502543d6-bf0a-4743-b2fc-bb68610c3a0f/pgbackrest/5.log
lr-x------. 1 root root 64 Jan 17 14:42 53 -> /var/log/pods/ci-idc-24412-rmu-scenario-fairo_fairo-f-wlm-split-function-0_da876b26-1fd0-47f4-bb4d-c1afd2f77bb1/pulsar-function/5.log
lr-x------. 1 root root 64 Jan 17 14:42 55 -> /var/log/pods/ci-idc-24412-int-scenario-fairo_fairo-aml-int-06-2-function-0_372c6c60-7d2b-457d-ad16-e55906c6d633/pulsar-function/5.log
lr-x------. 1 root root 64 Jan 17 14:42 56 -> /var/log/pods/ci-idc-24412-int-scenario-fairo_bae-mx-transformer-function-0_376dd1f2-5eea-4715-bb32-4eff4a0e25c2/pulsar-function/5.log
lr-x------. 1 root root 64 Jan 17 14:42 57 -> /var/log/pods/ci-idc-24412-ftr11-improv-fairo_fairo-f-frc-crs-function-0_28b80eb1-5222-47aa-a482-9da9ded42964/pulsar-function/5.log
lr-x------. 1 root root 64 Jan 17 14:42 58 -> /var/log/pods/dev01-cashpresso_cashpresso-f-wlm-split-function-0_89f2f691-5fc4-41fa-be22-9af92295e2d9/pulsar-function/5.log
lr-x------. 1 root root 64 Jan 17 14:42 59 -> /var/log/pods/ci-idc-24412-int-scenario-fairo_fairo-f-over-risk2-function-0_317282dc-cca4-41c5-bf1d-4e779eb4a05c/pulsar-function/5.log
lr-x------. 1 root root 64 Jan 17 14:42 6 -> 'pipe:[19432]'
lr-x------. 1 root root 64 Jan 17 14:42 60 -> /var/log/pods/dev01-cashpresso_apollo-f-aml-hr-trx-function-0_ff6bb273-fb99-4af0-a5d3-4b7d45947824/pulsar-function/5.log
lr-x------. 1 root root 64 Jan 17 14:42 61 -> /var/log/pods/dev01-neuron-filestreamer_eventbus-default-stan-0_2e497708-ee8a-43b3-aaaf-dfbe98565003/stan/5.log
lr-x------. 1 root root 64 Jan 17 14:42 62 -> /var/log/pods/dev01-neuron-filestreamer_eventbus-default-stan-0_2e497708-ee8a-43b3-aaaf-dfbe98565003/metrics/5.log
lr-x------. 1 root root 64 Jan 17 14:43 63 -> /var/log/pods/cortex-argo_cortex-argoworkflows-argo-workflows-server-75749b6ffd-22t55_f9b96ce3-f82e-4dce-a42a-d9e2c367a7c7/argo-server/5.log
lr-x------. 1 root root 64 Jan 17 14:42 65 -> /var/log/pods/ci-idc-24412-rmu-scenario-fairo_abc-calcproductrisk-function-0_6bd79182-91a9-4baf-9ce9-8e97d19b2bbe/pulsar-function/5.log
lr-x------. 1 root root 64 Jan 17 14:43 68 -> /var/log/pods/dev01-fcmi_cmi-repo-host-0_502543d6-bf0a-4743-b2fc-bb68610c3a0f/nss-wrapper-init/5.log
lr-x------. 1 root root 64 Jan 17 14:43 69 -> /var/log/pods/ci-idc-24412-rmu-scenario-fairo_fairo-ctry-risk-2-function-0_218a91de-52ac-4504-9214-091b128a2be9/pulsar-function/5.log
l-wx------. 1 root root 64 Jan 17 14:42 7 -> 'pipe:[19432]'
lr-x------. 1 root root 64 Jan 17 14:43 72 -> /var/log/pods/dev01-fcmi_cmi-repo-host-0_502543d6-bf0a-4743-b2fc-bb68610c3a0f/pgbackrest-config/5.log
lr-x------. 1 root root 64 Jan 17 14:43 73 -> /var/log/pods/cortex-certmanager_cortex-certmanager-webhook-5dd6c5d77c-2j2cb_4f97561b-3080-48c1-b72d-dacfc5a874c7/cert-manager/5.log
lr-x------. 1 root root 64 Jan 17 14:43 77 -> /var/log/pods/dev01-fairo_fairo-txn-risk-2-function-0_c3abbe43-46d0-49ae-b43c-52e5d468ae98/pulsar-function/5.log
lr-x------. 1 root root 64 Jan 17 14:42 78 -> /var/log/pods/dev01-neuron-pulsar_dev01-neuron-pulsar-bookie-1_124fc7cf-e13f-4f25-989a-866b7c653e97/dev01-neuron-pulsar-bookie/5.log
lr-x------. 1 root root 64 Jan 17 14:43 79 -> /var/log/pods/dev01-neuron-pulsar_dev01-neuron-pulsar-zookeeper-2_0f042e11-6194-471d-bcbd-1875560d4e50/dev01-neuron-pulsar-zookeeper/5.log
lrwx------. 1 root root 64 Jan 17 14:42 8 -> 'socket:[37726]'
l-wx------. 1 root root 64 Jan 17 14:42 9 -> /var/log/datadog/agent.log

Describe what you expected: In this case I would expect "agent" status to report correct number of files tailed.

Steps to reproduce the issue: Hard restart k8s worker node

Additional environment details (Operating System, Cloud provider, etc): Running Fedora on Amazon EC2 Running OKD version 4.11.0-0.okd-2022-12-02-145640

gh123man commented 1 year ago

Fixed in https://github.com/DataDog/datadog-agent/pull/15865