fluent-bit service fails due to segmentation fault

[diegoauad]

Describe the bug google-cloud-ops-agent-fluent-bit.service fails to start. A 'segmentation fault' error is shown in journalctl logs.

To Reproduce Steps to reproduce the behavior:

1. Start a GCE VM with Debian 10 image.
2. Install MongoDB Community Edition 5.0.11
3. Install Ops Agent 2.32.0
4. Apply the example configuration provided in the guide about Operations Suite for MongoDB
5. In our case, it worked fine for some hours. Then the fluent-bit.service failed and was unable to start again.

Expected behavior All Agent Ops services working correctly, or at least an informative error message about what's misconfigured.

Environment (please complete the following information):

• Project ID pulse-api-370218
• VM ID 7971989526812200110
• VM distro / OS: Debian 10
• Ops Agent version 2.32.0
• Ops Agent configuration MongoDB operations suite guide
• Ops Agent log

  May 23 16:46:26 mongodb-dev-1 google_cloud_ops_agent_engine[17334]: 2023/05/23 16:46:26 Built-in config:
  May 23 16:46:26 mongodb-dev-1 google_cloud_ops_agent_engine[17334]: logging:
  May 23 16:46:26 mongodb-dev-1 google_cloud_ops_agent_engine[17334]:   receivers:
  May 23 16:46:26 mongodb-dev-1 google_cloud_ops_agent_engine[17334]:     syslog:
  May 23 16:46:26 mongodb-dev-1 google_cloud_ops_agent_engine[17334]:       type: files
  May 23 16:46:26 mongodb-dev-1 google_cloud_ops_agent_engine[17334]:       include_paths:
  May 23 16:46:26 mongodb-dev-1 google_cloud_ops_agent_engine[17334]:       - /var/log/messages
  May 23 16:46:26 mongodb-dev-1 google_cloud_ops_agent_engine[17334]:       - /var/log/syslog
  May 23 16:46:26 mongodb-dev-1 google_cloud_ops_agent_engine[17334]:   service:
  May 23 16:46:26 mongodb-dev-1 google_cloud_ops_agent_engine[17334]:     pipelines:
  May 23 16:46:26 mongodb-dev-1 google_cloud_ops_agent_engine[17334]:       default_pipeline:
  May 23 16:46:26 mongodb-dev-1 google_cloud_ops_agent_engine[17334]:         receivers: [syslog]
  May 23 16:46:26 mongodb-dev-1 google_cloud_ops_agent_engine[17334]: metrics:
  May 23 16:46:26 mongodb-dev-1 google_cloud_ops_agent_engine[17334]:   receivers:
  May 23 16:46:26 mongodb-dev-1 google_cloud_ops_agent_engine[17334]:     hostmetrics:
  May 23 16:46:26 mongodb-dev-1 google_cloud_ops_agent_engine[17334]:       type: hostmetrics
  May 23 16:46:26 mongodb-dev-1 google_cloud_ops_agent_engine[17334]:       collection_interval: 60s
  May 23 16:46:26 mongodb-dev-1 google_cloud_ops_agent_engine[17334]:   processors:
  May 23 16:46:26 mongodb-dev-1 google_cloud_ops_agent_engine[17334]:     metrics_filter:
  May 23 16:46:26 mongodb-dev-1 google_cloud_ops_agent_engine[17334]:       type: exclude_metrics
  May 23 16:46:26 mongodb-dev-1 google_cloud_ops_agent_engine[17334]:       metrics_pattern: []
  May 23 16:46:26 mongodb-dev-1 google_cloud_ops_agent_engine[17334]:   service:
  May 23 16:46:26 mongodb-dev-1 google_cloud_ops_agent_engine[17334]:     pipelines:
  May 23 16:46:26 mongodb-dev-1 google_cloud_ops_agent_engine[17334]:       default_pipeline:
  May 23 16:46:26 mongodb-dev-1 google_cloud_ops_agent_engine[17334]:         receivers: [hostmetrics]
  May 23 16:46:26 mongodb-dev-1 google_cloud_ops_agent_engine[17334]:         processors: [metrics_filter]
  May 23 16:46:26 mongodb-dev-1 google_cloud_ops_agent_engine[17334]: 2023/05/23 16:46:26 Merged config:
  May 23 16:46:26 mongodb-dev-1 google_cloud_ops_agent_engine[17334]: logging:
  May 23 16:46:26 mongodb-dev-1 google_cloud_ops_agent_engine[17334]:   receivers:
  May 23 16:46:26 mongodb-dev-1 google_cloud_ops_agent_engine[17334]:     mongodb:
  May 23 16:46:26 mongodb-dev-1 google_cloud_ops_agent_engine[17334]:       type: mongodb
  May 23 16:46:26 mongodb-dev-1 google_cloud_ops_agent_engine[17334]:     syslog:
  May 23 16:46:26 mongodb-dev-1 google_cloud_ops_agent_engine[17334]:       type: files
  May 23 16:46:26 mongodb-dev-1 google_cloud_ops_agent_engine[17334]:       include_paths:
  May 23 16:46:26 mongodb-dev-1 google_cloud_ops_agent_engine[17334]:       - /var/log/messages
  May 23 16:46:26 mongodb-dev-1 google_cloud_ops_agent_engine[17334]:       - /var/log/syslog
  May 23 16:46:26 mongodb-dev-1 google_cloud_ops_agent_engine[17334]:   service:
  May 23 16:46:26 mongodb-dev-1 google_cloud_ops_agent_engine[17334]:     pipelines:
  May 23 16:46:26 mongodb-dev-1 google_cloud_ops_agent_engine[17334]:       default_pipeline:
  May 23 16:46:26 mongodb-dev-1 google_cloud_ops_agent_engine[17334]:         receivers: [syslog]
  May 23 16:46:26 mongodb-dev-1 google_cloud_ops_agent_engine[17334]:       mongo:
  May 23 16:46:26 mongodb-dev-1 google_cloud_ops_agent_engine[17334]:         receivers: [mongodb]
  May 23 16:46:26 mongodb-dev-1 google_cloud_ops_agent_engine[17334]: metrics:
  May 23 16:46:26 mongodb-dev-1 google_cloud_ops_agent_engine[17334]:   receivers:
  May 23 16:46:26 mongodb-dev-1 google_cloud_ops_agent_engine[17334]:     hostmetrics:
  May 23 16:46:26 mongodb-dev-1 google_cloud_ops_agent_engine[17334]:       type: hostmetrics
  May 23 16:46:26 mongodb-dev-1 google_cloud_ops_agent_engine[17334]:       collection_interval: 60s
  May 23 16:46:26 mongodb-dev-1 google_cloud_ops_agent_engine[17334]:     mongodb:
  May 23 16:46:26 mongodb-dev-1 google_cloud_ops_agent_engine[17334]:       type: mongodb
  May 23 16:46:26 mongodb-dev-1 google_cloud_ops_agent_engine[17334]:       insecure: true
  May 23 16:46:26 mongodb-dev-1 google_cloud_ops_agent_engine[17334]:       insecure_skip_verify: null
  May 23 16:46:26 mongodb-dev-1 google_cloud_ops_agent_engine[17334]:       cert_file: ""
  May 23 16:46:26 mongodb-dev-1 google_cloud_ops_agent_engine[17334]:       key_file: ""
  May 23 16:46:26 mongodb-dev-1 google_cloud_ops_agent_engine[17334]:       ca_file: ""
  May 23 16:46:26 mongodb-dev-1 google_cloud_ops_agent_engine[17334]:       collection_interval: ""
  May 23 16:46:26 mongodb-dev-1 google_cloud_ops_agent_engine[17334]:   processors:
  May 23 16:46:26 mongodb-dev-1 google_cloud_ops_agent_engine[17334]:     metrics_filter:
  May 23 16:46:26 mongodb-dev-1 google_cloud_ops_agent_engine[17334]:       type: exclude_metrics
  May 23 16:46:26 mongodb-dev-1 google_cloud_ops_agent_engine[17334]:       metrics_pattern: []
  May 23 16:46:26 mongodb-dev-1 google_cloud_ops_agent_engine[17334]:   service:
  May 23 16:46:26 mongodb-dev-1 google_cloud_ops_agent_engine[17334]:     pipelines:
  May 23 16:46:26 mongodb-dev-1 google_cloud_ops_agent_engine[17334]:       default_pipeline:
  May 23 16:46:26 mongodb-dev-1 google_cloud_ops_agent_engine[17334]:         receivers: [hostmetrics]
  May 23 16:46:26 mongodb-dev-1 google_cloud_ops_agent_engine[17334]:         processors: [metrics_filter]
  May 23 16:46:26 mongodb-dev-1 google_cloud_ops_agent_engine[17334]:       mongo:
  May 23 16:46:26 mongodb-dev-1 google_cloud_ops_agent_engine[17334]:         receivers: [mongodb]
  May 23 16:46:26 mongodb-dev-1 systemd[1]: Started Google Cloud Ops Agent - Logging Agent.
  May 23 16:46:27 mongodb-dev-1 google_cloud_ops_agent_wrapper[17343]: 2023/05/23 16:46:27 signal: segmentation fault
  May 23 16:46:27 mongodb-dev-1 systemd[1]: google-cloud-ops-agent-fluent-bit.service: Main process exited, code=exited, status=255/EXCEPTION
  May 23 16:46:27 mongodb-dev-1 systemd[1]: google-cloud-ops-agent-fluent-bit.service: Failed with result 'exit-code'.
  May 23 16:46:27 mongodb-dev-1 systemd[1]: google-cloud-ops-agent-fluent-bit.service: Service RestartSec=100ms expired, scheduling restart.
  May 23 16:46:27 mongodb-dev-1 systemd[1]: google-cloud-ops-agent-fluent-bit.service: Scheduled restart job, restart counter is at 4.
  May 23 16:46:27 mongodb-dev-1 systemd[1]: Stopped Google Cloud Ops Agent - Logging Agent.

Additional context We are running MongoDB as a standalone replica, with authentication enabled. Monitoring works fine, we are experiencing issues with logging only.

[braydonk]

Hi @diegoauad, thank you for opening an issue. I have been investigating some similar issues other users have reported with MongoDB logging. Particularly we have seen issues with particularly complicated nested logs coming from MongoDB.

Would you please open a support case so that we can assist in greater detail? In the support case, if you could include the output from our diagnostic tool as well as some representative Mongo logs if possible. You can also mention this GitHub Issue in the case.

If you do open a support case, please respond here with the case number so we can investigate.

[RafikFarhad]

I think this is not solved yet. We have almost 200+ VMs with Mongo workload. On some new VMs, we have the latest ops-agent 2.48.0, where we see a similar issue, and the other VMs with the earlier versions do not show this kind of issue.