search

amazonlinux / amazon-linux-2023

Amazon Linux 2023
https://aws.amazon.com/linux/amazon-linux-2023/
Other
500 stars 38 forks source link

filebeat service is auto FAILING in Amazon Linux 2023 without any error logs #680

Open rushivpay opened 3 months ago

rushivpay commented 3 months ago

Filebeat Service with same configuration is running properly on Amazon Linux 2 while on Amazon Linux 2023 first it get started and working as expected and then suddenly service goes in FAILED state without any logs.

Below is the details from server.

[root@ip-10-113-52-238 filebeat]# service filebeat status × filebeat.service - Filebeat sends log files to Logstash or directly to Elasticsearch. Loaded: loaded (/usr/lib/systemd/system/filebeat.service; enabled; preset: disabled) Active: failed (Result: exit-code) since Sat 2024-03-30 17:46:51 IST; 27min ago Duration: 87ms Docs: https://www.elastic.co/products/beats/filebeat Process: 5412 ExecStart=/usr/share/filebeat/bin/filebeat --environment systemd $BEAT_LOG_OPTS $BEAT_CONFIG_OPTS $BEAT_PATH_OPTS (code=exited, status=2) Main PID: 5412 (code=exited, status=2) CPU: 66ms

Mar 30 17:46:51 ip-10-113-52-238.ap-south-1.compute.internal filebeat[5412]: cs 0x33 Mar 30 17:46:51 ip-10-113-52-238.ap-south-1.compute.internal filebeat[5412]: fs 0x0 Mar 30 17:46:51 ip-10-113-52-238.ap-south-1.compute.internal filebeat[5412]: gs 0x0 Mar 30 17:46:51 ip-10-113-52-238.ap-south-1.compute.internal systemd[1]: filebeat.service: Main process exited, code=exited, status=2/INVALIDARGUMENT Mar 30 17:46:51 ip-10-113-52-238.ap-south-1.compute.internal systemd[1]: filebeat.service: Failed with result 'exit-code'. Mar 30 17:46:51 ip-10-113-52-238.ap-south-1.compute.internal systemd[1]: filebeat.service: Scheduled restart job, restart counter is at 14. Mar 30 17:46:51 ip-10-113-52-238.ap-south-1.compute.internal systemd[1]: Stopped filebeat.service - Filebeat sends log files to Logstash or directly to Elasticsearch.. Mar 30 17:46:51 ip-10-113-52-238.ap-south-1.compute.internal systemd[1]: filebeat.service: Start request repeated too quickly. Mar 30 17:46:51 ip-10-113-52-238.ap-south-1.compute.internal systemd[1]: filebeat.service: Failed with result 'exit-code'. Mar 30 17:46:51 ip-10-113-52-238.ap-south-1.compute.internal systemd[1]: Failed to start filebeat.service - Filebeat sends log files to Logstash or directly to Elasticsearch..

elsaco commented 3 months ago

@rushivpay use journalctl --unit=filebeat to get more info. journalctl -f -u filebeat will monitor the output on your console. systemctl will tell you the state of the application only.

rushivpay commented 3 months ago

Hello @elsaco ,

Thanks for the info.

PFB Output when i check with journalctl --unit=filebeat

[root@ip-192-168-247-32 filebeat]# journalctl --unit=filebeat Apr 01 16:52:36 ip-192-168-247-32.ap-south-1.compute.internal systemd[1]: Started filebeat.service - Filebeat sends log files to Logstash or directly to Elasticsearch.. Apr 01 16:52:36 ip-192-168-247-32.ap-south-1.compute.internal filebeat[1978]: runtime/cgo: runtime/cgo: pthread_create failed: Operation not permitted Apr 01 16:52:36 ip-192-168-247-32.ap-south-1.compute.internal filebeat[1978]: pthread_create failed: Operation not permitted Apr 01 16:52:36 ip-192-168-247-32.ap-south-1.compute.internal filebeat[1978]: SIGABRT: abort Apr 01 16:52:36 ip-192-168-247-32.ap-south-1.compute.internal filebeat[1978]: PC=0x7f7ebd8a153c m=6 sigcode=18446744073709551610 Apr 01 16:52:36 ip-192-168-247-32.ap-south-1.compute.internal filebeat[1978]: goroutine 0 [idle]: Apr 01 16:52:36 ip-192-168-247-32.ap-south-1.compute.internal filebeat[1978]: runtime: unknown pc 0x7f7ebd8a153c Apr 01 16:52:36 ip-192-168-247-32.ap-south-1.compute.internal filebeat[1978]: stack: frame={sp:0x7f7e8fffe8a0, fp:0x0} stack=[0x7f7e8f5ff1e8,0x7f7e8fffede8) Apr 01 16:52:36 ip-192-168-247-32.ap-south-1.compute.internal filebeat[1978]: 00007f7e8fffe7a0: 0000000000000000 0000000000000000 Apr 01 16:52:36 ip-192-168-247-32.ap-south-1.compute.internal filebeat[1978]: 00007f7e8fffe7b0: 0000000000000000 0000000000000000 Apr 01 16:52:36 ip-192-168-247-32.ap-south-1.compute.internal filebeat[1978]: 00007f7e8fffe7c0: 0000000000000000 0000000000000000 Apr 01 16:52:36 ip-192-168-247-32.ap-south-1.compute.internal filebeat[1978]: 00007f7e8fffe7d0: 0000000000000000 0000000000000000 Apr 01 16:52:36 ip-192-168-247-32.ap-south-1.compute.internal filebeat[1978]: 00007f7e8fffe7e0: 0000000000000000 0000000000000000 Apr 01 16:52:36 ip-192-168-247-32.ap-south-1.compute.internal filebeat[1978]: 00007f7e8fffe7f0: 0000000000000000 0000000000000000 Apr 01 16:52:36 ip-192-168-247-32.ap-south-1.compute.internal filebeat[1978]: 00007f7e8fffe800: 0000000000000000 0000000000000000 Apr 01 16:52:36 ip-192-168-247-32.ap-south-1.compute.internal filebeat[1978]: 00007f7e8fffe810: 0000000000000000 0000000000000000 Apr 01 16:52:36 ip-192-168-247-32.ap-south-1.compute.internal filebeat[1978]: 00007f7e8fffe820: 0000000000000000 0000000000000000 Apr 01 16:52:36 ip-192-168-247-32.ap-south-1.compute.internal filebeat[1978]: 00007f7e8fffe830: 0000000000000000 0000000000000000 Apr 01 16:52:36 ip-192-168-247-32.ap-south-1.compute.internal filebeat[1978]: 00007f7e8fffe840: 0000000000000000 0000000000000000 Apr 01 16:52:36 ip-192-168-247-32.ap-south-1.compute.internal filebeat[1978]: 00007f7e8fffe850: 0000000000000000 0000000000000000 Apr 01 16:52:36 ip-192-168-247-32.ap-south-1.compute.internal filebeat[1978]: 00007f7e8fffe860: 0000000000000000 0000000000000000 Apr 01 16:52:36 ip-192-168-247-32.ap-south-1.compute.internal filebeat[1978]: 00007f7e8fffe870: 0000000000000000 0000000000000000 Apr 01 16:52:36 ip-192-168-247-32.ap-south-1.compute.internal filebeat[1978]: 00007f7e8fffe880: 0000000000000000 0000000000000000 Apr 01 16:52:36 ip-192-168-247-32.ap-south-1.compute.internal filebeat[1978]: 00007f7e8fffe890: 0000000000000000 00007f7ebd8a152e Apr 01 16:52:36 ip-192-168-247-32.ap-south-1.compute.internal filebeat[1978]: 00007f7e8fffe8a0: <0000000000000000 0000000000000000 Apr 01 16:52:36 ip-192-168-247-32.ap-south-1.compute.internal filebeat[1978]: 00007f7e8fffe8b0: 0000000000000000 0000000000000000 Apr 01 16:52:36 ip-192-168-247-32.ap-south-1.compute.internal filebeat[1978]: 00007f7e8fffe8c0: 0000000000000000 0000000000000000 Apr 01 16:52:36 ip-192-168-247-32.ap-south-1.compute.internal filebeat[1978]: 00007f7e8fffe8d0: 0000000000000000 0000000000000000 Apr 01 16:52:36 ip-192-168-247-32.ap-south-1.compute.internal filebeat[1978]: 00007f7e8fffe8e0: 0000000000000000 00007f7ebd94f2a1 Apr 01 16:52:36 ip-192-168-247-32.ap-south-1.compute.internal filebeat[1978]: 00007f7e8fffe8f0: 00007f7e8f5fe640 00007f7e8fffebb0 Apr 01 16:52:36 ip-192-168-247-32.ap-south-1.compute.internal filebeat[1978]: 00007f7e8fffe900: 00007f7e8fffea2e 00007f7e8fffea2f Apr 01 16:52:36 ip-192-168-247-32.ap-south-1.compute.internal filebeat[1978]: 00007f7e8fffe910: 00007f7e8f5fe640 00007f7ebd89f425 Apr 01 16:52:36 ip-192-168-247-32.ap-south-1.compute.internal filebeat[1978]: 00007f7e8fffe920: 0000000000000000 6343b7675d807300 Apr 01 16:52:36 ip-192-168-247-32.ap-south-1.compute.internal filebeat[1978]: 00007f7e8fffe930: 00007f7e8ffff640 0000000000000006 Apr 01 16:52:36 ip-192-168-247-32.ap-south-1.compute.internal filebeat[1978]: 00007f7e8fffe940: 0000000004404820 0000000000000016 Apr 01 16:52:36 ip-192-168-247-32.ap-south-1.compute.internal filebeat[1978]: 00007f7e8fffe950: 00007f7ebd89f520 00007f7ebd854d26

any suggestions for troubleshooting ?