aws / aws-for-fluent-bit

The source of the amazon/aws-for-fluent-bit container image
Apache License 2.0
437 stars 130 forks source link

ECS Fargate fluent-bit side-car container exit with code 139 #840

Open zinho9 opened 3 days ago

zinho9 commented 3 days ago
### Describe the question/issue Hi , I am using aws fluent-bit image as sidecar for my server container. I had to send logs to s3 either opensearch ingestion, so I add configs (for OUTPUT section to s3 and opensearch ingestion) and then re-build the aws-for-fluent-bit image. Here is the config I had added, ``` [INPUT] Name forward unix_path /var/run/fluent.sock Mem_Buf_Limit 200MB [OUTPUT] Name http Match * Host Port 443 URI format json aws_auth true aws_region ap-northeast-2 aws_service osis aws_role_arn tls On net.keepalive false [OUTPUT] Name s3 Match * bucket region us-west-2 total_file_size 50M upload_timeout 60m use_put_object Off static_file_path On s3_key_format /$TAG[0]-$TAG[1]-$TAG[2]-$TAG[3]/%Y%m%d/%H%M%S.log s3_key_format_tag_delimiters ".-" store_dir /var/fluent-bit/state/flb-storage/s3 compression gzip net.keepalive false ``` And this is my Dockerfile ``` FROM amazon/aws-for-fluent-bit:stable ADD ./logDestinations.conf /fluent-bit/alt/fluent-bit.conf CMD ["/fluent-bit/bin/fluent-bit", "-c", "/fluent-bit/alt/fluent-bit.conf"] ``` Finally, my application logs are successfully send to s3 and opensearch. But there is some problems. This fluent-bit container is terminating in 2~4 hours after running. For debug logs , I added FLB_LOG_LEVEL key while value is debug. And then I add net.keepalive false in fluentbit config OUTPUT section. But exit code 139 is not resolved. This is the logs when fluent-bit container terminated with 139 code. ``` #1 0x4fd429 in mk_list_del() at lib/monkey/include/monkey/mk_core/mk_list.h:147 #2 0x4fe33d in prepare_destroy_conn() at src/flb_upstream.c:466 #3 0x4fe3c0 in prepare_destroy_conn_safe() at src/flb_upstream.c:492 #4 0x4fec1b in cb_upstream_conn_ka_dropped() at src/flb_upstream.c:752 #5 0x4e7a7c in output_thread() at src/flb_output_thread.c:300 #6 0x5002da in step_callback() at src/flb_worker.c:43 #7 0x7f0aaf19b44a in ???() at ???:0 #8 0x7f0aad59552e in ???() at ???:0 #9 0xffffffffffffffff in ???() at ???:0 [2024/07/02 05:36:17] [engine] caught signal (SIGSEGV) [2024/07/02 05:36:17] [debug] [upstream] KA connection #40 to sts.ap-northeast-2.amazonaws.com:443 has been disconnected by the remote service [2024/07/02 05:36:17] [debug] [socket] could not validate socket status for #40 (don't worry) [2024/07/02 05:36:17] [debug] [upstream] drop keepalive connection #-1 to sts.ap-northeast-2.amazonaws.com:443 (keepalive idle timeout) [2024/07/02 05:35:47] [ info] [output:http:http.0] :443, HTTP status=200 200 OK [2024/07/02 05:35:47] [debug] [out flush] cb_destroy coro_id=14 [2024/07/02 05:35:47] [debug] [task] destroy task=0x7f0aa6a91a80 (task_id=0) [2024/07/02 05:35:47] [debug] [upstream] KA connection #40 to sts.ap-northeast-2.amazonaws.com:443 is now available [2024/07/02 05:35:46] [debug] [http_client] not using http_proxy for header [2024/07/02 05:35:46] [debug] [aws_credentials] Retrieving credentials from the HTTP provider.. [2024/07/02 05:35:46] [debug] [http_client] not using http_proxy for header [2024/07/02 05:35:46] [debug] [output:http:http.0] signing request with AWS Sigv4 [2024/07/02 05:35:46] [debug] [aws_credentials] Requesting credentials from the STS provider.. [2024/07/02 05:35:46] [debug] [aws_credentials] STS Provider: Refreshing credential cache. [2024/07/02 05:35:46] [debug] [aws_credentials] Calling STS.. [2024/07/02 05:35:46] [debug] [task] created task=0x7f0aa6a91a80 id=0 OK [2024/07/02 05:35:46] [debug] [output:http:http.0] task_id=0 assigned to thread #1 [2024/07/02 05:35:46] [debug] [output:s3:s3.1] task_id=0 assigned to thread #0 [2024/07/02 05:35:46] [debug] [out flush] cb_destroy coro_id=29 [2024/07/02 05:35:45] [debug] [input chunk] update output instances with new chunk size diff=938, records=1, input=forward.0 [2024/07/02 05:35:41] [debug] [output:s3:s3.1] Running upload timer callback (cb_s3_upload).. ``` Please check my problem , help me.. ### Configuration

Fluent Bit Log Output

Fluent Bit Version Info

Cluster Details

Application Details

Steps to reproduce issue

Related Issues