Splunk HEC Forwarder - getaddrinfo Error - Domain name not found

[kumarpmd]

Bug Report

Describe the bug amazon/aws-for-fluent-bit:2.21.5 deployed in eks 1.22 cluster fails to resolve splunkcollector domain name.

Manual event trigger from the fluent-bit container was successful.

bash-4.2# curl -k -H "Authorization: Splunk <removed>" https://splunk<removed>.net/services/collector/event -d '{"event":"hello splunk2 world"}'
{"text":"Success","code":0}

To Reproduce configmap/fluent-bit-config

[FILTER]                                                                                                                                                                                                                                    
    Name                modify                                                                                                                                                                                                              
    Match               application.*                                                                                                                                                                                                       
    Set index           <removed>_dev_log                                                                                                                                                                                                     
    Set source          eks                                                                                                                                                                                                                 
    Set sourcetype      <removed>_logs                                                                                                                                                                                                    
    Set host            ${HOST_NAME}                                                                                                                                                                                                        

[OUTPUT]                                                                                                                                                                                                                                    
    name                splunk                                                                                                                                                                                                              
    match               application.*                                                                                                                                                                                                       
    host                https://splunk<removed>.net/services/collector                                                                                                                                                     
    splunk_token        <removed>                                                                                                                                                                                
    tls                 on                                                                                                                                                                                                                  
    tls.verify          off

[net] getaddrinfo(host='https://splunkcollector<removed>.net/services/collector', err=4):

• Steps to reproduce the problem: not sure if this is eks issue. But steps include deploy an eks 1.22 cluster with amazon linux worker instances. Deploy aws-for-fluentbit:2.21.5 chart, and configure fluentbit configmaps with splunk filter and output..

Expected behavior container logs forwarded to splunk index; searchable in splunk.

Your Environment aws eks 1.22 cluster, amazon/aws-for-fluent-bit:2.21.5

Additional context Since manual event trigger is successful, is this an issue with the host resolve logic with/without Authentication Splunk HEC token? Notice fluentd has an fullUrl option. was able to use fluentd to forward logs, but ideally would like fluent-bit to work..

[edsiper]

try out:

• removing https:// and /services/collector from the host parameter

[kumarpmd]

Thanks @edsiper.
Using the host name timeout at upstream. pod hits the alb for splunk collector on 8088, and gets a timeout. Can we do something equivalent to the full_Url, to get around the host, port and path? Here is the output for 443 and 8088 within the fluentbit container -

bash-4.2# curl -k -H "Authorization: Splunk <removed>" https://splunk<removed>.net:8088/services/collector/event -d '{"event":"hello splunk2 world"}'
^C
bash-4.2# curl -k -H "Authorization: Splunk <removed>" https://splunk<removed>.net:443/services/collector/event -d '{"event":"hello splunk2 world"}'
{"text":"Success","code":0}

[OUTPUT] 
    name                splunk
    match               application.*
    host                splunk<removed>.net
    splunk_token        <removed>
    tls                 on
    tls.verify          off

[2022/09/04 20:41:43] [error] [upstream] connection #222 to splunk<removed>.net:8088 timed out after 10 seconds
[2022/09/04 20:41:43] [error] [upstream] connection #225 to splunk<removed>.net:8088 timed out after 10 seconds
[2022/09/04 20:41:47] [error] [upstream] connection #223 to splunk<removed>.net:8088 timed out after 10 seconds

[kumarpmd]

@edsiper any thoughts on this approach - enabling full_URL for Splunk OUTPUT ?

[kumarpmd]

Was able to utilize host and port to stream logs to splunk.. Thank you..