Splunk Universal Forwarder trying to setup HEC

[PA7R14RCH]

Attempting to install a universal forwarder on a host and it continues to fail on task [splunk_universal_forwarder : Setup global HEC]

TASK [splunk_universal_forwarder : Setup global HEC] ***************************
fatal: [localhost]: FAILED! => {
    "cache_control": "no-store, no-cache, must-revalidate, max-age=0",
    "changed": false,
    "connection": "Close",
    "content_length": "168",
    "content_type": "text/xml; charset=UTF-8",
    "date": "Thu, 23 Nov 2023 17:33:25 GMT",
    "elapsed": 0,
    "expires": "Thu, 26 Oct 1978 00:00:00 GMT",
    "redirected": false,
    "server": "Splunkd",
    "status": 400,
    "url": "https://127.0.0.1:8089/services/data/inputs/http/http",
    "vary": "Cookie, Authorization",
    "x_content_type_options": "nosniff",
    "x_frame_options": "SAMEORIGIN"
}

MSG:

Status code was 400 and not [200]: HTTP Error 400: Bad Request

According to Splunk, Universal Forwarders are not setup for HEC for input/output

Splunk Community

Splunk Doc

Is there a chance we could add a conditional to that HEC task if it does need to be there and allow for flush handlers afterwards? I tested removing the task itself and was successful running the universal forwarder container. It took a bit more finesse to get the handlers to run, but I think it's because I don't understand the code enough. Again, I reserve the right to be completely wrong.

Thoughts, Comments, Jokes?

[Iammusa18]

I have also been faicing the same issue for a while now on all 9.x version of image. 8.2.x works fine. Can someone please look into this. Its really proving difficult. I faced similar issues in July too https://github.com/splunk/docker-splunk/issues/557

I am using the image in K8s so i always trigger failures whenever i try to mount custom configs via configMap.

Someone please help. I have support case open but that's giving little traction. Will update this issue if theres a breakthrough there.

ConfigMap

apiVersion: v1
data:
  inputs.conf: |
    # watch all files in <path>
    [monitor:///var/log/containers/app*.log]
    sourcetype = changeme1
    index = changeme

kind: ConfigMap
metadata:
 namespace: dev
 name: splunk-configs
 labels:
   app: splunk-forwarder
   component: agent

Daemonset Manifest Now if i comment out inputs.conf mount in volumeMounts section, it works. Forwarder fails when i try to mount ANY custom configs. Worked perfectly fine before 9.x!

....
     volumeMounts:
            - mountPath: "/opt/splunkforwarder/etc/apps/data/local/inputs.conf"
              subPath: inputs.conf
              readOnly: false
              name: splunk-forwarder-config

...
 volumes:
        - name: splunk-forwarder-config
          configMap:
            name: splunk-configs

[adityapinglesf]

thanks for reporting. looking into the issue @PA7R14RCH @Iammusa18

[ruomeiy-splunk]

Hello @PA7R14RCH @lachmatt, may I ask if this issue still happens? And if possible, could you provide steps for reproducing it?