Source :: splunk_hec not accepting splunk logs

atbagan commented 2 years ago

Community Note

Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
If you are interested in working on this issue or have submitted a pull request, please leave a comment

Vector Version

0.19.1

Vector Configuration File

live infra test setup
[sources.splunk_hec]
  type = "splunk_hec"
  address = "0.0.0.0:8088"
  store_hec_token = true

[sinks.hec]
  inputs = ["splunk_hec"]
  type = "splunk_hec_logs"
  healthcheck = true
  endpoint = "_REDACTED_"
  host_key = ""

  default_token = "_REDACTED_"
  index = "{{index}}"
  encoding.codec = "json"
  compression = "none"

---------------------------
ec2 test setup
[sources.splunk_hec]
type = "splunk_hec"
address = "0.0.0.0:8080"

[sinks.hec]
  inputs = ["splunk_hec"]
  type = "splunk_hec_logs"
  healthcheck = true
  endpoint = "_REDACTED_"
  host_key = ""

  default_token = "_REDACTED_"
  index = "_REDACTED_"
  encoding.codec = "json"
  compression = "none"

Debug Output

Full debug output shows nothing of note.

Expected Behavior

Logs sent via Universal Forwarder should have been correctly accepted by the vector source splunk_hec and then routed to our splunk endpoint using splunk_hec_logs sink.

Actual Behavior

Vector source splunk_hec seems to not be properly accepting traffic from a Splunk UF

Example Data

Additional Context

Full Context, and the scenarios tested: Our actual live infrastructure for our obs pipeline is setup in AWS. Running in a ecs cluster on fargate. The initial situation that made me aware of this is the following:

I was sending logs via a Splunk Universal Forwarder (UF) through an application load balancer(ALB) that sits in front of the before mentioned infrastructure. I know the traffic was correctly hitting the ALB from the metrics on the load balancer. I kept receiving 4xx count increases so I checked with a curl and received a 404 error when trying to curl the ALB dns on the port that vector was listening on for the splunk source. In my case I bound it to 8088.

To expand on testing I installed both vector and a UF on an ec2 instance. I setup vector with the splunk_hec source bound to localhost:8080 with its sink being our splunk hec endpoint directly (which I know works when I send logs from vector using other sources, i.e. socket). I setup the UF to send httpout to localhost:8080 and was met with the same results as in my live infrastructure. Nothing picked up via vector, and no errors. vector top showed N/A for all sources/sinks.

To test once again I decided to just send a singular log via cURL to localhost:8080 where vector was listening with the splunk_hec source. here is the output from that

bout to connect() to 127.0.0.1 port 8080 (#0)
*   Trying 127.0.0.1...
* Connected to 127.0.0.1 (127.0.0.1) port 8080 (#0)
> POST / HTTP/1.1
> User-Agent: curl/7.29.0
> Host: 127.0.0.1:8080
> Accept: */*
> Authorization: Splunk _REDACTED_
> Content-Length: 78
> Content-Type: application/x-www-form-urlencoded
>
* upload completely sent off: 78 out of 78 bytes
< HTTP/1.1 404 Not Found
< content-length: 0
< date: Thu, 10 Feb 2022 00:22:37 GMT
<
* Connection #0 to host 127.0.0.1 left intact
*

The version of UF's used today were 8.1 and 8.2.4 I can try a pre 8 version tomorrow.

References

Note

If you need more information I can provide whatever you need. Thanks, AB

atbagan commented 2 years ago

@jszwedko here you go

atbagan commented 2 years ago

So a small update. This seems to work fine when being sent to vector via a splunk Lambda that is designed to send to an HEC endpoint. I haven't had a chance to try an older UF version (pre 8.x). Can still confirm that a 8.x UF just doesn't work

dastardlyquasi commented 2 years ago

When testing with the universal http output: (outputs.conf from the Splunk Forwarder). `[default] sendCookedData = false

[httpout] httpEventCollectorToken = my_token uri = endpoint_value`

We were not able to see data coming into vector using the splunk hec source. As a workaround we tried configuring http source.

{ "sources": { "s_http_endpoint": { "type": "http", "address": "0.0.0.0:7389", "strict_path": false, "path": "" } } }

We were then able to see data coming in from the universal forwarder to "path": path: /services/collector/s2s

However, the message seems to have a unique splunk encoding. To try and resolve this, we added "sendCookedData" set to false in the outputs.conf (Splunk Universal Forwarder config).

atbagan commented 2 years ago

Yeah, that sort of makes sense. I'm pretty sure their HEC source is only acting like a receiving splunk would act so that is why when you try to just send it to the http source you end up with all the splunk encoding. Splunk being Splunk is very restrictive on where their stuff can send data and also restrictive on what can receive that data. The HEC source once fixed should resolve any of that.

Also, trying to send from a UF via tcpout doesn't work either when trying to send to socket. It's just splunk doing splunk things and expects that the receiving end is a splunk endpoint to accept the traffic.

When you turned that value to false did it do anything for you?

spencergilbert commented 2 years ago

So, as far as I can tell httpout from the Universal Forwarder requires support for s2s which we don't have (see: https://github.com/vectordotdev/vector/issues/2537 and https://github.com/vectordotdev/vector/issues/3848). We could possibly leverage a socket source and consume from their tcpout, this could also be an enhancement to the existing splunk source, or even a new component... (see: https://github.com/vectordotdev/vector/issues/4562).

Had this worked on a previous version of Vector? Also, it seems like sendCookedData is only applied to tcpout configurations, not httpout.

atbagan commented 2 years ago

I would love to see tcpout support for the socket source. That would be a huge enhancement. That would be great and would be ideal for my situation.

Back to HEC. What is the HEC source intended use then? If it isn't for UF sending httpout HEC? I know there are other ways and other methods of sending HEC, but I guess I just assumed it was supposed to be used that way.

I am not sure if this worked on previous versions of vector. I would assume not. As earlier versions of UF don't even have HEC support.

jszwedko commented 2 years ago

I believe httpout for Splunk prior to 8.1 was using HEC. It appears this changed in Splunk 8.1 to use their custom cooked encoding:

The release of version 8.1.0 of the Splunk Universal Forwarder introduced a brand new feature to support sending data over HTTP. Traditionally, a Splunk Universal Forwarder uses the proprietary Splunk-to-Splunk (S2S) protocol for communicating with the Indexers. Using the ‘HTTP Out Sender for Universal Forwarder’ it can now send data to a Splunk Indexer using HTTP. What this feature does is effectively encapsulates the S2S message within a HTTP payload. Additionally, this now enables the use of a 3rd party load-balancer between Universal Forwarders and Splunk Receivers. To date, this is a practice which has not been recommended, or supported, for traditional S2S based data forwarding.

https://discoveredintelligence.ca/solving-roaming-users-http-out-for-the-splunk-universal-forwarder/

atbagan commented 2 years ago

I’m still not sure. Did this work prior? And break with their new changes, or was this hec source not intended to be used with a UF? Which to me wouldn’t make much sense as a UF is the backbone of many consumers of splunk.

I would just have everyone use vector and not even bother with this, but some groups and teams are slow to change so I have to accommodate them within our pipeline. In a company as big as ours our landscape is very diverse

last edit, I’m having someone on my team test splunk logging driver to the hec source this week and should have the results of that soon to report back here.

atbagan commented 2 years ago

Confirmed working with Splunk Logging Driver. Running an application on ECS Fargate sending to a Vector sidecar with the splunk_hec source and routing that through our pipeline to it's end location.

jszwedko commented 2 years ago

@atbagan Apologies, for some reason I thought the httpout output predated Splunk 8, but I can't find it documented for Splunk 7 now in which case, you are correct, that the splunk_hec source will not work with the Splunk UF which seems to only be able to write its proprietary cooked format for HTTP. The source should work with any other data source capable of writing data using the HEC protocol (e.g. the log drivers for Fargate, log driver for Docker, fluentd, etc.).

I think Spencer appropriately summarized in https://github.com/vectordotdev/vector/issues/11292#issuecomment-1048262964 . I think the paths forward here are:

Supporting the Splunk cooked protocol. This is difficult since it is not a documented protocol, but is being tracked by #3848
Verifying that the UF's tcpout with sendCookedData: false works with Vector's socket source and documenting it if so. This is being tracked by https://github.com/vectordotdev/vector/issues/4562

Given that I think this issue is represented by the above two, I'll close this out, but please feel free to leave any additional comments on those issues (or here if you think there is something not covered by those ones).

atbagan commented 2 years ago

@jszwedko Thanks for this. I can confirm that sending sendCookedData: false to vector's socket indeed works. I will look at the linked issue to see if I can provide any context. I will say that while it works, you are unable to use the _meta argument from within the UF, adding tags, which poses a problem when you route based on tagged information.

vectordotdev / vector