Bad Request from Splunk HEC

[jimdodson]

I'm trying to use the connector with Confluent Platform.

The connector loads and appears to receive the events from the kafka topic, but when sending to Splunk I always get HTTP 400 Bad Request error in the kafka connect log.

How can I get more detailed information about what the actual error is? Or is there some issue with my configuration?

I've tried using a kafka producer sending events that look like:

{"eventKey":"0", "uuid":"21999a4b-65a7-4a0d-9a27-0d632e1b8126", "message":"wboweuzyvh"}

I've tried the console producer with something simple like:

echo event 123 | confluent produce kafka-demo

My connector configuration:

topics=kafka-demo
tasks.max=1
connector.class=com.splunk.kafka.connect.SplunkSinkConnector
splunk.indexes=main
splunk.hec.uri=http://localhost:8088
splunk.hec.token=<token>
confluent.topic.bootstrap.servers=localhost:9092
confluent.topic.replication.factor=1
key.converter=org.apache.kafka.connect.storage.StringConverter
value.converter=org.apache.kafka.connect.storage.StringConverter
key.converter.schemas.enable=false
value.converter.schemas.enable=false
name=SplunkSink

The Splunk HEC successfully receives this test msg with no error:

curl -k http://localhost:8088/services/collector/event -H "Authorization: Splunk <token>" -d '{"event": "hello world"}'

I've tried Confluent Platform 5.2.1 and 4.1.3

I've tried a local install of Splunk 7.3.0 and docker images of Splunk 7.3.0 and 6.5.0

To set up my HEC, I followed the docs here https://docs.splunk.com/Documentation/Splunk/latest/Data/UsetheHTTPEventCollector

[dtregonning]

Hi @jimdodson - sorry to hear your having issues.

Couple of things we can do:

Enable debug logging - this should provide some more context on what the error is: https://docs.splunk.com/Documentation/KafkaConnect/latest/User/Troubleshootyourdeployment#Enable_verbose_logging

Check on Indexer Acknowledgement whether it is enabled or disabled in Splunk. By default the connector will not use indexer-ack: https://docs.splunk.com/Documentation/KafkaConnect/latest/User/Troubleshootyourdeployment#Error:_ACK_is_disabled

If you are still having issues you can post through a sample of your logs to dataedge@splunk.com with any sensitive information removed.

[jimdodson]

Thanks for the reply!

The HEC has Indexer Acknowledgement disabled.

Enabling debug logging for com.splunk did not give any more helpful info. I enabled debug for org.apache.http and was able to see the payload being sent.

it was sending {"time":1561143416.832000,"index":"main","event":"{\"eventKey\":\"0\", \"uuid\":\"81e28fa1-993e-42f8-bd58-fe3efe73d3e9\", \"message\":\"iiqynobkzs\"}"}

so I put that into curl, and got {"text":"Invalid data format","code":6,"invalid-event-number":0}

Googling that gives some hits but nothing too helpful. But one thing did mention the need for a sourcetype field, which wasn't getting sent.

so I added splunk.sourcetypes=some_sourcetype to the config, and the connector sent {"time":1561144452.611000,"sourcetype":"some_sourcetype","index":"main","event":"{\"eventKey\":\"0\", \"uuid\":\"88b5e9fd-ebe0-4fe1-aeeb-b3d583ec9cfd\", \"message\":\"dnnpfbbwic\"}"}

which still got the Bad Request back... but, I put that into curl:

curl -k  http://localhost:8088/services/collector/event -H "Authorization: Splunk <token>" -d '{"time":1561144452.611000,"sourcetype":"some_sourcetype","index":"main","event":"{\"eventKey\":\"0\", \"uuid\":\"88b5e9fd-ebe0-4fe1-aeeb-b3d583ec9cfd\", \"message\":\"dnnpfbbwic\"}"}'
{"text":"Success","code":0}%

I'm not sure this proves anything, since curl with {"event": "hello world"} works, but I'm hoping it might be helpful.

I can send the logs to you if you would like - I really appreciate the help!

[jimdodson]

I sent a kafka connect log from when the error happens.

Please let me know if any other info would be helpful.

Thanks!

[dtregonning]

Thanks @jimdodson for sending that through yesterday and thanks for working with the team to test out some tweaks to your environment.

Summary of Issue: Confluent KSQL service shares a default port with Splunk's HTTP Event Collector - 8088.

• KSQL: https://docs.confluent.io/current/ksql/docs/operations.html#check-the-ksql-server-configuration
• Splunk: https://docs.splunk.com/Documentation/Splunk/7.3.0/Data/UsetheHTTPEventCollector#Send_data_to_HTTP_Event_Collector_on_Splunk_Enterprise

Solution: Modify default port for Splunk HEC or Confluent KSQL service to ensure they are different. And ensure no service is running on the Splunk HEC port selected.