Event data does not adhere to Splunk Logging Best Practices

[ghost]

Currently this modular input sends data to Splunk with a payload in the following basic shape:

{"push_status": {}, "body": "", "timeout": 60, "id": "6252861133115955122", "reserved_count": 1}

The above does not adhere to Splunk's logging best practices in that it does not include timestamp.

Rather than simply adding a timestamp it seems more appropriate to allow users of this modular input the ability to create a custom python script to transform event data while writing events to the queue. To this end I've created a custom fork with an example transform script integrated.

Would it be possible to make something extensible we can use to transform data without hacking the modular input script at /bin/splunk-ironmq-mi.py?

[treeder]

Would be great to get a PR for your changes, sounds like a good thing to have in this lib.

[ghost]

@treeder I'm glad you agree. I created an example script to reshape the data in a fork of this Modular Input. Things have been renamed a little in attempt to make the prototype modular input I created in the fork installable without affecting existing installations of the IronMQ modular input.

Could you help me find someone to test what I have for functionality-wise before I make it more extensible and worthy of a PR?

To use simply clone the repo and create the gzipped tarball expected by Splunk app uploads with the following command after cloning:

git archive -o ironmq-modular-input_10-ta-segment.tar.gz --prefix=ironmq_ta-segment/ master

The above will use the git archive command to produce an archive file suitable for uploading to Splunk from the Manage apps area of the Splunk UI for testing.

[ghost]

@treeder thoughts? /cc @john-paul-keenan

[john-paul-keenan]

@jhabdas it worked perfectly for me the first time through, I'd say submit the PR.

Thanks!

[ghost]

@john-paul-keenan Thanks for testing. I was having difficulty doing so, so it's great to know the payload shaper is working. I'll submit my PR using the current code used verbatim with some necessary name changes so users can choose to modify their installations prior to uploading to Splunk.

[john-paul-keenan]

Sounds great! Out of curiosity, what was happening on your testing @jhabdas ?

[ghost]

I wasn't seeing data flowing into Splunk from IronMQ. I believe it may have been an integration issue, or something I did to goof up my current Splunk install while developing. If you're seeing data flow I'm feeling good about moving forward and addressing my current Splunk environment setup later.

[john-paul-keenan]

Just ran again to be safe and still working without any problems on my end.

[ghost]

@john-paul-keenan PR submitted and linked. Looking for feedback based on what I've put in place so far. Thanks for keeping this moving!

[john-paul-keenan]

Thanks @jhabdas! I saw that come through, I'm just waiting on the engineering team to review/comment

[ghost]

@john-paul-keenan cool. I'm guessing the feedback we'll see is to have a UI toggle in Splunk to enable/disable the feature. That's a little outside my skillset here but I'm happy to see additional commits from your eng team on my PR.