search

microsoft / AzureFunctionforSplunkVS

An Azure Function to make Azure Monitor telemetry available to a Splunk monitoring system.
MIT License
50 stars 75 forks source link

Time field needs to be forced to appear at beginning of event #9

Closed jijulukose closed 5 years ago

jijulukose commented 6 years ago

Splunk's default MAX_TIMESTAMP_LOOKAHEAD is 128 bytes. Unless the event's time field comes within that, Splunk is going to take HEC event received time as event's time which may not be desirable in some cases. Time field must be forced to appear at the beginning of all event/source types.

sebastus commented 6 years ago

@jijulukose Apologies for the delay getting to this. I'll take a look. Can you give me a before/after example? Are you asking that each new message be refactored to move the time field?

ScottyMac52 commented 5 years ago

Any word on this?

sebastus commented 5 years ago

No - never heard back. It's back on my radar screen.

On Thu, Feb 7, 2019 at 8:47 PM Scott McIntosh notifications@github.com wrote:

Any word on this?

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/Microsoft/AzureFunctionforSplunkVS/issues/9#issuecomment-461589673, or mute the thread https://github.com/notifications/unsubscribe-auth/ABOrrPh5E1abb35zE3Lu0VxXTbILW2YCks5vLJDsgaJpZM4WrnhC .

sebastus commented 5 years ago

This isn't as easy as you might think. The messages come from many different resources and each has its own format. On top of that, new resources emit into the same event hub as they come online. This means that I can't write a dotnet object for each of them to de-serialize into, placing "time" at the beginning. Does it matter the name of the time property? I can add an extra time field and put it up front. Or - maybe better - I can add "time" to the event metadata - as mentioned on this page: http://dev.splunk.com/view/event-collector/SP-CAAAE6P

sebastus commented 5 years ago

I added time to the splunk event metadata. The time of the event is extracted from the message itself and converted to epoch according to HEC metadata definition. This is in the master branch.