Alerts not showing in Incident Posture due to broken extractions...

[jdeer0618]

155

There was not response to my last issue on the same topic but there appears to still be a problem in the latest version of AM on Splunk 6.5.2. Alert Manager does not properly extract all data in the data model. Am I the only one having this issue? Alert do not show up in the Incident Posture because these fields don't get extracted.

Seems pretty important...

[johnfromthefuture]

Are you on the latest version of AM from splunkbase or github? Just want to be sure.

I'm running 6.5.2 and I ran a similar search to what you posted in the screenshot. Of the 44,627 events found over the last 30 days, I am showing 100% coverage of those key fields.

So my initial thought: do you have the TA-alert_manager addon installed? It has the props file containing search-time extractions relative to this sourcetype.

[jdeer0618]

I am running the latest off of Splunkbase, TA is on all the indexers and search heads. Alert Manager is running on the search heads.

I had the same issue with the last version and wrote a bunch of Rex evals to fix it.

Best Regards,

James A. Deer Jr. (J.D.)

205.789.6025 - Cell* 866.601.4374 - Fax

"A good plan violently executed right now is far better than a perfect plan executed next week." -- George S. Patton

On Mar 1, 2017, 13:21 -0600, johnfromthefuture notifications@github.com, wrote:

Are you on the latest version of AM from splunkbase or github? Just want to be sure.

I'm running 6.5.2 and I ran a similar search to what you posted in the screenshot. Of the 44,627 events found over the last 30 days, I am showing 100% coverage of those key fields.

So my initial thought: do you have the TA-alert_manager addon installed? It has the props file containing search-time extractions relative to this sourcetype.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub (https://github.com/simcen/alert_manager/issues/173#issuecomment-283441496), or mute the thread (https://github.com/notifications/unsubscribe-auth/AEb4qCM1mBr_FXIJtBow_nctsU-83QJoks5rhcVWgaJpZM4MQB0P).

[johnfromthefuture]

Thanks, that's helpful!

So this is the props.conf from the splunkbase app:

[alert_metadata]
TIME_PREFIX = "updated": "
MAX_TIMESTAMP_LOOKAHEAD = 35
SHOULD_LINEMERGE = false
FIELDALIAS-alert_metadata-app = "entry{}.acl.app" AS app
FIELDALIAS-alert_metadata-owner = "entry{}.acl.owner" AS owner
FIELDALIAS-alert_metadata-label = "entry{}.content.label" AS label
FIELDALIAS-alert_metadata-ttl = "entry{}.content.ttl" AS ttl
FIELDALIAS-alert_metadata-eventSearch = "entry{}.content.eventSearch" AS eventSearch
FIELDALIAS-alert_metadata-earliest = "entry{}.content.earliestTime" AS earliest
FIELDALIAS-alert_metadata-latest = "entry{}.content.latestTime" AS latest
FIELDALIAS-alert_metadata-severity_id = severity AS severity_id
FIELDALIAS-alert_metadata-name = "entry{}.name" AS name
TRUNCATE = 104857600

And this is what I have in my production environment:

[alert_metadata]
TIME_PREFIX = "updated": "
MAX_TIMESTAMP_LOOKAHEAD = 35
SHOULD_LINEMERGE = false
#FIELDALIAS-alert_metadata-app = "entry{}.acl.app" AS app
#FIELDALIAS-alert_metadata-owner = "entry{}.acl.owner" AS owner
FIELDALIAS-alert_metadata-label = "entry{}.content.label" AS label
#FIELDALIAS-alert_metadata-ttl = "entry{}.content.ttl" AS ttl
FIELDALIAS-alert_metadata-eventSearch = "entry{}.content.eventSearch" AS eventSearch
#FIELDALIAS-alert_metadata-earliest = "entry{}.content.earliestTime" AS earliest
#FIELDALIAS-alert_metadata-latest = "entry{}.content.latestTime" AS latest
#FIELDALIAS-alert_metadata-severity_id = severity AS severity_id
#FIELDALIAS-alert_metadata-name = "entry{}.name" AS name
EXTRACT-alertmgr01 = \"job_id\":\s+\"(?<job_id>[^\"]+)\" 
EXTRACT-alertmgr02 = \"result_id\":\s+\"(?<result_id>[^\"]+)\" 
EXTRACT-alertmgr03 = \"incident_id\":\s+\"(?<incident_id>[^\"]+)\" 
EXTRACT-alertmgr04 = \"alert\":\s+\"(?<alert>[^\"]+)\"
EXTRACT-alertmgr05 = \"app\":\s+\"(?<app>[^\"]+)\"
EXTRACT-alertmgr06 = \"earliestTime\":\s+\"(?<earliest>[^\"]+)\"
EXTRACT-alertmgr07 = \"impact\":\s+\"(?<impact>[^\"]+)\"
EXTRACT-alertmgr08 = \"latestTime\":\s+\"(?<latest>[^\"]+)\"
EXTRACT-alertmgr09 = \"alert\":\s+\"(?<name>[^\"]+)\"
EXTRACT-alertmgr10 = \"title\":\s+\"(?<title>[^\"]+)\" 
EXTRACT-alertmgr11 = \"ttl\":\s+(?<ttl>\d+)
EXTRACT-alertmgr12 = \"owner\":\s+\"(?<owner>[^\"]+)\"
EXTRACT-alertmgr13 = \"urgency\":\s+\"(?<urgency>[^\"]+)\" 
EXTRACT-alertmgr14 = \"eventSearch\":\s+\"(?<eventSearch>((\\\"|[^\"])+))\"
TRUNCATE = 0

I've last track of what comes from the dev branch verse what I've done to fix issues I've experienced in the past but recommend you drop these into your TA-alert_manager props.conf to see if it corrects the issue. If so, it should be a simple fix to include it in the next splunkbase release.

[jdeer0618]

No dice... It seems like the GIANT "reportSearch" and "normalizedSearch" JSON keys might be the issue but I'm not sure. I don't understand why the getJob in alert_manager.py doesn't just keep the keys from the entry array or dict (not a python guy but I think one of those terms is correct). My JSON objects that get indexed are gigantic.

[johnfromthefuture]

reportSearch and normalizedSearch don't extract well for me either.

So what is happening is alert_manager.py does create the dictionary and the dictionary is sent (json.dumps) to be indexed. Here's a sample of the raw text indexed for a test alert on my test box:

{"result_id": "0", "alert": "top src in pfsense for last hour", "urgency": "medium", "entry": [{"id": "https://127.0.0.1:8089/services/search/jobs/scheduler__admin_VEEtc2VjdXJpdHk__RMD53632494b6e0b2da0_at_1488423600_18", "acl": {"modifiable": true, "perms": {"read": ["admin", "alert_manager", "security"], "write": ["admin", "alert_manager", "security"]}, "app": "TA-security", "can_write": true, "ttl": "240", "sharing": "global", "owner": "admin"}, "name": "search index=pfsense | top limit=1 src", "published": "2017-03-01T22:00:01.000-05:00", "links": {"events": "/services/search/jobs/scheduler__admin_VEEtc2VjdXJpdHk__RMD53632494b6e0b2da0_at_1488423600_18/events", "results": "/services/search/jobs/scheduler__admin_VEEtc2VjdXJpdHk__RMD53632494b6e0b2da0_at_1488423600_18/results", "summary": "/services/search/jobs/scheduler__admin_VEEtc2VjdXJpdHk__RMD53632494b6e0b2da0_at_1488423600_18/summary", "search.log": "/services/search/jobs/scheduler__admin_VEEtc2VjdXJpdHk__RMD53632494b6e0b2da0_at_1488423600_18/search.log", "timeline": "/services/search/jobs/scheduler__admin_VEEtc2VjdXJpdHk__RMD53632494b6e0b2da0_at_1488423600_18/timeline", "results_preview": "/services/search/jobs/scheduler__admin_VEEtc2VjdXJpdHk__RMD53632494b6e0b2da0_at_1488423600_18/results_preview", "alternate": "/services/search/jobs/scheduler__admin_VEEtc2VjdXJpdHk__RMD53632494b6e0b2da0_at_1488423600_18", "control": "/services/search/jobs/scheduler__admin_VEEtc2VjdXJpdHk__RMD53632494b6e0b2da0_at_1488423600_18/control"}, "content": {"doneProgress": 1, "dropCount": 0, "normalizedSearch": "litsearch index=pfsense | addinfo type=count label=prereport_events | fields keepcolorder=t \"cvp_reserved_count\" \"src\" | pretop 1 src", "pid": "30312", "searchProviders": ["pluto"], "sid": "scheduler__admin_VEEtc2VjdXJpdHk__RMD53632494b6e0b2da0_at_1488423600_18", "isZombie": false, "remoteSearch": "litsearch index=pfsense | addinfo  type=count label=prereport_events | fields  keepcolorder=t \"cvp_reserved_count\" \"src\" | pretop  1 src", "diskUsage": 114688, "keywords": "index::pfsense", "isSavedSearch": true, "isTimeCursored": true, "searchLatestTime": 1488423600, "searchEarliestTime": 1488420000, "resultCount": 1, "defaultTTL": "600", "ttl": 240, "reportSearch": "top  limit=1 src", "optimizedSearch": "| search index=pfsense | top limit=1 src", "eventFieldCount": 0, "scanCount": 4649, "canSummarize": true, "resultPreviewCount": 1, "eventCount": 4649, "dispatchState": "DONE", "fieldMetadataResults": {"percent": {"type": "unknown", "type_special": "percent"}, "count": {"type": "unknown", "type_special": "count"}}, "isDone": true, "savedSearchLabel": "{\"owner\":\"admin\",\"app\":\"TA-security\",\"sharing\":\"global\"}", "eventIsStreaming": true, "earliestTime": "2017-03-01T21:00:00.000-05:00", "delegate": "scheduler", "isPaused": false, "searchTotalBucketsCount": 3, "isFinalized": false, "latestTime": "2017-03-01T22:00:00.000-05:00", "eventIsTruncated": true, "fieldMetadataStatic": {"percent": {"type": "unknown", "type_special": "percent"}, "count": {"type": "unknown", "type_special": "count"}}, "isSaved": false, "performance": {"command.search.rawdata": {"duration_secs": 0.013, "invocations": 3}, "command.search.index": {"duration_secs": 0.005, "invocations": 3}, "command.top.execute_input": {"duration_secs": 0.005, "invocations": 5, "output_count": 0, "input_count": 161}, "dispatch.optimize.FinalEval": {"duration_secs": 0.045, "invocations": 1}, "dispatch.check_disk_usage": {"duration_secs": 0.001, "invocations": 1}, "dispatch.fetch": {"duration_secs": 0.14, "invocations": 5}, "command.search.summary": {"duration_secs": 0.001, "invocations": 4}, "command.fields": {"duration_secs": 0.002, "invocations": 4, "output_count": 4649, "input_count": 4649}, "command.search.tags": {"duration_secs": 0.003, "invocations": 3, "output_count": 4649, "input_count": 4649}, "command.search.fieldalias": {"duration_secs": 0.006, "invocations": 3, "output_count": 4649, "input_count": 4649}, "command.search.kv": {"duration_secs": 0.064, "invocations": 3}, "dispatch.evaluate": {"duration_secs": 0.044, "invocations": 1}, "command.search.lookups": {"duration_secs": 0.001, "invocations": 3, "output_count": 4649, "input_count": 4649}, "dispatch.writeStatus": {"duration_secs": 0.008, "invocations": 7}, "command.search.expand_search": {"duration_secs": 0.025, "invocations": 1}, "command.search.batch.sort": {"duration_secs": 0.002, "invocations": 2}, "command.addinfo": {"duration_secs": 0.003, "invocations": 4, "output_count": 4649, "input_count": 4649}, "dispatch.optimize.toSpl": {"duration_secs": 0.001, "invocations": 1}, "command.search.index.usec_1_8": {"invocations": 2}, "command.top.execute_output": {"duration_secs": 0.001, "invocations": 1, "output_count": 0, "input_count": 0}, "command.search.index.usec_8_64": {"invocations": 1}, "command.search": {"duration_secs": 0.085, "invocations": 4, "output_count": 4649, "input_count": 0}, "dispatch.optimize.optimization": {"duration_secs": 0.001, "invocations": 1}, "command.top": {"duration_secs": 0.006, "invocations": 6, "output_count": 0, "input_count": 161}, "dispatch.evaluate.search": {"duration_secs": 0.044, "invocations": 1}, "command.search.batch.cache_setup": {"duration_secs": 0.004, "invocations": 2}, "command.search.typer": {"duration_secs": 0.001, "invocations": 3, "output_count": 4649, "input_count": 4649}, "dispatch.optimize.toJson": {"duration_secs": 0.001, "invocations": 1}, "startup.configuration": {"duration_secs": 0.02, "invocations": 1}, "dispatch.localSearch": {"duration_secs": 0.09, "invocations": 1}, "dispatch.optimize.matchReportAcceleration": {"duration_secs": 0.132, "invocations": 1}, "command.search.calcfields": {"duration_secs": 0.002, "invocations": 3, "output_count": 4649, "input_count": 4649}, "startup.handoff": {"duration_secs": 0.048, "invocations": 1}, "dispatch.evaluate.top": {"duration_secs": 0.001, "invocations": 1}, "dispatch.stream.local": {"duration_secs": 0.091, "invocations": 4}, "command.pretop": {"duration_secs": 0.009, "invocations": 4, "output_count": 161, "input_count": 4649}, "dispatch.createdSearchResultInfrastructure": {"duration_secs": 0.002, "invocations": 1}, "dispatch.optimize.reparse": {"duration_secs": 0.001, "invocations": 1}}, "runDuration": 0.371, "statusBuckets": 0, "eventSearch": "search index=pfsense ", "isBatchModeSearch": true, "numPreviews": 0, "isPreviewEnabled": false, "isEventsPreviewEnabled": false, "sampleSeed": "0", "cursorTime": "1969-12-31T19:00:00.000-05:00", "request": {"ui_dispatch_view": "search", "earliest_time": "-60m@m", "indexedRealtimeMinSpan": "", "ui_dispatch_app": "TA-security", "time_format": "%FT%T.%Q%:z", "buckets": "0", "latest_time": "@m", "sample_ratio": "1", "max_time": "0", "indexedRealtime": "", "lookups": "1", "auto_cancel": "0", "reduce_freq": "10", "max_count": "500000", "rt_maximum_span": "", "auto_pause": "0", "rt_backfill": "0", "index_earliest": "", "indexedRealtimeOffset": "", "spawn_process": "1", "index_latest": ""}, "searchTotalEliminatedBucketsCount": 0, "isFailed": false, "searchCanBeEventType": false, "sampleRatio": "1", "fieldMetadataEvents": {"percent": {"type": "unknown", "type_special": "percent"}, "count": {"type": "unknown", "type_special": "count"}}, "eventSorting": "none", "label": "top src in pfsense for last hour", "messages": [], "isGoodSummarizationCandidate": true, "defaultSaveTTL": "604800", "eventAvailableCount": 0, "resultIsStreaming": false, "isRemoteTimeline": false, "reduceSearch": "sitop  limit=1 src", "priority": 5, "isRealTimeSearch": false}, "author": "admin", "updated": "2017-03-01T22:00:01.688-05:00"}], "owner": "unassigned", "title": "top src in pfsense for last hour", "impact": "medium", "job_id": "scheduler__admin_VEEtc2VjdXJpdHk__RMD53632494b6e0b2da0_at_1488423600_18", "app": "TA-security", "name": "top src in pfsense for last hour", "alert_time": "2017-03-01T22:00:01.000-05:00", "incident_id": "09ecc7ba-17a6-47c1-a4e5-60e339b5f40d", "ttl": 86400, "priority": "medium"}

Splunk typically recognizes this as JSON formatting. It will automatically extract key pairs based on the "key": "value" construct and give you a pretty display when searching the raw data. This is probably why the TA-alert_manager does not have extractions out of the box because it shouldn't be necessary. I've found that it breaks down a bit when the alert searches get complex which is why I introduced the extractions above. It still doesn't work 100% for the eventSearch and normalizedSearch fields - I just need to go back and make a better regex.

So, question: Do all the other fields work (not the normalizedSearch and reportSearch) after adding the above extractions on your search head? The only lost functionality at that point is the ability to click the magnifying glass in the Incident Posture page...

[jdeer0618]

I get that Splunk will pretty print JSON but it still seems necessary to return every single field back from the jobs endpoint. Also, it looks like the keys don't return in the same order from the json.dumps or it could just look that way. It is hard to tell between the large amount of data returned and the browser choking up as well because of event size.

Doing some Googleing it looks like setting sort_keys=True for the dumps might keep the raw data in order if the order is in fact not set when returned.

So this is what I have so far and seems to be working, though it has not been thoroughly tested. index=alerts eventtype="alert_metadata" | rex "\"eventSearch\":\s+\"(?((\\\"|[^\"])+))\"" | rex "\"name\":\s+\"(?

(?:\s?search\s?|\s?[|]\s?)(?:\\\"|[^\"])*?)\"" | rex field=search mode=sed "s/\\n/ /g" | rex "\"urgency\":\s+\"(?[^\"]+)\"" | rex "\"alert_time\":\s+\"(?[^\"]+)\"" | rex "\"owner\":\s+\"(?[^\"]+)\"" | rex "\"ttl\":\s+(?\d+)" | rex "\"title\":\s+\"(?