splunk / splunk-add-on-microsoft-azure

Splunk Add-on for Microsoft Azure
Apache License 2.0
11 stars 7 forks source link

Difficulty getting Alerts pulled #39

Closed kcphilipm closed 3 months ago

kcphilipm commented 1 year ago

I have configured several inputs with the add-on but have not been able to retrieve alerts from the Security Center. I have successfully been able to pull tasks. What could be the cause of this? I have enabled debugging in the logging. What is the best way to manually kick off an 'Alerts' pull from the command line using the python scripts?

Debug Log: 2023-01-09 10:45:17,220 INFO pid=25023 tid=MainThread file=splunk_rest_client.py:_request_handler:99 | Use HTTP connection pooling 2023-01-09 10:45:17,222 DEBUG pid=25023 tid=MainThread file=binding.py:get:695 | GET request to https://127.0.0.1:8089/servicesNS/nobody/TA-MS-AAD/storage/collections/config/TA_MS_AAD_checkpointer (body: {}) 2023-01-09 10:45:17,224 DEBUG pid=25023 tid=MainThread file=connectionpool.py:_new_conn:1007 | Starting new HTTPS connection (1): 127.0.0.1:8089 2023-01-09 10:45:17,229 DEBUG pid=25023 tid=MainThread file=connectionpool.py:_make_request:465 | https://127.0.0.1:8089 "GET /servicesNS/nobody/TA-MS-AAD/storage/collections/config/TA_MS_AAD_checkpointer HTTP/1.1" 200 5307 2023-01-09 10:45:17,230 DEBUG pid=25023 tid=MainThread file=binding.py:new_f:74 | Operation took 0:00:00.007901 2023-01-09 10:45:17,230 DEBUG pid=25023 tid=MainThread file=binding.py:get:695 | GET request to https://127.0.0.1:8089/servicesNS/nobody/TA-MS-AAD/storage/collections/config/ (body: {'count': -1, 'offset': 0, 'search': 'TA_MS_AAD_checkpointer'}) 2023-01-09 10:45:17,234 DEBUG pid=25023 tid=MainThread file=connectionpool.py:_make_request:465 | https://127.0.0.1:8089 "GET /servicesNS/nobody/TA-MS-AAD/storage/collections/config/?count=-1&offset=0&search=TA_MS_AAD_checkpointer HTTP/1.1" 200 4505 2023-01-09 10:45:17,234 DEBUG pid=25023 tid=MainThread file=binding.py:new_f:74 | Operation took 0:00:00.004127 2023-01-09 10:45:17,235 DEBUG pid=25023 tid=MainThread file=binding.py:get:695 | GET request to https://127.0.0.1:8089/servicesNS/nobody/TA-MS-AAD/storage/collections/data/TA_MS_AAD_checkpointer/asc_alert_last_date_Azure_Security_Center_Alerts (body: {}) 2023-01-09 10:45:17,238 DEBUG pid=25023 tid=MainThread file=connectionpool.py:_make_request:465 | https://127.0.0.1:8089 "GET /servicesNS/nobody/TA-MS-AAD/storage/collections/data/TA_MS_AAD_checkpointer/asc_alert_last_date_Azure_Security_Center_Alerts HTTP/1.1" 404 140 2023-01-09 10:45:17,238 DEBUG pid=25023 tid=MainThread file=binding.py:get:695 | GET request to https://127.0.0.1:8089/servicesNS/nobody/TA-MS-AAD/storage/collections/data/TA_MS_AAD_checkpointer/asc_task_last_date_Azure_Security_Center_Alerts (body: {}) 2023-01-09 10:45:17,241 DEBUG pid=25023 tid=MainThread file=connectionpool.py:_make_request:465 | https://127.0.0.1:8089 "GET /servicesNS/nobody/TA-MS-AAD/storage/collections/data/TA_MS_AAD_checkpointer/asc_task_last_date_Azure_Security_Center_Alerts HTTP/1.1" 200 128 2023-01-09 10:45:17,241 DEBUG pid=25023 tid=MainThread file=binding.py:new_f:74 | Operation took 0:00:00.002672 2023-01-09 10:45:17,241 DEBUG pid=25023 tid=MainThread file=base_modinput.py:log_debug:298 | Splunk Getting proxy server. 2023-01-09 10:45:17,242 INFO pid=25023 tid=MainThread file=setup_util.py:log_info:142 | Proxy is not enabled! 2023-01-09 10:45:17,243 DEBUG pid=25023 tid=MainThread file=connectionpool.py:_new_conn:1007 | Starting new HTTPS connection (1): login.microsoftonline.com:443 2023-01-09 10:45:17,463 DEBUG pid=25023 tid=MainThread file=connectionpool.py:_make_request:465 | https://login.microsoftonline.com:443 "POST /###/oauth2/token HTTP/1.1" 200 1468 2023-01-09 10:45:17,465 DEBUG pid=25023 tid=MainThread file=base_modinput.py:log_debug:298 | Splunk Getting proxy server. 2023-01-09 10:45:17,466 INFO pid=25023 tid=MainThread file=setup_util.py:log_info:142 | Proxy is not enabled! 2023-01-09 10:45:17,466 DEBUG pid=25023 tid=MainThread file=base_modinput.py:log_debug:298 | Splunk input_name=Azure_Security_Center_Alerts Collecting security alert data. sourcetype='azure:securityCenter:alert' 2023-01-09 10:45:17,466 DEBUG pid=25023 tid=MainThread file=base_modinput.py:log_debug:298 | Splunk input_name=Azure_Security_Center_Alerts No security center alert data checkpoint. Collecting all current alerts. 2023-01-09 10:45:17,467 DEBUG pid=25023 tid=MainThread file=connectionpool.py:_new_conn:1007 | Starting new HTTPS connection (1): management.azure.com:443 2023-01-09 10:45:17,827 DEBUG pid=25023 tid=MainThread file=connectionpool.py:_make_request:465 | https://management.azure.com:443 "GET /subscriptions/###/providers/Microsoft.Security/alerts?api-version=2021-01-01 HTTP/1.1" 200 133 2023-01-09 10:45:17,829 DEBUG pid=25023 tid=MainThread file=base_modinput.py:log_debug:298 | Splunk filename: /opt/splunk/etc/apps/TA-MS-AAD/bin/ta_azure_utils/utils.py, module: get_items_batch_session, execution time: 0.36301112174987793

linsmeyerh commented 1 year ago

Hi @kcphilipm I'm experiencing the same issue.

It seems not getting any ingestion for sourcerype azure:securityCenter:alert however no issues with azure:securityCenter:task, which is part of the same input configuration.

Can we please have some urgent assistance on this issue..

Thank you.

kcphilipm commented 1 year ago

So what I did discover is what I assumed would be pulling Security Center Alerts is in fact from what I can tell trying to pull alerts from Azure Defender security alerts what is an additional service we are not licensed for. I was thinking this alert counter was coming from [a](https://security.microsoft.com/ "Alerts" along with "Incidents". We are going to look at modifying the add-on to use this metric instead