ITDG: Integrate with Splunk

[halprin]

Backlog Task

ITDG: Integrate with Splunk

Completion Criteria

• [x] Integrate with Splunk in our CDC environments.

Tasks

• [x] Wait until an environment is created and then integrate with Splunk.
• [x] Re-read documentation.
• [x] Reach out to Jeremy and Josiah on how they set-up their function app's diagnostic settings to forward logs to event hub CSPO-Prod-Hubs-2/cspo-hub-function_app. Through an Azure Policy? Through manual creation or Terraform?
• [x] Fill out a service desk ticket to set our Azure Policy so our diagnostic settings forward logs to the correct event hub?
• [x] Wait for response from Service Desk and do what they say
• [x] Document this process for future devs

Other Notes

• CDC documentation on how to integrate with Splunk.
• Is there a way to be notified or extract info out of splunk to be useful for us?
• Does this fit into any of our other plans for observability?

[scleary1cs]

Use the cloud option.

[scleary1cs]

What is FISMA? The Federal Information Security Modernization Act (FISMA) defines a framework of guidelines and security standards to protect government information and operations.

[scleary1cs]

Reach out to Yu (Boris) Ning for scope/config help.

[jcrichlake]

This has been requested to the CDC service desk RITM0275666

[jcrichlake]

I reached out to the CDC help desk on this issue and they are meeting with the SPLUNK team on Wednesday and will let me know once the steps are complete.

[jcrichlake]

Update: The meeting that was supposed to be today (03/20) is now on Monday (03/25) Peter and I have been invited. The help desk person said that they are blocking off time after that meeting to do the integration for us because of the wait.

[scleary1cs]

CDC team met on Monday 04/25, Splunk team will give the service desk team an update soon. Jeff will receive a comm from the service desk team.

[jcrichlake]

I reached out to the Splunk team and we are in their queue, they are hoping to get to it this week

[scleary1cs]

Here are the resource log categories we'd need configured for collection in that resource group.

Resource Type Diagnostic Setting Categories App Services None App Service Plan None Azure Database for PostgreSQL Flexible Server PostgreSQL Server Logs, PostgreSQL Sessions data Container Registry All categories DNS Private Resolver N/A Key Vault Audit Logs Log Analytics Query Pack N/A Log Analytics Workspace None Network Interface None Network Security Group None (NSG Flow automatically configured for collection by OMHS) Private DNS Zone N/A Private Endpoint N/A Public IP None Recovery Services Vault Addon Azure Backup Alert Data Route Table N/A SQL Database Errors, Blocks, Timeouts, Deadlocks, Database Wait Statistics SQL Server N/A Storage Account None Virtual Network None Virtual Network Gateway N/A

For the diagnostic settings that need to be configured, please use the following destination: Subscription: OCIO-CSPO-PROD-C1 Event Hub Namespace: CSPO-Prod-Splunk-eastus-1-ehn Event Hub Name: resource_logs Event Hub Policy Name: resource_logs_and_metrics

[halprin]

All four of the values in the above comment cannot be set because we do not have permission to set the subscription. @jcrichlake has reached out to the Splunk team to find out how we can get access. Marking as blocked.

[scleary1cs]

Documentation added to Notion here.

[jcrichlake]

I have reached out to 2 folks from the CDC to receive the permissions necessary to set up the appropriate diagnostic settings. They informed me several weeks ago that a helpdesk ticket has been entered but I still don't have the permissions required. I am putting this card back into the backlog at the direction of @sfradkin

[JohnNKing]

@JohnNKing , @scleary1cs , or @sfradkin to raise with CDC. Either we need to know the configuration needed for Splunk soon or it will not be part of our production go live for CA.

[jcrichlake]

I reached out to the CDC folks who are in charge of the Splunk set up again and prodded them about getting the staging and prod resource groups established with Splunk logging. I also caught up @saquino0827 on where this ticket is at so that he has the context should I not be in the office. I also sent the Splunk settings to all of the Stream 1 engineers so that they have the information.

[jcrichlake]

I.... I think this may be complete? I asked @saquino0827 to double check