Protective Monitoring: Integration

SimonPPledger commented 7 months ago

User Story

As a product manager I need to ensure that we have an appropriate level of protective monitoring So that we have as secure a service as possible.

This ticket is to work with the protective monitoring team to capture the following levels of detail - for PRODUCTION only. We are looking to provide the following info:

VPC flow logs Networking live data Network Firewall logs Route 53

First step will me a conversation with Leo

Value / Purpose

This is to help security highlight issues

Useful Contacts

leonardo.marini@justice.gov.uk

Additional Information

For the integration work: 1) Route 53 - this should just require a configuration change in route 53

2) We then need to get the following 2 logs into a singel group in the tool (see below for previously used terraform):

VPC flow logs
Networking live data

3) And these logs into a separate group

Network Firewall logs

This is the terraform code used by the LAN team. module

Proposal / Unknowns

No response

Definition of Done

[ ] Route 53 config config changes made
[ ] Route 53 info ingested by monitoring tool
[ ] Log files made available to the the monitoring tool
[ ] Log files ingested by monitoring tool
[ ] User docs have been updated
[ ] Another team member has reviewed

davidkelliott commented 7 months ago

Route53 logs are also in cloudwatch, so we might want to do one of the other cloudwatch integrations first, then when we start to do route53 we will have the knowledge of the cloudwatch integration already so it may help decide which method is better.

dms1981 commented 7 months ago

Hi @SimonPPledger. Just so I'm clear, this would cover us exporting log information into a remote tool?

We can do that for the VPC flow logs with a little bit of coordination with the requestor.

For Route53 I'd need to know exactly what they're after. There are a few different kinds of Route 53 logs - resolve log queries and public dns queries are the sort of thing I'd assume the requestor would want.

Likewise I'm confident we can give them the Network Firewall logs they're after, but it's worth understanding what kind of granularity they want from the logs. They may, for example, want logs for traffic that we deny (but do not log at present).

mikereiddigital commented 7 months ago

I've been through and identified the following resources that are relevant:

Flow Logs: (core-vpc-production) (excluding default)

(also ref https://github.com/ministryofjustice/modernisation-platform/blob/42fe9153176eaa9fec75485561a1604955712f63/source/runbooks/querying-vpc-flow-logs.html.md.erb#L3)

arn:aws:logs:eu-west-2:278663825216:log-group:hmpps-production-vpc-flow-logs-b0500bc4
arn:aws:logs:eu-west-2:278663825216:log-group:hmcts-production-vpc-flow-logs-a2064786
arn:aws:logs:eu-west-2:278663825216:log-group:hq-production-vpc-flow-logs-b4c5e7fc
arn:aws:logs:eu-west-2:278663825216:log-group:laa-production-vpc-flow-logs-e380d929
arn:aws:logs:eu-west-2:278663825216:log-group:platforms-production-vpc-flow-logs-817e4e1c
arn:aws:logs:eu-west-2:278663825216:log-group:cjse-production-vpc-flow-logs-99fb731b
arn:aws:logs:eu-west-2:278663825216:log-group:cica-production-vpc-flow-logs-ad30cffc
arn:aws:logs:eu-west-2:278663825216:log-group:opg-production-vpc-flow-logs-5fbb18c1

Networking Live Data - Flow Logs (core-network-services)

fl-04e17b9b7ee09e6c7 - live_data-vpc-flow-logs-qoa52v4z

Network Firewall Logs (core-network-services)

fw-live-data-inline-inspection-logs-ulpiqfaq
NOMS-Transit-Live-DR-VPN-VNG-1-vpn-attachment-logs
NOMS-Transit-Live-DR-VPN-VNG-2-vpn-attachment-logs  

Route 53 DNS Logs 

- core-vpc-production for intra-vpc requests only
- not logged in core-network-services

I'm out tomorrow morning so I'll talk to @dms1981 to confirm this scope is correct.

mikereiddigital commented 7 months ago

Spoke to Leo via slack. Short initial chat organised for tomorrow at 2.30pm

Teams call - https://teams.microsoft.com/l/meetup-join/19%3ameeting_MGY1NjgwYTgtY2E4OC00NTM0LTgyYzgtYjQ5YzFkZTdiZTU1%40thread.v2/0?context=%7b%22Tid%22%3a%22c6874728-71e6-41fe-a9e1-2e8c36776ad8%22%2c%22Oid%22%3a%226e4e862b-b9a9-4735-8386-a706da19fddf%22%7d

mikereiddigital commented 7 months ago

Content of email from Leo:

Hi Mike,

In preparation for the meeting we have today I suggest you start looking into the two integration technologies we will have to implement:

Route53 - Logs are essentially sent over to an S3 bucket that is then paired with XSIAM.
We need to work a bit together and we can do this activity in our XSIAM PreProduction/Lab instance. Specific details of the activity are here:
Ingest Network Route 53 Logs from Amazon S3 • Cortex XSIAM Administrator Guide • Reader • Palo Alto Networks documentation portal https://docs-cortex.paloaltonetworks.com/r/Cortex-XSIAM/Cortex-XSIAM-Administrator-Guide/Ingest-Network-Route-53-Logs-from-Amazon-S3

AWS with CloudWatch – Logs aggregated in CloudWatch are streamed to XSIAM using an Amazon Kinesis Firehose.
Step by step implementation is described at the link below and we are happy to work on this with you, also we are trying to collect some terraform code to share with you.
https://docs-cortex.paloaltonetworks.com/r/Cortex-XSIAM/Cortex-XSIAM-Administrator-Guide/Ingest-Logs-from-Amazon-CloudWatch
We will need 2 streams, one for the network data and the other for the firewalls.

Also:

The way data will be parsed means that different delivery streams will be set up on their side for different categories of data.

mikereiddigital commented 7 months ago

Questions:

Which account for vpc flow log data? core-vpc-production or core-logging?

mikereiddigital commented 7 months ago

Email from Leo as a follow up to the meeting:

Hi Mike,

It was a pleasure meeting you today.

We created the connectors you will use for XSIAM

Preproduction/Lab
name: Modernisation Platforms Network
api: https://api-justiceukpreprod.xdr.uk.paloaltonetworks.com/logs/v1/aws
secret_key: XSIAM-PP-MODP-NETWORK-CLOUD-WATCH-KEY

name: Modernisation Platforms Firewalls
api: https://api-justiceukpreprod.xdr.uk.paloaltonetworks.com/logs/v1/aws
secret_key: XSIAM-PP-MODP-FIREWALL-CLOUD-WATCH-KEY

Live
name: Modernisation Platforms Network
api: https://api-justiceuk.xdr.uk.paloaltonetworks.com/logs/v1/aws
secret_key: XSIAM-LIVE-MODP-NETWORK-CLOUD-WATCH-KEY

name: Modernisation Platforms Firewalls
api: https://api-justiceuk.xdr.uk.paloaltonetworks.com/logs/v1/aws
secret_key: XSIAM-LIVE-MODP-FIREWALL-CLOUD-WATCH-KEY

The secret keys are stored in our Azure Key Vault, if you could provide us with the list of users and emails that will need access we will grant permission and provide the final link to them.

Kind Regards,
Leo

mikereiddigital commented 7 months ago

PR with the initial set of changes for firehose - https://github.com/ministryofjustice/modernisation-platform-terraform-member-vpc/pull/328

SecOps have granted us access to the security keys and added manually a secrets manager resource in core-vpc-development to hold the key for their preprod network endpoint.

mikereiddigital commented 7 months ago

Both https://github.com/ministryofjustice/modernisation-platform/tree/adds-firehose-resources and https://github.com/ministryofjustice/modernisation-platform-terraform-member-vpc/tree/add-aws-firehose are producing a clean plan. Next steps:

Clean commits
Set the scope to just one member vpc
new PR, plan & approvals and apply as a test.
Once confirmed working, new PR to include other vpcs and apply.
Then move onto network firewalls logs.

mikereiddigital commented 7 months ago

Working build in core-vpc-sandbox - https://github.com/ministryofjustice/modernisation-platform-terraform-member-vpc/tree/add-aws-firehose

Next step to identify the scope of what's needed to apply to development.

mikereiddigital commented 7 months ago

Note: │ Warning: Deprecated attribute │ │ on .terraform/modules/vpc/firehose.tf line 190, in resource "aws_cloudwatch_log_subscription_filter" "nacs_server_xsiam_subscription": │ 190: log_group_name = aws_flow_log.cloudwatch.log_group_name

This is from the plan & shows that property in question is deprecated however the terraform documentation shows otherwise - https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/cloudwatch_log_subscription_filter

mikereiddigital commented 6 months ago

Latest PR of the module - https://github.com/ministryofjustice/modernisation-platform-terraform-member-vpc/pull/338. We still use the build_firehose variable as this determines which environ ments the resources are built in or not.

mikereiddigital commented 6 months ago

New release for the VPC module -https://github.com/ministryofjustice/modernisation-platform-terraform-member-vpc/releases/tag/v2.3.0

mikereiddigital commented 6 months ago

PR for the modernisation-platform repo changes - https://github.com/ministryofjustice/modernisation-platform/pull/6449

mikereiddigital commented 6 months ago

PR applied and resources built in dev & production. Will clear down sandbox before end of today.

mikereiddigital commented 6 months ago

Sandbox cleared.

mikereiddigital commented 6 months ago

PR for the 2nd part of the changes - https://github.com/ministryofjustice/modernisation-platform/pull/6485

mikereiddigital commented 6 months ago

PR for secret - https://github.com/ministryofjustice/modernisation-platform/pull/6505. This single secret will hold all of the endpoint and key details.

mikereiddigital commented 6 months ago

New commit that covers a just the firewall inspection logs & review comments.

mikereiddigital commented 6 months ago

Firewall inspection log data now being transferred. Tomorrow will be looking at the firewall vpc flow log transfers - which will not take long, and then we can move onto R53 data.

mikereiddigital commented 6 months ago

PR applied for the firewall vpc flow log data - https://github.com/ministryofjustice/modernisation-platform/pull/6545

mikereiddigital commented 6 months ago

Chat with Leo at 12.30 to go through the R53 records.

mikereiddigital commented 6 months ago

Additional VPC flow logs:

core-logging - 2 vpcs - see branch link below. core-security - 2 vpcs (live & non-live core-shared-services - 2 vpcs ""

For each of the above:

2 data items in secrets.tf for the secret reference
1 local item for the secrets
new file - firehose.tf with the firehose module call & locals.

A single iteration of the module creates 19 resources.

https://github.com/ministryofjustice/modernisation-platform/tree/firehose-additional-vpcs

any others? Modernisation has just default vpc. Accounts that don't use shared vpcs such as bichard7?

mikereiddigital commented 6 months ago

Meeting with Leo:

R53 can be sent via firehose. They will send us the new endpoints & secrets. Update - we have the new keys for the R53 transfers. The endpoints are the same as networks.
Non-prod envs are not in scope currently.
Need to confirm how & when we will get the VPC flow log data for the unrestricted member accounts.
Cloudtrail logs is also a requirement which was recently added and needs to be discussed as part of a separate ticket?

mikereiddigital commented 6 months ago

Tidied up the added data & locals in core-vpc to reduce the number of items used.

mikereiddigital commented 6 months ago

We have added the transfer of R53 resolver logs & have confirmed they are transmitting. We need to trigger a small update to the firewall inspection & flow logs plus those for shared so the module changes are applied. Once that is done, we will create a new ticket for the cloudtrail logs as that is a new requirement that wasn't covered in the original scope.

mikereiddigital commented 6 months ago

Had a final chat with Leo before he goes on leave. Summarised where we are:

Member shared VPC flow logs & r53 resolver logs (core-vpc)
Firewall inspection logs & their VPC flow logs (core-network-services)
Shared account VPC flow logs (core-logging, core-security, core-shared-services)

He knows that the cloudtrail events are being covered in a separate ticket. Also he mentioned to contact this team - monitoring-and-integration-platform@justice.gov.uk - if anything changes.

Leo is back on Wednesday next week.

dms1981 commented 3 months ago

Cloudtrail integration discussed in https://github.com/ministryofjustice/modernisation-platform/issues/6614

ministryofjustice / modernisation-platform