search

ministryofjustice / modernisation-platform

A place for the core work of the Modernisation Platform • This repository is defined and managed in Terraform
https://user-guide.modernisation-platform.service.justice.gov.uk
MIT License
683 stars 290 forks source link

AWS Firehose for production vpc-flow logs in core-logging, shared-servies and security using wrong endpoint. #7424

Closed mikereiddigital closed 2 months ago

mikereiddigital commented 3 months ago

Expected Behavior

The xsiam endpoints for the production firehose transfers for core-shared-services, core-logging and core-security are using the wrong endpoint - preprod rather than prod. The source code is missing the local "xsiam" that should be in locals.tf which references the data items for the secret. As such the terraform build defaults to nonprod.

core-shared-services - https://github.com/ministryofjustice/modernisation-platform/blob/main/terraform/environments/core-shared-services/firehose.tf

core-logging - https://github.com/ministryofjustice/modernisation-platform/blob/main/terraform/environments/core-logging/firehose.tf

core-security - https://github.com/ministryofjustice/modernisation-platform/blob/main/terraform/environments/core-security/firehose.tf

Actual Behavior

The firehose resources are deployed referencing non-prod endpoints.

Steps to Reproduce the Problem

No response

Version

No response

Modules

No response

Account

core-logging, core-shared-services and core-security.

mikereiddigital commented 3 months ago

PR for core-logging - https://github.com/ministryofjustice/modernisation-platform/pull/7431

mikereiddigital commented 3 months ago

PR for core-shared-services and core-security - https://github.com/ministryofjustice/modernisation-platform/pull/7433

mikereiddigital commented 3 months ago

Have informed Leo Marini (Leonardo.Marini@justice.gov.uk) and will keep this open until he confirms the feeds are being received.

mikereiddigital commented 3 months ago

Fixed an issue with the transfers from core-network-services - https://github.com/ministryofjustice/modernisation-platform/pull/7438

SimonPPledger commented 2 months ago

Need to add info to user docs - for each feed firehose resource should send info about data set

mikereiddigital commented 2 months ago

Also:

mikereiddigital commented 2 months ago

Spoke with AWS who confirmed the commonAttribute pair of strings is in the header (metadata) of the transfer, not the payload body. Have asked Leo to confirm whether this is accessible at the Corext Xsiam endpoints.

Edit - the AWS doc with the detail -https://docs.aws.amazon.com/firehose/latest/dev/httpdeliveryrequestresponse.html

mikereiddigital commented 2 months ago

PR for the runbook - https://github.com/ministryofjustice/modernisation-platform/pull/7456

mikereiddigital commented 2 months ago

This is the link to the page that I've shared with Leo - https://user-guide.modernisation-platform.service.justice.gov.uk/runbooks/integration-with-protective-monitoring.html#sharing-of-platform-operational-data-with-security-operations-via-aws-data-firehose