🤝 Implement AWS Transfer Family Server

Gary-H9 commented 5 months ago

User Story

As a… User of the Analytical Platform I want to be able to ingest data into the platform So that… I can use all of the tooling etc within the platform

As a AP Product Engineer I want to the platform to be able to ingest data in a controlled, precise and monitored fashion. So that we can provide a better service for our users and build a foundational offering within the platform.

This ticket builds on this previously raised Feature Request.

Value / Purpose

Data ingestion will be a foundational part of the AP offering going forward. This piece of work will create the foundations which this offering will be built on.

Useful Contacts

Jacob W / Julia / Gary

Proposal

Create the ingestion route as outlined here.

Additional Information

For the AWS Transfer Family Server - Electronic Monitoring already use this functionality.

Definition of Done

[x] 📝 Documentation has been written / updated
[x] Modernisation Platform AWS Account created (WIP)
[x] AWS Transfer Family Server Terraform'ed
[x] IAM Roles defined + Terraform'ed
[x] User testing arranged
[x] Testing completed

jacobwoffenden commented 5 months ago

29/02/24 summary:

@Gary-H9 requested new Modernisation Platform environment https://github.com/ministryofjustice/modernisation-platform/issues/6364

jacobwoffenden commented 5 months ago

05/03/24 summary:

@jacobwoffenden #3602
@Gary-H9
- initial configuration of ingestion scanning container - WIP PR
- Addition of CloudWatch EventBridge, lambda function and associated testings of the above in concert with these
- Modernisation Platform environment build in progress

jacobwoffenden commented 5 months ago

06/03/24:

Successfully tested flow for updating and then consuming ClamAV definitions
Successfully tested scanning files using S3 bucket notifications
- Clean files moved to a processed bucket, while infected are moved to quarantine
- Files are deleted from landing once moved

TODO:

[x] Add object policy to deny access to quarantined
[x] Configure Transfer Family server
- [x] Home directory
- [x] Move into VPC and give an EIP
- [x] Custom hostname?
- [x] Restrict users to specific IP ranges
[ ] ~Notification for quarantined objects~
- ~Can we use a tag from the transfer user to send an email? and in the future a message to AP dashboard~ Split into #3770

Notes:

Electronic Monitoring create a server per provider, we don't want to do this, could be hard to scale and manager via AP dashboard

EDIT @jacobwoffenden:

I've managed to get this working by editing the IAM policy for the user to include more S3 permissions, and added KMS permissions

EDIT 2 @jacobwoffenden:

[ ] ~figure out how/where to run aws transfer update-server to set DirectoryListingOptimization to ENABLED. this is not exposed in Terraform~ MP ClickOps'd it

Gary-H9 commented 5 months ago

07/03/24:

Converted the lambda function to bash from python. Noted faster runs as a result. Tested both scenarios successfully.
Planned architecture relating to notifications in variety of scenarios.

jacobwoffenden commented 4 months ago

11/04/24 summary:

More work on scan functionality
- sends a message to SNS when quarantining a file
Started work on notify functionality
- Triggers on SNS publish
- Plumbed into Data Platform's GOV.UK Notify account, have tested basic functionality
Drafted architecture for "supplier data", or information about supplier we don't want to store in public Terraform, this will be superseded by AP dashboard eventually
Liaised with GDS about getting access to Analytical Platform's notify account

TODO:

[ ] ~SNS redrive logic~
[x] clarify naming/terminology (data contact, data owner, etc.)

Gary-H9 commented 4 months ago

14th March summary:

Refactored existing code + added code for new lambda and required components
Removed calls to SNS for this iteration of the product - this can be looked at after MVP - ticket created for this
Created ingestion-transfer handler.py
Created a test bucket in analytical-platform-dev called dev-ingestion-testing to test cross account file move from the above lambda. Manually updated the policy on this but it needs further work.

jacobwoffenden commented 4 months ago

18/03/2024 summary (plus a bit more):

Gary:

continued development on the ingestion-transfer image.

Gary-H9 commented 4 months ago

Closed development branch/PR in data-platform repository
Migrating code from the above branch into Modernisation Platform Environments repository. Merged - development and production built.
Updated DNS record for both environments in analytical-platform-production R53.

Gary-H9 commented 4 months ago

Created documentation relating to the solution in user-guidance (🚧) and in our new runbooks documentation.

jacobwoffenden commented 4 months ago

@ministryofjustice/modernisation-platform have enabled optimised directories manually in both transfer servers (dev and prod)

jacobwoffenden commented 4 months ago

https://github.com/hashicorp/terraform-provider-aws/issues/35851

Gary-H9 commented 4 months ago

Awaiting user information to allow testing. In the meantime egress has been completed.

Gary-H9 commented 4 months ago

Locked down S3 Bucket Permissions on the Quarantine bucket
Egress functionality
No update from users RE testing.

jacobwoffenden commented 4 months ago

Pending details from BOLD to begin onboarding