Adding Monitoring agents to EC2 instances

[kenoir]

We should add the CrowdStrike agent to EC2 instances in Wellcome Collection AWS accounts as a requirement from InfoSec to enable centralised monitoring of these instances for potential malicious activity.

Our estate is primarily containers in the form of ECS services & lambdas, however we have at least 4 services that make use of EC2 instances:

Digital Engagement Workflow account:

• Archivematica: Maintained by Wellcome Collection developers

Digital Engagement DLCS account:

• DLCS: Maintained by a third party Digirati

Digital Engagement Platform account:

• Buildkite CI instances: CloudFormation stack provided by Buildkite but maintained by Wellcome Collection developers. The EC2 instances in this stack scale in and out regularly and may pose a problem with CrowdStrike licensing.
  • Catalogue Pipeline: at least one step of the pipeline (the inferrer) relies on autoscaling EC2 instances.

Apart from the Buildkite service the base AMIs we use are based on Amazon Linux. We propose baking an AMI based on Amazon Linux with the CrowdStrike agent pre-installed and configured with the appropriate customer id. We’d then seek to use that as standard in our launch templates to incorporate the CrowdStrike agent and any future requirements of a similar nature.

We will plan and prioritise some work in our next sprint to test this approach with one of our services. If successful we will look to roll this out where feasible to the other services described above.

We need to:

• [x] Create a base AMI with the crowdstrike & qualys agent installed & ensure we can distribute the AMIs to all appropriate accounts
  • [x] https://github.com/wellcomecollection/platform-infrastructure/pull/416
  • [x] https://github.com/wellcomecollection/aws-account-infrastructure/pull/17
  • [ ] ~(optional) encrypt base AMIs~ not doing this, risk is minimal
• [x] Create AMIs for all appropriate instance types (for Archivematica, bastion hosts & anything for intranda):
  • [x] Amazon Linux 2 ECS optimised
  • [x] Plain Amazon Linux 2
• [x] Update all stacks and stages of EC2 instance to use the new AMIs
  • [x] https://github.com/wellcomecollection/goobi-infrastructure/pull/454
  • [x] https://github.com/wellcomecollection/archivematica-infrastructure/pull/149
  • [x] https://github.com/wellcomecollection/catalogue-pipeline/pull/2554
  • [x] #422

[!NOTE] Buildkite EC2 instances are a harder problem to solve and are a lower risk as they are ephemeral and harder to target, it may be we move away from these to GitHub actions at some point so leaving those for now.

Finally:

• [ ] Confirm monitoring agents are reporting home with InfoSec

[kenoir]

See https://www.crowdstrike.com/blog/tech-center/install-falcon-sensor-for-linux/ for agent installation instructions. The CrowdStrike installer and customer identifier can be requested from InfoSec.

[kenoir]

We could look at https://aws.amazon.com/image-builder/ to automate this process if we get success with a manual test. In general having a way to keep base AMIs up to date is a good idea.

We are using fixed AMI IDs in some instances (archivematica) so we are missing any updates that are taking place from the supplier to cover vulnerabilities etc - using system parameters to reference AMIs is preferred: https://docs.aws.amazon.com/systems-manager/latest/userguide/parameter-store-public-parameters-ami.html