DC1: Encrypt storage gateway EC2 AMI root(non-EBS) /dev/xvda volume to adhere AWS SEA deny rule on encryption: false

[obriensystems]

This is one of the encryption flags - the other root (non-EBS) volume is hidden because it comes with the unencrypted snapshot backing the AMI see https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/AMIEncryption.html

https://github.com/aws-quickstart/quickstart-uipath-orchestrator/blob/main/templates/storage.template.yaml#L296

API: ec2:RunInstances You are not authorized to perform this operation. "DecodedMessage": "{\"allowed\":false,\"explicitDeny\":true,\"matchedStatements\": {\"items\":[{\"statementId\":\ "PreventEc2MountUnencryptedVolume\" ,\"effect\":\"DENY\",\"principals\": {\"items\":[{\"value\":\"ARO....HWH\"}]},\"principalGroups\":{\"items\":[]},\"actions\": {\"items\":[{\"value\":\"ec2:RunInstances\"}]},\"resources\":{\"items\" :[{\"value\":\"arn:aws:ec2:::volume/*\"}]},\"conditions\":{\"items\": [{\"key\":\"ec2:Encrypted\",\"values\":{\"items\": [{\"value\":\"false\" }]}}]}}]},

other fixes - ha template add to ha template HAMaster: Type: 'AWS::EC2::Instance' Properties:

  BlockDeviceMappings:
    - DeviceName: /dev/sda1
      Ebs:
        DeleteOnTermination: true
        Encrypted: true
        VolumeSize: 100
        VolumeType: gp2

[obriensystems]

the unencrypted snapshot backing the AMI see (my personal uipath deployment has the ec2 volume encryption DENY rule set to false - to bring up the system in a less secure environment) https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/AMIEncryption.html

"In addition, when you launch an instance from an AMI backed by unencrypted EBS snapshots, you can encrypt some or all of the volumes during launch."

fix to check Turn on at account level https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/EBSEncryption.html#encryption-by-default

EC2 Settings - EBS encryption Modify EBS encryption EBS encryption Info Set the default encryption status of all new EBS volumes and copies of snapshots created in your account. Always encrypt new EBS volumes Enables encryption by default for newly created EBS volumes and snapshots. Enable (default false)

never mind - the account in question with the DENY rule on the unencrypted EBS volume already has it set to true - so the issue is flipping an AMI volume to encrypted

fix 2 to check Instance with both Encrypted and KmsKeyId set An unencrypted snapshot is restored to an EBS volume encrypted by the specified KMS key.

ami-081419a3f54890191

Volume ID | Device name | Volume size (GiB) | Attachment status | Attachment time | Encrypted | KMS key ID | Delete on termination -- | -- | -- | -- | -- | -- | -- | -- vol-03a8688d236eed4fb | /dev/xvda | 80 | Attached | Tue Oct 26 2021 11:11:21 GMT-0400 (Eastern Daylight Time) | No | – | Yes vol-05ac349c22fda6756 | /dev/sdb | 150 | Attached | Tue Oct 26 2021 11:11:21 GMT-0400 (Eastern Daylight Time) | No | –

[obriensystems]

experiment: ami existing ec2 - turn on encryption by default - restore ami to new vm (although this should have worked during the cf deployment on the DENY account) Noticed that the offending drive is greyed on encrypting via AMI creation

launching of ami - allows for encryption (although these should be auto) with the enable global flag on

default encryption does not kick in - but the selection did

I will re- ami this one and try to use in the cloudformation ami search override

[obriensystems]

Weird, they are already encrypted - in the 60 day old version before the deny rule - looks like the global flag was flipped in the middle also the root drive is now ebs

testing by raising the original ami to check encryption status worst case we save the existing encrypted ami for later use in the cf script

tried ami does not work

2021-11-07 00:56:21 UTC-0400 | uipath306-michael-OrchestratorStack-18PNLJIMURS7R-StorageStack-UGUCW08YRU9M | CREATE_FAILED | The following resource(s) failed to create: [ActivationKey]. -- | -- | -- | -- 2021-11-07 00:56:20 UTC-0400 | ActivationKey | CREATE_FAILED | Received response status [FAILED] from custom resource. Message returned: See the details in CloudWatch Log Stream: 2021/11/07/[$LATEST]9f726642b7674601bd2e0cee503bcebe (RequestId: ff40fe28-c852-4218-9889-5d4727d88415)   | 2021-11-07T00:56:18.532-04:00 | Described instances: [{'AvailabilityZone': 'ca-central-1a', 'InstanceId': 'i-0aaf7a32cef8316da', 'InstanceState': {'Code': 16, 'Name': 'running'}, 'InstanceStatus': {'Details': [{'Name': 'reachability', 'Status': 'passed'}], 'Status': 'ok'}, 'SystemStatus': {'Details': [{'Name': 'reachability', 'Status': 'passed'}], 'Status': 'ok'}}] -- | -- | --   | 2021-11-07T00:56:18.532-04:00 | Retrieving activation key ...   | 2021-11-07T00:56:18.743-04:00 | curl: (7) Failed to connect to 10.200.138.150 port 80: Connection refused   | 2021-11-07T00:56:18.764-04:00 | Exception: No redirect url returned for ip: 10.200.138.150

[obriensystems]

Anyway both drives are EBS and encrypted (as per the ami). It looks like the account DENY rule only blocks cf ec2 create - but if we allowed it through - then the account level encryption=true override would encrypt all volumes anyway.

will reuse the previous activation key from 145

ActivationKey	J635P-LII2P-FHF20-32FBR-DCTUV	Custom::ActivationKey	CREATE_COMPLETE

309

Exception: An error occurred (InvalidGatewayRequestException) when calling the ActivateGateway operation: The specified activation key was not found.

rerun with normal ami search shows

ami-04109bbae95017363 --

which is different than uipath145's ami ami-05b96677f6afd44be

312 use ami-05 hardcoded and turn off default encryption

however we are failing later on activation key

313 - theory - try encryption on the yaml - not the default ec2 setting

uipath313 try with default ami - global encryption off

  ImageId: !Ref StorageGatewayAMI
  # copy of uipath145 double encryped existing ami copy
  #ImageId: ami-05b96677f6afd44be
  # uipath145 original - passed in the past
  # used in 312 with default encyrption off - ec2 came in double encrypted but activation failed later
  #ImageId: ami-05b96677f6afd44be
  # uipath310 this dev original - failing
  # ami-04109bbae95017363
  InstanceType: m4.xlarge

encoded ec2 again using ami-04109bbae95017363

1604 noticed account encryption back on - turned off, reran with all defaults 315 verified, encoded error message

316 turn back on default account encryption - run defaults result - encryption error

317: encryption global on and add below Gateway: Type: 'AWS::EC2::Instance' Properties: BlockDeviceMappings:

• DeviceName: /dev/xvda Ebs: DeleteOnTermination: true Encrypted: true VolumeSize: 80 VolumeType: gp3

2021-11-07 17:31:52 UTC-0500 | FileShare | CREATE_FAILED | Received response status [FAILED] from custom resource. Message returned: See the details in CloudWatch Log Stream: 2021/11/07/[$LATEST]e32e8ed09d464796ae9ab130b6dc0326 (RequestId: ffab79a8-932d-4cf7-ac48-bc54b8543bf7) -- | -- | -- | -- 2021-11-07T17:31:51.505-05:00 | Creating NFS File Share for Gateway arn:aws:storagegateway:ca-central-1:045590688108:gateway/sgw-F792729E ... -- | --   | 2021-11-07T17:31:51.505-05:00 | Exception: name 'true' is not defined

good partial news - the drives are now encrypted but we get the above fileshare error

activation key good

ActivationKey | GG1LF-C989U-DRJUI-M3MQH-495LR | Custom::ActivationKey | CREATE_COMPLETE -- | -- | -- | --

using ami

StorageGatewayAMI | ami-04109bbae95017363 | Custom::StorageGatewayAMI | CREATE_COMPLETE -- | -- | -- | --

3xxx keep default encryption off - re ami 04 with ebs encryption on - hardcode ami to copy

318: pending

[obriensystems]

Summary: When the account is set for EC2 | Dashboard | Encryption | encryption = True (for new volumes during runinstance ops) - all EBS volumes are encrypted - even those that came in through any AMI that had un-encrypted volumes The IAM DENY rule on EC2:createinstance where the encryption flag is set to false - has no effect when the global encrypt flag is on - all volumes are encrypted anyway.

I see however there would be case where the regional encryption global flag is turned off and and EC2 was created (indirectly via ec2 api via CloudFormation) - that would now come in un-encrypted.

For the purposes of the dev account as long as the regional ec2 encryption flag is always false - we can relax the rule on the EC2 deny during development of the storage gateway AMI retrofit.

The remaining issue is that the root drive on the AMI used for the SG is unencrypted - however the latest run of 317 flipped the encryption to true as soon as the AMI was used via the EC2 global flag. If you check all the volumes for the entire account you will see that except for 2 test instances from another user last quarter are encrypted - specifically the pre-prod 145 from 6 weeks ago actually already had encrypted volumes across all the EC2 VMs

reference

[ubikusss]

Hey, thank you very much for reporting this. If I understand correctly you need a feature where the storage gateway will have encryption enabled for the root volume. Is this correct?