Perimeter Security - Default Endpoint Policy Issues

[satyamohanty2024]

When we added the policy default_endpoint_policy.json provided under data-perimeter-policy-examples\vpc_endpoint_policies with our org-id to the endpoints, we encountered the below two issues:

Patching Issue- We noticed buttons for manual patching of instance through patch manager were frozen. When we investigated, then we found a lot of JS errors in the browser using the developer tools. These errors were related to SSM endpoint which has this endpoint policy added.

Inconsistence kubelet behavior - kubelet services running in EKS nodes are dependent on ec2 and ec2messages endpoints. The kubelet services couldn’t access these endpoints once this default endpoint policy was added. This led to inconsistence in kubelet behavior causing unavailability of these nodes for running kubenetes workload.

For now, we have reverted back to original endpoint policy that allows all actions by all principals on all resources as a temporary fix to address the above patching and Inconsistence kubelet behavior issues.

Please investigate and let us know the change required in default_endpoint_policy.json that will address the above two issues

[liwadman]

Hello, thank you for raising this issue. Can you confirm all the different VPC endpoints you applied this policy to? As well can you confirm whether or not you used the vpc_endpoint_policies/ssm_endpoint_policy.json for the SSM VPC endpoint?

We're investigating this internally with members of your account team who have also escalated this in parallel.

Thank you,

Liam

[satyamohanty2024]

We applied the policy to all the below endpoints. Also, we did not used the vpc_endpoint_policies/ssm_endpoint_policy.json for the SSM VPC endpoint

    {"name": "ssm"},
    {"name": "ssmmessages"},
    {"name": "states"},
    {"name": "sts"},
    {"name": "sqs"},
    {"name": "sns"},
    {"name": "secretsmanager"},
    {"name": "sagemaker.api"},
    {"name": "rds"},
    {"name": "monitoring"},
    {"name": "logs"},
    {"name": "lambda"},
    {"name": "kms"},
    {"name": "glue"},
    {"name": "events"},
    {"name": "elasticloadbalancing"},
    {"name": "ecs"},
    {"name": "ec2messages"},
    {"name": "ec2"},
    {"name": "config"},
    {"name": "cloudformation"},
    {"name": "autoscaling"},
    {"name": "athena"},
    {"name": "application-autoscaling"},
    {"name": "sagemaker.runtime"},
    {"name": "ecr.api"},
    {"name": "ecr.dkr"},
    {"name": "elasticfilesystem"},
    {"name": "kinesis-streams"},
    {"name": "kinesis-firehose"},
    {"name": "ecs-agent"},
    {"name": "lakeformation"},
    {"name": "ecs-telemetry"},
    {"name": "redshift"},
    {"name": "textract"},
    {"name": "imagebuilder"},
    {
        "name": "s3",
        "private_dns_enabled": False,
        "lookup_supported_azs": False,
    },
    {"name": "xray"}

[liwadman]

acknowledged. For the SSM endpoint, and all other endpoints for which we provided a specific example, we recommend using those ones. These tailored endpoint policies for specific services allow access to resources which may be required to use the service or some functionality of it.

[liwadman]

Please let us know when you're able to test again using the provided endpoint policies. For all endpoints where we provide an example policy, we recommend starting with that as there as some exceptions required for these services to work within your vpc.

Today we're providing sample policies for SSM, S3, EC2, ECR.API, ImageBuilder VPC endpoints, which allow services to consume service owned resources such as the S3 bucket that hosts amazon linux updates, or SSM documents.