search

duo-labs / cloudmapper

CloudMapper helps you analyze your Amazon Web Services (AWS) environments.
BSD 3-Clause "New" or "Revised" License
5.97k stars 802 forks source link

Map network resources beyond EC2, ELB, and RDS #365

Closed 0xdabbad00 closed 5 years ago

0xdabbad00 commented 5 years ago

CloudMapper should be able to display a map of all AWS network resources, which I'll define as those resources that can have Security Groups applied to them.

This includes:

There are probably more. My plan is to use aws ec2 describe-network-interfaces which provides a list of network interfaces with their:

This issue partially resolves #339, #306, #89, adn #51

0xdabbad00 commented 5 years ago

Looking at the output of ec2 describe-network-interfaces there are some issues.

  1. An ALB (ELBv2) only appears in one subnet. I'm guessing AWS must only keep one interface for an ALB alive at a time? So if the AZ with the subnet disappears the interface is moved to a healthy subnet maybe?
  2. An old ELB does not appear at all.
  3. An RDS also appears in only one subnet.
0xdabbad00 commented 5 years ago

Using aws ec2 describe-vpc-endpoints I can see the PrivateLink (VPC Endpoints) and Gateway endpoints (S3 + DynamoDB). These just go to the services, and then you have to parse the policies associated with them to identify the actual resources they allow access to. As such, for an initial release, I'm only going to show a connection as going to a service and not name the specific resources. Further, there are all sorts of other things that can restrict access to the resources, such as IAM or resource policies.

This is explained in the UI screen when you go to edit the policy for one of the endpoints.

Screen Shot 2019-04-30 at 9 23 37 AM

An interesting thing is the VPC endpoints end up with interfaces (as found from aws ec2 describe-network-interfaces) in each of the subnets, so a single VPC endpoint for accessing SQS results in 6 new interfaces appearing for me in us-east-1.

Here is an example:

{
  "Status": "in-use",
  "MacAddress": "06:61:a7:1f:d0:dc",
  "SourceDestCheck": true,
  "AvailabilityZone": "us-east-1e",
  "Description": "VPC Endpoint Interface vpce-05ed16ac73726e737",
  "NetworkInterfaceId": "eni-09785271abed2b16d",
  "VpcId": "vpc-4b52dd32",
  "PrivateIpAddresses": [
    {
      "PrivateDnsName": "ip-172-31-60-106.ec2.internal",
      "Primary": true,
      "PrivateIpAddress": "172.31.60.106"
    }
  ],
  "RequesterManaged": true,
  "PrivateDnsName": "ip-172-31-60-106.ec2.internal",
  "RequesterId": "727180483921",
  "InterfaceType": "vpc_endpoint",
  "Attachment": {
    "Status": "attached",
    "DeviceIndex": 1,
    "DeleteOnTermination": false,
    "AttachmentId": "ela-attach-23af6704",
    "InstanceOwnerId": "amazon-aws"
  },
  "Groups": [
    {
      "GroupName": "default",
      "GroupId": "sg-09920178"
    }
  ],
  "Ipv6Addresses": [],
  "OwnerId": "954574370272",
  "SubnetId": "subnet-07d3ea3b",
  "TagSet": [],
  "PrivateIpAddress": "172.31.60.106"
}

The Gateway endpoints however do not have any associated interfaces.

$ aws ec2 describe-vpc-endpoints
{
    "VpcEndpoints": [
        {
            "PolicyDocument": "{\"Version\":\"2008-10-17\",\"Statement\":[{\"Effect\":\"Allow\",\"Principal\":\"*\",\"Action\":\"*\",\"Resource\":\"*\"}]}",
            "VpcId": "vpc-4b52dd32",
            "NetworkInterfaceIds": [],
            "SubnetIds": [],
            "PrivateDnsEnabled": false,
            "State": "available",
            "ServiceName": "com.amazonaws.us-east-1.s3",
            "RouteTableIds": [
                "rtb-ca95c4b2"
            ],
            "Groups": [],
            "VpcEndpointId": "vpce-05e0785124009dbbc",
            "VpcEndpointType": "Gateway",
            "CreationTimestamp": "2019-04-11T18:59:11.000Z",
            "DnsEntries": []
        },
        {
            "PolicyDocument": "{\n    \"Statement\": [\n        {\n            \"Action\": \"*\",\n            \"Effect\": \"Allow\",\n            \"Resource\": \"*\",\n            \"Principal\": \"*\"\n        }\n    ]\n}",
            "VpcId": "vpc-4b52dd32",
            "NetworkInterfaceIds": [
                "eni-039a37a6ce940d5d2",
                "eni-0972ceecdba8c5df9",
                "eni-09785271abed2b16d",
                "eni-0afc1be0c66a42633",
                "eni-0dde8fc67086a2b22",
                "eni-03d75ab23bc6e04e4"
            ],
            "SubnetIds": [
                "subnet-7543383d",
                "subnet-5d7bc007",
                "subnet-1abda07f",
                "subnet-07d3ea3b",
                "subnet-a09b188c",
                "subnet-2e33ab22"
            ],
            "PrivateDnsEnabled": true,
            "State": "available",
            "ServiceName": "com.amazonaws.us-east-1.sqs",
            "RouteTableIds": [],
            "Groups": [
                {
                    "GroupName": "default",
                    "GroupId": "sg-09920178"
                }
            ],
            "VpcEndpointId": "vpce-05ed16ac73726e737",
            "VpcEndpointType": "Interface",
            "CreationTimestamp": "2019-04-30T15:19:35.000Z",
            "DnsEntries": [
                {
                    "HostedZoneId": "Z7HUB22UULQXV",
                    "DnsName": "vpce-05ed16ac73726e737-5xkr9dae.sqs.us-east-1.vpce.amazonaws.com"
                },
                {
                    "HostedZoneId": "Z7HUB22UULQXV",
                    "DnsName": "vpce-05ed16ac73726e737-5xkr9dae-us-east-1c.sqs.us-east-1.vpce.amazonaws.com"
                },
                {
                    "HostedZoneId": "Z7HUB22UULQXV",
                    "DnsName": "vpce-05ed16ac73726e737-5xkr9dae-us-east-1d.sqs.us-east-1.vpce.amazonaws.com"
                },
                {
                    "HostedZoneId": "Z7HUB22UULQXV",
                    "DnsName": "vpce-05ed16ac73726e737-5xkr9dae-us-east-1a.sqs.us-east-1.vpce.amazonaws.com"
                },
                {
                    "HostedZoneId": "Z7HUB22UULQXV",
                    "DnsName": "vpce-05ed16ac73726e737-5xkr9dae-us-east-1e.sqs.us-east-1.vpce.amazonaws.com"
                },
                {
                    "HostedZoneId": "Z7HUB22UULQXV",
                    "DnsName": "vpce-05ed16ac73726e737-5xkr9dae-us-east-1b.sqs.us-east-1.vpce.amazonaws.com"
                },
                {
                    "HostedZoneId": "Z7HUB22UULQXV",
                    "DnsName": "vpce-05ed16ac73726e737-5xkr9dae-us-east-1f.sqs.us-east-1.vpce.amazonaws.com"
                },
                {
                    "HostedZoneId": "ZCYZV1JQU4VH2",
                    "DnsName": "sqs.us-east-1.amazonaws.com"
                }
            ]
        }
    ]
}
0xdabbad00 commented 5 years ago

Because Gateway Endpoints do not have interfaces, they do not have Security Groups.

0xdabbad00 commented 5 years ago

I created a Lambda in a VPC. Doing aws lambda list-functions shows:

{
            "TracingConfig": {
                "Mode": "PassThrough"
            },
            "Version": "$LATEST",
            "CodeSha256": "H4QrZXTeJhAFtZ6dfi7NjMZiGHZfFgGpD38SjX+BaMw=",
            "FunctionName": "invpclambda",
            "VpcConfig": {
                "SubnetIds": [
                    "subnet-5d7bc007",
                    "subnet-7543383d"
                ],
                "VpcId": "vpc-4b52dd32",
                "SecurityGroupIds": [
                    "sg-09920178"
                ]
            },
            "MemorySize": 128,
            "RevisionId": "78a5eca3-193c-4f09-bed0-123f2d1c3ad3",
            "CodeSize": 262,
            "FunctionArn": "arn:aws:lambda:us-east-1:954574370272:function:invpclambda",
            "Handler": "index.handler",
            "Role": "arn:aws:iam::954574370272:role/service-role/invpclambda",
            "Timeout": 3,
            "LastModified": "2019-04-30T16:26:38.618+0000",
            "Runtime": "nodejs8.10",
            "Description": ""
        },

However aws ec2 describe-network-interfaces still does not show an interface for it. It therefore looks like I'm going to have to iterate through the different services to identify which ones are in VPCs.

0xdabbad00 commented 5 years ago

I need to do the following:

  1. Identify all services that have VPC support
  2. Get icons for all of them
  3. Ensure I collect info for all of them. Then iterate through all and build normalized nodes that include:
    • IP addresses: if exists, as some things like Lambda don't have one
    • Security Groups
    • Location: region, VPC, AZ, subnet
    • Tags
  4. Identify connections based on SGs. Could also add this point more easily add ACls and Route tables.
  5. Perform filtering based on tags, location, etc.
0xdabbad00 commented 5 years ago

I believe the following all support VPCs:

Through PrivateLink:

Through Gateway endpoints:

0xdabbad00 commented 5 years ago

I'm going to start with the Gateway endpoints, then the PrivateLink ones, then Lambda and EKS, and see how things are from there.

0xdabbad00 commented 5 years ago

I've added code to read from ec2-describe-vpc-endpoints.json.

I need to test it works correctly with Security Groups. Because the Gateway endpoints are associated with a VPC, these just sort of "float" in the VPC.

I am generically giving that purple icon (which is supposed to be for an ENI) to any of these end-points, and then for S3 I change it's type to S3.

Screen Shot 2019-05-01 at 4 30 59 PM

Some tasks for this

0xdabbad00 commented 5 years ago

The Gateway endpoints (S3) does not have Security Groups. Everything in the VPC is allowed access to it. I'm not sure if I should therefore draw lines from everything to it? That seems like the most correct choice, but it will be cluttered.

0xdabbad00 commented 5 years ago

This is what I have currently, and although technically correct I believe, it is misleading because I believe the Gateway and Interface endpoints can't call out, so they can't actually talk to each other.

Screen Shot 2019-05-01 at 9 23 39 PM
0xdabbad00 commented 5 years ago

My belief is correct that the endpoints cannot initiate traffic "Services cannot initiate requests to resources in your VPC through the endpoint. An endpoint only returns responses to traffic initiated from resources in your VPC." - https://docs.aws.amazon.com/vpc/latest/userguide/vpce-interface.html

So I'm going to have a check in the code that if the source is an endpoint, then don't build a connection.

Also, I'm thinking of creating a virtual/fake subnet for the VPC-level resources such as the S3 endpoint.

0xdabbad00 commented 5 years ago

VPC Endpoint support has been added in #376

Screen Shot 2019-05-02 at 11 19 22 AM
0xdabbad00 commented 5 years ago

I added endpoints to the demo account data. It looks cluttered. :(

Screen Shot 2019-05-02 at 2 02 15 PM

I cut a release for this https://github.com/duo-labs/cloudmapper/releases/tag/2.5.4

Going to work on ECS support now.

0xdabbad00 commented 5 years ago

For ECS, the info I need, using the flaws2 target account for data, comes from:

aws ecs describe-tasks --cluster level3 --tasks arn:aws:ecs:us-east-1:653711331788:task/d190d14a-2404-45d6-9113-4eda22d7f2c7
{
    "failures": [],
    "tasks": [
        {
            "launchType": "FARGATE",
            "attachments": [
                {
                    "status": "ATTACHED",
                    "type": "ElasticNetworkInterface",
                    "id": "ed8fed01-82d0-4bf6-86cf-fe3115c23ab8",
                    "details": [
                        {
                            "name": "subnetId",
                            "value": "subnet-e45602eb"
                        },
                        {
                            "name": "networkInterfaceId",
                            "value": "eni-200e9484"
                        },
                        {
                            "name": "macAddress",
                            "value": "16:2f:d0:d6:ed:28"
                        },
                        {
                            "name": "privateIPv4Address",
                            "value": "172.31.48.168"
                        }
                    ]
                }
            ],
            "clusterArn": "arn:aws:ecs:us-east-1:653711331788:cluster/level3",
            "desiredStatus": "RUNNING",
            "createdAt": 1543289790.71,
            "taskArn": "arn:aws:ecs:us-east-1:653711331788:task/d190d14a-2404-45d6-9113-4eda22d7f2c7",
            "group": "family:level3",
            "pullStartedAt": 1543289803.371,
            "version": 3,
            "memory": "512",
            "connectivityAt": 1543289795.422,
            "startedAt": 1543289825.371,
            "taskDefinitionArn": "arn:aws:ecs:us-east-1:653711331788:task-definition/level3:3",
            "containers": [
                {
                    "containerArn": "arn:aws:ecs:us-east-1:653711331788:container/f67ce604-5248-41a0-8a95-78ab4fb57a44",
                    "taskArn": "arn:aws:ecs:us-east-1:653711331788:task/d190d14a-2404-45d6-9113-4eda22d7f2c7",
                    "name": "level3",
                    "networkBindings": [],
                    "lastStatus": "RUNNING",
                    "healthStatus": "UNHEALTHY",
                    "cpu": "0",
                    "networkInterfaces": [
                        {
                            "privateIpv4Address": "172.31.48.168",
                            "attachmentId": "ed8fed01-82d0-4bf6-86cf-fe3115c23ab8"
                        }
                    ]
                }
            ],
            "tags": [],
            "lastStatus": "RUNNING",
            "connectivity": "CONNECTED",
            "healthStatus": "UNHEALTHY",
            "platformVersion": "1.2.0",
            "overrides": {
                "containerOverrides": [
                    {
                        "name": "level3"
                    }
                ]
            },
            "pullStoppedAt": 1543289824.371,
            "cpu": "256"
        }
    ]
}

That tells me the subnet and IP for the ECS, but then to get the Security Groups, I have to look at:

aws ec2 describe-network-interfaces
{
    "NetworkInterfaces": [
        {
            "Status": "in-use",
            "MacAddress": "16:2f:d0:d6:ed:28",
            "SourceDestCheck": true,
            "AvailabilityZone": "us-east-1f",
            "Description": "arn:aws:ecs:us-east-1:653711331788:attachment/ed8fed01-82d0-4bf6-86cf-fe3115c23ab8",
            "NetworkInterfaceId": "eni-200e9484",
            "VpcId": "vpc-d301aea9",
            "PrivateIpAddresses": [
                {
                    "PrivateDnsName": "ip-172-31-48-168.ec2.internal",
                    "PrivateIpAddress": "172.31.48.168",
                    "Primary": true,
                    "Association": {
                        "PublicIp": "3.80.3.41",
                        "PublicDnsName": "ec2-3-80-3-41.compute-1.amazonaws.com",
                        "IpOwnerId": "amazon"
                    }
                }
            ],
            "RequesterManaged": true,
            "PrivateDnsName": "ip-172-31-48-168.ec2.internal",
            "RequesterId": "578734482556",
            "InterfaceType": "interface",
            "Attachment": {
                "Status": "attached",
                "DeviceIndex": 1,
                "AttachTime": "2018-11-27T03:36:34.000Z",
                "DeleteOnTermination": false,
                "AttachmentId": "eni-attach-08ac3da5d33fc7a02",
                "InstanceOwnerId": "501673713797"
            },
            "Groups": [
                {
                    "GroupName": "level3-4493",
                    "GroupId": "sg-0e9a07e8609bce851"
                }
            ],
            "Ipv6Addresses": [],
            "OwnerId": "653711331788",
            "PrivateIpAddress": "172.31.48.168",
            "SubnetId": "subnet-e45602eb",
            "TagSet": [],
            "Association": {
                "PublicIp": "3.80.3.41",
                "PublicDnsName": "ec2-3-80-3-41.compute-1.amazonaws.com",
                "IpOwnerId": "amazon"
            }
        }
    ]
}

So I need to improve my collect. My recent attempt at calling ecs-list-container-instances/ does nothing.

I already have:

$ aws ecs list-clusters
{
    "clusterArns": [
        "arn:aws:ecs:us-east-1:653711331788:cluster/level3"
    ]
}

So I need to call:

aws ecs list-tasks --cluster arn:aws:ecs:us-east-1:653711331788:cluster/level3
{
    "taskArns": [
        "arn:aws:ecs:us-east-1:653711331788:task/d190d14a-2404-45d6-9113-4eda22d7f2c7"
    ]
}

Then I need to use that to call 

aws ecs describe-tasks --cluster level3 --tasks arn:aws:ecs:us-east-1:653711331788:task/d190d14a-2404-45d6-9113-4eda22d7f2c7

So I'll need multiple parameters for collect

Once I have all this data, I'll need to combine the aws ecs describe-tasks with aws ec2 describe-network-interfaces as mentioned above to get the Security Groups.

0xdabbad00 commented 5 years ago

ECS support added in #378

Here is a sample from the flaws2 target account. This ECS I believe is connected to an API Gateway which I don't currently support showing.

Next I'll need to show Lambda and API Gateway

Screen Shot 2019-05-02 at 9 12 33 PM
0xdabbad00 commented 5 years ago

Getting API Gateway configs looks like this:

$ aws apigateway get-rest-apis
{
    "items": [
        {
            "apiKeySource": "HEADER",
            "description": "Created by AWS Lambda",
            "endpointConfiguration": {
                "types": [
                    "REGIONAL"
                ]
            },
            "createdDate": 1542734151,
            "id": "2rfismmoo8",
            "name": "level1-API"
        },
        {
            "apiKeySource": "HEADER",
            "description": "Created by AWS Lambda",
            "endpointConfiguration": {
                "types": [
                    "REGIONAL"
                ]
            },
            "createdDate": 1543342826,
            "id": "4y0944ja4h",
            "name": "level4-API"
        }
    ]
}

Then use the rest-api-id:

$ aws apigateway get-resources --rest-api-id 4y0944ja4h
{
    "items": [
        {
            "path": "/level4",
            "resourceMethods": {
                "ANY": {}
            },
            "id": "31jtug",
            "pathPart": "level4",
            "parentId": "tnf7fjnef6"
        },
        {
            "path": "/",
            "id": "tnf7fjnef6"
        }
    ]
}

Then use the rest-api-id, resource-id, and http-method:

$ aws apigateway get-integration --rest-api-id 4y0944ja4h --resource-id 31jtug --http-method ANY
{
    "integrationResponses": {
        "200": {
            "selectionPattern": ".*",
            "statusCode": "200"
        }
    },
    "passthroughBehavior": "WHEN_NO_MATCH",
    "timeoutInMillis": 29000,
    "uri": "arn:aws:apigateway:us-east-1:lambda:path/2015-03-31/functions/arn:aws:lambda:us-east-1:653711331788:function:level4/invocations",
    "httpMethod": "POST",
    "cacheNamespace": "31jtug",
    "type": "AWS_PROXY",
    "cacheKeyParameters": []
}

This will require a custom collection function. :(

Also, this still doesn't show how I accessed that Level3 ECS. Need to figure out how that is being called publicly.

0xdabbad00 commented 5 years ago

ECS was being exposed directly to the Internet, but I didn't realize that I had to look at the network interface to identify the public IP. This is resolved in #379

0xdabbad00 commented 5 years ago

The flaws2 target account is visualized as follows, as only ECS is supported right now.

Screen Shot 2019-05-03 at 11 56 27 AM

Next up will be Lambda and API Gateway. I also need to change the icons for the VPC Endpoints back to a generic endpoint icon now that I am actually supporting the real services for the icons.

0xdabbad00 commented 5 years ago

I was going crazy trying to find where the Level4 lambda was that I mentioned earlier. Turns out I had setup API Gateway for a level4 lambda, but then deleted that Lambda, so there is no level 4 in flaws2, and thus no lambda, so this is a good test case.

0xdabbad00 commented 5 years ago

There are no Security Groups associated with API Gateway or with a non-VPC Lambda, so I'm not sure if it really makes sense to map these with the current scope of how CloudMapper works. To map those, and have the maps make sense, I would need to parse their policies, which would be a big scope increase. So I'll just map VPC-configured Lambdas.

0xdabbad00 commented 5 years ago

Lambda support added in #381 I only show the Lambdas that are inside VPCs. This is going to confuse people when I don't show other Lambdas, but I don't know how to handle that.

0xdabbad00 commented 5 years ago

Here's a screenshot from a test account with Lambdas.

Screen Shot 2019-05-03 at 2 37 18 PM

I'm going to do ElasticSearch and then RedShift next.

0xdabbad00 commented 5 years ago

ElasticSearch and Redshift added in #382

Screenshot from my test environment. I'll move this data over to the demo data and add unit tests.

Screen Shot 2019-05-03 at 4 54 12 PM
0xdabbad00 commented 5 years ago

I need to fix what I'm doing for Redshift. Currently, I just Redshift as a VPC level node, but technically it does exist in a subnet. I need to do the following:

0xdabbad00 commented 5 years ago

Redshift subnet issue fixed in #383 and the demo data updated along with the tests.

Demo data looks like this currently.

Screen Shot 2019-05-06 at 10 50 14 AM
0xdabbad00 commented 5 years ago

ECS, Lambda, and ES added to demo data in #384

Screen Shot 2019-05-06 at 1 44 11 PM

I'm going to update the online demo, then close this ticket.

0xdabbad00 commented 5 years ago

Demo has been updated.

mfuellbier commented 5 years ago

Why are S3 and Lambda not displayed in the map of my own infrastructure? When preparing the demo-infrastructure, I can see all the different services. But after preparing my own infrastructure (with EC2, ELB, Lambda, S3), I only see EC2 and ELB.

0xdabbad00 commented 5 years ago

@mfuellbier Only VPC resources are displayed, so unless you set up a VPC Endpoint for the S3 service, or if your Lambda's are VPC Lambdas (not the default), then you won't see them. Basically, if you can't attach a Security Group to it, then it won't be displayed. I'll look into clarifying this better in the docs.