Missing `ec2:CreateNetworkInterface` perrmission when creating new deployment

[nadove-ucsc]

https://groups.google.com/a/ucsc.edu/g/azul-group/c/wGBpy99vvFA

This was the azul-service-nadove5 role, with policy lacking ec2:CreateNetworkInterface permissions.

[
    {
        "@timestamp": "2023-05-20 21:58:27.810",
        "@message": {
            "eventVersion": "1.08",
            "userIdentity": {
                "type": "AssumedRole",
                "principalId": "AROAUHATKQ7O7SMMXHF5B:749678902839",
                "arn": "arn:aws:sts::289950828509:assumed-role/azul-service-nadove5/749678902839",
                "accountId": "289950828509",
                "accessKeyId": "ASIAUHATKQ7OQHN7M55C",
                "sessionContext": {
                    "sessionIssuer": {
                        "type": "Role",
                        "principalId": "AROAUHATKQ7O7SMMXHF5B",
                        "arn": "arn:aws:iam::289950828509:role/azul-service-nadove5",
                        "accountId": "289950828509",
                        "userName": "azul-service-nadove5"
                    },
                    "webIdFederationData": {},
                    "attributes": {
                        "creationDate": "2023-05-20T21:57:50Z",
                        "mfaAuthenticated": "false"
                    }
                }
            },
            "eventTime": "2023-05-20T21:57:50Z",
            "eventSource": "[ec2.amazonaws.com](http://ec2.amazonaws.com/)",
            "eventName": "CreateNetworkInterface",
            "awsRegion": "us-east-1",
            "sourceIPAddress": "54.156.66.245",
            "userAgent": "aws-internal/3 aws-sdk-java/1.12.461 Linux/4.14.313-235.533.amzn2.x86_64 OpenJDK_64-Bit_Server_VM/11.0.19+7-LTS java/11.0.19 kotlin/1.6.21 vendor/Amazon.com_Inc. cfg/retry-mode/standard",
            "errorCode": "Client.UnauthorizedOperation",
            "errorMessage": "You are not authorized to perform this operation. Encoded authorization failure message: 4TnGqCs_bI1p7uoVUwG_tlodyV_bYAW_8V1DKdsNWklqhoPIXou5d6zm9iGHmHGedDaPLMajro9yz2UpoBrkkWHsqfY8EkrWsM9Xj_0sIxWsL-xEWtOTCMYS3DhwtuGb7FrX4TKx3GjM7XosCylMhTNb38OvdGMlGJYxkOHRs4Ls5o1gWZEUlE_nZuoDnXdrYSZ4xQximoqDuoR2K5m-MPkjuGvCmnjxsHIaRPsRi2cr_E8Vbwsz7COR3kde6vW6ApkRh1ic6-RBrNmbZ6FQs3IANx7WX7ZUw9UG1c5ap8zUevHGiT5pX5WhhcgtjYoUNPxwyLH7cnjj6WuTcDvL7LLFGmu1uGPxKYqy-nH6USKbkyAzK3Svqh9Juy5lFvTaJvXHk-ymwq5wItTv3fKRdSgKv9p05-GZXxY-hlvWcYCpf-aiKA1iGPSqEVvs6k-uSKf2P1GkAi162AdpmGm5W4814tqYCqA0ij1lMr0hf7PbLeHajDZ6WXEr_lWncivLVCbFoRW-vzH2MBTsYI0OBuz5sPilbiQEZVbMijjjcmQsFIWQ9Z2WvFYoZqu_iMZ2BAs41D0rBnG0s4FOlDQrgQ1nwpZdr9c6l3GQNcL4X3A7vBtjmlDvbEY2KRPRxg64hClqUQyvcVu2LEbDQxJTTV02kfAFdJDtC9Hvv5qwTRDfMRxuyii5_drnkBkY1KjPJzf275WaoShtL9RXJogM9O_wTD4HNVovxnnhL7ZrszXnz8hS09WTcUV1Bkw1pPFd6ratBSzSXwKMfBYXAymrra4L3uojevt9Bb_NkyivywgzrfoZhZ1xKYwzZ3jIPmEj1m8",
            "requestParameters": {
                "subnetId": "subnet-03ff70f9c17cabf3f",
                "description": "AWS Lambda VPC ENI-azul-service-nadove5-manifest-3ae83400-5ac0-4316-b51f-ee55ee974b58",
                "groupSet": {
                    "items": [
                        {
                            "groupId": "sg-0c057e200474391c7"
                        }
                    ]
                },
                "privateIpAddressesSet": {},
                "clientToken": "a661de39-4a20-46fb-94e4-a6e15f42d0e6"
            },
            "responseElements": null,
            "requestID": "ad8173ee-69e6-4f34-8a46-1e04b13085b0",
            "eventID": "8acbe0f4-0ddb-45da-a246-00f3cd847ee2",
            "readOnly": false,
            "eventType": "AwsApiCall",
            "managementEvent": true,
            "recipientAccountId": "289950828509",
            "eventCategory": "Management",
            "tlsDetails": {
                "tlsVersion": "TLSv1.2",
                "cipherSuite": "ECDHE-RSA-AES128-GCM-SHA256",
                "clientProvidedHostHeader": "[ec2.us-east-1.amazonaws.com](http://ec2.us-east-1.amazonaws.com/)"
            }
        }
    },
    {
        "@timestamp": "2023-05-20 22:00:00.170",
        "@message": {
            "eventVersion": "1.08",
            "userIdentity": {
                "type": "AssumedRole",
                "principalId": "AROAUHATKQ7O7SMMXHF5B:749678902839",
                "arn": "arn:aws:sts::289950828509:assumed-role/azul-service-nadove5/749678902839",
                "accountId": "289950828509",
                "accessKeyId": "ASIAUHATKQ7OVTEJZWN6",
                "sessionContext": {
                    "sessionIssuer": {
                        "type": "Role",
                        "principalId": "AROAUHATKQ7O7SMMXHF5B",
                        "arn": "arn:aws:iam::289950828509:role/azul-service-nadove5",
                        "accountId": "289950828509",
                        "userName": "azul-service-nadove5"
                    },
                    "webIdFederationData": {},
                    "attributes": {
                        "creationDate": "2023-05-20T21:57:50Z",
                        "mfaAuthenticated": "false"
                    }
                }
            },
            "eventTime": "2023-05-20T21:57:50Z",
            "eventSource": "[ec2.amazonaws.com](http://ec2.amazonaws.com/)",
            "eventName": "CreateNetworkInterface",
            "awsRegion": "us-east-1",
            "sourceIPAddress": "54.156.66.245",
            "userAgent": "aws-internal/3 aws-sdk-java/1.12.461 Linux/4.14.313-235.533.amzn2.x86_64 OpenJDK_64-Bit_Server_VM/11.0.19+7-LTS java/11.0.19 kotlin/1.6.21 vendor/Amazon.com_Inc. cfg/retry-mode/standard",
            "errorCode": "Client.UnauthorizedOperation",
            "errorMessage": "You are not authorized to perform this operation. Encoded authorization failure message: H1FtEqyUnvnT5Ic9qe9X1IDQn15hctSFCg9azw6UQ7GRUvDwg0wMYB7rx9Jfft33BxfkqbFJyOOSRSZzUjmKx3DK6_Cov71KKNtXGoyzzugBB0CxoM9k2ZCr_ZS_XGsHYoiPrq4XYlLmfk25SC6twk7KUjkT7_61Q5hPa19amMmkHI-OZsWNefpB9zZltahBND_k6kv0LorI0OykeRJxcGobRvorn6Ir5Nqd3DiEQO48Di0ZkLCXDKVU52RxNrS1T-fldOecjlgNxtj34bmbjWDmWLFekr1gOELZHUDj8gbVDcFLXX3bA4g8D6yzpylU8WPw_BjM7XrCt0XhFGh25OKZZYKVmdti2c1mlzg0jIdcBsR6x0eI8NuEBKfgeCx9ggviipXiZgwYZ8VOuOM7czkjSP_nCym3ruNxPIjZo01jI5NlTkqL7tRRytOnl41FYP_TD6NNwlePKLxrcShxGXawGOixEbYFbBvd0iqM3NSYD8KFfX3Ou5BSUmfLky0op2ANuDJ_mlCPBspv4UbB0LtnGCit5NulbPwWpQwYUZY7bqCS_-Yxzw-_DA8EFF4PoyJ8bdCEsPA5aPAqZWPqSqXC3rKS9ByDlssL9rgGc1ARAgCyLfEyTmCcp2dZMttYEOZ02zEeXrlEMidK_X9YXRUhneb0sCsab-be2bjJ1WSOdYcByxx9jBmHamqgw9uhdXSs2TmC1Gp28oqQsBY2M24wrCjx7WqizBIF3x38tJAerLVZTAZQl8SVqIoRlgI2MJRlwXuKEfqIU87z3MvILAqy8AJvVEnzZdLFlSEbc0ADdL9ruZZB_ZV_6AFOeI2w",
            "requestParameters": {
                "subnetId": "subnet-076a0d5de7b6ded91",
                "description": "AWS Lambda VPC ENI-azul-service-nadove5-manifest-26702181-c571-4f23-ac96-9209ab6a5719",
                "groupSet": {
                    "items": [
                        {
                            "groupId": "sg-0c057e200474391c7"
                        }
                    ]
                },
                "privateIpAddressesSet": {},
                "clientToken": "5e3a9917-f04b-4851-98d6-05e8cbec3c1c"
            },
            "responseElements": null,
            "requestID": "be6f4e2b-1c51-4662-91eb-130f73395750",
            "eventID": "ce644e1e-6396-4662-b6d3-d5513f45b2b3",
            "readOnly": false,
            "eventType": "AwsApiCall",
            "managementEvent": true,
            "recipientAccountId": "289950828509",
            "eventCategory": "Management",
            "tlsDetails": {
                "tlsVersion": "TLSv1.2",
                "cipherSuite": "ECDHE-RSA-AES128-GCM-SHA256",
                "clientProvidedHostHeader": "[ec2.us-east-1.amazonaws.com](http://ec2.us-east-1.amazonaws.com/)"
            }
        }
    }
]

May be a missing depends_on clause in the TF config.

• [ ] Security design review completed; the Resolution of this issue does not …
  • [ ] … affect authentication; for example:
  • OAuth 2.0 with the application (API or Swagger UI)
  • Authentication of developers with Google Cloud APIs
  • Authentication of developers with AWS APIs
  • Authentication with a GitLab instance in the system
  • Password and 2FA authentication with GitHub
  • API access token authentication with GitHub
  • Authentication with
  • [ ] … affect the permissions of internal users like access to
  • Cloud resources on AWS and GCP
  • GitLab repositories, projects and groups, administration
  • an EC2 instance via SSH
  • GitHub issues, pull requests, commits, commit statuses, wikis, repositories, organizations
  • [ ] … affect the permissions of external users like access to
  • TDR snapshots
  • [ ] … affect permissions of service or bot accounts
  • Cloud resources on AWS and GCP
  • [ ] … affect audit logging in the system, like
  • adding, removing or changing a log message that represents an auditable event
  • changing the routing of log messages through the system
  • [ ] … affect monitoring of the system
  • [ ] … introduce a new software dependency like
  • Python packages on PYPI
  • Command-line utilities
  • Docker images
  • Terraform providers
  • [ ] … add an interface that exposes sensitive or confidential data at the security boundary
  • [ ] … affect the encryption of data at rest
  • [ ] … require persistence of sensitive or confidential data that might require encryption at rest
  • [ ] … require unencrypted transmission of data within the security boundary
  • [ ] … affect the network security layer; for example by
  • modifying, adding or removing firewall rules
  • modifying, adding or removing security groups
  • changing or adding a port a service, proxy or load balancer listens on
• [ ] Documentation on any unchecked boxes is provided in comments below

[achave11-ucsc]

Spike to check for obviously missing dependency between policy and lambda resources.

[achave11-ucsc]

If a missing dependency exists, it doesn't stand out. However, from the logs one might conclude that this is due to a race condition. Item 6, 5 and 1 are of interest in this logs).

[achave11-ucsc]

@hannes-ucsc: "Looking at the CloudTrail event sequence, we know that the PutRolePolicy call precedes the CreateNetworkInterface call, but only by seven seconds. We have observed latency with which policy updates become effective in IAM and those latencies es were in the order of several tens of seconds. Furthermore, the dependency graph in our Terraform config has the lambda depend on the role (R) but not the role policy (RP). That should also be fixed. We can address both problems by inserting a sleep resource(Z) between the function (F) and the role policy (RP) resource."