Enable Cross-Account Capture

[chelma]

Description

This task is to update the AWS-AIO repo so that it's possible to perform cross-account capture (the GWLBE is in a different account than the GWLB).

See here for an explanation of how this will work: https://github.com/arkime/aws-aio/issues/109#issuecomment-1682526890

Acceptance Criteria

• Able to have traffic generated in account 111111111111 captured by an Arkime cluster in account 222222222222

[chelma]

After looking at the code/docs, here's a tentative plan-of-action for this:

• Update the VPC-Mirroring Stack to use its own EventBus
  • It's currently using a shared Cluster EventBus that Cluster and VPC-specific events use. I think we did that to save on resource count utilization (there's a cap of 100/account-region).
  • After the change, the VPC-Mirroring Stack would only be tied to the Capture Cluster account via the GWLBE and it's association with the GWLB; everything else should be independent and work regardless of if it's in the same or another account.
• Make CLI Commands to register/deregister source accounts w/ the capture account
  • Once you have created a GWLB, you can manage linking of endpoints to it via both permissions and manual approval. In our case, we'll forgo the manual approval but still need to add permissions for an endpoint to be created in the source account that directs to the LB in the capture account.
  • The permission setup/teardown is just a API call, so that part is easy. There will likely be some updates to how we handle state in Parameter Store for both the source and capture sides.
    • https://docs.aws.amazon.com/cli/latest/reference/ec2/modify-vpc-endpoint-service-permissions.html
• Make CLI Commands to register/deregister the capture account w/ the source account
  • This should just be creating some state in the source account (e.g. Parameter Store) so that it has some of the details it needs about the cluster it's working with.
• Update how VNI provisioning works
  • Currently, the VNI selection process relies on a central store in the capture account. However, vpc-add needs to be called using the source account's credentials. We could solve this by providing cross-account permissions to hit the VNI store, which is currently just two Param values. Alternatively, would could issue a block of VNIs to each source account, or something like that. Either way, we can do this set up during the account-account registration CLI calls.
• README update
  • Just describe how this process works

[chelma]

Additional consideration - we have some commands (like clusters-list) which should continue to operate normally when called on the capture account. This means we'll need to update our data store to somehow link the source/capture accounts in the capture account data store. Practically, this probably means that when we perform the registration dance we'll register a specific source VPC/account with the capture account rather than a capture account as a whole. This has the side benefit of also scoping down the whitelists we're applying to our GWLB.

[chelma]

Thinking about this more, we have a couple different approaches we can take with the data store.

One is to allow source accounts to interact with a central data store in the capture account. This will (likely) require the least number of code changes, simplifies some architecture decisions, and is a well-supported AWS paradigm. The flipside is that we'll need to be able to reasonably scope down the cross-account permissions. Ideally, we'd scope the permissions to an explicit list of actors in the source account but I think that will be quite burdensome in-practice. As a result, we'll probably grant the root principal of the source account perms to perform very limited actions on very limited resources.

The other approach is to bulkhead the two accounts. In the short term, this would require a lot more thought about precisely how to accomplish this effectively but would limit (or even eliminate) the need for cross-account data store calls. In the longer term, I believe this will have on ongoing impact on design avenues - which may be a good thing, and may not. It will likely create substantially more friction for users and be more complex to operate, but maybe we really do want to treat these accounts as separate entities.

Another consideration (somewhat) tied to this decision is how to handle multiple actors acting on the data store at the same time. We currently assume there's only one actor performing cluster-level operations at a given time, but that may not always be the case. The problem presents itself for both conjoined and bulkheaded accounts, but seems worth bringing up.

Overall, I think the right decision (for now) is to allow cross-account data store operations, as it feels like we can scope the permissions down pretty tightly in terms of operations/resources and seems to provide a better user experience. It's possible that some users need the bulkhead approach, but we don't have positive data points in that direction yet and can pivot later if needed.

[chelma]

It appears going the cross-account access route makes this fairly straightforward. Here's an updated plan of action, in the order a user would invoke them:

• Create the cluster-register-vpc CLI command
  • Run w/ capture account creds and all actions in capture account
  • Gives the source account root principal permission to make GWLBE's against the cluster's GWLB
  • Creates an IAM role to allow the source account root pricipal access to the cluster/vpc-specific Parameter Store entries
  • Creates an parameter store entry to capture this association
    • Path like: /arkime/clusters/MyCluster3/vpcs/vpc-0f08710cdbc32d58a/cross-account
    • Contents: capture account ID, source account ID, cluster name, VPC ID, IAM role
  • Spits out the vpc-associate-with-cluster CLI command the user needs to run next
• Create the vpc-register-cluster CLI command
  • Run w/ source account creds and all actions in source account
  • Create a duplicate entry of the association in the source account
    • /arkime/clusters/MyCluster3/vpcs/vpc-0f08710cdbc32d58a/cross-account
• Update vpc-add and vpc-remove
  • Run w/ source account creds and all actions in source account
  • When these commands are run, they look for an association parameter store entry in the source account
    • /arkime/clusters/MyCluster3/vpcs/vpc-0f08710cdbc32d58a/cross-account
  • If it is found, SSM operations are performed with a cross-account AWS Client created using STS assume-role; other operations remain the same and use an in-account AWS Client
• Update clusters-list
  • Run w/ capture account creds and all actions in capture account
  • Should look for this association parameter and report the source account ID and IAM role for each VPC
• Create the vpc-deregister-cluster CLI command
  • Run w/ source account creds and all actions in source account
  • Just deletes the associated parameter store entry
• Create the cluster-deregister-vpc CLI command
  • Run w/ capture account creds and all actions in capture account
  • Deletes the cross-account access IAM role and the associated parameter store entry
  • If it's safe to do so, remove the GWLB permissions. Here, safe means there's not another VPC that relies on those permissions, etc.

Overall, the cross-account user workflow would be:

1. cluster-create using capture account credentials
2. cluster-register-vpc using capture account credentials
3. vpc-register-cluster using source account credentials
4. vpc-add using source account credentials
5. vpc-remove using source account credentials
6. vpc-deregister-cluster using source account credentials
7. cluster-deregister-vpc using capture account credentials
8. cluster-destroy using capture account credentials

The same-account workflow would remain unchanged:

1. cluster-create using capture account credentials
2. vpc-add using capture account credentials
3. vpc-remove using capture account credentials
4. cluster-destroy using capture account credentials

[chelma]

PR posted for the vast majority of the work. Remaining effort:

• Update clusters-list to work correctly w/ cross-account VPCs
• Update clusters-deregister-vpc to remove GWLBE creation perms if there's no other VPCs using the,
• Update the README with instructions how to use

[chelma]

OK, so - clusters-list currently finds which VPCs are being monitored by looking for Parameter Store entries of the form /arkime/clusters/<cluster name/vpcs/<vpc id>. That entry is created by the CloudFormation stack that spins up the VPC-specific resources, which means in the cross-account scenario it isn't present in the Cluster account because that stack is created in the VPC account. This means we can either create a duplicate entry in the Cluster account or do a cross-account call from the Cluster account. The later option seems like the better approach to avoid additional bookkeeping but we'll need to make a new cross-account role the Cluster Account can use to get access to the VPC account (easy).

Plan of attack:

• Remove the roleArn entry from the cross-account link Param Store entry, as it's tied to a specific account and can be easily constructed for the correct account from the roleName we also store
• Update vpc-register-cluster to create a cross-account role the Cluster account can use to access Param Store in the VPC account
• Update vpc-deregister-cluster to delete that cross-account role
• Update clusters-list to check for cross-account Param Store link entries and pull VPC details (e.g. VNI) from the VPC account when it finds them

[chelma]

PR posted to cover the remaining work; task should be complete after it is merged.

[chelma]

PRs merged; acceptance criteria met. Resolving.