Can now specify the Viewer CIDR

chelma commented 12 months ago

Description

Adds the --viewer-cidr option to the CLI. When this is provided, the Viewer Nodes are split into a separate VPC that is peered with the Capture VPC using a TGW.
The CLI tries to ensure that the user's Capacity Plan will fit in the Capture/Viewer VPCs, that the two VPCs don't have overlapping IP space, and that the user isn't trying to change the CIDRs after initial creation - three particularly bad scenarios.

Tasks

Testing

Added unit tests
Used the feature to spin up a new Cluster with separate CIDRs for Capture/Viewer and added a VPC to it. The traffic shows up in the dashboard as normal. Confirmed in the AWS Console the VPC IP space split.

(.venv) chelma@3c22fba4e266 aws-aio % ./manage_arkime.py cluster-create --name MyCluster --capture-cidr 192.168.0.0/26 --viewer-cidr 10.0.0.0/26 --preconfirm-usage
2023-09-12 08:39:33 - Debug-level logs save to file: /Users/chelma/workspace/Arkime/aws-aio/manage_arkime/manage_arkime.log
2023-09-12 08:39:33 - Using AWS Credential Profile: default
2023-09-12 08:39:33 - Using AWS Region: default from AWS Config settings
2023-09-12 08:39:35 - Cost estimate report:
OnDemand us-east-1 cost estimate, your cost may be different based on region, discounts or reserve instances:
Allocated:
   Capture                         1 * $  30.3680/mo = $     30.37/mo
   Viewer                          1 * $  29.5504/mo = $     29.55/mo
   OS Master Node                  3 * $  26.2800/mo = $     78.84/mo
   OS Data Node                    2 * $  26.2800/mo = $     52.56/mo
   OS Storage                    200 * $   0.1000/GB = $     20.00/mo
Variable:
   PCAP Storage first 50TB       810 * $   0.0230/GB = $     18.63/mo
   GWLB                        3,285 * $   0.0040/GB = $     13.14/mo
   GWLBE                       3,285 * $   0.0035/GB = $     11.50/mo
   Traffic Mirror/ENI              1 * $  10.9500/mo = $     10.95/mo
Total:
                                                       $    265.54/mo

2023-09-12 08:39:35 - Usage report:
Arkime Metadata:
    Session Retention [days]: 30
    User History Retention [days]: 365
Capture Nodes:
    Max Count: 2
    Desired Count: 1
    Min Count: 1
    Type: t3.medium
Viewer Nodes:
    Max Count: 2
    Min Count: 1
OpenSearch Domain:
    Master Node Count: 3
    Master Node Type: t3.small.search
    Data Node Count: 2
    Data Node Type: t3.small.search
    Data Node Volume Size [GB]: 100
S3:
    PCAP Retention [days]: 30

2023-09-12 08:39:36 - Generating self-signed certificate...
2023-09-12 08:39:36 - Certificate generated
2023-09-12 08:39:37 - Ensuring Arkime Config dir exists for cluster: MyCluster
2023-09-12 08:39:37 - Arkime Config dir exists at: /Users/chelma/workspace/Arkime/aws-aio/config-MyCluster-XXXXXXXXXXXX-us-east-2
2023-09-12 08:39:37 - Copying default Arkime Config to dir: /Users/chelma/workspace/Arkime/aws-aio/config-MyCluster-XXXXXXXXXXXX-us-east-2
2023-09-12 08:39:37 - Determining the status of S3 bucket: arkimeconfig-XXXXXXXXXXXX-us-east-2-mycluster
2023-09-12 08:39:38 - S3 Bucket arkimeconfig-XXXXXXXXXXXX-us-east-2-mycluster does not exist; creating it to hold Arkime Configuration
2023-09-12 08:39:39 - Uploading Arkime config for Capture Nodes...
2023-09-12 08:39:40 - Turning Capture configuration at /Users/chelma/workspace/Arkime/aws-aio/config-MyCluster-XXXXXXXXXXXX-us-east-2/capture into archive at /Users/chelma/workspace/Arkime/aws-aio/config-MyCluster-XXXXXXXXXXXX-us-east-2/capture.zip
2023-09-12 08:39:40 - Uploading config archive to S3 bucket: arkimeconfig-XXXXXXXXXXXX-us-east-2-mycluster
2023-09-12 08:39:41 - Uploading Arkime config for Viewer Nodes...
2023-09-12 08:39:42 - Turning Viewer configuration at /Users/chelma/workspace/Arkime/aws-aio/config-MyCluster-XXXXXXXXXXXX-us-east-2/viewer into archive at /Users/chelma/workspace/Arkime/aws-aio/config-MyCluster-XXXXXXXXXXXX-us-east-2/viewer.zip
2023-09-12 08:39:42 - Uploading config archive to S3 bucket: arkimeconfig-XXXXXXXXXXXX-us-east-2-mycluster
2023-09-12 08:39:43 - Executing command: deploy MyCluster-CaptureBucket MyCluster-CaptureNodes MyCluster-CaptureVPC MyCluster-OSDomain MyCluster-ViewerNodes MyCluster-CaptureTGW MyCluster-ViewerVpc
2023-09-12 08:39:43 - NOTE: This operation can take a while.  You can 'tail -f' the logfile to track the status.
2023-09-12 09:29:55 - Deployment succeeded
(.venv) chelma@3c22fba4e266 aws-aio %
(.venv) chelma@3c22fba4e266 aws-aio %
(.venv) chelma@3c22fba4e266 aws-aio %

(.venv) chelma@3c22fba4e266 aws-aio % ./manage_arkime.py vpc-add --cluster-name MyCluster --vpc-id vpc-0f08710cdbc32d58a
2023-09-12 09:34:59 - Debug-level logs save to file: /Users/chelma/workspace/Arkime/aws-aio/manage_arkime/manage_arkime.log
2023-09-12 09:34:59 - Using AWS Credential Profile: default
2023-09-12 09:34:59 - Using AWS Region: default from AWS Config settings
2023-09-12 09:35:03 - Deploying shared mirroring components via CDK...
2023-09-12 09:35:03 - Executing command: deploy MyCluster-vpc-0f08710cdbc32d58a-Mirror
2023-09-12 09:35:03 - NOTE: This operation can take a while.  You can 'tail -f' the logfile to track the status.
2023-09-12 09:38:46 - Deployment succeeded
2023-09-12 09:38:48 - Initiating creation of mirroring session for ENI eni-0c8086cab8c0f6d7d
2023-09-12 09:38:48 - Initiating creation of mirroring session for ENI eni-0cec24eb6bd7eca23
2023-09-12 09:38:49 - Initiating creation of mirroring session for ENI eni-0de172a698bd98bd1
2023-09-12 09:38:50 - Initiating creation of mirroring session for ENI eni-00c6430d08b9aa3bb
2023-09-12 09:38:50 - Initiating creation of mirroring session for ENI eni-0f76be6c5ea4f6af8
2023-09-12 09:38:50 - Initiating creation of mirroring session for ENI eni-0d555c627e87ff41c
2023-09-12 09:38:51 - Initiating creation of mirroring session for ENI eni-0869c4edb794aa0e1

Screen Shot 2023-09-12 at 10 03 42 AM

Screen Shot 2023-09-12 at 10 05 28 AM

License

I confirm that this contribution is made under an Apache 2.0 license and that I have the authority necessary to make this contribution on behalf of its copyright owner.

awick commented 12 months ago

I think there should be a line in the cost estimate, looks like each attachment is $36.5 (0.05 * 730) a month, and there are two of them? https://aws.amazon.com/transit-gateway/pricing/

chelma commented 12 months ago

I think there should be a line in the cost estimate, looks like each attachment is $36.5 (0.05 * 730) a month, and there are two of them? https://aws.amazon.com/transit-gateway/pricing/

Good callout, will add

chelma commented 12 months ago

Posted a fix for a cluster-destroy deletion order bug and added TGWs to the pricing plan.

OnDemand us-east-1 cost estimate, your cost may be different based on region, discounts or reserve instances:
Allocated:
   Capture                         1 * $  30.3680/mo = $     30.37/mo
   Viewer                          1 * $  29.5504/mo = $     29.55/mo                                                                   OS Master Node                  3 * $  26.2800/mo = $     78.84/mo
   OS Data Node                    2 * $  26.2800/mo = $     52.56/mo
   OS Storage                    200 * $   0.1000/GB = $     20.00/mo
   TGW Attachments                 2 * $  36.5000/mo = $     73.00/mo
Variable:
   PCAP Storage first 50TB       810 * $   0.0230/GB = $     18.63/mo
   GWLB                        3,285 * $   0.0040/GB = $     13.14/mo
   GWLBE                       3,285 * $   0.0035/GB = $     11.50/mo
   Traffic Mirror/ENI              1 * $  10.9500/mo = $     10.95/mo
Total:
                                                       $    338.54/mo

arkime / aws-aio