Added a --just-print-cfn option to the cluster-create and vpc-add, which skips CloudFormation deployments and (most) AWS Operations that create resources. We can't skip all AWS operations because some are required in order to generate the CloudFormation templates.
The CloudFormation templates are written to the repo root in a cluster/account/region specific directory.
Also renamed the Cluster Arkime Configuration directory to use the same suffixes as the Cfn template directories. This will prevent users from seeing collisions when specifying Arkime configuration for Clusters with the same name in different accounts/regions
Ran the cluster-create and vpc-add commands locally
(.venv) chelma@3c22fba4e266 aws-aio % ./manage_arkime.py cluster-create --name MyCluster3 --preconfirm-usage --just-print-cfn
2023-08-15 11:46:03 - Debug-level logs save to file: /Users/chelma/workspace/Arkime/aws-aio/manage_arkime/manage_arkime.log
2023-08-15 11:46:03 - Using AWS Credential Profile: default
2023-08-15 11:46:03 - Using AWS Region: default from AWS Config settings
2023-08-15 11:46:05 - Cost estimate report:
OnDemand us-east-1 cost estimate, your cost may be different based on region, discounts or reserve instances:
Allocated:
Capture 1 * $ 30.3680/mo = $ 30.37/mo
Viewer 2 * $ 29.5504/mo = $ 59.10/mo
OS Master Node 3 * $ 26.2800/mo = $ 78.84/mo
OS Data Node 2 * $ 26.2800/mo = $ 52.56/mo
OS Storage 200 * $ 0.1000/GB = $ 20.00/mo
Variable:
PCAP Storage first 50TB 810 * $ 0.0230/GB = $ 18.63/mo
GWLB 3,285 * $ 0.0040/GB = $ 13.14/mo
GWLBE 3,285 * $ 0.0035/GB = $ 11.50/mo
Traffic Mirror/ENI 1 * $ 10.9500/mo = $ 10.95/mo
Total:
$ 295.09/mo
2023-08-15 11:46:05 - Usage report:
Arkime Metadata:
Session Retention [days]: 30
User History Retention [days]: 365
Capture Nodes:
Max Count: 2
Desired Count: 1
Min Count: 1
Type: t3.medium
OpenSearch Domain:
Master Node Count: 3
Master Node Type: t3.small.search
Data Node Count: 2
Data Node Type: t3.small.search
Data Node Volume Size [GB]: 100
S3:
PCAP Retention [days]: 30
2023-08-15 11:46:06 - Ensuring Arkime Config dir exists for cluster: MyCluster3
2023-08-15 11:46:06 - Arkime Config dir exists at: /Users/chelma/workspace/Arkime/aws-aio/config-MyCluster3-XXXXXXXXXXXX-us-east-2
2023-08-15 11:46:06 - Copying default Arkime Config to dir: /Users/chelma/workspace/Arkime/aws-aio/config-MyCluster3-XXXXXXXXXXXX-us-east-2
2023-08-15 11:46:06 - Cluster config directory not empty; skipping copy
2023-08-15 11:46:06 - Determining the status of S3 bucket: arkimeconfig-XXXXXXXXXXXX-us-east-2-mycluster3
2023-08-15 11:46:06 - S3 Bucket arkimeconfig-XXXXXXXXXXXX-us-east-2-mycluster3 already exists; no work needed
2023-08-15 11:46:06 - Uploading Arkime config for Capture Nodes...
2023-08-15 11:46:07 - Config has been uploaded previously; skipping
2023-08-15 11:46:07 - Uploading Arkime config for Viewer Nodes...
2023-08-15 11:46:07 - Config has been uploaded previously; skipping
2023-08-15 11:46:08 - Executing command: synthesize --quiet MyCluster3-CaptureBucket MyCluster3-CaptureNodes MyCluster3-CaptureVPC MyCluster3-OSDomain MyCluster3-ViewerNodes
2023-08-15 11:46:08 - NOTE: This operation can take a while. You can 'tail -f' the logfile to track the status.
2023-08-15 11:47:12 - Synthesize succeeded
2023-08-15 11:47:12 - Setting up the CloudFormation template directory for cluster: MyCluster3
2023-08-15 11:47:12 - Removing any existing CloudFormation templates...
2023-08-15 11:47:12 - Copying over CloudFormation templates for current command...
2023-08-15 11:47:12 - CloudFormation template dir exists at: /Users/chelma/workspace/Arkime/aws-aio/cfn-MyCluster3-XXXXXXXXXXXX-us-east-2
(.venv) chelma@3c22fba4e266 aws-aio % ls -al /Users/chelma/workspace/Arkime/aws-aio/cfn-MyCluster3-XXXXXXXXXXXX-us-east-2
total 208
drwxr-xr-x 7 chelma staff 224 Aug 15 11:47 .
drwxr-xr-x 33 chelma staff 1056 Aug 15 11:47 ..
-rw-r--r-- 1 chelma staff 4559 Aug 15 11:47 MyCluster3-CaptureBucket.template.json
-rw-r--r-- 1 chelma staff 36717 Aug 15 11:47 MyCluster3-CaptureNodes.template.json
-rw-r--r-- 1 chelma staff 14763 Aug 15 11:47 MyCluster3-CaptureVPC.template.json
-rw-r--r-- 1 chelma staff 17019 Aug 15 11:47 MyCluster3-OSDomain.template.json
-rw-r--r-- 1 chelma staff 21982 Aug 15 11:47 MyCluster3-ViewerNodes.template.json
(.venv) chelma@3c22fba4e266 aws-aio % ./manage_arkime.py vpc-add --cluster-name MyCluster3 --vpc-id vpc-008d258d7c536384b --just-print-cfn
2023-08-15 13:11:17 - Debug-level logs save to file: /Users/chelma/workspace/Arkime/aws-aio/manage_arkime/manage_arkime.log
2023-08-15 13:11:17 - Using AWS Credential Profile: default
2023-08-15 13:11:17 - Using AWS Region: default from AWS Config settings
2023-08-15 13:11:22 - Executing command: synthesize --quiet MyCluster3-vpc-008d258d7c536384b-Mirror
2023-08-15 13:11:22 - NOTE: This operation can take a while. You can 'tail -f' the logfile to track the status.
2023-08-15 13:12:45 - Synthesize succeeded
2023-08-15 13:12:45 - Setting up the CloudFormation template directory for cluster: MyCluster3
2023-08-15 13:12:45 - Removing any existing CloudFormation templates...
2023-08-15 13:12:45 - Copying over CloudFormation templates for current command...
2023-08-15 13:12:45 - CloudFormation template dir exists at: /Users/chelma/workspace/Arkime/aws-aio/cfn-MyCluster3-XXXXXXXXXXXX-us-east-2
(.venv) chelma@3c22fba4e266 aws-aio % ls -al /Users/chelma/workspace/Arkime/aws-aio/cfn-MyCluster3-XXXXXXXXXXXX-us-east-2
total 72
drwxr-xr-x 3 chelma staff 96 Aug 15 13:12 .
drwxr-xr-x 33 chelma staff 1056 Aug 15 13:12 ..
-rw-r--r-- 1 chelma staff 35983 Aug 15 13:12 MyCluster3-vpc-008d258d7c536384b-Mirror.template.json
## License
I confirm that this contribution is made under an Apache 2.0 license and that I have the authority necessary to make this contribution on behalf of its copyright owner.
Description
--just-print-cfnoption to thecluster-createandvpc-add, which skips CloudFormation deployments and (most) AWS Operations that create resources. We can't skip all AWS operations because some are required in order to generate the CloudFormation templates.Tasks
Testing
cluster-createandvpc-addcommands locally2023-08-15 11:46:05 - Usage report: Arkime Metadata: Session Retention [days]: 30 User History Retention [days]: 365 Capture Nodes: Max Count: 2 Desired Count: 1 Min Count: 1 Type: t3.medium OpenSearch Domain: Master Node Count: 3 Master Node Type: t3.small.search Data Node Count: 2 Data Node Type: t3.small.search Data Node Volume Size [GB]: 100 S3: PCAP Retention [days]: 30
2023-08-15 11:46:06 - Ensuring Arkime Config dir exists for cluster: MyCluster3 2023-08-15 11:46:06 - Arkime Config dir exists at: /Users/chelma/workspace/Arkime/aws-aio/config-MyCluster3-XXXXXXXXXXXX-us-east-2 2023-08-15 11:46:06 - Copying default Arkime Config to dir: /Users/chelma/workspace/Arkime/aws-aio/config-MyCluster3-XXXXXXXXXXXX-us-east-2 2023-08-15 11:46:06 - Cluster config directory not empty; skipping copy 2023-08-15 11:46:06 - Determining the status of S3 bucket: arkimeconfig-XXXXXXXXXXXX-us-east-2-mycluster3 2023-08-15 11:46:06 - S3 Bucket arkimeconfig-XXXXXXXXXXXX-us-east-2-mycluster3 already exists; no work needed 2023-08-15 11:46:06 - Uploading Arkime config for Capture Nodes... 2023-08-15 11:46:07 - Config has been uploaded previously; skipping 2023-08-15 11:46:07 - Uploading Arkime config for Viewer Nodes... 2023-08-15 11:46:07 - Config has been uploaded previously; skipping 2023-08-15 11:46:08 - Executing command: synthesize --quiet MyCluster3-CaptureBucket MyCluster3-CaptureNodes MyCluster3-CaptureVPC MyCluster3-OSDomain MyCluster3-ViewerNodes 2023-08-15 11:46:08 - NOTE: This operation can take a while. You can 'tail -f' the logfile to track the status. 2023-08-15 11:47:12 - Synthesize succeeded 2023-08-15 11:47:12 - Setting up the CloudFormation template directory for cluster: MyCluster3 2023-08-15 11:47:12 - Removing any existing CloudFormation templates... 2023-08-15 11:47:12 - Copying over CloudFormation templates for current command... 2023-08-15 11:47:12 - CloudFormation template dir exists at: /Users/chelma/workspace/Arkime/aws-aio/cfn-MyCluster3-XXXXXXXXXXXX-us-east-2
(.venv) chelma@3c22fba4e266 aws-aio % ls -al /Users/chelma/workspace/Arkime/aws-aio/cfn-MyCluster3-XXXXXXXXXXXX-us-east-2 total 208 drwxr-xr-x 7 chelma staff 224 Aug 15 11:47 . drwxr-xr-x 33 chelma staff 1056 Aug 15 11:47 .. -rw-r--r-- 1 chelma staff 4559 Aug 15 11:47 MyCluster3-CaptureBucket.template.json -rw-r--r-- 1 chelma staff 36717 Aug 15 11:47 MyCluster3-CaptureNodes.template.json -rw-r--r-- 1 chelma staff 14763 Aug 15 11:47 MyCluster3-CaptureVPC.template.json -rw-r--r-- 1 chelma staff 17019 Aug 15 11:47 MyCluster3-OSDomain.template.json -rw-r--r-- 1 chelma staff 21982 Aug 15 11:47 MyCluster3-ViewerNodes.template.json
(.venv) chelma@3c22fba4e266 aws-aio % ./manage_arkime.py vpc-add --cluster-name MyCluster3 --vpc-id vpc-008d258d7c536384b --just-print-cfn 2023-08-15 13:11:17 - Debug-level logs save to file: /Users/chelma/workspace/Arkime/aws-aio/manage_arkime/manage_arkime.log 2023-08-15 13:11:17 - Using AWS Credential Profile: default 2023-08-15 13:11:17 - Using AWS Region: default from AWS Config settings 2023-08-15 13:11:22 - Executing command: synthesize --quiet MyCluster3-vpc-008d258d7c536384b-Mirror 2023-08-15 13:11:22 - NOTE: This operation can take a while. You can 'tail -f' the logfile to track the status. 2023-08-15 13:12:45 - Synthesize succeeded 2023-08-15 13:12:45 - Setting up the CloudFormation template directory for cluster: MyCluster3 2023-08-15 13:12:45 - Removing any existing CloudFormation templates... 2023-08-15 13:12:45 - Copying over CloudFormation templates for current command... 2023-08-15 13:12:45 - CloudFormation template dir exists at: /Users/chelma/workspace/Arkime/aws-aio/cfn-MyCluster3-XXXXXXXXXXXX-us-east-2
(.venv) chelma@3c22fba4e266 aws-aio % ls -al /Users/chelma/workspace/Arkime/aws-aio/cfn-MyCluster3-XXXXXXXXXXXX-us-east-2 total 72 drwxr-xr-x 3 chelma staff 96 Aug 15 13:12 . drwxr-xr-x 33 chelma staff 1056 Aug 15 13:12 .. -rw-r--r-- 1 chelma staff 35983 Aug 15 13:12 MyCluster3-vpc-008d258d7c536384b-Mirror.template.json