search

wazuh / wazuh-qa

Wazuh - Quality Assurance
GNU General Public License v2.0
64 stars 30 forks source link

The allocator module fails to allocate instances in AWS due to volume size #5617

Closed rauldpm closed 1 month ago

rauldpm commented 1 month ago

Description

We found the following error when creating instances for Linux endpoints using the Allocator module and AWS resources

botocore.exceptions.ClientError: An error occurred (InvalidBlockDeviceMapping) when calling the RunInstances operation: Volume of size 20GB is smaller than snapshot 'snap-0e449fc6854107ee9', expect size>=30GB

This is happening for

It is necessary to investigate the error and fix the volume size so those systems can be used in AWS

Related

c-bordon commented 1 month ago

Update report

A change is made to obtain the default storage requested by the AMI and compare it with the storage defined by us. If the storage requested by the AMI is greater than the storage we defined, we use the storage requested by the AMI, in this way, we ensure that the instance can be created correctly.

cbordon@cbordon-MS-7C88:~/Documents/wazuh/repositorios/wazuh-qa$ python3 deployability/modules/allocation/main.py --provider aws --size medium --label-team devops --label-termination-date 1d --composite-name linux-debian-12-amd64
[2024-07-30 14:45:05] [INFO] ALLOCATOR: Creating instance at /tmp/wazuh-qa
[2024-07-30 14:45:06] [DEBUG] ALLOCATOR: No config provided. Generating from payload
[2024-07-30 14:45:06] [DEBUG] ALLOCATOR: Generating new key pair
[2024-07-30 14:45:06] [DEBUG] ALLOCATOR: Creating base directory: /tmp/wazuh-qa/AWS-76AC6808-D874-4198-8C4C-3914236DE1B7
Config storage: 20 is less than the AMI storage: 30.
[2024-07-30 14:45:26] [DEBUG] ALLOCATOR: Renaming temp /tmp/wazuh-qa/AWS-76AC6808-D874-4198-8C4C-3914236DE1B7 directory to /tmp/wazuh-qa/i-055b513de26cf97b0
[2024-07-30 14:45:26] [INFO] ALLOCATOR: Instance i-055b513de26cf97b0 created.
[2024-07-30 14:45:28] [INFO] ALLOCATOR: Instance i-055b513de26cf97b0 started.
[2024-07-30 14:45:28] [INFO] ALLOCATOR: The inventory file generated at /tmp/wazuh-qa/i-055b513de26cf97b0/inventory.yaml
[2024-07-30 14:45:28] [INFO] ALLOCATOR: The track file generated at /tmp/wazuh-qa/i-055b513de26cf97b0/track.yaml
[2024-07-30 14:45:28] [WARNING] ALLOCATOR: Error on attempt 1 of 30: [Errno None] Unable to connect to port 2200 on 54.162.193.132
[2024-07-30 14:46:00] [INFO] ALLOCATOR: SSH connection successful.
[2024-07-30 14:46:00] [INFO] ALLOCATOR: Instance i-055b513de26cf97b0 created successfully.
cbordon@cbordon-MS-7C88:~/Documents/wazuh/repositorios/wazuh-qa$ cat /tmp/wazuh-qa/i-055b513de26cf97b0/inventory.yaml
ansible_connection: ssh
ansible_host: ec2-54-162-193-132.compute-1.amazonaws.com
ansible_port: 2200
ansible_ssh_common_args: -o StrictHostKeyChecking=no
ansible_ssh_private_key_file: /tmp/wazuh-qa/i-055b513de26cf97b0/debian-12-amd64-key-4411
ansible_user: admin
cbordon@cbordon-MS-7C88:~/Documents/wazuh/repositorios/wazuh-qa$ ssh -i /tmp/wazuh-qa/i-055b513de26cf97b0/debian-12-amd64-key-4411 -o StrictHostKeyChecking=no -p 2200 admin@ec2-54-162-193-132.compute-1.amazonaws.com
Warning: Permanently added '[ec2-54-162-193-132.compute-1.amazonaws.com]:2200' (ED25519) to the list of known hosts.
Linux ip-172-31-32-230 6.1.0-13-cloud-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.55-1 (2023-09-29) x86_64

The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
Last login: Mon Jul  1 14:33:46 2024 from 79.117.226.210
admin@ip-172-31-32-230:~$ sudo su -
root@ip-172-31-32-230:~# df -h
Filesystem       Size  Used Avail Use% Mounted on
udev             1.9G     0  1.9G   0% /dev
tmpfs            389M  456K  388M   1% /run
/dev/nvme0n1p1    30G  1.1G   27G   4% /
tmpfs            1.9G     0  1.9G   0% /dev/shm
tmpfs            5.0M     0  5.0M   0% /run/lock
/dev/nvme0n1p15  124M   12M  113M  10% /boot/efi
tmpfs            389M     0  389M   0% /run/user/1000
c-bordon commented 1 month ago

Windows tests

cbordon@cbordon-MS-7C88:~/Documents/wazuh/repositorios/wazuh-qa$ python3 deployability/modules/allocation/main.py --provider aws --size medium --label-team devops --label-termination-date 1d --composite-name windows-desktop-10-amd64
[2024-07-30 14:53:27] [INFO] ALLOCATOR: Creating instance at /tmp/wazuh-qa
[2024-07-30 14:53:28] [DEBUG] ALLOCATOR: No config provided. Generating from payload
[2024-07-30 14:53:28] [DEBUG] ALLOCATOR: Creating temp directory: /tmp/wazuh-qa/AWS-F0BC3431-9158-4FC4-AE6A-6EC4272895D3
Config storage: 20 is less than the AMI storage: 40.
[2024-07-30 14:53:46] [DEBUG] ALLOCATOR: Renaming temp /tmp/wazuh-qa/AWS-F0BC3431-9158-4FC4-AE6A-6EC4272895D3 directory to /tmp/wazuh-qa/i-07078c248024bc99e
[2024-07-30 14:53:46] [INFO] ALLOCATOR: Instance i-07078c248024bc99e created.
[2024-07-30 14:53:48] [INFO] ALLOCATOR: Instance i-07078c248024bc99e started.
[2024-07-30 14:53:48] [INFO] ALLOCATOR: The inventory file generated at /tmp/wazuh-qa/i-07078c248024bc99e/inventory.yaml
[2024-07-30 14:53:48] [INFO] ALLOCATOR: The track file generated at /tmp/wazuh-qa/i-07078c248024bc99e/track.yaml
[2024-07-30 14:54:18] [WARNING] ALLOCATOR: Error on attempt 1 of 30: HTTPSConnectionPool(host='ec2-44-200-82-183.compute-1.amazonaws.com', port=5986): Max retries exceeded with url: /wsman (Caused by ConnectTimeoutError(<urllib3.connection.HTTPSConnection object at 0x7045fd714d30>, 'Connection to ec2-44-200-82-183.compute-1.amazonaws.com timed out. (connect timeout=30)'))
[2024-07-30 14:55:18] [WARNING] ALLOCATOR: Error on attempt 2 of 30: HTTPSConnectionPool(host='ec2-44-200-82-183.compute-1.amazonaws.com', port=5986): Max retries exceeded with url: /wsman (Caused by ConnectTimeoutError(<urllib3.connection.HTTPSConnection object at 0x7045fd7172b0>, 'Connection to ec2-44-200-82-183.compute-1.amazonaws.com timed out. (connect timeout=30)'))
[2024-07-30 14:56:18] [WARNING] ALLOCATOR: Error on attempt 3 of 30: HTTPSConnectionPool(host='ec2-44-200-82-183.compute-1.amazonaws.com', port=5986): Max retries exceeded with url: /wsman (Caused by ConnectTimeoutError(<urllib3.connection.HTTPSConnection object at 0x7045fd717ca0>, 'Connection to ec2-44-200-82-183.compute-1.amazonaws.com timed out. (connect timeout=30)'))
[2024-07-30 14:56:49] [WARNING] ALLOCATOR: Error on attempt 4 of 30: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))
[2024-07-30 14:57:19] [WARNING] ALLOCATOR: Error on attempt 5 of 30: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))
[2024-07-30 14:57:50] [WARNING] ALLOCATOR: Error on attempt 6 of 30: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))
[2024-07-30 14:58:20] [WARNING] ALLOCATOR: Error on attempt 7 of 30: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))
[2024-07-30 14:58:51] [WARNING] ALLOCATOR: Error on attempt 8 of 30: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))
[2024-07-30 14:59:21] [WARNING] ALLOCATOR: Error on attempt 9 of 30: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))
[2024-07-30 14:59:51] [WARNING] ALLOCATOR: Error on attempt 10 of 30: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))
[2024-07-30 15:00:22] [WARNING] ALLOCATOR: Error on attempt 11 of 30: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))
[2024-07-30 15:00:52] [WARNING] ALLOCATOR: Error on attempt 12 of 30: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))
[2024-07-30 15:01:22] [WARNING] ALLOCATOR: Error on attempt 13 of 30: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))
[2024-07-30 15:01:53] [WARNING] ALLOCATOR: Error on attempt 14 of 30: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))
[2024-07-30 15:02:23] [WARNING] ALLOCATOR: Error on attempt 15 of 30: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))
[2024-07-30 15:02:53] [WARNING] ALLOCATOR: Error on attempt 16 of 30: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))
[2024-07-30 15:03:24] [WARNING] ALLOCATOR: Error on attempt 17 of 30: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))
[2024-07-30 15:03:54] [WARNING] ALLOCATOR: Error on attempt 18 of 30: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))
[2024-07-30 15:04:24] [WARNING] ALLOCATOR: Error on attempt 19 of 30: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))
[2024-07-30 15:04:55] [WARNING] ALLOCATOR: Error on attempt 20 of 30: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))
[2024-07-30 15:05:25] [WARNING] ALLOCATOR: Error on attempt 21 of 30: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))
[2024-07-30 15:05:56] [WARNING] ALLOCATOR: Error on attempt 22 of 30: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))
[2024-07-30 15:06:56] [WARNING] ALLOCATOR: Error on attempt 23 of 30: HTTPSConnectionPool(host='ec2-44-200-82-183.compute-1.amazonaws.com', port=5986): Max retries exceeded with url: /wsman (Caused by ConnectTimeoutError(<urllib3.connection.HTTPSConnection object at 0x7045fd717790>, 'Connection to ec2-44-200-82-183.compute-1.amazonaws.com timed out. (connect timeout=30)'))
[2024-07-30 15:07:26] [WARNING] ALLOCATOR: Error on attempt 24 of 30: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))
[2024-07-30 15:07:56] [WARNING] ALLOCATOR: Error on attempt 25 of 30: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))
[2024-07-30 15:08:27] [WARNING] ALLOCATOR: Error on attempt 26 of 30: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))
[2024-07-30 15:08:57] [WARNING] ALLOCATOR: Error on attempt 27 of 30: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))
[2024-07-30 15:09:49] [WARNING] ALLOCATOR: Error on attempt 28 of 30: 
[2024-07-30 15:10:41] [WARNING] ALLOCATOR: Error on attempt 29 of 30: 
[2024-07-30 15:11:18] [INFO] ALLOCATOR: WinRM connection successful.
[2024-07-30 15:11:18] [INFO] ALLOCATOR: Instance i-07078c248024bc99e created successfully

Screenshot_20240730_151304

c-bordon commented 1 month ago

macOS test

cbordon@cbordon-MS-7C88:~/Documents/wazuh/repositorios/wazuh-qa$ python3 deployability/modules/allocation/main.py --provider aws --size medium --label-team devops --label-termination-date 1d --composite-name macos-sonoma-14-amd64
[2024-07-30 15:02:02] [INFO] ALLOCATOR: Creating instance at /tmp/wazuh-qa
[2024-07-30 15:02:03] [DEBUG] ALLOCATOR: No config provided. Generating from payload
[2024-07-30 15:02:03] [DEBUG] ALLOCATOR: Generating new key pair
[2024-07-30 15:02:03] [DEBUG] ALLOCATOR: Creating base directory: /tmp/wazuh-qa/AWS-BB2689E8-8E98-4818-A1E7-357CAD0D3FC9
[2024-07-30 15:02:05] [INFO] ALLOCATOR: Creating dedicated host: sonoma-14-amd64-Host-amd64
[2024-07-30 15:02:06] [INFO] ALLOCATOR: Dedicated host created: h-08062808a74443fb6
[2024-07-30 15:02:24] [DEBUG] ALLOCATOR: Renaming temp /tmp/wazuh-qa/AWS-BB2689E8-8E98-4818-A1E7-357CAD0D3FC9 directory to /tmp/wazuh-qa/i-0e97802433b2d874d
[2024-07-30 15:02:24] [INFO] ALLOCATOR: Instance i-0e97802433b2d874d created.
[2024-07-30 15:02:26] [INFO] ALLOCATOR: Instance i-0e97802433b2d874d started.
[2024-07-30 15:02:26] [INFO] ALLOCATOR: The inventory file generated at /tmp/wazuh-qa/i-0e97802433b2d874d/inventory.yaml
[2024-07-30 15:02:26] [INFO] ALLOCATOR: The track file generated at /tmp/wazuh-qa/i-0e97802433b2d874d/track.yaml
[2024-07-30 15:04:40] [WARNING] ALLOCATOR: Error on attempt 1 of 30: [Errno 110] Connection timed out
[2024-07-30 15:07:24] [WARNING] ALLOCATOR: Error on attempt 2 of 30: [Errno 110] Connection timed out
[2024-07-30 15:07:54] [WARNING] ALLOCATOR: Error on attempt 3 of 30: [Errno None] Unable to connect to port 2200 on 52.207.191.134
[2024-07-30 15:08:27] [INFO] ALLOCATOR: SSH connection successful.
[2024-07-30 15:08:27] [INFO] ALLOCATOR: Instance i-0e97802433b2d874d created successfully.

Screenshot_20240730_151247

c-bordon commented 1 month ago

Debian 12 arm

cbordon@cbordon-MS-7C88:~/Documents/wazuh/repositorios/wazuh-qa$ python3 deployability/modules/allocation/main.py --provider aws --size medium --label-team devops --label-termination-date 1d --composite-name linux-debian-12-arm64
[2024-07-30 15:19:13] [INFO] ALLOCATOR: Creating instance at /tmp/wazuh-qa
[2024-07-30 15:19:13] [DEBUG] ALLOCATOR: No config provided. Generating from payload
[2024-07-30 15:19:13] [DEBUG] ALLOCATOR: Generating new key pair
[2024-07-30 15:19:13] [DEBUG] ALLOCATOR: Creating base directory: /tmp/wazuh-qa/AWS-595EE7D1-2A01-456A-873B-EDF3D2924797
[2024-07-30 15:19:34] [DEBUG] ALLOCATOR: Renaming temp /tmp/wazuh-qa/AWS-595EE7D1-2A01-456A-873B-EDF3D2924797 directory to /tmp/wazuh-qa/i-0ceaf401e8b3e6659
[2024-07-30 15:19:34] [INFO] ALLOCATOR: Instance i-0ceaf401e8b3e6659 created.
[2024-07-30 15:19:35] [INFO] ALLOCATOR: Instance i-0ceaf401e8b3e6659 started.
[2024-07-30 15:19:35] [INFO] ALLOCATOR: The inventory file generated at /tmp/wazuh-qa/i-0ceaf401e8b3e6659/inventory.yaml
[2024-07-30 15:19:35] [INFO] ALLOCATOR: The track file generated at /tmp/wazuh-qa/i-0ceaf401e8b3e6659/track.yaml
[2024-07-30 15:19:43] [WARNING] ALLOCATOR: Error on attempt 1 of 30: [Errno None] Unable to connect to port 2200 on 3.216.126.171
[2024-07-30 15:20:14] [INFO] ALLOCATOR: SSH connection successful.
[2024-07-30 15:20:14] [INFO] ALLOCATOR: Instance i-0ceaf401e8b3e6659 created successfully.

Screenshot_20240730_152038

c-bordon commented 1 month ago

Oracle Linux 9

cbordon@cbordon-MS-7C88:~/Documents/wazuh/repositorios/wazuh-qa$ python3 deployability/modules/allocation/main.py --provider aws --size medium --label-team devops --label-termination-date 1d --composite-name linux-oracle-9-amd64
[2024-07-30 15:21:31] [INFO] ALLOCATOR: Creating instance at /tmp/wazuh-qa
[2024-07-30 15:21:32] [DEBUG] ALLOCATOR: No config provided. Generating from payload
[2024-07-30 15:21:32] [DEBUG] ALLOCATOR: Generating new key pair
[2024-07-30 15:21:32] [DEBUG] ALLOCATOR: Creating base directory: /tmp/wazuh-qa/AWS-B8071AC0-CEB2-4E13-86EE-6887D4249970
[2024-07-30 15:21:52] [DEBUG] ALLOCATOR: Renaming temp /tmp/wazuh-qa/AWS-B8071AC0-CEB2-4E13-86EE-6887D4249970 directory to /tmp/wazuh-qa/i-0e343abfea62eaff0
[2024-07-30 15:21:52] [INFO] ALLOCATOR: Instance i-0e343abfea62eaff0 created.
[2024-07-30 15:21:53] [INFO] ALLOCATOR: Instance i-0e343abfea62eaff0 started.
[2024-07-30 15:21:54] [INFO] ALLOCATOR: The inventory file generated at /tmp/wazuh-qa/i-0e343abfea62eaff0/inventory.yaml
[2024-07-30 15:21:54] [INFO] ALLOCATOR: The track file generated at /tmp/wazuh-qa/i-0e343abfea62eaff0/track.yaml
[2024-07-30 15:23:03] [INFO] ALLOCATOR: SSH connection successful.
[2024-07-30 15:23:03] [INFO] ALLOCATOR: Instance i-0e343abfea62eaff0 created successfully.

Screenshot_20240730_152324