DTT1 - Identify Allocator module resources uniquely

Description

We need to improve the Allocator module to allow identify each instance uniquely, as currently, each instance is being named with generic data

The instance should be identified in a way to be able to answer the following questions

Who created the resource
For what task is the resource created

We need to find a way to identify those resources for local and cloud (Jenkins/GitHub) deployments

Required changes

[x] The label-issue parameter must be used to set the instance name for all the resources (as in AWS).
[x] If the label-issue is not specified, the Allocator module must validate if the instance-name parameter is specified.
[x] If none of those parameters were specified, the Allocator module must ask for one of those parameters and exit.
[ ] The directory name (for the artifacts) must maintain the current UUID.

https://github.com/wazuh/wazuh-qa/issues/4495

Testing Vagrant local:

Testing without `--instance-name` or `--label-issue`

cbordon@cbordon-MS-7C88:~/Documents/wazuh/repositorios/wazuh-qa$ python3 deployability/modules/allocation/main.py --provider vagrant --size micro --composite-name linux-ubuntu-22.04-amd64 
[2024-05-06 14:09:27] [INFO] ALLOCATOR: Creating instance at /tmp/wazuh-qa
Traceback (most recent call last):
  File "/home/cbordon/Documents/wazuh/repositorios/wazuh-qa/deployability/modules/allocation/main.py", line 39, in <module>
    main()
  File "/home/cbordon/Documents/wazuh/repositorios/wazuh-qa/deployability/modules/allocation/main.py", line 35, in main
    Allocator.run(InputPayload(**vars(parse_arguments())))
  File "/home/cbordon/Documents/wazuh/repositorios/wazuh-qa/deployability/modules/allocation/allocation.py", line 37, in run
    return cls.__create(payload)
  File "/home/cbordon/Documents/wazuh/repositorios/wazuh-qa/deployability/modules/allocation/allocation.py", line 56, in __create
    instance = provider.create_instance(
  File "/home/cbordon/Documents/wazuh/repositorios/wazuh-qa/deployability/modules/allocation/generic/provider.py", line 70, in create_instance
    return cls._create_instance(base_dir, params, config, ssh_key)
  File "/home/cbordon/Documents/wazuh/repositorios/wazuh-qa/deployability/modules/allocation/vagrant/provider.py", line 60, in _create_instance
    raise ValueError("Instance name or issue label is required.")
ValueError: Instance name or issue label is required.

Testing with `--label-issue`

cbordon@cbordon-MS-7C88:~/Documents/wazuh/repositorios/wazuh-qa$ python3 deployability/modules/allocation/main.py --provider vagrant --size micro --composite-name linux-ubuntu-22.04-amd64 --label-issue https://github.com/wazuh/wazuh-qa/issues/5311
[2024-05-06 14:11:35] [INFO] ALLOCATOR: Creating instance at /tmp/wazuh-qa
[2024-05-06 14:11:35] [DEBUG] ALLOCATOR: No config provided. Generating from payload
[2024-05-06 14:11:35] [DEBUG] ALLOCATOR: Generating new key pair
[2024-05-06 14:11:39] [DEBUG] ALLOCATOR: Vagrantfile created. Creating instance.
[2024-05-06 14:11:39] [INFO] ALLOCATOR: Instance qa-5311-ubuntu-22.04-9438 created.
[2024-05-06 14:12:23] [INFO] ALLOCATOR: Instance qa-5311-ubuntu-22.04-9438 started.
[2024-05-06 14:12:26] [INFO] ALLOCATOR: Inventory file generated at /tmp/wazuh-qa/qa-5311-ubuntu-22.04-9438/inventory.yml
[2024-05-06 14:12:26] [INFO] ALLOCATOR: SSH connection successful.
[2024-05-06 14:12:28] [INFO] ALLOCATOR: Track file generated at /tmp/wazuh-qa/qa-5311-ubuntu-22.04-9438/track.yml
cbordon@cbordon-MS-7C88:~/Documents/wazuh/repositorios/wazuh-qa$ python3 deployability/modules/allocation/main.py --action delete --track-output /tmp/wazuh-qa/qa-5311-ubuntu-22.04-9438/track.yml
[2024-05-06 14:13:12] [INFO] ALLOCATOR: Deleting instance from trackfile /tmp/wazuh-qa/qa-5311-ubuntu-22.04-9438/track.yml
[2024-05-06 14:13:12] [DEBUG] ALLOCATOR: Destroying instance qa-5311-ubuntu-22.04-9438
[2024-05-06 14:13:17] [INFO] ALLOCATOR: Instance qa-5311-ubuntu-22.04-9438 deleted.

Screenshot_20240506_141251

Testing with `--instance-name`

cbordon@cbordon-MS-7C88:~/Documents/wazuh/repositorios/wazuh-qa$ python3 deployability/modules/allocation/main.py --provider vagrant --size micro --composite-name linux-ubuntu-22.04-amd64 --instance-name cbordon-testing-ubuntu
[2024-05-06 14:13:52] [INFO] ALLOCATOR: Creating instance at /tmp/wazuh-qa
[2024-05-06 14:13:52] [DEBUG] ALLOCATOR: No config provided. Generating from payload
[2024-05-06 14:13:52] [DEBUG] ALLOCATOR: Generating new key pair
[2024-05-06 14:13:55] [DEBUG] ALLOCATOR: Vagrantfile created. Creating instance.
[2024-05-06 14:13:55] [INFO] ALLOCATOR: Instance cbordon-testing-ubuntu-9104 created.
[2024-05-06 14:14:36] [INFO] ALLOCATOR: Instance cbordon-testing-ubuntu-9104 started.
[2024-05-06 14:14:38] [INFO] ALLOCATOR: Inventory file generated at /tmp/wazuh-qa/cbordon-testing-ubuntu-9104/inventory.yml
[2024-05-06 14:14:38] [INFO] ALLOCATOR: SSH connection successful.
[2024-05-06 14:14:41] [INFO] ALLOCATOR: Track file generated at /tmp/wazuh-qa/cbordon-testing-ubuntu-9104/track.yml
cbordon@cbordon-MS-7C88:~/Documents/wazuh/repositorios/wazuh-qa$ python3 deployability/modules/allocation/main.py --action delete --track-output /tmp/wazuh-qa/cbordon-testing-ubuntu-9104/track.yml 
[2024-05-06 14:17:27] [INFO] ALLOCATOR: Deleting instance from trackfile /tmp/wazuh-qa/cbordon-testing-ubuntu-9104/track.yml
[2024-05-06 14:17:27] [DEBUG] ALLOCATOR: Destroying instance cbordon-testing-ubuntu-9104
[2024-05-06 14:17:32] [INFO] ALLOCATOR: Instance cbordon-testing-ubuntu-9104 deleted.

Screenshot_20240506_141701

Testing validation for github Wazuh repository

cbordon@cbordon-MS-7C88:~/Documents/wazuh/repositorios/wazuh-qa$ python3 deployability/modules/allocation/main.py --provider vagrant --size micro --composite-name linux-ubuntu-22.04-amd64 --label-issue https://github.com/cbordon/wazuh-qa/issues/5311
[2024-05-06 14:18:44] [INFO] ALLOCATOR: Creating instance at /tmp/wazuh-qa
Traceback (most recent call last):
  File "/home/cbordon/Documents/wazuh/repositorios/wazuh-qa/deployability/modules/allocation/main.py", line 39, in <module>
    main()
  File "/home/cbordon/Documents/wazuh/repositorios/wazuh-qa/deployability/modules/allocation/main.py", line 35, in main
    Allocator.run(InputPayload(**vars(parse_arguments())))
  File "/home/cbordon/Documents/wazuh/repositorios/wazuh-qa/deployability/modules/allocation/allocation.py", line 37, in run
    return cls.__create(payload)
  File "/home/cbordon/Documents/wazuh/repositorios/wazuh-qa/deployability/modules/allocation/allocation.py", line 56, in __create
    instance = provider.create_instance(
  File "/home/cbordon/Documents/wazuh/repositorios/wazuh-qa/deployability/modules/allocation/generic/provider.py", line 71, in create_instance
    return cls._create_instance(base_dir, params, config, ssh_key)
  File "/home/cbordon/Documents/wazuh/repositorios/wazuh-qa/deployability/modules/allocation/vagrant/provider.py", line 55, in _create_instance
    raise ValueError(f"The issue label was not provided or is of incorrect format, example: https://github.com/wazuh/<repository>/issues/<issue-number>")
ValueError: The issue label was not provided or is of incorrect format, example: https://github.com/wazuh/<repository>/issues/<issue-number>
cbordon@cbordon-MS-7C88:~/Documents/wazuh/repositorios/wazuh-qa$ python3 deployability/modules/allocation/main.py --provider vagrant --size micro --composite-name linux-ubuntu-22.04-amd64 --label-issue https://github.com/wazuh/wazuh-qa/pull/5311
[2024-05-06 14:18:57] [INFO] ALLOCATOR: Creating instance at /tmp/wazuh-qa
Traceback (most recent call last):
  File "/home/cbordon/Documents/wazuh/repositorios/wazuh-qa/deployability/modules/allocation/main.py", line 39, in <module>
    main()
  File "/home/cbordon/Documents/wazuh/repositorios/wazuh-qa/deployability/modules/allocation/main.py", line 35, in main
    Allocator.run(InputPayload(**vars(parse_arguments())))
  File "/home/cbordon/Documents/wazuh/repositorios/wazuh-qa/deployability/modules/allocation/allocation.py", line 37, in run
    return cls.__create(payload)
  File "/home/cbordon/Documents/wazuh/repositorios/wazuh-qa/deployability/modules/allocation/allocation.py", line 56, in __create
    instance = provider.create_instance(
  File "/home/cbordon/Documents/wazuh/repositorios/wazuh-qa/deployability/modules/allocation/generic/provider.py", line 71, in create_instance
    return cls._create_instance(base_dir, params, config, ssh_key)
  File "/home/cbordon/Documents/wazuh/repositorios/wazuh-qa/deployability/modules/allocation/vagrant/provider.py", line 55, in _create_instance
    raise ValueError(f"The issue label was not provided or is of incorrect format, example: https://github.com/wazuh/<repository>/issues/<issue-number>")
ValueError: The issue label was not provided or is of incorrect format, example: https://github.com/wazuh/<repository>/issues/<issue-number>

Test in AWS

cbordon@cbordon-MS-7C88:~/Documents/wazuh/repositorios/wazuh-qa$ python3 deployability/modules/allocation/main.py --provider aws --size micro --composite-name linux-ubuntu-22.04-amd64 --label-issue https://github.com/wazuh/wazuh-qa/issues/5311 --label-team devops --label-termination-date 1d
[2024-05-06 14:21:16] [INFO] ALLOCATOR: Creating instance at /tmp/wazuh-qa
[2024-05-06 14:21:16] [DEBUG] ALLOCATOR: No config provided. Generating from payload
[2024-05-06 14:21:16] [DEBUG] ALLOCATOR: Generating new key pair
[2024-05-06 14:21:16] [DEBUG] ALLOCATOR: Creating base directory: /tmp/wazuh-qa/AWS-EEB0F126-993C-4769-8082-68753C90CAA0
[2024-05-06 14:21:51] [DEBUG] ALLOCATOR: Renaming temp /tmp/wazuh-qa/AWS-EEB0F126-993C-4769-8082-68753C90CAA0 directory to /tmp/wazuh-qa/i-0234e7c80116e3c61
[2024-05-06 14:21:51] [INFO] ALLOCATOR: Instance i-0234e7c80116e3c61 created.
[2024-05-06 14:21:53] [INFO] ALLOCATOR: Instance i-0234e7c80116e3c61 started.
[2024-05-06 14:21:53] [INFO] ALLOCATOR: Inventory file generated at /tmp/wazuh-qa/i-0234e7c80116e3c61/inventory.yml
[2024-05-06 14:21:58] [WARNING] ALLOCATOR: Error on attempt 1 of 30: [Errno None] Unable to connect to port 2200 on 3.88.26.8
[2024-05-06 14:22:29] [INFO] ALLOCATOR: SSH connection successful.
[2024-05-06 14:22:29] [INFO] ALLOCATOR: Track file generated at /tmp/wazuh-qa/i-0234e7c80116e3c61/track.yml
cbordon@cbordon-MS-7C88:~/Documents/wazuh/repositorios/wazuh-qa$ python3 deployability/modules/allocation/main.py --action delete --track-output /tmp/wazuh-qa/i-0234e7c80116e3c61/track.yml 
[2024-05-06 14:24:34] [INFO] ALLOCATOR: Deleting instance from trackfile /tmp/wazuh-qa/i-0234e7c80116e3c61/track.yml
[2024-05-06 14:24:35] [DEBUG] ALLOCATOR: Deleting credentials: qa-5311-ubuntu-22.04-key-5195
[2024-05-06 14:25:22] [INFO] ALLOCATOR: Instance i-0234e7c80116e3c61 deleted.

Screenshot_20240506_142419

Testing macOS ARM

cbordon@cbordon-MS-7C88:~/Documents/wazuh/repositorios/wazuh-qa$ python3 deployability/modules/allocation/main.py --provider vagrant --size micro --composite-name macos-ventura-13.4.1-arm64 --instance-name cbordon-testing-macos
[2024-05-06 14:37:35] [INFO] ALLOCATOR: Creating instance at /tmp/wazuh-qa
[2024-05-06 14:37:40] [INFO] ALLOCATOR: macStadium server has less than 2 VMs running, deploying in this host.
[2024-05-06 14:37:40] [DEBUG] ALLOCATOR: Checking if instance directory exists on remote host
[2024-05-06 14:37:44] [DEBUG] ALLOCATOR: Creating instance directory on remote host
[2024-05-06 14:37:47] [DEBUG] ALLOCATOR: No config provided. Generating from payload
[2024-05-06 14:37:47] [DEBUG] ALLOCATOR: Generating new key pair
[2024-05-06 14:37:51] [DEBUG] ALLOCATOR: Vagrantfile created. Creating instance.
[2024-05-06 14:37:56] [INFO] ALLOCATOR: Instance cbordon-testing-macos-8137 created.
[2024-05-06 14:39:20] [INFO] ALLOCATOR: Instance cbordon-testing-macos-8137 started.
[2024-05-06 14:39:38] [INFO] ALLOCATOR: Inventory file generated at /tmp/wazuh-qa/cbordon-testing-macos-8137/inventory.yml
[2024-05-06 14:39:40] [INFO] ALLOCATOR: SSH connection successful.
[2024-05-06 14:39:51] [INFO] ALLOCATOR: Track file generated at /tmp/wazuh-qa/cbordon-testing-macos-8137/track.yml
cbordon@cbordon-MS-7C88:~/Documents/wazuh/repositorios/wazuh-qa$ python3 deployability/modules/allocation/main.py --action delete --track-output /tmp/wazuh-qa/cbordon-testing-macos-8137/track.yml 
[2024-05-06 14:41:13] [INFO] ALLOCATOR: Deleting instance from trackfile /tmp/wazuh-qa/cbordon-testing-macos-8137/track.yml
[2024-05-06 14:41:14] [DEBUG] ALLOCATOR: Destroying instance cbordon-testing-macos-8137
[2024-05-06 14:41:28] [DEBUG] ALLOCATOR: Deleting remote directory /Users/jenkins/testing/cbordon-testing-macos-8137
[2024-05-06 14:41:31] [DEBUG] ALLOCATOR: Killing remote process on port 43220
[2024-05-06 14:41:38] [INFO] ALLOCATOR: Instance cbordon-testing-macos-8137 deleted.

Remote host

sh-3.2# ls -la
total 0
drwxr-xr-x   8 jenkins  staff   256 May  6 10:37 .
drwxr-x---+ 34 jenkins  staff  1088 May  6 07:10 ..
drwxr-xr-x   4 root     staff   128 Jan 15 04:56 689
drwxr-xr-x   4 jenkins  staff   128 May  6 10:37 cbordon-testing-macos-8137
drwxr-xr-x   4 root     staff   128 Apr 19 07:07 macos-testing
drwxr-xr-x   4 jenkins  staff   128 Apr 26 10:57 sign-test
drwxr-xr-x   2 root     staff    64 Feb 21 05:17 testing_dir
drwxr-xr-x   4 root     staff   128 Aug 22  2023 wpk-testing
sh-3.2# prlctl list --all
UUID                                    STATUS       IP_ADDR         NAME
{e2b36366-bc32-4eba-8d56-213d0b1db078}  running      -               Test_e2e_system_283_20240506170351_macos_1400_7_0
{7c5230be-ef0d-42f6-a37e-5c5989c1af13}  running      -               cbordon-testing-macos-8137

Forcing instance dir concurrency

cbordon@cbordon-MS-7C88:~/Documents/wazuh/repositorios/wazuh-qa$ python3 deployability/modules/allocation/main.py --provider vagrant --size micro --composite-name linux-ubuntu-22.04-amd64 --label-issue https://github.com/wazuh/wazuh-qa/issues/5311
[2024-05-06 14:45:03] [INFO] ALLOCATOR: Creating instance at /tmp/wazuh-qa
[2024-05-06 14:45:03] [DEBUG] ALLOCATOR: No config provided. Generating from payload
[2024-05-06 14:45:03] [DEBUG] ALLOCATOR: Generating new key pair
[2024-05-06 14:45:07] [DEBUG] ALLOCATOR: Vagrantfile created. Creating instance.
[2024-05-06 14:45:07] [INFO] ALLOCATOR: Instance qa-5311-ubuntu-22.04-3347 created.
[2024-05-06 14:45:46] [INFO] ALLOCATOR: Instance qa-5311-ubuntu-22.04-3347 started.
[2024-05-06 14:45:49] [INFO] ALLOCATOR: Inventory file generated at /tmp/wazuh-qa/qa-5311-ubuntu-22.04-3347/inventory.yml
[2024-05-06 14:45:49] [INFO] ALLOCATOR: SSH connection successful.
[2024-05-06 14:45:51] [INFO] ALLOCATOR: Track file generated at /tmp/wazuh-qa/qa-5311-ubuntu-22.04-3347/track.yml
cbordon@cbordon-MS-7C88:~/Documents/wazuh/repositorios/wazuh-qa$ python3 deployability/modules/allocation/main.py --provider vagrant --size micro --composite-name linux-ubuntu-22.04-amd64 --label-issue https://github.com/wazuh/wazuh-qa/issues/5311
[2024-05-06 14:48:14] [INFO] ALLOCATOR: Creating instance at /tmp/wazuh-qa
[2024-05-06 14:48:14] [DEBUG] ALLOCATOR: Instance directory /tmp/wazuh-qa/qa-5311-ubuntu-22.04-3347 already exists. Assigning new suffix.
[2024-05-06 14:48:14] [DEBUG] ALLOCATOR: No config provided. Generating from payload
[2024-05-06 14:48:14] [DEBUG] ALLOCATOR: Generating new key pair
[2024-05-06 14:48:18] [DEBUG] ALLOCATOR: Vagrantfile created. Creating instance.
[2024-05-06 14:48:18] [INFO] ALLOCATOR: Instance qa-5311-ubuntu-22.04-3126 created.
[2024-05-06 14:48:59] [INFO] ALLOCATOR: Instance qa-5311-ubuntu-22.04-3126 started.
[2024-05-06 14:49:02] [INFO] ALLOCATOR: Inventory file generated at /tmp/wazuh-qa/qa-5311-ubuntu-22.04-3126/inventory.yml
[2024-05-06 14:49:02] [INFO] ALLOCATOR: SSH connection successful.
[2024-05-06 14:49:05] [INFO] ALLOCATOR: Track file generated at /tmp/wazuh-qa/qa-5311-ubuntu-22.04-3126/track.yml

In the case where an attempt was made to create 2 or more instances with the same characteristics, that is, the same issue and the same system and version, or the same --instance-name, the module will throw the following debug message and will retry creating the directory by appealing to that In the second attempt there was no concurrency, taking into account that the suffix we added is a random number between 0000 and 9999, so the probability of the same number being repeated is low.

[2024-05-06 14:48:14] [DEBUG] ALLOCATOR: Instance directory /tmp/wazuh-qa/qa-5311-ubuntu-22.04-3347 already exists. Assigning new suffix.

wazuh / wazuh-qa