QUADS automates the future scheduling, end-to-end provisioning and delivery of bare-metal servers and networks.
Python3, Flask, SqlAlchemy, PostrgreSQL, Jinja2
Step | Documentation | Details |
---|---|---|
General Architecture Overview | docs | Architecture overview |
Install and Setup Foreman/Satellite | docs | Not covered here |
Setup Foreman/Satellite Validation Templates | examples | Templates for internal interface configs |
Prepare Host and Network Environment | docs | Covers Juniper Environments, IPMI, Foreman |
Install QUADS | docs | Install via RPM |
Enable SSL | docs | Optionally enable SSL for API, Web |
Install Wiki | docs | Use Ansible to Deploy Wordpress |
Configure Wiki | docs | Configure Wordpress Pages and Plugins |
Configure your QUADS Move Command | docs | Configure your provisioning and move actions |
Configure QUADS Crons | docs | Tell QUADS how to manage your infrastructure |
Add Clouds and Hosts | docs | Configure your hosts and environments in QUADS |
Host Metadata Model and Search | docs | Host metadata info and filtering |
Using JIRA with QUADS | docs | Optional JIRA tools and library for QUADS |
You can read about QUADS architecture, provisioning, visuals and workflow in our documentation examples and screenshots
dnf copr enable quadsdev/python3-quads -y
dnf install quads -y
systemctl start quads-{db,server,web}
quads-db
needs to be initialized via the following command:flask --app quads.server.app init-db
quads --help
This step is optional but may be welcoming due to recent HSTS enforcement in most browsers.
To enable TLS/SSL on QUADS (API, Web) you'll need to generate your own certificates, if you're cool with self-signed cerificates you can use this one-liner:
servername=$(hostname)
mkdir -p /etc/pki/tls/certs ; cd /etc/pki/tls/certs
openssl req -x509 -newkey rsa:4096 -keyout $servername.key -out $servername.pem -sha256 -days 3650 -nodes -subj "/C=XX/ST=StateName/L=CityName/O=CompanyName/OU=CompanySectionName/CN=CommonNameOrHostname"
This generates your certificate and key here:
SSL Component | File System Path |
---|---|
Nginx Cert | /etc/pki/tls/certs/servername.pem |
Nginx Key | /etc/pki/tls/certs/servername.key |
Next you'll need to copy apiv3_ssl.conf.example
into place as apiv3_ssl.conf
cd /etc/nginx/conf.d
cp apiv3_ssl.conf.example apiv3_ssl.conf
Lastly, in-line edit the configuration file to point to your generated cert/key pair.
servername=$(hostname) ; sed -i -e "s/quads.example.com/$servername/" /etc/nginx/conf.d/apiv3_ssl.conf
Lastly, restart nginx:
sytemctl restart nginx
git clone https://github.com/redhat-performance/ops-tools
cd ansible/wiki-wordpress-nginx-mariadb
wiki.example.com
with your host in the hosts
file.
# cat hosts
[wordpress_server]
wiki.example.com
group_vars/all
for any configuration settings you want to change first.
ansible-playbook -i hosts site.yml
seboolean
usage you'll also need to install the ansible.posix
collection then re-run the playbook.
dnf install ansible-collection-ansible-posix
ansible-galaxy
without RPM.
ansible-galaxy collection install ansible.posix
infrastructure
page and assignments
page and denote their page id
for use in automation. This is set in conf/quads.yml
Classic Editor
plugin is also useful. All themes and plugins can be activated from settings.wp-config.php
file to limit the amount of page revisions that are kept in the database.
ABSPATH
in wp-config.php
add:define('WP_POST_REVISIONS', 100);
wp-cli
utility as well, QUADS regenerates all content as it changes so there is no need to keep around old revisions of pages unless you want to.yum install wp-cli -y
su - wordpress -s /bin/bash
wp post delete --force $(wp post list --post_type='revision' --format=ids)
Service Command | Category | Purpose |
---|---|---|
quads --move-hosts | provisioning | checks for hosts to move/reclaim as scheduled |
quads --validate-env | validation | checks clouds pending to be released for all enabled validation checks |
quads --regen-wiki | documentation | keeps your infra wiki updated based on current state of environment |
quads --regen-heatmap | visualization | keeps your systems availability and usage visualization up to date |
quads --regen-instack | openshift/openstack | keeps optional openstack triple-o installation files up-to-date |
quads --notify | notifications | check and send email or webhook/IRC notifications on events and releases |
Service Command | Category | Purpose |
---|---|---|
quads --foreman-rbac | RBAC | ensures environment user ownership maps to systems in Foreman |
quads --define-cloud --cloud cloud01 --description "Primary Cloud Environment"
quads --define-cloud --cloud cloud02 --description "02 Cloud Environment"
quads --define-cloud --cloud cloud03 --description "03 Cloud Environment"
--host-type
parameter, this is a mandatory, free-form label that can be anything. It will be used later for post-config
automation and categorization.--no-wipe
to the define command.for h in $(hammer host list --per-page 1000 | egrep -v "mgmt|c08-h30"| grep r630 | awk '{ print $3 }') ; do quads --define-host $h --default-cloud cloud01 --host-type general; done
quads --define-host --host <hostname> --default-cloud cloud01 --host-type general
default_pxe_interface
on the quads.yml will set the default value of pxe_boot=True
for that interface while any other interface will have a default value of False
unless specified via --pxe-boot
or --no-pxe-boot
. This can be later modified via --mod-interface
.quads --add-interface --interface-name em1 --interface-mac 52:54:00:d9:5d:df --interface-switch-ip 10.12.22.201 --interface-port xe-0/0/1:0 --interface-vendor "Intel" --interface-speed 1000 --host <hostname>
quads --add-interface --interface-name em2 --interface-mac 52:54:00:d9:5d:dg --interface-switch-ip 10.12.22.201 --interface-port xe-0/0/1:1 --interface-vendor "Intel" --interface-speed 1000 --pxe-boot --host <hostname>
quads --add-interface --interface-name em3 --interface-mac 52:54:00:d9:5d:dh --interface-switch-ip 10.12.22.201 --interface-port xe-0/0/1:2 --interface-vendor "Intel" --interface-speed 1000 --host <hostname>
quads --add-interface --interface-name em4 --interface-mac 52:54:00:d9:5d:d1 --interface-switch-ip 10.12.22.201 --interface-port xe-0/0/1:3 --interface-vendor "Intel" --interface-speed 1000 --host <hostname>
quads --ls-hosts
You will now see the list of full hosts.
c08-h21-r630.example.com
c08-h22-r630.example.com
c08-h23-r630.example.com
c08-h24-r630.example.com
c08-h25-r630.example.com
c08-h26-r630.example.com
c08-h27-r630.example.com
c08-h28-r630.example.com
c08-h29-r630.example.com
c09-h01-r630.example.com
c09-h02-r630.example.com
c09-h03-r630.example.com
quads --ls-interface --host c08-h21-r630.example.com
{"name": "em1", "mac_address": "52:54:00:d9:5d:df", "switch_ip": "10.12.22.201", "switch_port": "xe-0/0/1:0"}
{"name": "em2", "mac_address": "52:54:00:d9:5d:dg", "switch_ip": "10.12.22.201", "switch_port": "xe-0/0/1:1"}
{"name": "em3", "mac_address": "52:54:00:d9:5d:dh", "switch_ip": "10.12.22.201", "switch_port": "xe-0/0/1:2"}
{"name": "em4", "mac_address": "52:54:00:d9:5d:d1", "switch_ip": "10.12.22.201", "switch_port": "xe-0/0/1:3"}
quads --summary
cloud01 : 45 (Primary Cloud Environment)
cloud02 : 22 (02 Cloud Environment)
--detail
quads --summary --detail
cloud01 (quads): 45 (Primary Cloud Environment) - 451
cloud02 (jdoe): 22 (02 Cloud Environment) - 462
--all
quads --summary --detail --all
cloud01 (quads): 45 (Primary Cloud Environment) - 451
cloud02 (jdoe): 22 (02 Cloud Environment) - 462
cloud03 (jhoffa): 0 (03 Cloud Environment) - 367
NOTE:
The format here is based on the following:
{cloud_name} ({owner}): {count} ({description}) - {ticket_number}
c08-h21
to the workload/cloud cloud02
quads --add-schedule --host c08-h21-r630.example.com --schedule-start "2016-07-11 08:00" --schedule-end "2016-07-12 08:00" --schedule-cloud cloud02
quads --ls-schedule --host c08-h21-r630.example.com
You'll see the schedule output below
Default cloud: cloud01
Current cloud: cloud02
Defined schedules:
0:
start: 2016-07-11 08:00
end: 2016-07-12 08:00
cloud: cloud02
quads --move-hosts
You should see the following verbosity from a move operation
INFO: Moving c08-h21-r630.example.com from cloud01 to cloud02 c08-h21-r630.example.com cloud01 cloud02
In QUADS, a move-command
is the actionable call that provisions and moves a set of systems from one cloud environment to the other. Via cron, QUADS routinely queries the existing schedules and when it comes time for a set of systems to move to a new environment or be reclaimed and moved back to the spare pool it will run the appropriate varation of your move-command
.
In the above example the default move command called /bin/echo
for illustration purposes. In order for this to do something more meaningful you should invoke a script with the --move-command
option, which should be the path to a valid command or provisioning script/workflow.
hostname current-cloud new-cloud
.quads --move-hosts --move-command quads/tools/move_and_rebuild_hosts.py
You can modify the default settings via the default_move_command
setting in quads.
You can look at the move-and-rebuild-hosts script as an example. It's useful to note that with quads/tools/move_and_rebuild.py
passing a fourth argument will result in only the network automation running and the actual host provisioning will be skipped. You should review this script and adapt it to your needs, we try to make variables for everything but some assumptions are made to fit our running environments.
As of QUADS 1.1.6
we now have the --report-detailed
command which will list all upcoming future assignments that are scheduled.
You can also specify custom start and end dates via --schedule-start "YYYY-MM-DD HH:MM"
and --schedule-stop "YYYY-MM-DD HH:MM"
quads --report-detailed
Example Output
Owner | Ticket| Cloud| Description| Systems| Scheduled| Duration|
tcruise | 1034| cloud20| Openshift| 6| 2022-02-06| 14|
cwalken | 1031| cloud19| Openstack| 6| 2022-02-06| 14|
bhicks | 1029| cloud18| Openstack-B| 4| 2022-02-06| 14|
nreeves | 1028| cloud11| Openshift-P| 2| 2022-02-06| 14|
gcarlin | 1026| cloud08| Ceph| 4| 2022-02-06| 14|
Generate a report with a list of server types with total count of systems and their current and future availability plus an average build time delta overall
quads --report-available
Example output
Quads report for 2019-12-01 to 2019-12-31:
Percentage Utilized: 60%
Average build delta: 0:00:26.703556
Server Type | Total| Free| Scheduled| 2 weeks| 4 weeks
r620 | 5| 0| 100%| 0| 0
1029p | 3| 3| 0%| 3| 3
Additionally, you can pass --schedule-start
and --schedule-end
dates for reports in the past. 2 weeks and 4 weeks free calculate starting days from the first Sunday following when the command was run, or return current day at 22:01 if run on Sunday.
Generate a report detailing systems and scheduling utilization over the course of months or years.
quads --report-scheduled --months 6
Example Output
Month | Scheduled| Systems| % Utilized|
2022-02 | 1| 1268| 42%|
2022-01 | 9| 1268| 66%|
2022-02 | 1| 1268| 42%|
2021-09 | 10| 1226| 83%|
2021-08 | 14| 1215| 77%|
2021-07 | 3| 1215| 87%|
Generate statistics on the number of assigned clouds in quads over a period of months in the past starting today or on a specific year.
quads --report-scheduled --months 6
Example output
Month | Scheduled| Systems| % Utilized|
2019-12 | 0| 8| 58%|
2019-11 | 2| 8| 62%|
2019-10 | 15| 8| 20%|
2019-09 | 0| 0| 0%|
2019-08 | 0| 0| 0%|
Additionally, you can pass --year
instead for a report for every month in that year.
Creating a new schedule and assigning machines is currently done through the QUADS CLI. There are a few options you'll want to utilize. Mandatory options are in bold and optional are in italics.
This pertains to the internal interfaces that QUADS will manage for you to move sets of hosts between environments based on a schedule. For setting up optional publicly routable VLANS please see the QUADS public vlan setup steps
VLAN design (optional, will default to qinq: 0
below)
qinq: 0
(default) qinq VLAN separation by interface: primary, secondary and beyond QUADS-managed interfaces all match the same VLAN membership across other hosts in the same cloud allocation. Each interface per host is in its own VLAN, and these match across the rest of your allocated hosts by interface (all nic1, all nic2, all nic3, all nic4 etc).
qinq: 1
all QUADS-managed interfaces in the same qinq VLAN. For this to take effect you need to pass the optional argument of --qinq 1
to the --define-cloud
command.
You can use the command quads --ls-qinq
to view your current assignment VLAN configuration:
quads --ls-qinq
cloud01: 0 (Isolated)
cloud02: 1 (Combined)
cloud03: 0 (Isolated)
cloud04: 1 (Combined)
If you need to associate a public vlan (routable) with your cloud, quads currently supports associating your last NIC per host with one of your defined public VLANs (see the QUADS public vlan setup steps).
To define your cloud with a public VLAN, use the following syntax:
quads --define-cloud --cloud cloud03 [ other define-cloud options ] --vlan 601
If you need to clear the vlan association with your cloud, you can pass any string to the --vlan
argument in --mod-cloud
quads --mod-cloud --cloud cloud03 --vlan none
quads --define-cloud --cloud cloud03 --description "Messaging AMQ" --cloud-owner epresley --cc-users "jdoe jhoffa" --cloud-ticket 423625 --qinq 1
Note: in QUADS 1.1.4
you can change any of these values selectively via the --mod-cloud
command described below.
Note: in QUADS 2.0
the --force
command is no longer needed for defining inactive environments future use.
Now that you've defined your new cloud you'll want to allocate machines and a schedule.
quads --cloud-only --cloud cloud01 | grep r620 | head -20 > /tmp/RT423624
quads --host-list /tmp/RT423624 --add-schedule --schedule-start "2016-10-17 00:00" --schedule-end "2016-11-14 17:00" --schedule-cloud cloud03
--host-list
along with a list of hosts and it will take care of updating your --cloud-ticket
in JIRA for you in one swoop.quads --add-schedule --host-list /tmp/hosts --schedule-start "2021-04-20 22:00" --schedule-end "2021-05-02 22:00" --schedule-cloud cloud20
That's it. At this point your hosts will be queued for provision and move operations, we check once a minute if there are any pending provisioning tasks. To check manually:
quads --move-hosts --dry-run
After your hosts are provisioned and moved you should see them populate under the cloud list.
quads --cloud-only --cloud cloud03
Starting with 1.1.4
QUADS can manage broken or faulty hosts for you and ensure they are ommitted from being added to a future schedule or listed as available. Prior to 1.1.4
this is managed via the Foreman host parameter broken_state
(true/false).
Listing all broken systems.
# quads --ls-broken
f18-h22-000-r620.stage.example.com
Marking a system as faulty
# quads --mark-broken --host f18-h23-000-r620.example.com
Host f18-h23-000-r620.example.com is now marked as broken
Marking a system as repaired or no longer faulty.
# quads --mark-repaired --host f18-h23-000-r620.example.com
Host f18-h23-000-r620.example.com is now marked as repaired.
Hosts marked as faulty will be ommitted from --ls-available
Hosts marked as faulty are not able to be scheduled until they are marked as repaired again.
broken_state
Foreman host parameter to manage your broken or out-of-service systems within your fleet you'll want to migrate to using the new methodology of the QUADS database handling this for you for versions 1.1.4
and higher.broken_state
host parameters and status into QUADS:for h in $(hammer host list --per-page 1000 --search params.broken_state=true | grep $(egrep ^domain /opt/quads/conf/quads.yml | awk '{ print $NF }') | awk '{ print $3 }') ; do quads --mark-broken --host $h ; done
1.1.5
and higher we now have the --retire
, --unretire
and --ls-retire
features to manage decomissioning or reviving hosts.quads --ls-retire
quads --retire --host host01.example.com
quads --unretire --host host01.example.com
cloud01
.for host in $(cat /tmp/retired_hosts); do yes | quads --shrink --host $host --now; done
--move-host
command will want to move these back to your resource pool and power them off.retired
hosts will remain officially in your resource pool but not show up in any visualizations or usage reporting, however their past usage history will all be available for record keeping and data requirements.Occasionally you'll want to extend the lifetime of a particular assignment. QUADS lets you do this with one command but you'll want to double-check things first. In this example we'll be extending the assignment end date for cloud02
In QUADS version 1.1.4
or higher or the current master
branch you can extend a cloud environment with a simple command.
quads --extend --cloud cloud02 --weeks 2 --check
This will check whether or not the environment can be extended without conflicts.
To go ahead and extend it remove the --check
quads --extend --cloud cloud02 --weeks 2
You might also want to extend the lifetime of a specific host. In this example we'll be extending the assignment end date for host01.
quads --extend --host host01 --weeks 2 --check
This will check whether or not the environment can be extended without conflicts.
To go ahead and extend it remove the --check
quads --extend --host host01 --weeks 2
Occasionally you'll want to shrink the lifetime of a particular assignment. In this example we'll be shrinking the assignment end date for cloud02
quads --shrink --cloud cloud02 --weeks 2 --check
This will check whether or not the environment can be shrunk without conflicts.
To go ahead and shrink it remove the --check
quads --shrink --cloud cloud02 --weeks 2
You might also want to shrink the lifetime of a specific host. In this example we'll be shrinking the assignment end date for host01.
quads --shrink --host host01 --weeks 2 --check
This will check whether or not the host schedule can be shrunk without conflicts.
To go ahead and shrink it remove the --check
quads --shrink --host host01 --weeks 2
If you would like to terminate the lifetime of a schedule at either a host or cloud level, you can pass the --now
argument instead of --weeks
which will set the schedules end date to now.
In this example we'll be terminating the assignment end date for cloud02.
quads --shrink --cloud cloud02 --now --check
This will check whether or not the environment can be terminated without conflicts.
To go ahead and terminate it remove the --check
quads --shrink --cloud cloud02 --now
QUADS also supports adding new machines into an existing workload (cloud).
2019-03-11 22:00
until 2019-04-22 22:00
quads --ls-available --schedule-start "2016-12-05 08:00" --schedule-end "2016-12-15 08:00"
c03-h11-r620.rdu.openstack.example.com
c03-h13-r620.rdu.openstack.example.com
c03-h14-r620.rdu.openstack.example.com
c03-h15-r620.rdu.openstack.example.com
Above we see all the free servers during our timeframe, let's move them into cloud10
quads --host c03-h11-r620.rdu.openstack.example.com --add-schedule --schedule-start "2016-12-05 08:00" --schedule-end "2016-12-15 08:00" --schedule-cloud cloud10
quads --host c03-h13-r620.rdu.openstack.example.com --add-schedule --schedule-start "2016-12-05 08:00" --schedule-end "2016-12-15 08:00" --schedule-cloud cloud10
quads --host c03-h14-r620.rdu.openstack.example.com --add-schedule --schedule-start "2016-12-05 08:00" --schedule-end "2016-12-15 08:00" --schedule-cloud cloud10
quads --host c03-h15-r620.rdu.openstack.example.com --add-schedule --schedule-start "2016-12-05 08:00" --schedule-end "2016-12-15 08:00" --schedule-cloud cloud10
You can remove an existing schedule across a set of hosts using the --rm-schedule
flag against the schedule ID for each particular machine of that assignment.
Obtain the schedule ID via quads --ls-schedule --host
quads --rm-schedule --schedule-id 2
quads --rm-schedule --schedule-id 3
quads --rm-schedule --schedule-id 4
--host
. This is not required on QUADS 2.0 onwards as the schedule Ids are now unique.You should search for either the start or end dates to select the right schedule ID to remove when performing schedule removals across a large set of hosts.
for host in $(cat /tmp/452851); do quads --rm-schedule --schedule-id $(quads --ls-schedule --host $host | grep cloud08 | grep "start=2017-08-06" | tail -1 | awk -F\| '{ print $1 }'); echo Done. ; done
To remove a host entirely from QUADS management you can use the --rm-host
command.
quads --rm-host f03-h30-000-r720xd.rdu2.example.com
Removed: {'host': 'f03-h30-000-r720xd.rdu2.example.com'}
--ls-schedule
command.--mod-schedule --schedule-id
command and either --schedule-start
or --schedule-end
or both as needed.--shrink
or --extend
and sub-commands across the entire cloud/environments or even per-host first.quads --mod-schedule --host host01.example.com --mod-schedule --schedule-id 31 --schedule-start "2023-05-22 22 :00" --schedule-end "2023-06-22 22:00"
cloud06
and start=2023-03-13
to make sure our --mod-schedule
command is unique.echo
to quads
to ensure that the schedule-id
are matched.for host in $(cat /tmp/2491); do quads --mod-schedule --schedule-id $(quads --ls-schedule --host $host | grep cloud06 | grep "start=2023-03-13" | tail -1 | awk -F\| '{ print $1 }') --host $host --schedule-start "2023-03-12 22:00" ; done
To remove a host entirely from QUADS management you can use the --rm-host
command.
quads --mod-interface --interface-name em1 --host f03-h30-000-r720xd.rdu2.example.com --no-pxe-boot
Interface successfully updated
To remove a host entirely from QUADS management you can use the --rm-host
command.
quads --rm-interface --interface-name em1 --host f03-h30-000-r720xd.rdu2.example.com
Resource properly removed
TCP/8080
/opt/quads/quads/verify_switchconf.py
can be used to both validate and correct network switch configs.--change
to see if it would fix something./opt/quads/quads/tools/verify_switchconf.py --cloud cloud10
--change
/opt/quads/quads/tools/verify_switchconf.py --cloud cloud10 --change
To validate a singular hosts network switch configuration:
/opt/quads/quads/tools/verify_switchconf.py --host host01.example.com
To validate and fix a single hosts network config use --change
/opt/quads/quads/tools/verify_switchconf.py --host host01.example.com --change
/opt/quads/quads/tools/verify_switchconf.py --host host01.example.com --cloud cloud10
Note, if host01.example.com is not in cloud10, but rather cloud20, you will see the following output:
WARNING - Both --cloud and --host have been specified.
WARNING -
WARNING - Host: host01.example.com
WARNING - Cloud: cloud10
WARNING -
WARNING - However, host01.example.com is a member of cloud20
WARNING -
WARNING - !!!!! Be certain this is what you want to do. !!!!!
WARNING -
modify_switch_conf.py
tool you can set up each individual network interface to a specific vlan id.--change
argument will make the changes effective in the switch. Not passing this will only verify the configuration is set to the desired./opt/quads/quads/tools/modify_switch_conf.py --host host01.example.com --nic1 1400 --nic2 1401 --nic3 1400 --nic4 1402 --nic5 1400
--nic*
arguments are optional so this can be also done individually for all nics.em
interface in the QUADS --ls-interfaces
information we now include the following tool:
./opt/quads/quads/tools/ls_switch_conf.py --cloud cloud32
INFO - Cloud qinq: 1
INFO - Interface em1 appears to be a member of VLAN 1410
INFO - Interface em2 appears to be a member of VLAN 1410
This tool also accepts the --all
argument which will list a detail for all hosts in the cloud.
Additional you can achieve the same with the following shell one-liner, setting cloud=XX
for the cloud and adjusting $(seq 1 4)
for your interface ranges available on the host.
cloud=32 ; origin=1100 ; offset=$(expr $(expr $cloud - 1) \* 10); vl=$(expr $origin + $offset) ;for i in $(seq 1 4) ; do vlan=$(expr $vl + $i - 1) ; echo "em$i is interface VLAN $vlan in cloud$cloud" ; done
em1 is interface VLAN 1400 in cloud32
em2 is interface VLAN 1401 in cloud32
em3 is interface VLAN 1402 in cloud32
em4 is interface VLAN 1403 in cloud32
Q-in-Q 0 (isolated)
VLAN configuration. The Q-in-Q 1 (combined)
configuration would simple be VLAN1400
for all interfaces above respectively.1.1.4
with the --mod-cloud
command.quads --mod-cloud --cloud cloud02 --cloud-owner jhoffa
quads --mod-cloud --cloud cloud04 --cc-users "tpetty fmercury"
quads --mod-cloud --cloud cloud06 --vlan 604 --wipe
quads --mod-cloud --cloud cloud50 --no-wipe
quads --mod-cloud --cloud cloud50 --vlan none
Because QUADS knows about all future schedules you can display what your environment will look like at any point in time using the --date
command.
Looking into a specific environment by date
quads --cloud-only --cloud cloud08 --date "2019-06-04 22:00"
f16-h01-000-1029u.rdu2.example.com
f16-h02-000-1029u.rdu2.example.com
f16-h03-000-1029u.rdu2.example.com
f16-h05-000-1029u.rdu2.example.com
f16-h06-000-1029u.rdu2.example.com
quads --ls-schedule --date "2020-06-04 22:00"
--dry-run
sub-flag of --move-hosts
quads --move-hosts --dry-run
INFO: Moving b10-h27-r620.rdu.openstack.example.com from cloud01 to cloud03
INFO: Moving c02-h18-r620.rdu.openstack.example.com from cloud01 to cloud03
INFO: Moving c02-h19-r620.rdu.openstack.example.com from cloud01 to cloud03
INFO: Moving c02-h21-r620.rdu.openstack.example.com from cloud01 to cloud03
INFO: Moving c02-h25-r620.rdu.openstack.example.com from cloud01 to cloud03
INFO: Moving c02-h26-r620.rdu.openstack.example.com from cloud01 to cloud03
quads --find-free-cloud
to suggest a cloud environment to use that does not have any future hosts scheduled to use it.quads --find-free-cloud
cloud12
cloud16
cloud17
cloud18
The --ls-available
functionality lets you search for available hosts in the future based on a date range or other criteria.
quads --ls-available --schedule-start "2019-12-05 08:00" --schedule-end "2019-12-15 08:00"
quads --ls-available --schedule-end "2019-06-02 22:00"
1.1.4
and higher you can now filter your availability search based on hardware capabilities or model type.--filter "model==1029U-TRTP"
quads --ls-available --schedule-start "2020-08-02 22:00" --schedule-end "2020-08-16 22:00" --filter "model==1029U-TRTP"
--filter
feature:quads --ls-hosts --filter "retired==True"
quads --cloud-only --cloud cloud13 --filter "model==FC640"
--ls-available
web interface available on quadshost
.quads-web
systemd service.models
data for your systems using the new host metadata feature--filter
command in 1.1.4
and above along with --ls-hosts
to search for a system by MAC Address.quads --ls-hosts --filter "interfaces.mac_address==ac:1f:6b:2d:19:48"
ip_address
(soon to be switch_ip
in 1.1.7) information from the interfaces datasource.quads --ls-hosts --filter "interfaces.ip_address==10.1.34.210"
We do provide some best-effort tooling and a JIRA library to bridge automation gaps.
We do not implement backups for QUADS for you, but it's really easy to do on your own via Postgres pg_dumpall
su - postgres -c "pg_dumpall --clean > /tmp/quadsdb.sql"
Configuration files are all kept in /opt/quads/conf
/opt/quads/conf
files.su - postgres -c "psql -d quads -f /tmp/quadsdb.sql"
su - postgres -c "psql -f /tmp/quadsdb.sql"
A useful part of QUADS is the functionality for automated systems/network validation. Below you'll find some steps to help understand why systems/networks might not pass validation so you can address any issues.
There are two main validation tests that occur before a cloud environment is automatically released:
All of these validations are run from --validate-env
and we also ship a few useful tools to help you figure out validation failures.
--validate-env
is run from cron, see our example cron entry
You should run through each of these steps manually to determine what systems/networks might need attention of automated validation does not pass in a reasonable timeframe. Typically, admin_cc:
will receieve email notifications of trouble hosts as well.
fping
command, this should be run first.quads --cloud-only --cloud cloud23 > /tmp/cloud23
fping -u -f /tmp/cloud23
gem install hammer_cli_foreman_admin hammer_cli
for host in $(quads --cloud-only --cloud cloud15) ; do echo $host $(hammer host info --name $host | grep -i build); done
No systems should be left marked for build.
NOTE Automated validation will not start until 2 hours after the assignment is scheduled to go out, until this point --validate-env
will not attempt to validate any systems if run and they have started less than 2 hours ago.
validation_grace_period:
setting in /opt/quads/conf/quads.yml
--validate-env
now has a --debug
option which tells you what's happening during validation.
This will test the backend network connectivity part and the entire set of checks.
Successful Validation looks like this:
quads --validate-env --debug
Validating cloud23
Using selector: EpollSelector
:Initializing Foreman object:
GET: /status
GET: /hosts?search=build=true
Command executed successfully: fping -u f12-h01-000-1029u.rdu2.scalelab.example.com f12-h02-000-1029u.rdu2.scalelab.example.com f12-h03-000-1029u.rdu2.scalelab.example.com
Command executed successfully: fping -u 172.16.38.126 172.20.38.126 172.16.36.206
Command executed successfully: fping -u 172.17.38.126 172.21.38.126 172.17.36.206
Command executed successfully: fping -u 172.18.38.126 172.22.38.126 172.18.36.206
Command executed successfully: fping -u 172.19.38.126 172.23.38.126 172.19.36.206
Subject: Validation check succeeded for cloud23
From: RDU2 Scale Lab <quads@example.com>
To: dev-null@example.com
Cc: wfoster@example.com, kambiz@example.com, jtaleric@example.com,
abond@example.com, grafuls@example.com, natashba@example.com
Reply-To: dev-null@example.com
User-Agent: Rufus Postman 1.0.99
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: 7bit
MIME-Version: 1.0
A post allocation check previously failed for:
cloud: cloud23
owner: ipinto
ticket: 498569
has successfully passed the verification test(s)! The owner
should receive a notification that the environment is ready
for use.
DevOps Team
cloud23 / ipinto / 498569
quads --validate-env --debug
Validating cloud23
Using selector: EpollSelector
:Initializing Foreman object:
GET: /status
GET: /hosts?search=build=true
There was something wrong with your request
ICMP Host Unreachable from 10.1.38.126 for ICMP Echo sent to f12-h14-000-1029u.rdu2.scalelab.example.com (10.1.38.43)
ICMP Host Unreachable from 10.1.38.126 for ICMP Echo sent to f12-h14-000-1029u.rdu2.scalelab.example.com (10.1.38.43)
ICMP Host Unreachable from 10.1.38.126 for ICMP Echo sent to f12-h14-000-1029u.rdu2.scalelab.example.com (10.1.38.43)
ICMP Host Unreachable from 10.1.38.126 for ICMP Echo sent to f12-h14-000-1029u.rdu2.scalelab.example.com (10.1.38.43)
QUADS 1.1.6+
you can skip past network validation via:quads --validate-env --skip-network
QUADS 1.1.8
you can skip past systems and host validation (Foreman) via:/opt/quads/quads/tools/validate_env.py --skip-system
QUADS 1.1.8
you can skip past both systems and network checks per host via:/opt/quads/quads/tools/validate_env.py --skip-hosts host01.example.com host02.example.com
--skip-hosts
will pass it completely through validation.--skip-system
and --skip-network
as well.--cloud
flag.
quads --validate-env --cloud cloud01
You might have noticed that we configure our Foreman templates to drop 172.{16,17,18,19}.x
internal VLAN interfaces which correspond to the internal, QUADS-managed multi-tenant interfaces across a set of hosts in a cloud assignment.
The first two octets here can be substituted by the first two octets of your systems public network in order to determine from validate_env.py --debug
which host internal interfaces have issues or are unreachable.
host
command to determine what these machines map to by substituting 10.1
for the first two octects:# for host in 10.1.37.231 10.1.38.150; do host $host; done
231.37.1.10.in-addr.arpa domain name pointer e17-h26-b04-fc640.example.com.
150.38.1.10.in-addr.arpa domain name pointer e17-h26-b03-fc640.example.com.
This mapping feeds into our VLAN network validation code
Besides Github we're also on IRC via irc.libera.chat
. You can click here to join in your browser.