search

redhat-performance / quads

:calendar: The infrastructure deployment time machine
https://quads.dev
GNU General Public License v3.0
86 stars 36 forks source link
assignment automation baremetal cloud network provisioning quads scheduler systems

readme

QUADS (quick and dirty scheduler)

quads

QUADS automates the future scheduling, end-to-end provisioning and delivery of bare-metal servers and networks.

quads-rpm-build

What does it do?

Design

quadsarchitecture

Requirements

Setup Overview

Step Documentation Details
General Architecture Overview docs Architecture overview
Install and Setup Foreman/Satellite docs Not covered here
Setup Foreman/Satellite Validation Templates examples Templates for internal interface configs
Prepare Host and Network Environment docs Covers Juniper Environments, IPMI, Foreman
Install QUADS docs Install via RPM
Enable SSL docs Optionally enable SSL for API, Web
Install Wiki docs Use Ansible to Deploy Wordpress
Configure Wiki docs Configure Wordpress Pages and Plugins
Configure your QUADS Move Command docs Configure your provisioning and move actions
Configure QUADS Crons docs Tell QUADS how to manage your infrastructure
Add Clouds and Hosts docs Configure your hosts and environments in QUADS
Host Metadata Model and Search docs Host metadata info and filtering
Using JIRA with QUADS docs Optional JIRA tools and library for QUADS

QUADS Workflow

You can read about QUADS architecture, provisioning, visuals and workflow in our documentation examples and screenshots

QUADS Switch and Host Setup

Installing QUADS

Installing QUADS from RPM

dnf copr enable quadsdev/python3-quads  -y
dnf install quads -y
systemctl start quads-{db,server,web}
flask --app quads.server.app init-db

Using SSL with Flask API and QUADS

This step is optional but may be welcoming due to recent HSTS enforcement in most browsers.

To enable TLS/SSL on QUADS (API, Web) you'll need to generate your own certificates, if you're cool with self-signed cerificates you can use this one-liner:

servername=$(hostname)
mkdir -p /etc/pki/tls/certs ; cd /etc/pki/tls/certs
openssl req -x509 -newkey rsa:4096 -keyout $servername.key -out $servername.pem -sha256 -days 3650 -nodes -subj "/C=XX/ST=StateName/L=CityName/O=CompanyName/OU=CompanySectionName/CN=CommonNameOrHostname"

This generates your certificate and key here:

SSL Component File System Path
Nginx Cert /etc/pki/tls/certs/servername.pem
Nginx Key /etc/pki/tls/certs/servername.key

Next you'll need to copy apiv3_ssl.conf.example into place as apiv3_ssl.conf

cd /etc/nginx/conf.d
cp apiv3_ssl.conf.example apiv3_ssl.conf

Lastly, in-line edit the configuration file to point to your generated cert/key pair.

servername=$(hostname) ; sed -i -e "s/quads.example.com/$servername/" /etc/nginx/conf.d/apiv3_ssl.conf

Lastly, restart nginx:

sytemctl restart nginx

Installing other QUADS Components

QUADS Wiki

Installing Wordpress via Ansible
Limit Page Revisions in Wordpress
define('WP_POST_REVISIONS', 100);
yum install wp-cli -y
su - wordpress -s /bin/bash
wp post delete --force $(wp post list --post_type='revision' --format=ids)

QUADS Move Command

Making QUADS Run

Major Components

Service Command Category Purpose
quads --move-hosts provisioning checks for hosts to move/reclaim as scheduled
quads --validate-env validation checks clouds pending to be released for all enabled validation checks
quads --regen-wiki documentation keeps your infra wiki updated based on current state of environment
quads --regen-heatmap visualization keeps your systems availability and usage visualization up to date
quads --regen-instack openshift/openstack keeps optional openstack triple-o installation files up-to-date
quads --notify notifications check and send email or webhook/IRC notifications on events and releases

External Services

Service Command Category Purpose
quads --foreman-rbac RBAC ensures environment user ownership maps to systems in Foreman

QUADS Usage Documentation

Adding New Hosts to QUADS

Define Initial Cloud Environments

quads --define-cloud --cloud cloud01 --description "Primary Cloud Environment"
quads --define-cloud --cloud cloud02 --description "02 Cloud Environment"
quads --define-cloud --cloud cloud03 --description "03 Cloud Environment"

Define Host in QUADS and Foreman

for h in $(hammer host list --per-page 1000 | egrep -v "mgmt|c08-h30"| grep r630 | awk '{ print $3 }') ; do quads --define-host $h --default-cloud cloud01 --host-type general; done
quads --define-host --host <hostname> --default-cloud cloud01 --host-type general

Define Host Interfaces in QUADS

quads --add-interface --interface-name em1 --interface-mac 52:54:00:d9:5d:df --interface-switch-ip 10.12.22.201 --interface-port xe-0/0/1:0 --interface-vendor "Intel" --interface-speed 1000 --host <hostname>
quads --add-interface --interface-name em2 --interface-mac 52:54:00:d9:5d:dg --interface-switch-ip 10.12.22.201 --interface-port xe-0/0/1:1 --interface-vendor "Intel" --interface-speed 1000 --pxe-boot --host <hostname>
quads --add-interface --interface-name em3 --interface-mac 52:54:00:d9:5d:dh --interface-switch-ip 10.12.22.201 --interface-port xe-0/0/1:2 --interface-vendor "Intel" --interface-speed 1000 --host <hostname>
quads --add-interface --interface-name em4 --interface-mac 52:54:00:d9:5d:d1 --interface-switch-ip 10.12.22.201 --interface-port xe-0/0/1:3 --interface-vendor "Intel" --interface-speed 1000 --host <hostname>
quads --ls-hosts

You will now see the list of full hosts.

c08-h21-r630.example.com
c08-h22-r630.example.com
c08-h23-r630.example.com
c08-h24-r630.example.com
c08-h25-r630.example.com
c08-h26-r630.example.com
c08-h27-r630.example.com
c08-h28-r630.example.com
c08-h29-r630.example.com
c09-h01-r630.example.com
c09-h02-r630.example.com
c09-h03-r630.example.com
quads --ls-interface --host c08-h21-r630.example.com

{"name": "em1", "mac_address": "52:54:00:d9:5d:df", "switch_ip": "10.12.22.201", "switch_port": "xe-0/0/1:0"}
{"name": "em2", "mac_address": "52:54:00:d9:5d:dg", "switch_ip": "10.12.22.201", "switch_port": "xe-0/0/1:1"}
{"name": "em3", "mac_address": "52:54:00:d9:5d:dh", "switch_ip": "10.12.22.201", "switch_port": "xe-0/0/1:2"}
{"name": "em4", "mac_address": "52:54:00:d9:5d:d1", "switch_ip": "10.12.22.201", "switch_port": "xe-0/0/1:3"}
quads --summary
cloud01 : 45 (Primary Cloud Environment)
cloud02 : 22 (02 Cloud Environment)
quads --summary --detail
cloud01 (quads): 45 (Primary Cloud Environment) - 451
cloud02 (jdoe): 22 (02 Cloud Environment) - 462
quads --summary --detail --all
cloud01 (quads): 45 (Primary Cloud Environment) - 451
cloud02 (jdoe): 22 (02 Cloud Environment) - 462
cloud03 (jhoffa): 0 (03 Cloud Environment) - 367

NOTE:

The format here is based on the following: {cloud_name} ({owner}): {count} ({description}) - {ticket_number}

quads --add-schedule --host c08-h21-r630.example.com --schedule-start "2016-07-11 08:00" --schedule-end "2016-07-12 08:00" --schedule-cloud cloud02
quads --ls-schedule --host c08-h21-r630.example.com

You'll see the schedule output below

Default cloud: cloud01
Current cloud: cloud02
Defined schedules:
  0:
    start: 2016-07-11 08:00
    end: 2016-07-12 08:00
    cloud: cloud02
quads --move-hosts

You should see the following verbosity from a move operation

INFO: Moving c08-h21-r630.example.com from cloud01 to cloud02 c08-h21-r630.example.com cloud01 cloud02

How Provisioning Works

QUADS move host command

In QUADS, a move-command is the actionable call that provisions and moves a set of systems from one cloud environment to the other. Via cron, QUADS routinely queries the existing schedules and when it comes time for a set of systems to move to a new environment or be reclaimed and moved back to the spare pool it will run the appropriate varation of your move-command.

In the above example the default move command called /bin/echo for illustration purposes. In order for this to do something more meaningful you should invoke a script with the --move-command option, which should be the path to a valid command or provisioning script/workflow.

quads --move-hosts --move-command quads/tools/move_and_rebuild_hosts.py

QUADS Reporting

Future Assignment Reporting

As of QUADS 1.1.6 we now have the --report-detailed command which will list all upcoming future assignments that are scheduled. You can also specify custom start and end dates via --schedule-start "YYYY-MM-DD HH:MM" and --schedule-stop "YYYY-MM-DD HH:MM"

quads --report-detailed

Example Output

Owner    |    Ticket|    Cloud| Description| Systems|  Scheduled| Duration|
tcruise  |      1034|  cloud20|   Openshift|       6| 2022-02-06|       14|
cwalken  |      1031|  cloud19|   Openstack|       6| 2022-02-06|       14|
bhicks   |      1029|  cloud18| Openstack-B|       4| 2022-02-06|       14|
nreeves  |      1028|  cloud11| Openshift-P|       2| 2022-02-06|       14|
gcarlin  |      1026|  cloud08|        Ceph|       4| 2022-02-06|       14|

Server Availability Overview Report

Generate a report with a list of server types with total count of systems and their current and future availability plus an average build time delta overall

quads --report-available

Example output

Quads report for 2019-12-01 to 2019-12-31:
Percentage Utilized: 60%
Average build delta: 0:00:26.703556
Server Type | Total|  Free| Scheduled| 2 weeks| 4 weeks
r620        |     5|     0|      100%|       0|       0
1029p       |     3|     3|        0%|       3|       3

Additionally, you can pass --schedule-start and --schedule-end dates for reports in the past. 2 weeks and 4 weeks free calculate starting days from the first Sunday following when the command was run, or return current day at 22:01 if run on Sunday.

Assignment Scheduling Statistics

Generate a report detailing systems and scheduling utilization over the course of months or years.

quads --report-scheduled --months 6

Example Output

Month   | Scheduled|  Systems|  % Utilized|
2022-02 |         1|     1268|         42%|
2022-01 |         9|     1268|         66%|
2022-02 |         1|     1268|         42%|
2021-09 |        10|     1226|         83%|
2021-08 |        14|     1215|         77%|
2021-07 |         3|     1215|         87%|

Upcoming Scheduled Assignments Report

Generate statistics on the number of assigned clouds in quads over a period of months in the past starting today or on a specific year.

quads --report-scheduled --months 6

Example output

Month   | Scheduled|  Systems|  % Utilized|
2019-12 |         0|        8|         58%|
2019-11 |         2|        8|         62%|
2019-10 |        15|        8|         20%|
2019-09 |         0|        0|          0%|
2019-08 |         0|        0|          0%|

Additionally, you can pass --year instead for a report for every month in that year.

Common Administration Tasks

Creating a New Cloud Assignment and Schedule

Creating a new schedule and assigning machines is currently done through the QUADS CLI. There are a few options you'll want to utilize. Mandatory options are in bold and optional are in italics.

QUADS VLAN Options

This pertains to the internal interfaces that QUADS will manage for you to move sets of hosts between environments based on a schedule. For setting up optional publicly routable VLANS please see the QUADS public vlan setup steps

quads --ls-qinq
cloud01: 0 (Isolated)
cloud02: 1 (Combined)
cloud03: 0 (Isolated)
cloud04: 1 (Combined)

Optional QUADS Public VLAN

If you need to associate a public vlan (routable) with your cloud, quads currently supports associating your last NIC per host with one of your defined public VLANs (see the QUADS public vlan setup steps).

To define your cloud with a public VLAN, use the following syntax:

quads --define-cloud --cloud cloud03 [ other define-cloud options ] --vlan 601

If you need to clear the vlan association with your cloud, you can pass any string to the --vlan argument in --mod-cloud

quads --mod-cloud --cloud cloud03 --vlan none

Defining a New Cloud

quads --define-cloud --cloud cloud03 --description "Messaging AMQ" --cloud-owner epresley --cc-users "jdoe jhoffa" --cloud-ticket 423625 --qinq 1

Adding New Hosts to your Cloud

quads --cloud-only --cloud cloud01 | grep r620 | head -20 > /tmp/RT423624
quads --host-list /tmp/RT423624 --add-schedule --schedule-start "2016-10-17 00:00" --schedule-end "2016-11-14 17:00" --schedule-cloud cloud03

Adding New Hosts to your Cloud with JIRA Integration

quads --add-schedule --host-list /tmp/hosts --schedule-start "2021-04-20 22:00" --schedule-end "2021-05-02 22:00" --schedule-cloud cloud20

That's it. At this point your hosts will be queued for provision and move operations, we check once a minute if there are any pending provisioning tasks. To check manually:

quads --move-hosts --dry-run

After your hosts are provisioned and moved you should see them populate under the cloud list.

quads --cloud-only --cloud cloud03

Managing Faulty Hosts

Starting with 1.1.4 QUADS can manage broken or faulty hosts for you and ensure they are ommitted from being added to a future schedule or listed as available. Prior to 1.1.4 this is managed via the Foreman host parameter broken_state (true/false).

Migrating to QUADS-managed Host Health

for h in $(hammer host list --per-page 1000 --search params.broken_state=true | grep $(egrep ^domain /opt/quads/conf/quads.yml | awk '{ print $NF }') | awk '{ print $3 }') ; do quads --mark-broken --host $h ; done

Managing Retired Hosts

quads --ls-retire

Retiring Hosts

for host in $(cat /tmp/retired_hosts); do yes | quads --shrink --host $host --now; done

Extending the Schedule of an Existing Cloud

Occasionally you'll want to extend the lifetime of a particular assignment. QUADS lets you do this with one command but you'll want to double-check things first. In this example we'll be extending the assignment end date for cloud02

In QUADS version 1.1.4 or higher or the current master branch you can extend a cloud environment with a simple command.

quads --extend --cloud cloud02 --weeks 2 --check

This will check whether or not the environment can be extended without conflicts.

To go ahead and extend it remove the --check

quads --extend --cloud cloud02 --weeks 2

Extending the Schedule of a Specific Host

You might also want to extend the lifetime of a specific host. In this example we'll be extending the assignment end date for host01.

quads --extend --host host01 --weeks 2 --check

This will check whether or not the environment can be extended without conflicts.

To go ahead and extend it remove the --check

quads --extend --host host01 --weeks 2

Shrinking the Schedule of an Existing Cloud

Occasionally you'll want to shrink the lifetime of a particular assignment. In this example we'll be shrinking the assignment end date for cloud02

quads --shrink --cloud cloud02 --weeks 2 --check

This will check whether or not the environment can be shrunk without conflicts.

To go ahead and shrink it remove the --check

quads --shrink --cloud cloud02 --weeks 2

Shrinking the Schedule of a Specific Host

You might also want to shrink the lifetime of a specific host. In this example we'll be shrinking the assignment end date for host01.

quads --shrink --host host01 --weeks 2 --check

This will check whether or not the host schedule can be shrunk without conflicts.

To go ahead and shrink it remove the --check

quads --shrink --host host01 --weeks 2

Terminating a Schedule

If you would like to terminate the lifetime of a schedule at either a host or cloud level, you can pass the --now argument instead of --weeks which will set the schedules end date to now. In this example we'll be terminating the assignment end date for cloud02.

quads --shrink --cloud cloud02 --now --check

This will check whether or not the environment can be terminated without conflicts.

To go ahead and terminate it remove the --check

quads --shrink --cloud cloud02 --now

Adding Hosts to an existing Cloud

QUADS also supports adding new machines into an existing workload (cloud).

quads --ls-available --schedule-start "2016-12-05 08:00" --schedule-end "2016-12-15 08:00"

c03-h11-r620.rdu.openstack.example.com
c03-h13-r620.rdu.openstack.example.com
c03-h14-r620.rdu.openstack.example.com
c03-h15-r620.rdu.openstack.example.com

Above we see all the free servers during our timeframe, let's move them into cloud10

quads --host c03-h11-r620.rdu.openstack.example.com --add-schedule --schedule-start "2016-12-05 08:00" --schedule-end "2016-12-15 08:00" --schedule-cloud cloud10
quads --host c03-h13-r620.rdu.openstack.example.com --add-schedule --schedule-start "2016-12-05 08:00" --schedule-end "2016-12-15 08:00" --schedule-cloud cloud10
quads --host c03-h14-r620.rdu.openstack.example.com --add-schedule --schedule-start "2016-12-05 08:00" --schedule-end "2016-12-15 08:00" --schedule-cloud cloud10
quads --host c03-h15-r620.rdu.openstack.example.com --add-schedule --schedule-start "2016-12-05 08:00" --schedule-end "2016-12-15 08:00" --schedule-cloud cloud10

Removing a Schedule

You can remove an existing schedule across a set of hosts using the --rm-schedule flag against the schedule ID for each particular machine of that assignment.

Removing a Schedule across a large set of hosts

You should search for either the start or end dates to select the right schedule ID to remove when performing schedule removals across a large set of hosts.

for host in $(cat /tmp/452851); do quads --rm-schedule --schedule-id $(quads --ls-schedule --host $host | grep cloud08 | grep "start=2017-08-06" | tail -1 | awk -F\| '{ print $1 }'); echo Done. ; done

Removing a Host from QUADS

To remove a host entirely from QUADS management you can use the --rm-host command.

quads --rm-host f03-h30-000-r720xd.rdu2.example.com
Removed: {'host': 'f03-h30-000-r720xd.rdu2.example.com'}

Modifying a Host Schedule

quads --mod-schedule --host host01.example.com --mod-schedule --schedule-id 31 --schedule-start "2023-05-22 22 :00" --schedule-end "2023-06-22 22:00"

Modifying a Host Schedule across a large set of hosts

for host in $(cat /tmp/2491); do quads --mod-schedule --schedule-id $(quads --ls-schedule --host $host | grep cloud06 | grep "start=2023-03-13" | tail -1 | awk -F\| '{ print $1 }') --host $host --schedule-start "2023-03-12 22:00" ; done

Modify a Host Interface

To remove a host entirely from QUADS management you can use the --rm-host command.

quads --mod-interface --interface-name em1 --host f03-h30-000-r720xd.rdu2.example.com --no-pxe-boot
Interface successfully updated

Remove a Host Interface

To remove a host entirely from QUADS management you can use the --rm-host command.

quads --rm-interface --interface-name em1 --host f03-h30-000-r720xd.rdu2.example.com
Resource properly removed

Using the QUADS JSON API

Additional Tools and Commands

Verify or Correct Cloud and Host Network Switch Settings

/opt/quads/quads/tools/verify_switchconf.py --cloud cloud10
/opt/quads/quads/tools/verify_switchconf.py --cloud cloud10 --change
/opt/quads/quads/tools/verify_switchconf.py --host host01.example.com --change

Note, if host01.example.com is not in cloud10, but rather cloud20, you will see the following output:

WARNING - Both --cloud and --host have been specified.
WARNING -
WARNING - Host: host01.example.com
WARNING - Cloud: cloud10
WARNING -
WARNING - However, host01.example.com is a member of cloud20
WARNING -
WARNING - !!!!! Be certain this is what you want to do. !!!!!
WARNING -

Modify or check a specific Host Network Switch Settings

/opt/quads/quads/tools/modify_switch_conf.py --host host01.example.com --nic1 1400 --nic2 1401 --nic3 1400 --nic4 1402 --nic5 1400

Mapping Interface to VLAN ID

This tool also accepts the --all argument which will list a detail for all hosts in the cloud.

Additional you can achieve the same with the following shell one-liner, setting cloud=XX for the cloud and adjusting $(seq 1 4) for your interface ranges available on the host.

cloud=32 ; origin=1100 ; offset=$(expr $(expr $cloud - 1) \* 10); vl=$(expr $origin + $offset) ;for i in $(seq 1 4) ; do vlan=$(expr $vl + $i - 1) ; echo "em$i is interface VLAN $vlan in cloud$cloud" ; done
em1 is interface VLAN 1400 in cloud32
em2 is interface VLAN 1401 in cloud32
em3 is interface VLAN 1402 in cloud32
em4 is interface VLAN 1403 in cloud32

Modifying Cloud-level Attributes

quads --mod-cloud --cloud cloud02 --cloud-owner jhoffa
quads --mod-cloud --cloud cloud04 --cc-users "tpetty fmercury"
quads --mod-cloud --cloud cloud06 --vlan 604 --wipe
quads --mod-cloud --cloud cloud50 --no-wipe
quads --mod-cloud --cloud cloud50 --vlan none

Looking into the Future

quads --cloud-only --cloud cloud08 --date "2019-06-04 22:00"
f16-h01-000-1029u.rdu2.example.com
f16-h02-000-1029u.rdu2.example.com
f16-h03-000-1029u.rdu2.example.com
f16-h05-000-1029u.rdu2.example.com
f16-h06-000-1029u.rdu2.example.com
quads --ls-schedule --date "2020-06-04 22:00"

Dry Run for Pending Actions

quads --move-hosts --dry-run
INFO: Moving b10-h27-r620.rdu.openstack.example.com from cloud01 to cloud03
INFO: Moving c02-h18-r620.rdu.openstack.example.com from cloud01 to cloud03
INFO: Moving c02-h19-r620.rdu.openstack.example.com from cloud01 to cloud03
INFO: Moving c02-h21-r620.rdu.openstack.example.com from cloud01 to cloud03
INFO: Moving c02-h25-r620.rdu.openstack.example.com from cloud01 to cloud03
INFO: Moving c02-h26-r620.rdu.openstack.example.com from cloud01 to cloud03

Find Free Cloud Environment

quads --find-free-cloud
cloud12
cloud16
cloud17
cloud18

Find Available Hosts

quads --ls-available --schedule-start "2019-12-05 08:00" --schedule-end "2019-12-15 08:00"
quads --ls-available --schedule-end "2019-06-02 22:00"

Find Available Hosts based on Hardware or Model

quads --ls-available --schedule-start "2020-08-02 22:00" --schedule-end "2020-08-16 22:00" --filter "model==1029U-TRTP"
quads --ls-hosts --filter "retired==True"
quads --cloud-only --cloud cloud13 --filter "model==FC640"

Find Available Web Interface

quads-available-web

Find a System by MAC Address

quads --ls-hosts --filter "interfaces.mac_address==ac:1f:6b:2d:19:48"

Find Systems by Switch IP Address

quads --ls-hosts --filter "interfaces.ip_address==10.1.34.210"

Using JIRA with QUADS

Backing up QUADS

Restoring QUADS from Backup

su - postgres -c "psql -d quads -f /tmp/quadsdb.sql"
su - postgres -c "psql -f /tmp/quadsdb.sql"

Troubleshooting Validation Failures

A useful part of QUADS is the functionality for automated systems/network validation. Below you'll find some steps to help understand why systems/networks might not pass validation so you can address any issues.

Understanding Validation Structure

There are two main validation tests that occur before a cloud environment is automatically released:

All of these validations are run from --validate-env and we also ship a few useful tools to help you figure out validation failures.

--validate-env is run from cron, see our example cron entry

Troubleshooting Steps

You should run through each of these steps manually to determine what systems/networks might need attention of automated validation does not pass in a reasonable timeframe. Typically, admin_cc: will receieve email notifications of trouble hosts as well.

quads --cloud-only --cloud cloud23 > /tmp/cloud23
fping -u -f /tmp/cloud23
for host in $(quads --cloud-only --cloud cloud15) ; do echo $host $(hammer host info --name $host | grep -i build); done

No systems should be left marked for build.

Validation using Debug Mode

quads --validate-env --debug
Validating cloud23
Using selector: EpollSelector
:Initializing Foreman object:
GET: /status
GET: /hosts?search=build=true
Command executed successfully: fping -u f12-h01-000-1029u.rdu2.scalelab.example.com f12-h02-000-1029u.rdu2.scalelab.example.com f12-h03-000-1029u.rdu2.scalelab.example.com
Command executed successfully: fping -u 172.16.38.126 172.20.38.126 172.16.36.206
Command executed successfully: fping -u 172.17.38.126 172.21.38.126 172.17.36.206
Command executed successfully: fping -u 172.18.38.126 172.22.38.126 172.18.36.206
Command executed successfully: fping -u 172.19.38.126 172.23.38.126 172.19.36.206
Subject: Validation check succeeded for cloud23
From: RDU2 Scale Lab <quads@example.com>
To: dev-null@example.com
Cc: wfoster@example.com, kambiz@example.com, jtaleric@example.com,
 abond@example.com, grafuls@example.com, natashba@example.com
Reply-To: dev-null@example.com
User-Agent: Rufus Postman 1.0.99
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: 7bit
MIME-Version: 1.0

A post allocation check previously failed for:

   cloud: cloud23
   owner: ipinto
   ticket: 498569

has successfully passed the verification test(s)!  The owner
should receive a notification that the environment is ready
for use.

DevOps Team

 cloud23 / ipinto / 498569
quads --validate-env --debug
Validating cloud23
Using selector: EpollSelector
:Initializing Foreman object:
GET: /status
GET: /hosts?search=build=true
There was something wrong with your request
ICMP Host Unreachable from 10.1.38.126 for ICMP Echo sent to f12-h14-000-1029u.rdu2.scalelab.example.com (10.1.38.43)

ICMP Host Unreachable from 10.1.38.126 for ICMP Echo sent to f12-h14-000-1029u.rdu2.scalelab.example.com (10.1.38.43)

ICMP Host Unreachable from 10.1.38.126 for ICMP Echo sent to f12-h14-000-1029u.rdu2.scalelab.example.com (10.1.38.43)

ICMP Host Unreachable from 10.1.38.126 for ICMP Echo sent to f12-h14-000-1029u.rdu2.scalelab.example.com (10.1.38.43)

Skipping Past Network Validation

quads --validate-env --skip-network

Skipping Past Host and Systems Validation

/opt/quads/quads/tools/validate_env.py --skip-system

Skipping Past Network and Systems Validation per Host

/opt/quads/quads/tools/validate_env.py --skip-hosts host01.example.com host02.example.com

Validate Only a Specific Cloud

Mapping Internal VLAN Interfaces to Problem Hosts

You might have noticed that we configure our Foreman templates to drop 172.{16,17,18,19}.x internal VLAN interfaces which correspond to the internal, QUADS-managed multi-tenant interfaces across a set of hosts in a cloud assignment.

The first two octets here can be substituted by the first two octets of your systems public network in order to determine from validate_env.py --debug which host internal interfaces have issues or are unreachable.

validation_1

# for host in 10.1.37.231 10.1.38.150; do host $host; done
231.37.1.10.in-addr.arpa domain name pointer e17-h26-b04-fc640.example.com.
150.38.1.10.in-addr.arpa domain name pointer e17-h26-b03-fc640.example.com.

validation_2

This mapping feeds into our VLAN network validation code

Contact QUADS Developers

Besides Github we're also on IRC via irc.libera.chat. You can click here to join in your browser.

QUADS Talks and Media

Skynet your Infrastructure with QUADS @ EuroPython 2017