JetSki inherits roles from upstream and aims to provide a consistent, seamless OpenShift installation experience on bare metal in Red Hat's Shared Labs.
Table of Contents
This Ansible playbook and set of Ansible roles are aimed at providing a cluster of Red Hat OpenShift 4 (IPI
) in the Red Hat shared labs with as little user input and intervention as possible.
The playbook is intended to be run from outside the cluster of machines you wish to deploy on, from a host we will refer to as jumphost
for the purposes of this discussion. It could even be a user's laptop or some Virtual Machine. The host from which the the playbook is run from (jumphost
) must satisfy the following requirements
Passwordless sudo can be setup as below:
echo "username ALL=(root) NOPASSWD:ALL" | tee -a /etc/sudoers.d/username
chmod 0440 /etc/sudoers.d/username
The username
should be the user with which the playbook is run.
The playbook has been most extensively tested running from a Fedora 30 jumphost
.
The servers used for the OpenShift deployment itself are recommended to satisfy the following requirements.
jumphost
For end-to-end automation and easy deployment, JetSki makes certain assumptions. The first node in your lab allocation is deployed as the provisioner host and the next 3 nodes are deployed as masters. The rest of the nodes are deployed as workers depending on how many workers were requested by user (by default all remaining nodes are deployed as workers unless otherwise specified by worker_count
variable in ansible-ipi-install/group_vars/all.yml
). dnsmasq
is also setup on the provisioner to provide DNS
and DHCP
for the baremetal interfaces of the OpenShift nodes.
The ansible-ipi-install
directory consists of three main sub-directories in addition to the main playbook playbook-jetski.yml
that is used to kick off the installation. They are:
group_vars
- Contains the all.yml
which holds the bare minimum variables needed for installinventory
- contains the file jetski/hosts
that has advanced variables for customized installationroles
- contains 11 roles: bootstrap
, prepare-kni
, add-provisioner
, network-discovery
, set-deployment-facts
, shared-labs-prep
,node-prep
installer
, scale-bootstrap
, scale-node-prep
and scale-worker
. node-prep
handles all the prerequisites that the provisioner node requires prior to running the installer. The installer
role handles extracting the installer, setting up the manifests, and running the Red Hat OpenShift installation.The purpose served by each role can be summarized as follows:
bootstrap
- This role does a lot of heavy lifting for seamless deployment in the shared labs. On a high level, this role is responsible for installing needed packages on the jumphost
, obtaining the list of nodes in your lab allocation dynamically, setting some variables required in inventory as ansible facts (like list of master nodes, worker nodes, mgmt interfaces), copying keys of the jumphost
to the provisioner, rebuilding the provisioner if needed and finally adding the master and worker nodes to the in-memory dynamic inventory of ansible. This role runs on the jumphost
aka localhost
.prepare-kni
- Prepares the kni
user and related artifacts on the provisioner node. This role runs on the provisioner host.add-provisioner
- Adds provsioner host to the dynamic in-memory inventory. This role runs on the jumphost
aka localhost
.network-discovery
- Set several important variables for the inventory including the NICs and MACs to be used for the provisioning and baremetal networks. Some of the MAC details are obtained from an inventory automatically generated on the Lab Wiki which the network-discovery role uses to further set all variables needed for proper networking. This role runs on the provisioner host.set-deployment-facts
- This role is used to set some of the facts registered on the jumphost on to the provisioner host for use in future roles. This role runs on the provisioner host.shared-labs-prep
- Creates the BM bridge, powers on nodes, sets boot order etc. This role runs on the provisioner host.node-prep
- Prepares the provisioner node for the OpenShift Installer by installing needed packages, creating necessary directories etc. This role runs on the provisioner host.installer
- Actually drives the OpenShift Installer. This role runs on the provisioner host.Scale Up worker roles
scale-bootstrap
- This role is similar to boostrap, but runs only during scale up worker playbook execution, it is responsible for gathering inventory details from the file ocpdeployednodeinv.json
and ocpnondeployednodeinv.json
. It validates the input worker_count
with the available non deployed nodes.scale-node-prep
- This role is similar to shared-labs-prep, runs only during scale up worker playbook execution, it is responsible for setting up the boot order on the new worker nodes.scale-worker
- This is the main role which does all openshift operations, it creates BMC scecret, BMH objects for new host and scale worker machinesets. The tree structure is shown below:
├── ansible.cfg
├── filter_plugins
│ ├── deprecate_me.py
│ └── warn_me.py
├── group_vars
│ └── all.yml
├── inventory
│ ├── jetski
│ │ └── hosts
│ └── upstream
│ └── hosts
├── OWNERS
├── playbook-jetski.yml
├── playbook-jetski-scaleup.yml
├── playbook.yml
└── roles
├── add-provisioner
│ └── tasks
│ └── main.yml
├── bootstrap
│ ├── files
│ │ ├── Dockerfile
│ │ └── foreman.yml.j2
│ ├── tasks
│ │ ├── 01_install_packages.yml
│ │ ├── 05_ssh_keys.yml
│ │ ├── 10_load_inv.yml
│ │ ├── 20_reprovision_nodes.yml
│ │ ├── 25_copykeys.yml
│ │ ├── 30_get_interpreter.yml
│ │ ├── 40_prepare_provisioning.yml
│ │ ├── 50_add_ocp_inventory.yml
│ │ ├── 55_add_ocp_masters.yml
│ │ ├── 60_add_ocp_workers.yml
│ │ └── main.yml
│ ├── templates
│ │ └── node_inv.j2
│ └── vars
│ └── main.yml
├── installer
│ ├── defaults
│ │ └── main.yml
│ ├── files
│ │ ├── customize_filesystem
│ │ │ ├── master
│ │ │ └── worker -> master
│ │ ├── filetranspile-1.1.1.py
│ │ └── manifests
│ ├── handlers
│ │ └── main.yml
│ ├── library
│ │ └── podman_container.py
│ ├── meta
│ │ └── main.yml
│ ├── tasks
│ │ ├── 10_get_oc.yml
│ │ ├── 15_disconnected_registry_create.yml
│ │ ├── 15_disconnected_registry_existing.yml
│ │ ├── 20_extract_installer.yml
│ │ ├── 23_rhcos_image_paths.yml
│ │ ├── 24_rhcos_image_cache.yml
│ │ ├── 25_create-install-config.yml
│ │ ├── 30_create_metal3.yml
│ │ ├── 40_create_manifest.yml
│ │ ├── 50_extramanifests.yml
│ │ ├── 55_customize_filesystem.yml
│ │ ├── 59_cleanup_bootstrap.yml
│ │ ├── 60_deploy_ocp.yml
│ │ ├── 70_cleanup_sub_man_registeration.yml
│ │ └── main.yml
│ ├── templates
│ │ ├── chrony.conf.j2
│ │ ├── etc-chrony.conf.j2
│ │ ├── httpd_conf.j2
│ │ ├── install-config-appends.j2
│ │ ├── install-config.j2
│ │ ├── magic.j2
│ │ └── metal3-config.j2
│ ├── tests
│ │ ├── inventory
│ │ └── test.yml
│ └── vars
│ └── main.yml
├── network-discovery
│ └── tasks
│ └── main.yml
├── node-prep
│ ├── defaults
│ │ └── main.yml
│ ├── handlers
│ │ └── main.yml
│ ├── library
│ │ └── nmcli.py
│ ├── meta
│ │ └── main.yml
│ ├── tasks
│ │ ├── 100_power_off_cluster_servers.yml
│ │ ├── 10_validation.yml
│ │ ├── 15_validation_disconnected_registry.yml
│ │ ├── 20_sub_man_register.yml
│ │ ├── 30_req_packages.yml
│ │ ├── 40_bridge.yml
│ │ ├── 45_networking_facts.yml
│ │ ├── 50_modify_sudo_user.yml
│ │ ├── 60_enabled_services.yml
│ │ ├── 70_enabled_fw_services.yml
│ │ ├── 80_libvirt_pool.yml
│ │ ├── 90_create_config_install_dirs.yml
│ │ └── main.yml
│ ├── templates
│ │ └── dir.xml.j2
│ ├── tests
│ │ ├── inventory
│ │ └── test.yml
│ └── vars
│ └── main.yml
├── prepare-kni
│ └── tasks
│ └── main.yml
├── scale-bootstrap
│ └── tasks
│ ├── main.yml
│ └── 10_load_old_inv.yml
├── scale-node-prep
│ ├── tasks
│ │ └── main.yml
│ └── vars
│ └── main.yml
├── scale-worker
│ ├── defaults
│ │ └── main.yml
│ └── tasks
│ ├── 01_dns_update.yml
│ ├── 10_bmc_secrets.yml
│ ├── 20_create_bmh.yml
│ ├── 30_scale_machinesets.yml
│ └── main.yml
├── set-deployment-facts
│ └── tasks
│ └── main.yml
└── shared-labs-prep
├── defaults
│ └── main.yml
├── library
│ └── nmcli.py -> ../../node-prep/library/nmcli.py
├── tasks
│ ├── 10_redfish_queue.yml
│ └── main.yml
├── templates
│ ├── ocp4-lab.dnsmasq.conf.j2
│ └── ocp4-lab.ifcfg-template.j2
├── tests
│ ├── inventory
│ └── test.yml
└── vars
└── main.yml
The TL;DR version is
$ ansible-playbook -i inventory/jetski/hosts playbook-jetski.yml
However, for the playbook to successfully execute certain variables have to be set at a minimum in ansible-ipi-install/group_vars/all.yml
.
The following are the detailed steps to successfully run the Ansible playbook.
ansible.cfg
fileWhile the ansible.cfg
may vary upon your environment a sample is provided in the repository. The default ansible.cfg
supplied in this repository should work in the shared labs. This is purely infromational, modifications are not necessary.
[defaults]
inventory=./inventory
remote_user=kni
callback_whitelist = profile_tasks
[privilege_escalation]
become_method=sudo
ansible-ipi-install/group_vars/all.yml
fileThis is the most important file to modify for a successful install of OpenShift in the Red Hat Shared Labs. Some variables can be left default, but the most important ones to be filled out are
cloud_name
lab_name
ansible_ssh_password
version
build
pullsecret
foreman_url
(Used to rebuild provisioning host aka first host in your allocation with RHEL 8.1)Here's a sample:
# Your allocation name/number in the shared labs
cloud_name: cloud00
# Lab name, typically can be alias or scale
lab_name: scale
# Default root password to your nodes in the lab allocation so that keys can be added automatically for ansible to run
ansible_ssh_pass: password
# Location of the private key of the user running the ansible playbook, leave default
ansible_ssh_key: "{{ ansible_user_dir }}/.ssh/id_rsa"
# The version of the openshift-installer, undefined or empty results in the playbook failing with error message.
# Values accepted: 'latest-4.3', 'latest-4.4', explicit version i.e. 4.3.0-0.nightly-2019-12-09-035405
# For reference, https://amd64.ocp.releases.ci.openshift.org/ and https://mirror.openshift.com/pub/openshift-v4/x86_64/clients/ocp/
version: "4.4.4"
# Enter whether the build should use 'dev' (nightly builds) or 'ga' for Generally Available version of OpenShift
# Empty value results in playbook failing with error message.
build: "ga"
# Your pull secret as is, https://cloud.redhat.com/openshift/install
pullsecret:
# This variable is used to point to the foreman server that is used to reimage nodes. This variables is useful in two cases
# 1. When the first node in your allocation (provisioning host) is not having RHEL 8.1 OS, it is automatically rebuilt with
# RHEL 8.1 as the OCP installer expects the provisioning host to be RHEL 8.1. In some other cases, maybe when you have an
# existing cluster and are trying to reinstall etc, it might be better to start with a clean provisioning host, in which case this variable
# is also used to reimage the provisioning host in spite of it having RHEL 8.1 on it. This variable changes depending on the lab you are
# using as each lab has its own foreman server. This URL can be deduced from the lab allocation email you receive when your allocation is
# ready. It will be under the paragraph "You can also view/manage your hosts via Foreman:" in the email. It is important to use an https
# prefix even if url in the email has http. Also, the url in email might have the '/hosts' path appended, we can remove 'hosts' from url
# and have it be https://foreman.example.com for example. If you are having trouble figuring out this variable please look for the
# pastebins under the "Modifying the ansible-ipi-install/group_vars/all.yml file" section in README.md
foreman_url: https://foreman.example.com/
# Choose a provisioner node from the allocated cloud, this is optional.
# Same thing can also be achieved by editing ocpinv.json file and reorder the nodes list.
# https://github.com/redhat-performance/JetSki#changing-node-rolesexcluding-nodes-in-your-allocation-from-the-deployment
# If not specified, it will pick the first node in your cloud as provisioner(preferred)
# Make sure the specified server hostname doesn't not contain 'mgmt-' or '-drac'.
# provisioner_hostname: ""
# The automation automatically rebuilds provisioner node to RHEL 8.1 if not already RHEL 8.1 (see foreman_url variable)
# However you can also force a reprovsioning of the provisioner node for redeployment scenarios
rebuild_provisioner: false
# Number of workers desired, by default all hosts in your allocation except 1 provisioner and 3 masters are used workers
# However that behaviour can be overrided by explicitly setting the desired number of workers here. For a masters only deploy,
# set worker_count to 0
# Update this variable to scale up your existing cluster, provided lab allocation is sufficient to scale up to this count.
# If not mentioned for a scale up execution, it includes all node available in the inventory `ocpnondeployednodeinv.json`
# If mentioned, this value should be final worker count and cannot be less than existing worker_count.
worker_count: 0
# set to true to deploy with jumbo frames
jumbo_mtu: false
# set to true only if you requested a public routable VLAN for your cloud in scale lab
routable_api: false
Here's a sample all.yml for the scale lab with the pull secret and password scraped: http://pastebin.test.redhat.com/962541
Here's a sample all.yml for the ALIAS lab with the pull secret and password scraped: http://pastebin.test.redhat.com/962544
If you're a part of the redhat-performance GitHub organization, you can also access the samples here: https://github.com/redhat-performance/JetSki-Configs/tree/master/jetski
ansible-ipi-install/inventory/jetski/hosts
fileThe bare minimum variables to get a successful install are listed in ansible-ipi-install/group_vars/all.yml
. Typically, correctly filling ansible-ipi-install/group_vars/all.yml
should suffice for the shared labs use case, but in cases where some advanced configuration is needed and to fully utilize the options supported by the installer and the upstream playbooks
, the ansible-ipi-install/inventory/jetski/hosts
can be edited and used as an inventory file. For example, the SDN
for OpenShift can be set using the network_type
variable in the inventory. While editing the hosts file is optional the file needs to be passed to the ansible playbook invocation command using -i inventory/jetski/hosts
. This is because it contains some default variables required for the installation.
Below is a sample of the ansible-ipi-install/inventory/jetski/hosts
:
[all:vars]
###############################################################################
# Required configuration variables for IPI on Baremetal Installations #
###############################################################################
# (Optional) Set the provisioning bridge name. Default value is 'provisioning'.
#provisioning_bridge=provisioning
# (Optional) Set the baremetal bridge name. Default value is 'baremetal'.
#baremetal_bridge=baremetal
# (Optional) Activation-key for proper setup of subscription-manager, empty value skips registration
#activation_key=""
# (Optional) Activation-key org_id for proper setup of subscription-manager, empty value skips registration
#org_id=""
# The directory used to store the cluster configuration files (install-config.yaml, pull-secret.txt, metal3-config.yaml)
dir="{{ ansible_user_dir }}/clusterconfigs"
# (Optional) Fully disconnected installs require manually downloading the release.txt file and hosting the file
# on a webserver accessible to the provision host. The release.txt file can be downloaded at
# https://mirror.openshift.com/pub/openshift-v4/clients/ocp-dev-preview/{{ version }}/release.txt (for DEV version)
# https://mirror.openshift.com/pub/openshift-v4/clients/ocp/{{ version }}/release.txt (for GA version)
# Example of hosting the release.txt on your example.com webserver under the 'latest-4.3' version directory.
# http://example.com:<port>/latest-4.3/release.txt
# Provide the webserver URL as shown below if using fully disconnected
#webserver_url=http://example.com:<port>
# Provisioning IP address (default value)
prov_ip=172.22.0.3
# (Optional) Provisioning network configuration overrides (4.4+)
prov_bootstrap_ip=172.22.0.2
prov_dhcp_range="172.22.0.10,172.22.0.254"
# (Optional) Enable playbook to pre-download RHCOS images prior to cluster deployment and use them as a local
# cache. Default is false.
#cache_enabled=True
# (Optional) The port exposed on the caching webserver. Default is port 8080.
#webserver_caching_port=8080
# (Optional) Enable IPv6 addressing instead of IPv4 addressing on both provisioning and baremetal network
ipv6_enabled=False
# (Optional) When ipv6_enabled is set to True, but want IPv4 addressing on provisioning network
# Default is false.
#ipv4_provisioning=True
# (Optional) When ipv6_enabled is set to True, but want IPv4 addressing on baremetal network
#ipv4_baremetal=True
# (Optional) A list of clock servers to be used in chrony by the masters and workers
#clock_servers=["pool.ntp.org","clock.redhat.com"]
# (Optional) Provide HTTP proxy settings
#http_proxy=http://USERNAME:PASSWORD@proxy.example.com:8080
# (Optional) Provide HTTPS proxy settings
#https_proxy=https://USERNAME:PASSWORD@proxy.example.com:8080
# (Optional) comma-separated list of hosts, IP Addresses, or IP ranges in CIDR format
# excluded from proxying
# NOTE: OpenShift does not accept '*' as a wildcard attached to a domain suffix
# i.e. *.example.com
# Use '.' as the wildcard for a domain suffix as shown in the example below.
# i.e. .example.com
#no_proxy_list="172.22.0.0/24,.example.com"
# The default installer timeouts for the bootstrap and install processes may be too short for some baremetal
# deployments. The variables below can be used to extend those timeouts.
# (Optional) Increase bootstrap process timeout by N iterations.
increase_bootstrap_timeout=2
# (Optional) Increase install process timeout by N iterations.
increase_install_timeout=2
######################################
# Vars regarding install-config.yaml #
######################################
# Base domain, i.e. example.com
domain="myocp4.com"
# Name of the cluster, i.e. openshift
cluster="test"
# Note: Under some conditions, it may be useful to randomize the cluster name. For instance,
# when redeploying an existing environment this can help avoid VRID conflicts. You can
# set the cluster_random boolean below to true to append a random number to you cluster name.
cluster_random=true
# The public CIDR address, i.e. 10.1.1.0/21
extcidrnet="192.168.222.0/24"
# NOTE: For the RH shared labs, the VIPs below are automated w/ variables
# based on the extcidrnet above.
# An IP reserved on the baremetal network.
dnsvip="{{ extcidrnet | next_nth_usable(2) }}"
# An IP reserved on the baremetal network for the API endpoint.
# (Optional) If not set, a DNS lookup verifies that api.<clustername>.<domain> provides an IP
apivip="{{ extcidrnet | next_nth_usable(3) }}"
# An IP reserved on the baremetal network for the Ingress endpoint.
# (Optional) If not set, a DNS lookup verifies that *.apps.<clustername>.<domain> provides an IP
ingressvip="{{ extcidrnet | next_nth_usable(4) }}"
# The master hosts provisioning nic
# (Optional) If not set, the prov_nic will be used
#masters_prov_nic=""
# Network Type (OpenShiftSDN or OVNKubernetes). Playbook defaults to OVNKubernetes.
# Uncomment below for OpenShiftSDN
network_type="OVNKubernetes"
# (Optional) A URL to override the default operating system image for the bootstrap node.
# The URL must contain a sha256 hash of the image.
# See https://github.com/openshift/installer/blob/master/docs/user/metal/customization_ipi.md
# Example https://mirror.example.com/images/qemu.qcow2.gz?sha256=a07bd...
#bootstraposimage=""
# (Optional) A URL to override the default operating system image for the cluster nodes.
# The URL must contain a sha256 hash of the image.
# See https://github.com/openshift/installer/blob/master/docs/user/metal/customization_ipi.md
# Example https://mirror.example.com/images/metal.qcow2.gz?sha256=3b5a8...
#clusterosimage=""
# Registry Host
# Define a host here to create or use a local copy of the installation registry
# Used for disconnected installation
# [registry_host]
# registry.example.com
# [registry_host:vars]
# The following cert_* variables are needed to create the certificates
# when creating a disconnected registry. They are not needed to use
# an existing disconnected registry.
# cert_country=US # two letters country
# cert_state=MyState
# cert_locality=MyCity
# cert_organization=MyCompany
# cert_organizational_unit=MyDepartment
# The port exposed on the disconnected registry host can be changed from
# the default 5000 to something else by changing the following variable.
# registry_port=5000
# The directory the mirrored registry files are written to can be modified from teh default /opt/registry by changing the following variable.
# registry_dir="/opt/registry"
# The following two variables must be set to use an existing disconnected registry.
#
# Specify a file that contains extra auth tokens to include in the
# pull-secret if they are not already there.
# disconnected_registry_auths_file=/path/to/registry-auths.json
# Specify a file that contains the addition trust bundle and image
# content sources for the local registry. The contents of this file
# will be appended to the install-config.yml file.
# disconnected_registry_mirrors_file=/path/to/install-config-appends.json
playbook-jetski.yml
The Ansible playbook generates the list of hosts it is operating against automatically based on your lab allocation and runs through the roles
to give you an OpenShift on baremetal install. No modification of these roles or the playboo is necessary. All modifications of variables may be done within the ansible-ipi-install/group_vars/all.yml
and ansible-ipi-install/inventory/hosts
files. Please note that if the same variable is defined in ansible-ipi-install/group_vars/all.yml
and ansible-ipi-install/inventory/hosts
, the value in ansible-ipi-install/group_vars/all.yml
will take precedence. A sample file for inventory is located at ansible-ipi-install/inventory/jetski/hosts
.
Sample playbook-jetski.yml
:
---
- name: IPI on Baremetal Installation Playbook -- Red Hat Shared Labs Edition
hosts: localhost
roles:
- { role: bootstrap }
- hosts: provisioner
roles:
- { role: prepare-kni, ssh_path: /root/.ssh}
- hosts: localhost
roles:
- { role: add-provisioner }
- hosts: provisioner
roles:
- { role: network-discovery }
- { role: set-deployment-facts }
- { role: shared-labs-prep }
- { role: node-prep }
- { role: installer }
environment:
http_proxy: "{{ http_proxy }}"
https_proxy: "{{ https_proxy }}"
no_proxy: "{{ no_proxy_list }}"
- name: Finishing IPI on Baremetal Installation
hosts: provisioner
tasks:
- name: Create directory
file:
state: directory
path: "{{ ansible_facts['user_dir'] }}/scale-worker"
- name: Writing Deployed node content to a file
copy:
dest: "{{ ansible_facts['user_dir'] }}/scale-worker/ocpdeployednodeinv.json"
content: "{{ ocp_deploying_node_content | to_nice_json }}"
when: ocp_deploying_node_content.nodes | length > 0
- name: Writing available non Deployed node content to a file
copy:
dest: "{{ ansible_facts['user_dir'] }}/scale-worker/ocpnondeployednodeinv.json"
content: "{{ nondeploying_worker_nodes_content | to_nice_json }}"
when: nondeploying_worker_nodes_content.nodes | length > 0
- name: Deleting non Deployed node content
file:
dest: "{{ ansible_facts['user_dir'] }}/scale-worker/ocpnondeployednodeinv.json"
state: absent
when: nondeploying_worker_nodes_content.nodes | length == 0
playbook-jetski.yml
With the playbook-jetski.yml
set and in-place, run the playbook-jetski.yml
$ cd ansible-ipi-install/
$ ansible-playbook -i inventory/jetski/hosts playbook-jetski.yml
Once the playbook has successfully completed, verify that your environment is up and running.
ssh kni@provisioner.example.com
kubeconfig
file located in the ~/clusterconfigs/auth
directoryexport KUBECONFIG=~/clusterconfigs/auth/kubeconfig
[kni@provioner~]$ oc get nodes
NAME STATUS ROLES AGE VERSION
master-0.openshift.example.com Ready master 19h v1.16.2
master-1.openshift.example.com Ready master 19h v1.16.2
master-2.openshift.example.com Ready master 19h v1.16.2
worker-0.openshift.example.com Ready worker 19h v1.16.2
playbook-jetski-scaleup.yml
This playbook scales up worker nodes to the desired worker_count
mentioned in ansible-ipi-install/group_vars/all.yml
make sure to update the worker_count before execution. This playbook reads the current cluster size and available nodes from ocpdeployednodeinv.json
and ocpnondeployednodeinv.json
(originally created by playbook-jetski.yml
) stored in provisioner node /home/kni/scale-worker/
directory.
Sample playbook-jetski-scaleup.yml
:
---
- name: IPI on Baremetal Installation Playbook -- Red Hat Shared Labs JetSki Edition
hosts: orchestration
roles:
- { role: scale-bootstrap }
- hosts: provisioner
tasks:
- name: set facts needed for OCP deployment on the provisioning host
set_fact:
"{{ item }}": "{{ hostvars[groups['orchestration'][0]][item] }}"
with_items:
- worker_fqdns
- lab_ipmi_user
- lab_ipmi_password
- scale_worker_node
- worker_count
- dell_nodes
- supermicro_nodes
- ocp_deploying_node_content
- nondeploying_worker_nodes_content
- hosts: provisioner
roles:
- { role: scale-node-prep }
- { role: scale-worker }
- name: Finishing IPI on Baremetal Installation
hosts: provisioner
tasks:
- block:
- name: Update ocp_nondeplopyed_node_content
set_fact:
ocp_deploying_node_content: "{{ ocp_deploying_node_content | combine({'nodes': ocp_deploying_node_content.nodes|difference(failed_nodes)}, recursive=True) }}"
- name: Update ocp_nondeplopyed_node_content
set_fact:
nondeploying_worker_nodes_content: "{{ nondeploying_worker_nodes_content | combine({'nodes': nondeploying_worker_nodes_content.nodes|union(failed_nodes)}, recursive=True) }}"
when:
- failed_nodes is defined
- failed_nodes | length > 0
- name: Writing Deployed node content to a file
copy:
dest: "{{ ansible_facts['user_dir'] }}/scale-worker/ocpdeployednodeinv.json"
content: "{{ ocp_deploying_node_content | to_nice_json }}"
when: ocp_deploying_node_content.nodes | length > 0
- name: Writing available non Deployed node content to a file
copy:
dest: "{{ ansible_facts['user_dir'] }}/scale-worker/ocpnondeployednodeinv.json"
content: "{{ nondeploying_worker_nodes_content | to_nice_json }}"
when: nondeploying_worker_nodes_content.nodes | length > 0
- name: Deleting non Deployed node content
file:
dest: "{{ ansible_facts['user_dir'] }}/scale-worker/ocpnondeployednodeinv.json"
state: absent
when: nondeploying_worker_nodes_content.nodes | length == 0
playbook-jetski.yml
With the playbook-jetski-scaleup.yml
set and in-place, run the playbook-jetski-scaleup.yml
$ ansible-playbook -i inventory/jetski/hosts playbook-jetski-scaleup.yml
Once the playbook has successfully completed, verify that your environment is up and running.
ssh kni@provisioner.example.com
kubeconfig
file located in the ~/clusterconfigs/auth
directoryexport KUBECONFIG=~/clusterconfigs/auth/kubeconfig
[kni@provioner~]$ oc get nodes
NAME STATUS ROLES AGE VERSION
master-0.openshift.example.com Ready master 19h v1.16.2
master-1.openshift.example.com Ready master 19h v1.16.2
master-2.openshift.example.com Ready master 19h v1.16.2
worker-0.openshift.example.com Ready worker 19h v1.16.2
worker-1.openshift.example.com Ready worker 19h v1.16.2
To run JetSki in a container, you need the image.
You can build and run the image with podman. You only need podman installed. If you have Docker installed instead of Podman, you can create an alias that makes the podman command run Docker.
The convenience script command to build it is:
./build-jetski-container.sh
If that does not work for you, you can build the image manually using the Dockerfile in the project root directory.
Alternatively, if the container is on an image repository, you can pull it from there.
The run script mentioned below uses the local computer's copy of the file ansible-ipi-install/group-vars/all.yml
and ansible-ipi-install/inventory/jetski/hosts
You must configure that file as requested in the above section. Containerized JetSki
The convenience script command run the image is:
./run-jetski-container.sh
You may manually run it with Docker or Podman, as long as you pass in the applicable configuration files.
Deployment of OCP 4.3, 4.4, 4.5 and 4.6 has been tested with this playbook.
We follow the standard GitHub process for PRs. One thing you need to note is that if your changes affect the node-prep
or installer
roles, the PR needs to be submitted against baremetal-deploy as JetSki uses those roles as is by rebasing, for maintainability.
For additional reading material and advanced usage of all the options provided by ansible-ipi-install/inventory/jetski/hosts
please refer to https://github.com/openshift-kni/baremetal-deploy/tree/master/ansible-ipi-install and upstream docs. The playbook provided in this repo contantly aims to support everything supported upstream which is made possible by the modular architecture of using upstream roles as is without change and only having extra roles that run before the upstream roles.
By default, JetSki deploys a minimum of 3 masters and number of workers defined by worker_count
in ansible-ipi-install/group_vars/all.yml
. If worker_count
is not defined, it deploys all the remaining nodes in your allocation (apart from the 1 provisioner host and 3 master nodes) as worker nodes. The nodes assigned to each role are also fixed in the sense that the first node in your allocation always becomes the provisioner host, the next 3 become the OpenShift masters and the remaining become workers. However, if you want to assign specific nodes to specific roles or keep a few nodes of your allocation out of the deployment, it can be achieved by changing the ocpinv.json
file that is downloaded tothe ansible-ipi-install
directory, when you are running the playbook from localhost
. It's a little bit hackish, but here's what you would do
Initially, kick off the JetSki playbook run and after running for a few seconds you would see these tasks
TASK [bootstrap : check ocpinv file available] ***********************************************************************************************************************************************************************************************
Wednesday 01 July 2020 13:52:42 -0400 (0:00:00.199) 0:00:06.601 ********
ok: [localhost]
TASK [bootstrap : Download ocpninv.json] *****************************************************************************************************************************************************************************************************
Wednesday 01 July 2020 13:52:42 -0400 (0:00:00.332) 0:00:06.933 ********
skipping: [localhost]
TASK [bootstrap : Download ocpinv.json] ******************************************************************************************************************************************************************************************************
Wednesday 01 July 2020 13:52:42 -0400 (0:00:00.193) 0:00:07.127 ********
changed: [localhost]
As soon as the Download ocpinv.json
task completes successfully, you can exit out of the playbook by using ctrl+c
and then manually edit the ocpinv.json
file. The file is a JSON dictionary which has a list of dictionaries under the nodes
key, each of which represents a node in your lab allocation. By changing the order of nodes/removing nodes from here you are able to assign nodes to a particular role/remove them from deployment. Be careful to ensure that it is still a valid JSON after your edits. For example, if I want to ensure that a specific node in my allocation becomes the provisioner, I would move the dictionary representing that node to the first position in the nodes
list, or if I wanted to ensure specific nodes become OpenShift masters, I would put them in the 2,3 and 4 positions in the list, or if I wanted to exclude a few nodes, I would totally remove them from the list of nodes
. After these changes, JetSki will consume the edited ocpinv.json
instead of quads inventory order on every execution.
In case of any failure during a scaleup play, make sure you remove all faulty nodes as well as other nodes which are not provisioned
yet in the cluster and re-run the script from ansible node.
Scaling down from 2 to 1,
oc get nodes
oc adm cordon <node_name>
oc adm drain <node_name> --force=true
oc get machinesets -n openshift-machine-api
oc scale --replicas=1 machineset <machineset> -n openshift-machine-api
# Check active worker nodes, it would have been reduced to 1
oc get nodes
oc get machinesets
oc delete bmh <host_name> -n openshift-machine-api
Wait till you the see the cluster back to normal state with reduced worker node, then re-run the playbook.
This approach will not work if you want to rebuild a node after a successful playbook execution, because after a successful execution ocpnondeployednodeinv.json
will be update to latest or removed.
Re-run might use the update inventory.
By default the the Kuberentes API is only accessible from the provisioner node as the cluster does not have any lab routable IPs and the IPs are only routable from within the cluster. Along with your allocation in Scale Lab, you are able to request for public routable IP addresses, in which case you can use the routable_api
option to access your baremetal cluster from outside on the lab/provosioner host as long as you are on VPN. To do this, you need to set routable_api
in group_vars/all.yml
to true
and along with that the inventory/jetski/hosts
has to use the information from http://wiki.rdu2.scalelab.redhat.com/vlans/ for your cloud and look like below.
For example, if the the VLAN wiki page has details as follow:
605 10.1.54.0/24 255.255.255.0 10.1.54.254 254 smalleni 746 cloud37
My inventory/jetski/hosts
should have the following content
######################################
# Vars regarding install-config.yaml #
######################################
# Base domain, i.e. example.com
domain="rdu2.scalelab.redhat.com"
# Name of the cluster, i.e. openshift
cluster="vlan605"
# Note: Under some conditions, it may be useful to randomize the cluster name. For instance,
# when redeploying an existing environment this can help avoid VRID conflicts. You can
# set the cluster_random boolean below to true to append a random number to you cluster name.
cluster_random=false
# The public CIDR address, i.e. 10.1.1.0/21
extcidrnet="10.1.54.0/24"
cloud_name
field in ansible-ipi-install/group_vars/all.yml
, it would otherwise be trivial for someone to accidentally re-install over your already installed cluster by supplying the wrong cloud_name
in ansible-ipi-install/group_vars/all.yml
.Once the playbook reaches the task
TASK [installer : Deploy OpenShift Cluster]
The control is passed to the OpenShift installer and any failure in the deployment of OpenShift after that point is either due to hardware issues or installer issues itself. It is not a JetSki failure.
Something that has worked out well in the past is to keep the consoles of all three of your master nodes open during deployment, so that you can observe if any one node is misbehaving when compared to the others. There are 3 boots total before the masters fully become operational and cluster operators start rolling out.
Deployments usually fail if the above three steps do not happen. Sometimes, a node might not boot to disk and be stuck in a PXE loop even when asked to boot to hard disk by the installer or it might be stuck at the OS selection menu when asked to boot to disk. These among several other issues have been seen with hardware in the past and it is worth keeping an eye on the console and intervening if needed for a successful deploy. This is beyond the scope of the playbook or for that matter even the installer.