Install three-node ACM cluster to replace existing single-node cluster

cdoan1 commented 3 years ago

With RHACM 2.0, bare metal support was technical preview. So, bare metal was disabled by default. Here are the steps to enable the feature in the ui. reference: https://github.com/open-cluster-management/deploy#enabling-bare-metal-and-vmware-consoles

Enabling Bare metal consoles

To work with bare metal, two flags need to be flipped activated

console-header

Run the following command to enable the bare metal assets on the navigation menu

oc -n open-cluster-management patch deploy console-header -p \
'{"spec":{"template":{"spec":{"containers":[{"name":"console-header","env": [{"name": "featureFlags_baremetal","value":"true"}]}]}}}}'

console-ui

Run the following commands to enable the bare metal button on the create cluster page

DEPLOY_NAME=`oc -n open-cluster-management get deploy -o name | grep consoleui`
oc -n open-cluster-management patch ${DEPLOY_NAME} -p \
'{"spec":{"template":{"spec":{"containers":[{"name":"hcm-ui","env": [{"name": "featureFlags_baremetal","value":"true"}]}]}}}}'

ipolonsk commented 3 years ago

Open question about the deployment. Two option to deploy the infra cluster -

Using the cloud.redhat.com tool to deploy the cluster
Using the Openshift install binary (KNI) to deploy the cluster (IPI).

@ooichman let each write in details his point of view

ipolonsk commented 3 years ago

Pros and Cons for cloud.redhat.com (Assisted Installer): Pros:

Assisted Installer is a growing RedHat tool to install a new cluster, getting family with it will have future benefits.
This tool will be part of the ACM and will be used in deploying new clusters via the ACM.
It's a good opportunity to provided feedback to the developers.
Can be easily automated, use API calls.

Cons:

A dev preview tool, not completed product

ooichman commented 3 years ago

I think we should keep the installation as close as we can to a disconnected environment without using external tools which may be available only internally (at this point in time). I thinks we should use the KNI (openshift-install) method to deploy the 3 nodes cluster for our "initial infra cluster" and once it is up and running we can deploy other clusters with the other methods

ipolonsk commented 3 years ago

@ooichman I agree with you, we should go with the KNI (openshift install) option, it's a more easy and light deployment.

ipolonsk commented 3 years ago

@larsks I want to create a new cluster with the name 'moc-infra' under the 'massopen.cloud' domain. What do you think? we will need to change the record of our previous ('cnv2.massopen.cloud') cluster to a new name.

tumido commented 3 years ago

I like the KNI more as of right now as well. Lightweight, easy and available to general public are huge pros to me.

@ipolonsk can you please convert this to an ADR PR against https://github.com/operate-first/blueprint ?

I think this comment of yours https://github.com/open-infrastructure-labs/ops-issues/issues/22#issuecomment-779991950 is precisely the ADR content we're after :+1:

ipolonsk commented 3 years ago

@tumido sure, we created a ADR of the high-level design of the new infra cluster, I will add those comments as a new section in the ADR, by the end of the day I will submit it.

larsks commented 3 years ago

@ipolonsk re: creating a moc-infra hostname, that shouldn't be a problem. I've created https://github.com/CCI-MOC/ops-issues/issues/168 to track the request.

cdoan1 commented 3 years ago

@ipolonsk for additional context, the openshift kni documentations references the openshift-installer-baremetal binary. This is the same binary that ACM will invoke through the hive component to provision a bare metal cluster, be it virtual bare metal or real bare metal. So we will need to ensure we satisfy the same requirements--this means network and provision host.

larsks commented 3 years ago

@ilana I've closed https://github.com/CCI-MOC/ops-issues/issues/168. I've made the following hostname changes:

openshift-master-0.cnv2.massopen.cloud -> os-ctrl-0.moc-infra.massopen.cloud
openshift-master-1.cnv2.massopen.cloud -> os-ctrl-1.moc-infra.massopen.cloud
openshift-master-2.cnv2.massopen.cloud -> os-ctrl-2.moc-infra.massopen.cloud

The complete dnsmasq configuration for the domain looks like:

local=/moc-infra.massopen.cloud/

dhcp-option=tag:moc-infra,option:classless-static-route,10.0.0.0/19,192.12.185.101
dhcp-option=tag:moc-infra,option:domain-name,moc-infra.massopen.cloud

host-record=api.moc-infra.massopen.cloud,192.12.185.141
host-record=ns1.moc-infra.massopen.cloud,192.12.185.142
host-record=lb.moc-infra.massopen.cloud,192.12.185.143
address=/apps.moc-infra.massopen.cloud/192.12.185.143

# Host records
# These entries create forward and reverse entries for each host
host-record=192.12.185.140,provisioner.moc-infra.massopen.cloud
host-record=192.12.185.145,os-ctrl-0.moc-infra.massopen.cloud
host-record=192.12.185.146,os-ctrl-1.moc-infra.massopen.cloud
host-record=192.12.185.147,os-ctrl-2.moc-infra.massopen.cloud

# ipmi helpers
host-record=10.0.5.13,provisioner-ipmi.moc-infra.massopen.cloud
host-record=10.0.15.2,os-ctrl-0-ipmi.moc-infra.massopen.cloud
host-record=10.0.17.36,os-ctrl-1-ipmi.moc-infra.massopen.cloud
host-record=10.0.3.6,os-ctrl-2-ipmi.moc-infra.massopen.cloud

# DHCP records
# These entries create static DHCP leases for each host
dhcp-host=00:0f:53:0d:cd:31,192.12.185.140,set:moc-infra,set:pxe,provisioner,24h
dhcp-host=d4:ae:52:94:d5:a5,192.12.185.145,set:moc-infra,os-ctrl-0,24h
dhcp-host=d4:ae:52:98:15:0f,192.12.185.146,set:moc-infra,os-ctrl-1,24h
dhcp-host=90:b1:1c:21:af:4a,192.12.185.147,set:moc-infra,os-ctrl-2,24h

Any hostnames in the cnv2.massopen.cloud domain have been removed.

larsks commented 3 years ago

@ipolonsk et al did this and the result is the moc-infra cluster.

open-infrastructure-labs / ops-issues