openshift-instruqt / instruqt

Repository for the Red Hat and OpenShift Developers Learning Portal, powered by Instruqt.
https://developers.redhat.com/learn/openshift
32 stars 36 forks source link

Document image build process #94

Closed ryanj closed 2 weeks ago

ryanj commented 2 years ago

Document the image build process (https://github.com/openshift-instruqt/instruqt/issues/88)

CRC tarball -> crc binary -> crc.qcow2 -> disk.raw -> crc.tar.gz -> GCP Image (build) -> GCP Image (GUID) -> GCP instance (launched via instruqt) -> GCP instance (READY)

Solution Outline:

GCP setup

  1. Log in to GCP and create a new project for this work
  2. Install the gcloud command line tools: https://cloud.google.com/sdk/docs/install
  3. Log in to GCP via the command line: gcloud auth login
  4. Set your default project scope: gcloud config set project rhd-devx-instruqt

CRC VM image extraction

Image Extraction workflow (tested on Fedora35):

  1. Install RH VPN services: https://redhat.service-now.com/help?id=kb_article_view&sys_kb_id=04168d681b830990839e32a3cc4bcbb9
  2. Download the latest CRC binary from the cdk build server (VPN required): http://cdk-builds.usersys.redhat.com/builds/crc/releases/
  3. Extract the CRC installation tarball tar xvf ~/Downloads/crc-linux-amd64.tar.xz
  4. Establish a $PATH-accessible folder for your local binaries: mkdir -p ~/.local/bin
  5. Export CRC_RELEASE export CRC_RELEASE=2.10.1 and OS_RELEASE export OPENSHIFT_RELEASE=4.11.7
  6. Move the CRC executable to a PATH accessible location: mv ~/Downloads/crc-linux-${CRC_RELEASE}-amd64/crc ~/.local/bin
  7. Run crc setup to download the latest CRC qcow2 VM image to your local machine
  8. The extracted VM disk image should be visible here: cd ~/.crc/cache/crc_libvirt_${OPENSHIFT_RELEASE}_amd64/ && ls -alh
  9. Convert the image to RAW format: qemu-img convert -O raw ~/.crc/cache/crc_libvirt_${OPENSHIFT_RELEASE}_amd64/crc.qcow2 ~/.crc/cache/crc_libvirt_${OPENSHIFT_RELEASE}_amd64/disk.raw
  10. Tar the raw disk image tar cvf crc.tar disk.raw
  11. Compress the tarball: gzip crc.tar
  12. Check to make sure the tarball includes a single file named disk.raw: tar tfz crc.tar.gz

Import the raw CRC disk image to GCP

  1. Try listing images from the command line. This step should give you an opportunity to enable the Compute API for your project: gcloud compute images list --no-standard-images | grep openshift.

  2. export CRC_DASH_RELEASE=2-10-1

  3. export OPENSHIFT_DASH_RELEASE=4-11-7

  4. Use Google Cloud Storage to create a new storage bucket for the raw image: gsutil mb -l europe-west1 gs://openshift-${OPENSHIFT_DASH_RELEASE}

  5. Upload the raw image to the new cloud storage bucket gsutil cp ~/.crc/cache/crc_libvirt_${OPENSHIFT_RELEASE}_amd64/crc.tar.gz gs://openshift-${OPENSHIFT_DASH_RELEASE}

  6. Review what you've uploaded to the bucket: gcloud alpha storage ls gs://openshift-${OPENSHIFT_DASH_RELEASE}

  7. After the upload has completed, create a new Image in region europe-west1 by selecting it from the bucket gs://openshift-${OPENSHIFT_DASH_RELEASE}: gcloud compute images create --storage-location europe-west1 openshift-${OPENSHIFT_DASH_RELEASE}-disk --source-uri gs://openshift-${OPENSHIFT_DASH_RELEASE}/crc.tar.gz

  8. Confirm that the new image is available in GCP: gcloud compute images list --no-standard-images | grep openshift

GCP Image Build

Establish a build script and automation for converting the image ending with "-disk" into a pre-booted image with a name that ends with "-GUID" (use GCP cloud build, packer, or ssh?). The "-GUID" version will be booted by instruqt

  1. Launch a GCE instace using the new image: gcloud compute instances create openshift-${OPENSHIFT_DASH_RELEASE}-build --image openshift-${OPENSHIFT_DASH_RELEASE}-disk --zone=europe-west1-c --machine-type n1-standard-8
  2. ssh -i ~/.crc/cache/crc_libvirt_${OPENSHIFT_RELEASE}_amd64/id_ecdsa_crc core@EXTERNAL_IP
  3. sudo su - then run the following script: https://github.com/openshift-labs/learn-instruqt/blob/main/scripts/build/setup.sh
  4. systemctl reset-failed
  5. exit
  6. Add pull secret to openshift-config namespace: https://github.com/openshift-labs/learn-instruqt/blob/main/scripts/pullsecret.json
  7. sudo chown core.core /var/lib/kubelet/config.json
  8. https://docs.google.com/document/d/1Iz2nAfhz4_G0W1L50CelDckdXjx6WByD4D4lRE4eEko/edit
  9. export KUBECONFIG=/opt/kubeconfig
  10. oc delete secret pull-secret -n openshift-config && oc create secret generic pull-secret --from-file=.dockerconfigjson=/var/lib/kubelet/config.json --type=kubernetes.io/dockerconfigjson -n openshift-config
  11. Add admin, developer, and user1 accounts: https://github.com/openshift-instruqt/instruqt/pull/117/files#diff-e438bfd9dad2f72ff1626b329d6d466d2afc779b8209f3866498f64fd1742dc0R42-R68
  12. history -c
  13. sudo shutdown -h now
  14. Wait for the VM to shut down, then generate a new image from the updated VM disk gcloud compute images create openshift-${OPENSHIFT_DASH_RELEASE}-GUID --project=rhd-devx-instruqt --source-disk=openshift-${OPENSHIFT_DASH_RELEASE}-build --source-disk-zone=europe-west1-c --storage-location=europe-west1

Optimization goals

Client Image Build

Image startup optimizations

ryanj commented 2 years ago

^ these notes should be moved to a markdown doc somewhere in the repo

blues-man commented 2 years ago

@ryanj I was able to keep the same hostname while recreating the new image like this:

hostnamectl set-hostname --static crc-gh9wd-master-0
hostnamectl set-hostname --transient crc-gh9wd-master-0

@ryanj actually the hostname then mismatches with kubelet:

[root@crc-gh9wd-master-0 ~]# oc get nodes
NAME                 STATUS     ROLES           AGE   VERSION
crc-5nvrm-master-0   NotReady   master,worker   11d   v1.23.5+3afdacb
praveenkumar commented 2 years ago

About CRC VM image extraction step I think you don't need the vpn to download the bundle because after crc release it is publicly available and if you want some different bundle then only use vpn and get the linux bundle from internal server http://cdk-builds.usersys.redhat.com/builds/crc/bundles/openshift/ location.

Downloading bundle for a latest crc release should be simply.

$ openshiftVersion=$(curl -sL https://mirror.openshift.com/pub/openshift-v4/clients/crc/latest/release-info.json | jq -r .version.openshiftVersion)
$ curl -L -O https://mirror.openshift.com/pub/openshift-v4/clients/crc/bundles/openshift/"${openshiftVersion}"/crc_libvirt_"${openshiftVersion}"_amd64.crcbundle
ryanj commented 2 weeks ago

Ping me for access to the final build doc