search

crc-org / crc

CRC is a tool to help you run containers. It manages a local OpenShift 4.x cluster, Microshift or a Podman VM optimized for testing and development purposes
https://crc.dev
Apache License 2.0
1.26k stars 241 forks source link

[BUG] x509 ingress controller error when connecting from macOS #3375

Closed KevinMGranger closed 2 years ago

KevinMGranger commented 2 years ago

Note: crc-related information is on the linux machine, and oc-related information is on the macOS machine.

I also embedded various files and log outputs for convenience. I can move them to a gist if you wish.

I actually wrote this up for the first time on September 12th but never posted it. Now that the M1 is finally supported, I tried running it locally on my mac-- and I'm getting the same error!

General information

CRC version

linux$ crc version
CRC version: 2.8.0+217b3bd
OpenShift version: 4.11.1
Podman version: 4.1.1

OC version

mac$ oc version
Client Version: 4.11.1
Kustomize Version: v4.5.4
error: You must be logged in to the server (Unauthorized)

CRC status

linux$ crc status --log-level debug
level=debug msg="CRC version: 2.8.0+217b3bd\n"
level=debug msg="OpenShift version: 4.11.1\n"
level=debug msg="Podman version: 4.1.1\n"
level=debug msg="Running 'crc status'"
level=debug msg="Checking file: /home/kevin/.crc/machines/crc/.crc-exist"
level=debug msg="Checking file: /home/kevin/.crc/machines/crc/.crc-exist"
level=debug msg="Found binary path at /home/kevin/.crc/bin/crc-driver-libvirt"
level=debug msg="Launching plugin server for driver libvirt"
level=debug msg="Plugin server listening at address 127.0.0.1:41339"
level=debug msg="() Calling .GetVersion"
level=debug msg="Using API Version 1"
level=debug msg="() Calling .SetConfigRaw"
level=debug msg="() Calling .GetMachineName"
level=debug msg="(crc) Calling .GetBundleName"
level=debug msg="(crc) Calling .GetState"
level=debug msg="(crc) DBG | time=\"2022-09-12T16:28:22-04:00\" level=debug msg=\"Getting current state...\""
level=debug msg="(crc) DBG | time=\"2022-09-12T16:28:22-04:00\" level=debug msg=\"Fetching VM...\""
level=debug msg="(crc) Calling .GetIP"
level=debug msg="(crc) DBG | time=\"2022-09-12T16:28:22-04:00\" level=debug msg=\"GetIP called for crc\""
level=debug msg="(crc) DBG | time=\"2022-09-12T16:28:22-04:00\" level=debug msg=\"Getting current state...\""
level=debug msg="(crc) DBG | time=\"2022-09-12T16:28:22-04:00\" level=debug msg=\"IP address: 192.168.130.11\""
level=debug msg="(crc) Calling .GetIP"
level=debug msg="(crc) DBG | time=\"2022-09-12T16:28:22-04:00\" level=debug msg=\"GetIP called for crc\""
level=debug msg="(crc) DBG | time=\"2022-09-12T16:28:22-04:00\" level=debug msg=\"Getting current state...\""
level=debug msg="(crc) DBG | time=\"2022-09-12T16:28:22-04:00\" level=debug msg=\"IP address: 192.168.130.11\""
level=debug msg="Running SSH command: df -B1 --output=size,used,target /sysroot | tail -1"
level=debug msg="Using ssh private keys: [/home/kevin/.crc/machines/crc/id_ecdsa /home/kevin/.crc/cache/crc_libvirt_4.11.1_amd64/id_ecdsa_crc]"
level=debug msg="SSH command results: err: <nil>, output: 68171051008 15629176832 /sysroot\n"
level=debug msg="Making call to close driver server"
level=debug msg="(crc) Calling .Close"
level=debug msg="Successfully made call to close driver server"
level=debug msg="Making call to close connection to plugin binary"
CRC VM:          Running
OpenShift:       Running (v4.11.1)
Podman:          
Disk Usage:      15.63GB of 68.17GB (Inside the CRC VM)
Cache Usage:     17.04GB
Cache Directory: /home/kevin/.crc/cache

CRC config

linux$ crc config view
- consent-telemetry                     : yes
- disk-size                             : 64
- memory                                : 20480
- pull-secret-file                      : /home/kevin/crc_pull_secret

Host Operating System

Linux

linux$ cat /etc/os-release
NAME="Fedora Linux"
VERSION="35 (Server Edition)"
ID=fedora
VERSION_ID=35
VERSION_CODENAME=""
PLATFORM_ID="platform:f35"
PRETTY_NAME="Fedora Linux 35 (Server Edition)"
ANSI_COLOR="0;38;2;60;110;180"
LOGO=fedora-logo-icon
CPE_NAME="cpe:/o:fedoraproject:fedora:35"
HOME_URL="https://fedoraproject.org/"
DOCUMENTATION_URL="https://docs.fedoraproject.org/en-US/fedora/f35/system-administrators-guide/"
SUPPORT_URL="https://ask.fedoraproject.org/"
BUG_REPORT_URL="https://bugzilla.redhat.com/"
REDHAT_BUGZILLA_PRODUCT="Fedora"
REDHAT_BUGZILLA_PRODUCT_VERSION=35
REDHAT_SUPPORT_PRODUCT="Fedora"
REDHAT_SUPPORT_PRODUCT_VERSION=35
PRIVACY_POLICY_URL="https://fedoraproject.org/wiki/Legal:PrivacyPolicy"
VARIANT="Server Edition"
VARIANT_ID=server

macOS

mac$ sw_vers
ProductName:    macOS
ProductVersion: 12.5.1
BuildVersion:   21G83

topology details

There are three machines, each with different purposes:

  1. a macOS machine, used for day-to-day work. I run oc from it. Let's call this one mac
  2. a powerful Linux desktop, used to run crc. Let's call this one linux.
  3. a low-power Linux computer, used to run DNS (dnsmasq). Let's call this one dns-linux.

All three machines use the third for DNS configuration.

They are connected over tailscale, although that shouldn't matter much.

Other relevant config files

kubeconfig on linux

~/.crc/machines/crc/kubeconfig ```yaml apiVersion: v1 clusters: - cluster: certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURXekNDQWtPZ0F3SUJBZ0lJVEU2NHBCWlQzVzR3RFFZSktvWklodmNOQVFFTEJRQXdKakVrTUNJR0ExVUUKQXd3YmFXNW5jbVZ6Y3kxdmNHVnlZWFJ2Y2tBeE5qWXhNekUxTURnek1CNFhEVEl5TURneU5EQTBNalEwTkZvWApEVEkwTURneU16QTBNalEwTlZvd0hURWJNQmtHQTFVRUF3d1NLaTVoY0hCekxXTnlZeTUwWlhOMGFXNW5NSUlCCklqQU5CZ2txaGtpRzl3MEJBUUVGQUFPQ0FROEFNSUlCQ2dLQ0FRRUF6MW5kK2tWL090VmRITzVQbkxubkNGLzcKczBkMzkxY0J4TGkxQzcyS0lIVEtxWFJ1eXNYQjBYbFdiT1JRZ0o3b3dieTIxbGR6bVBFdWxoejZlZFdOcXZFMwp6SEloNDViSFpBSUdCaDN1UHB4SFhwa0VING5xY2NLd3hna3EyZEdwZVJhN3NVUUI1T1Z5NVVUellTYkxIU1JoCmduMjhlcDVtUE81MzBhK0FnRVVVMy84NTRlbVZyMm5XL29YZVJMZmpYZVUvSzBrNkJpQ2Y3dFpEcUtub0tHUHQKRC8xbjBBZmlHclNUK0NUNllWS281S0VSUTFPa2V0Mm56Mnd3Y1krWkZBTCt0L2trMW52WXI5WVlLUlJJRngxRQp6M0t0TUEzTk9ldnEwZ3VGeTVKZkdHN3dKZ2lMRWNtZDZZVjNzbjY1RllqNjJ5dTZ6ajdDMkNjOUFIR1NJd0lECkFRQUJvNEdWTUlHU01BNEdBMVVkRHdFQi93UUVBd0lGb0RBVEJnTlZIU1VFRERBS0JnZ3JCZ0VGQlFjREFUQU0KQmdOVkhSTUJBZjhFQWpBQU1CMEdBMVVkRGdRV0JCUTNBL24zQk5nRXMrWm9pdEZFc0d6cVFCSExYekFmQmdOVgpIU01FR0RBV2dCUWllaTRIZG5BVzgzVVh2Um1UdllsYkJ4ZmlqVEFkQmdOVkhSRUVGakFVZ2hJcUxtRndjSE10ClkzSmpMblJsYzNScGJtY3dEUVlKS29aSWh2Y05BUUVMQlFBRGdnRUJBQ2k2aDgyYVRiQ1RpNU9UV09TWTk3bTIKb1FUckJGQ3BLWHdBYWMvdEtjc1V4ZUZKZHBHaFhHa2ZnMklPWXMvRldORzdGNU4wZTJkQ2lROXQvL2x2UlVhTAp3NkRnYWZYZ1BldFdvb2svMGpJakJKWi9oK3RMNjlhOE5qcWcxb2ZWem5JcmM5eE5kRldCSzdpVklraUpRV0k0Cmwyc2xBWElnQUVQUnJLYmtyWXN4UkRXcXlkL3l6eDNOZTJQaldOZUFpV1h1MS9XNTRyWC9QRVg1cmxpSVR4WkcKWU1tSG4vWnp3VXk1UlFwTjZhMEhkTTR3akNGdTFxUFFJdjd3S21nVXJRQnVURFp4ZSt1VmpabVlGYXdtSkQ0VApPdVZLK2h1NjdJTnZsdzJhT2I4dDJROU5JVmRzRWJjTStuRVQxNjh3ZEZpL3BvSFMyOFk5WGV0LzIrSlcvdUE9Ci0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0KLS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURERENDQWZTZ0F3SUJBZ0lCQVRBTkJna3Foa2lHOXcwQkFRc0ZBREFtTVNRd0lnWURWUVFEREJ0cGJtZHkKWlhOekxXOXdaWEpoZEc5eVFERTJOakV6TVRVd09ETXdIaGNOTWpJd09ESTBNRFF5TkRReVdoY05NalF3T0RJegpNRFF5TkRReldqQW1NU1F3SWdZRFZRUUREQnRwYm1keVpYTnpMVzl3WlhKaGRHOXlRREUyTmpFek1UVXdPRE13CmdnRWlNQTBHQ1NxR1NJYjNEUUVCQVFVQUE0SUJEd0F3Z2dFS0FvSUJBUURmUVRMQWNFdExNYXdnaVQyU1FOLzcKdEttbm9rN1gwMll0S0NVU3ZRVjBuUWhMZGZ0c3RocGZVdTdqa3lCRXRhSTI4ajlJbXU1L1Q0QU1yUE5vOEhpcgpwNkhLbHB3cDVubjM3REpwUFJ0ZHc3dmd5UmdqRmpxcmR0Vk5VWXJidWFTdEJ3WTl2c2ppNFhvNTR1OHMxWkpGCjNOOWJ2c20rMm15SnZEd0VyNG5OZXZwZDV6WDlqR1g1UU1oS1RhdHFsMTJoaXNVYWNJRTcvaFgzdUoyZXZxcU8KcGtqVU93Ri8yVlNVbkd4Qmg5NHpXRVNUNlI2SVcrUFo5Zlo0d25kRFVzK3laWVZCdmIxRGVaQ3NLQW9EU0l1Ugpnc2pvbzYyKzZqR0Q2bDl1QklQVlE5bnozRFgzeW1FTEdjbzR5TFh0eGlkS282Y0sraUwyZTRMblB2QWFZRVRUCkFnTUJBQUdqUlRCRE1BNEdBMVVkRHdFQi93UUVBd0lDcERBU0JnTlZIUk1CQWY4RUNEQUdBUUgvQWdFQU1CMEcKQTFVZERnUVdCQlFpZWk0SGRuQVc4M1VYdlJtVHZZbGJCeGZpalRBTkJna3Foa2lHOXcwQkFRc0ZBQU9DQVFFQQpGTjhZRDhFeTNUMVpROGdrS0Y4VFZSVEdwZG5aTmlLUjhtVjZaOEt5WUtyNVpETUJHd29tTXVWZUw3di85NlBOCmtRSUhGUzZ4QzhnSkxrUjFpdkZJZFlHdmxqN3dvbXcxS0p4V0k5SlBGZnVWTzhIbU9NaDNzNCtJUXRyV1QvYlgKSDZMWVlLLzByWXNiMDIzTGtYUTJGdU1iRzNxeWJEcnJCc3crNDlFVmkya3lRbHF2ZExBSUEwNEtKdExuL0RaagpOZEJyc3k4a0tzaEhyOGJpdnpsT2E4V0xDTDBlcTBUb3NrT1hOUUlYMmNXVVRGMlNvcUV3QmF4a2ZzdDUxTUduCjZHM1FrZmhYRDdXYk8rTm5OTmJjbTFxeHhQd2EzdUFBUXo3N3BiMUNJTG91MHVqdWtLM3h5MTYwWlVaVzNGSWIKQWV2ejFxYzJqckJTU1NLYjFSdTNVUT09Ci0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0KLS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURXekNDQWtPZ0F3SUJBZ0lJVEU2NHBCWlQzVzR3RFFZSktvWklodmNOQVFFTEJRQXdKakVrTUNJR0ExVUUKQXd3YmFXNW5jbVZ6Y3kxdmNHVnlZWFJ2Y2tBeE5qWXhNekUxTURnek1CNFhEVEl5TURneU5EQTBNalEwTkZvWApEVEkwTURneU16QTBNalEwTlZvd0hURWJNQmtHQTFVRUF3d1NLaTVoY0hCekxXTnlZeTUwWlhOMGFXNW5NSUlCCklqQU5CZ2txaGtpRzl3MEJBUUVGQUFPQ0FROEFNSUlCQ2dLQ0FRRUF6MW5kK2tWL090VmRITzVQbkxubkNGLzcKczBkMzkxY0J4TGkxQzcyS0lIVEtxWFJ1eXNYQjBYbFdiT1JRZ0o3b3dieTIxbGR6bVBFdWxoejZlZFdOcXZFMwp6SEloNDViSFpBSUdCaDN1UHB4SFhwa0VING5xY2NLd3hna3EyZEdwZVJhN3NVUUI1T1Z5NVVUellTYkxIU1JoCmduMjhlcDVtUE81MzBhK0FnRVVVMy84NTRlbVZyMm5XL29YZVJMZmpYZVUvSzBrNkJpQ2Y3dFpEcUtub0tHUHQKRC8xbjBBZmlHclNUK0NUNllWS281S0VSUTFPa2V0Mm56Mnd3Y1krWkZBTCt0L2trMW52WXI5WVlLUlJJRngxRQp6M0t0TUEzTk9ldnEwZ3VGeTVKZkdHN3dKZ2lMRWNtZDZZVjNzbjY1RllqNjJ5dTZ6ajdDMkNjOUFIR1NJd0lECkFRQUJvNEdWTUlHU01BNEdBMVVkRHdFQi93UUVBd0lGb0RBVEJnTlZIU1VFRERBS0JnZ3JCZ0VGQlFjREFUQU0KQmdOVkhSTUJBZjhFQWpBQU1CMEdBMVVkRGdRV0JCUTNBL24zQk5nRXMrWm9pdEZFc0d6cVFCSExYekFmQmdOVgpIU01FR0RBV2dCUWllaTRIZG5BVzgzVVh2Um1UdllsYkJ4ZmlqVEFkQmdOVkhSRUVGakFVZ2hJcUxtRndjSE10ClkzSmpMblJsYzNScGJtY3dEUVlKS29aSWh2Y05BUUVMQlFBRGdnRUJBQ2k2aDgyYVRiQ1RpNU9UV09TWTk3bTIKb1FUckJGQ3BLWHdBYWMvdEtjc1V4ZUZKZHBHaFhHa2ZnMklPWXMvRldORzdGNU4wZTJkQ2lROXQvL2x2UlVhTAp3NkRnYWZYZ1BldFdvb2svMGpJakJKWi9oK3RMNjlhOE5qcWcxb2ZWem5JcmM5eE5kRldCSzdpVklraUpRV0k0Cmwyc2xBWElnQUVQUnJLYmtyWXN4UkRXcXlkL3l6eDNOZTJQaldOZUFpV1h1MS9XNTRyWC9QRVg1cmxpSVR4WkcKWU1tSG4vWnp3VXk1UlFwTjZhMEhkTTR3akNGdTFxUFFJdjd3S21nVXJRQnVURFp4ZSt1VmpabVlGYXdtSkQ0VApPdVZLK2h1NjdJTnZsdzJhT2I4dDJROU5JVmRzRWJjTStuRVQxNjh3ZEZpL3BvSFMyOFk5WGV0LzIrSlcvdUE9Ci0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0KLS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURERENDQWZTZ0F3SUJBZ0lCQVRBTkJna3Foa2lHOXcwQkFRc0ZBREFtTVNRd0lnWURWUVFEREJ0cGJtZHkKWlhOekxXOXdaWEpoZEc5eVFERTJOakV6TVRVd09ETXdIaGNOTWpJd09ESTBNRFF5TkRReVdoY05NalF3T0RJegpNRFF5TkRReldqQW1NU1F3SWdZRFZRUUREQnRwYm1keVpYTnpMVzl3WlhKaGRHOXlRREUyTmpFek1UVXdPRE13CmdnRWlNQTBHQ1NxR1NJYjNEUUVCQVFVQUE0SUJEd0F3Z2dFS0FvSUJBUURmUVRMQWNFdExNYXdnaVQyU1FOLzcKdEttbm9rN1gwMll0S0NVU3ZRVjBuUWhMZGZ0c3RocGZVdTdqa3lCRXRhSTI4ajlJbXU1L1Q0QU1yUE5vOEhpcgpwNkhLbHB3cDVubjM3REpwUFJ0ZHc3dmd5UmdqRmpxcmR0Vk5VWXJidWFTdEJ3WTl2c2ppNFhvNTR1OHMxWkpGCjNOOWJ2c20rMm15SnZEd0VyNG5OZXZwZDV6WDlqR1g1UU1oS1RhdHFsMTJoaXNVYWNJRTcvaFgzdUoyZXZxcU8KcGtqVU93Ri8yVlNVbkd4Qmg5NHpXRVNUNlI2SVcrUFo5Zlo0d25kRFVzK3laWVZCdmIxRGVaQ3NLQW9EU0l1Ugpnc2pvbzYyKzZqR0Q2bDl1QklQVlE5bnozRFgzeW1FTEdjbzR5TFh0eGlkS282Y0sraUwyZTRMblB2QWFZRVRUCkFnTUJBQUdqUlRCRE1BNEdBMVVkRHdFQi93UUVBd0lDcERBU0JnTlZIUk1CQWY4RUNEQUdBUUgvQWdFQU1CMEcKQTFVZERnUVdCQlFpZWk0SGRuQVc4M1VYdlJtVHZZbGJCeGZpalRBTkJna3Foa2lHOXcwQkFRc0ZBQU9DQVFFQQpGTjhZRDhFeTNUMVpROGdrS0Y4VFZSVEdwZG5aTmlLUjhtVjZaOEt5WUtyNVpETUJHd29tTXVWZUw3di85NlBOCmtRSUhGUzZ4QzhnSkxrUjFpdkZJZFlHdmxqN3dvbXcxS0p4V0k5SlBGZnVWTzhIbU9NaDNzNCtJUXRyV1QvYlgKSDZMWVlLLzByWXNiMDIzTGtYUTJGdU1iRzNxeWJEcnJCc3crNDlFVmkya3lRbHF2ZExBSUEwNEtKdExuL0RaagpOZEJyc3k4a0tzaEhyOGJpdnpsT2E4V0xDTDBlcTBUb3NrT1hOUUlYMmNXVVRGMlNvcUV3QmF4a2ZzdDUxTUduCjZHM1FrZmhYRDdXYk8rTm5OTmJjbTFxeHhQd2EzdUFBUXo3N3BiMUNJTG91MHVqdWtLM3h5MTYwWlVaVzNGSWIKQWV2ejFxYzJqckJTU1NLYjFSdTNVUT09Ci0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0KLS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURRRENDQWlpZ0F3SUJBZ0lJSFFXRC9nRU9YWkl3RFFZSktvWklodmNOQVFFTEJRQXdQakVTTUJBR0ExVUUKQ3hNSmIzQmxibk5vYVdaME1TZ3dKZ1lEVlFRREV4OXJkV0psTFdGd2FYTmxjblpsY2kxc2IyTmhiR2h2YzNRdApjMmxuYm1WeU1CNFhEVEl5TURneU5EQTBNVEF5TVZvWERUTXlNRGd5TVRBME1UQXlNVm93UGpFU01CQUdBMVVFCkN4TUpiM0JsYm5Ob2FXWjBNU2d3SmdZRFZRUURFeDlyZFdKbExXRndhWE5sY25abGNpMXNiMk5oYkdodmMzUXQKYzJsbmJtVnlNSUlCSWpBTkJna3Foa2lHOXcwQkFRRUZBQU9DQVE4QU1JSUJDZ0tDQVFFQXBpTnNaa3pWYUxTMgpFMjNmVDl4bHFRVGlQdUxnUkU0MDFjNFVsVE9jTXI2UjhqNVo2cnYwbjlvYTEzdzRocFFUZjJNeVl1cmg4WCs5CnliL3diL2wvRVBENzByS09hUHhqbUhJWVpQZXkrZHJaT3BwTm5DMUwzUHZlcVlzYVNYL2NQTm10cFRBbGZEZUkKazNoZE4vaUlreDhSemNFMU1qbWxhWjgxTUN3M1BieWV2d1hHZUV1aEtvRmE5NVhlcnU1UnR1WlAvS2V1UWZCUApoR2h2T1B5d3Q2SWd0OHlrK09LdXd2blU3SEd2dWpjelJEZDVYT1ExbHZ6eUttNXJIMTRXRjM5Y0N6cjFpNnFsCjBlQisrVWdyTXYwaWp1WjQ4Y2dZalE3R0daT2VJMzZhelJqUVFiamtLbU5xRHlvWGJCdkxKZzdqRUFwdnYvVEkKRnMyWjJzMjBBUUlEQVFBQm8wSXdRREFPQmdOVkhROEJBZjhFQkFNQ0FxUXdEd1lEVlIwVEFRSC9CQVV3QXdFQgovekFkQmdOVkhRNEVGZ1FVNlFkc3ZYbzdDZlhOejEvaW8rWG80TyszRVl3d0RRWUpLb1pJaHZjTkFRRUxCUUFECmdnRUJBRUpYamlIMjNJMTlFOGpmcmlvV3hJbWZLTFpaRFFiYVRpSVJaTXZPb3ltWHcyNnhZL2RMTm1JK3doZmsKYjNZWWpDdjFJZWc5S3lFaDNXTlVraytZQ3hQMCtzUkRwaHNmcUNhSWI0RG9NTnR6VTl6RmRoMmYwOWpxR2FDaApCVDM0T1pGQVQ4Um5MTXVrTTYzc1NpV0VVRlhrMWtucXVCSEpBU0NkbzBvM2ZGOUt1VzcwTXhtVlRodFJzaTBpCnAvMC8vVVNFZDhLVXQ4M2YvMzhQTWI5TW13T1RQUjFlR0ZOOW5uek95NjlSUm1aSW5tc3E3cE9qSGZ3eVVVZXQKbzhXMmhoYU9qVnMxWDNrdHA4YTdnaCtRYTZkQ09GQWdPR01xTXREbXAyNTIxbVhzTXdlWm9iQzU2K0NjLzhocgo5MVptN3dyMFYyQ3NWd0dIc2I0NTl0S1ZQRzA9Ci0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0KLS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURURENDQWpTZ0F3SUJBZ0lJSm5nWmVhcENKTGN3RFFZSktvWklodmNOQVFFTEJRQXdSREVTTUJBR0ExVUUKQ3hNSmIzQmxibk5vYVdaME1TNHdMQVlEVlFRREV5VnJkV0psTFdGd2FYTmxjblpsY2kxelpYSjJhV05sTFc1bApkSGR2Y21zdGMybG5ibVZ5TUI0WERUSXlNRGd5TkRBME1UQXlNbG9YRFRNeU1EZ3lNVEEwTVRBeU1sb3dSREVTCk1CQUdBMVVFQ3hNSmIzQmxibk5vYVdaME1TNHdMQVlEVlFRREV5VnJkV0psTFdGd2FYTmxjblpsY2kxelpYSjIKYVdObExXNWxkSGR2Y21zdGMybG5ibVZ5TUlJQklqQU5CZ2txaGtpRzl3MEJBUUVGQUFPQ0FROEFNSUlCQ2dLQwpBUUVBM2NnTlF1ajVRbVhXRkxUMFFmTkF0blJxTks5RUlQWFBiSGFQdUJzWjRXV2YrWFdJK3dUUEhLNWlIYTRpCjZCM2xqWnlBUFd4UDhzSmNQbjl3ODRtZGJpd3VzdzIyVXMxdlo2Umw1dVZ0ME1DUGR4YUhqWGo3UThJSEc3UG8KSEJhWTRwOGdhMVFDS2docmpRMHJ2clEwZ2lZejR0cU81cGpFdSt0RmVQR3hnK01DeE03YmFRVUNNYUtQdnRldgpONDBiTVVNYy9raTNTRVV3QnRQelRxTlV0Y08wa0xDeEkyVGdBQ3NpU0daZlVoZ1kxMUxxMTlieEd3MnBQMHFkCnhrMjM1ZFcwTFBTYVZlY3FGelVEbGJqd3VWQXRweHhmRS9uc0ZzTVMvOWdwdjUvM21XdkpIUGc0WUc3M3Q0RWQKUGxhOGJSc3g4WFh2Nk4yN0cveGZMUy9kcHdJREFRQUJvMEl3UURBT0JnTlZIUThCQWY4RUJBTUNBcVF3RHdZRApWUjBUQVFIL0JBVXdBd0VCL3pBZEJnTlZIUTRFRmdRVXpTb3BnMURmOFZacjdXeGZ4Y1JiV1NjWjZIWXdEUVlKCktvWklodmNOQVFFTEJRQURnZ0VCQUZhdzJsV3ZqNTlrc3FTb2xMN2xCYWU5TXkrWUF4S0NOWjhQTGg4SUx5dWoKY3c4a0lDNUt0KzQ4cjNZWWZVcUlNd3B6QTV4cStVNnJNa3FRSnk0aSswbmsrdEtCZkplV3hPTlQ0Ui9xVVNEOQpKVFN5R1dwK2hTYkF2alRieHNKZnRrMjFkTFVSOXdCNGluZWdsbXkrQ1FyeFFzaGJMb1VCVVVtSmhjYTd3VmFHClZjK2FRdmZPZ3lGc2VQVFQrbVNFWWYwTEJEWnUzR2xMQnMvRElCTVAyNHYzV2hEZDRXeW4yd0RzSWdEd0c4UVMKZlFmaTFZdkE2T3c0NWsyb2dFODluM3kyUW5LMHVDNDlDeGt5Mnhmc3RTdmpoWWdEOGdwZkxhMU1wL2d2VVpRQgpGclg3T2NVTWMyTzFqeDlGam51NllxSEFzeUZpc2h6Um05K3pjRXVWaG4wPQotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCi0tLS0tQkVHSU4gQ0VSVElGSUNBVEUtLS0tLQpNSUlETWpDQ0FocWdBd0lCQWdJSVhCY0tZcytqbFdFd0RRWUpLb1pJaHZjTkFRRUxCUUF3TnpFU01CQUdBMVVFCkN4TUpiM0JsYm5Ob2FXWjBNU0V3SHdZRFZRUURFeGhyZFdKbExXRndhWE5sY25abGNpMXNZaTF6YVdkdVpYSXcKSGhjTk1qSXdPREkwTURReE1ESXlXaGNOTXpJd09ESXhNRFF4TURJeVdqQTNNUkl3RUFZRFZRUUxFd2x2Y0dWdQpjMmhwWm5ReElUQWZCZ05WQkFNVEdHdDFZbVV0WVhCcGMyVnlkbVZ5TFd4aUxYTnBaMjVsY2pDQ0FTSXdEUVlKCktvWklodmNOQVFFQkJRQURnZ0VQQURDQ0FRb0NnZ0VCQU1PSGROaUZ3SnR6LzNXeVFvcUM2d1JjR3FnazRpTHQKSDNOYm15LzE1L1AvVVA5cWZvMDlYOTBGRUF4K043cUR2a29nd3BTTEZPWjVRMFdQYzBjVS9LVENkbWdhNW1YWgpsdFR6L1pDTWNGODN2OVdIMHdGcGtXUXBVRzE1OVZQdmNXTHpjTDJjeVJJb21SZld1R21Ca3FIUXNKdGNtV0s4Cmc2VjIzYncyc2FyR093ejgrVU9zNm44bDBReFJqSWovRmJJa3RUclk5VU1LQldPZlE5ODAweXdBd1pJU3FmbU8KVWZBa0tGUUwxR29aRXZaMkQ1Q1pjM0hGY1FrYmYwLzNidmMxZUZNYXI1MUNXRGNXNkZIbCtXYVhsTnlLUU1INQpnYmxFT3p4TkpXM2doNHhPdEhzczF6eC9qOFNFU3pxcldrbGR5YkZQT0diQTdQd2gzQTU0QTRVQ0F3RUFBYU5DCk1FQXdEZ1lEVlIwUEFRSC9CQVFEQWdLa01BOEdBMVVkRXdFQi93UUZNQU1CQWY4d0hRWURWUjBPQkJZRUZKejQKekVMNkJMTTRXb001UDFSSDBBOWtWM210TUEwR0NTcUdTSWIzRFFFQkN3VUFBNElCQVFDQXQ3ZUJRNXVuY21rTwp0SThPV251aGFxclpFbmtIcGdOelRYL1BWM0JXS2xaUlY1VTdYZmhjUkNXcUw1QitrVGJ1d0d2cytWV3lQKzZGCndRWWgrZndLTVcwYjA0V2pJa3QvOVFyV3V2dXU3YUdKZG5VNW0wbzJoWW9oU0JFUzVwUU9vbXJUR09MVXVDSWcKWmZEOHRGbXJZaGc2QWM3WWFJcUZWRldGcFgycCs3SlBZT1FLNnlaaWhMRnFVQTRYRU5hQ0taekFGcFVBdUQ3MgpxSUlTM2F2cGp1dStZdy9zSzFpZE5FZ1dLdHlYR2ZkTFZ5WldTZmJ5Vk5lcklIYWNBVVRzeldyTHQ4RmNiNGROClB4b0ErVmhycUZ3MFY4RHpiM1Y2bjJ6VWpYSmJmUjRlWW9SUjVKMlVrU0IzRC8xdTVGMllVRytROUhYRkxYQTMKOHhsV0lOdisKLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo= server: https://api.crc.testing:6443 name: crc contexts: - context: cluster: crc user: admin name: admin current-context: admin kind: Config preferences: {} users: - name: admin user: client-certificate-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURpRENDQW5DZ0F3SUJBZ0lJR1VhVllvMUw0b1F3RFFZSktvWklodmNOQVFFTEJRQXdQVEVTTUJBR0ExVUUKQ3hNSmIzQmxibk5vYVdaME1TY3dKUVlEVlFRREV4NWhaRzFwYmkxcmRXSmxZMjl1Wm1sbkxYTnBaMjVsY2kxagpkWE4wYjIwd0hoY05Nakl3T1RFeU1qQXdOVE0xV2hjTk16SXdPVEE1TWpBd05UTTFXakF3TVJjd0ZRWURWUVFMCkV3NXplWE4wWlcwNmJXRnpkR1Z5Y3pFVk1CTUdBMVVFQXhNTWMzbHpkR1Z0T21Ga2JXbHVNSUlCSWpBTkJna3EKaGtpRzl3MEJBUUVGQUFPQ0FROEFNSUlCQ2dLQ0FRRUF5VG54bHNHci9POExIWk1DUFEvcUd0YW8wMEtrZmRFbgoydC9wa0JsSVhSVEhoL2IxV2ptWTZ5VzZIdGRZTFFaRlNmczdRU0MvY0pqN2c2YTdHc2htbUtjWHh4ZGEwa3lZCkhKVG5vZ3lPRGVLVzVXVWYwYy9CL0pGSjhYZGJuQ3crZXd3NjBzMmxESUxDYTVOZjN3M2JKSm1EM2dQS3l0MlkKQ2g3dHNueWlsV0RrRFZoMjRRY09BcDlzeC9pMHF6M2x5b2tlcktZTXRFeThKaW1EaGxNMjlrWkZ6SDNSMy93NApkRHlLbXBUNUVIajZ1ajUzSEhwNnVZMk0rbUtjT01HU3liMjJwRFB6SXZueFNZZmZiNWcxQ3RzR2I5bXRJc2RJCmFIdVpUZ28rWjFRTUlTUzVRMSswV2NxbjJIZDErRzZ2T04vU3lpZjV1UDV1VzhQeGxxTzhGUUlEQVFBQm80R1kKTUlHVk1BNEdBMVVkRHdFQi93UUVBd0lGb0RBZEJnTlZIU1VFRmpBVUJnZ3JCZ0VGQlFjREFRWUlLd1lCQlFVSApBd0l3REFZRFZSMFRBUUgvQkFJd0FEQXBCZ05WSFE0RUlnUWc4bEZrNHhHd1JkYStsdDBzV1FGOEJueHVvcEdmClhEamU2OE5UQTVhazdCZ3dLd1lEVlIwakJDUXdJb0FnQnQzN21xZEE1bkZIMUxrWkFiMG0wanIxQXo5eUVmYTgKQU8yL3ZTMnVSdUl3RFFZSktvWklodmNOQVFFTEJRQURnZ0VCQUJrRjNydHR6Mmg4RElUd2JzL0luQmVsNkN2dApBK2tRSzkwbWJFTkY4OWp4SnM4dksxczZGSlhpVXdKVk9ZbU5meTczY2dTYmp5TnljcmFrVUhXS05mMDNkSW1yCmpXdVZHL0U4R0xkUEhZTDY0KzhwcW1uU2NvTG0rRFFSU3FROTBuQllRZUs4YnJsVGY0SG9KQkpUeXVGZGFjaUkKUzRXK3F5SEZsb3JZWGNBWEFxWVZQOHdOVmtXbXBqK0w5bEtaZGxmaFMyK2wwSzU2NjVNcUU1WUQzUm4xckNxUQpQY2JkNHhLL2J3KzJpRFFsMXdOK1N3dm9oS2JBcE1aOG1NdmpuMy9kTDBzMTQyMVgvc2crT3hoY2xXeWpRMC96CmFxN3orQk14ekh5RnlBQ0o1SVlqYlFJVTJvbEp1bmtEOHRyRDdCRTIvaXY0emdJdnM1UlU5eXRQTlFnPQotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCg== client-key-data: REDACTED ```

resolv.conf on linux machine (crc host)

linux /etc/resolv.conf ``` # resolv.conf(5) file generated by tailscale # DO NOT EDIT THIS FILE BY HAND -- CHANGES WILL BE OVERWRITTEN nameserver 100.100.100.100 search TAILSCALE_USERNAME_REDACTED.beta.tailscale.net apps-crc.testing api.crc.testing ```

macos dns config

mac /etc/resolv.conf ``` # # macOS Notice # # This file is not consulted for DNS hostname resolution, address # resolution, or the DNS query routing mechanism used by most # processes on this system. # # To view the DNS configuration used by this system, use: # scutil --dns # # SEE ALSO # dns-sd(1), scutil(8) # # This file is automatically generated. # search TAILSCALE_USERNAME_REDACTED.beta.tailscale.net apps-crc.testing api.crc.testing ISP_DNS_DOMAIN_REDACTED nameserver 100.100.100.100 ```
scutil --dns ```console mac$ scutil --dns DNS configuration resolver #1 search domain[0] : TAILSCALE_USERNAME_REDACTED.beta.tailscale.net search domain[1] : apps-crc.testing search domain[2] : api.crc.testing search domain[3] : ISP_DNS_DOMAIN_REDACTED nameserver[0] : 100.100.100.100 if_index : 23 (utun5) flags : Supplemental, Request A records, Request AAAA records reach : 0x00000003 (Reachable,Transient Connection) order : 102600 resolver #2 nameserver[0] : 192.168.1.1 if_index : 13 (en0) flags : Request A records reach : 0x00020002 (Reachable,Directly Reachable Address) order : 200000 resolver #3 domain : TAILSCALE_USERNAME_REDACTED.beta.tailscale.net. nameserver[0] : 100.100.100.100 if_index : 23 (utun5) flags : Supplemental, Request A records, Request AAAA records reach : 0x00000003 (Reachable,Transient Connection) order : 102601 resolver #4 domain : apps-crc.testing. nameserver[0] : 100.100.100.100 if_index : 23 (utun5) flags : Supplemental, Request A records, Request AAAA records reach : 0x00000003 (Reachable,Transient Connection) order : 102602 resolver #5 domain : api.crc.testing. nameserver[0] : 100.100.100.100 if_index : 23 (utun5) flags : Supplemental, Request A records, Request AAAA records reach : 0x00000003 (Reachable,Transient Connection) order : 102603 resolver #6 domain : local options : mdns timeout : 5 flags : Request A records reach : 0x00000000 (Not Reachable) order : 300000 resolver #7 domain : 254.169.in-addr.arpa options : mdns timeout : 5 flags : Request A records reach : 0x00000000 (Not Reachable) order : 300200 resolver #8 domain : 8.e.f.ip6.arpa options : mdns timeout : 5 flags : Request A records reach : 0x00000000 (Not Reachable) order : 300400 resolver #9 domain : 9.e.f.ip6.arpa options : mdns timeout : 5 flags : Request A records reach : 0x00000000 (Not Reachable) order : 300600 resolver #10 domain : a.e.f.ip6.arpa options : mdns timeout : 5 flags : Request A records reach : 0x00000000 (Not Reachable) order : 300800 resolver #11 domain : b.e.f.ip6.arpa options : mdns timeout : 5 flags : Request A records reach : 0x00000000 (Not Reachable) order : 301000 DNS configuration (for scoped queries) resolver #1 search domain[0] : ISP_DNS_DOMAIN_REDACTED nameserver[0] : 192.168.1.1 if_index : 13 (en0) flags : Scoped, Request A records reach : 0x00020002 (Reachable,Directly Reachable Address) resolver #2 search domain[0] : TAILSCALE_USERNAME_REDACTED.beta.tailscale.net search domain[1] : apps-crc.testing search domain[2] : api.crc.testing nameserver[0] : 100.100.100.100 if_index : 23 (utun5) flags : Scoped, Request A records, Request AAAA records reach : 0x00000003 (Reachable,Transient Connection) ```

dnsmasq on linux-dns

/etc/dnsmasq.conf ``` # If you don't want dnsmasq to read /etc/resolv.conf or any other # file, getting its servers from this file instead (see below), then # uncomment this. no-resolv server=192.168.1.1 # If you want dnsmasq to change uid and gid to something other # than the default, edit the following lines. user=dnsmasq group=dnsmasq # If you want dnsmasq to listen for DHCP and DNS requests only on # specified interfaces (and the loopback) give the name of the # interface (eg eth0) here. # Repeat the line for more than one interface. #interface= # Listen only on localhost by default #interface=lo #interface=tailscale0 # Or you can specify which interface _not_ to listen on #except-interface= # Or which to listen on by address (remember to include 127.0.0.1 if # you use this.) listen-address=127.0.0.1 listen-address=192.168.1.63 listen-address=192.168.1.220 listen-address=TAILSCALE_IP_REDACTED # Include all files in /etc/dnsmasq.d except RPM backup files conf-dir=/etc/dnsmasq.d,.rpmnew,.rpmsave,.rpmorig ```
/etc/dnsmasq.d/crc.conf ``` # tailscale address=/apps-crc.testing/TAILSCALE_IP_REDACTED address=/api.crc.testing/TAILSCALE_IP_REDACTED ```

haproxy on linux

/etc/haproxy/haproxy.cfg ``` global log /dev/log local0 defaults balance roundrobin log global maxconn 100 mode tcp timeout connect 5s timeout client 500s timeout server 500s listen apps bind 0.0.0.0:80 server crcvm 192.168.130.11:80 check listen apps_ssl bind 0.0.0.0:443 server crcvm 192.168.130.11:443 check listen api bind 0.0.0.0:6443 server crcvm 192.168.130.11:6443 check ```

Steps to reproduce

Note: these steps are simplified since it started happening locally on macOS too. I can rewrite this issue to reflect that, but it's a lot of work to expose the same issue.

  1. Set up with crc setup
  2. Log in with the login command provided by crc console --credentials

Expected behavior: I am successfully logged in. Actual behavior: x509 errors.

oc login on linux

linux$ oc login -u kubeadmin -p PASSWORD_REDACTED https://api.crc.testing:6443
Login successful.

You have access to 65 projects, the list has been suppressed. You can list all projects with 'oc projects'

Using project "default".
oc login on mac ```console mac$ rm ~/.kube/config mac$ oc login -u kubeadmin https://api.crc.testing:6443 -p PASSWORD_REDACTED --loglevel=9 --insecure-skip-tls-verify I0912 16:57:48.582512 23120 round_trippers.go:466] curl -v -XHEAD 'https://api.crc.testing:6443/' I0912 16:57:53.592736 23120 round_trippers.go:495] HTTP Trace: DNS Lookup for api.crc.testing resolved to [{TAILSCALE_IP_REDACTED }] I0912 16:57:53.597938 23120 round_trippers.go:510] HTTP Trace: Dial to tcp:TAILSCALE_IP_REDACTED:6443 succeed I0912 16:57:53.619555 23120 round_trippers.go:553] HEAD https://api.crc.testing:6443/ in 5036 milliseconds I0912 16:57:53.619593 23120 round_trippers.go:570] HTTP Statistics: DNSLookup 5009 ms Dial 5 ms TLSHandshake 20 ms Duration 5036 ms I0912 16:57:53.619603 23120 round_trippers.go:577] Response Headers: error: x509: “kube-apiserver-lb-signer” certificate is not trusted mac$ scp linux:~/.crc/machines/crc/kubeconfig ~/.kube/config # attempt to use certificate information from here kubeconfig mac$ oc login -u kubeadmin https://api.crc.testing:6443 -p PASSWORD_REDACTED --loglevel=9 --insecure-skip-tls-verify I0912 16:59:35.036954 23283 loader.go:372] Config loaded from file: /Users/kevin/.kube/config I0912 16:59:35.037341 23283 round_trippers.go:466] curl -v -XHEAD 'https://api.crc.testing:6443/' I0912 16:59:35.047021 23283 round_trippers.go:495] HTTP Trace: DNS Lookup for api.crc.testing resolved to [{TAILSCALE_IP_REDACTED }] I0912 16:59:35.050609 23283 round_trippers.go:510] HTTP Trace: Dial to tcp:TAILSCALE_IP_REDACTED:6443 succeed I0912 16:59:35.067353 23283 round_trippers.go:553] HEAD https://api.crc.testing:6443/ 403 Forbidden in 29 milliseconds I0912 16:59:35.067380 23283 round_trippers.go:570] HTTP Statistics: DNSLookup 9 ms Dial 3 ms TLSHandshake 7 ms ServerProcessing 8 ms Duration 29 ms I0912 16:59:35.067386 23283 round_trippers.go:577] Response Headers: I0912 16:59:35.067429 23283 round_trippers.go:580] X-Content-Type-Options: nosniff I0912 16:59:35.067437 23283 round_trippers.go:580] X-Kubernetes-Pf-Flowschema-Uid: 8e4390e2-7947-48f1-b7a5-aca69ea99dbc I0912 16:59:35.067442 23283 round_trippers.go:580] X-Kubernetes-Pf-Prioritylevel-Uid: bfdd957e-32ce-4b55-84aa-209ed9a9ef83 I0912 16:59:35.067446 23283 round_trippers.go:580] Content-Length: 186 I0912 16:59:35.067450 23283 round_trippers.go:580] Date: Mon, 12 Sep 2022 20:59:35 GMT I0912 16:59:35.067454 23283 round_trippers.go:580] Audit-Id: 066a7292-eaf6-480d-ba77-5f8f360bdb4a I0912 16:59:35.067458 23283 round_trippers.go:580] Cache-Control: no-cache, private I0912 16:59:35.067462 23283 round_trippers.go:580] Content-Type: application/json I0912 16:59:35.067725 23283 round_trippers.go:466] curl -v -XGET -H "X-Csrf-Token: 1" 'https://api.crc.testing:6443/.well-known/oauth-authorization-server' I0912 16:59:35.070964 23283 round_trippers.go:553] GET https://api.crc.testing:6443/.well-known/oauth-authorization-server 200 OK in 3 milliseconds I0912 16:59:35.070987 23283 round_trippers.go:570] HTTP Statistics: GetConnection 0 ms ServerProcessing 3 ms Duration 3 ms I0912 16:59:35.070992 23283 round_trippers.go:577] Response Headers: I0912 16:59:35.070998 23283 round_trippers.go:580] Content-Type: application/json I0912 16:59:35.071003 23283 round_trippers.go:580] X-Kubernetes-Pf-Flowschema-Uid: 8e4390e2-7947-48f1-b7a5-aca69ea99dbc I0912 16:59:35.071007 23283 round_trippers.go:580] X-Kubernetes-Pf-Prioritylevel-Uid: bfdd957e-32ce-4b55-84aa-209ed9a9ef83 I0912 16:59:35.071011 23283 round_trippers.go:580] Content-Length: 573 I0912 16:59:35.071015 23283 round_trippers.go:580] Date: Mon, 12 Sep 2022 20:59:35 GMT I0912 16:59:35.071019 23283 round_trippers.go:580] Audit-Id: bce7f502-78f2-4747-a3d1-bf137b79daa8 I0912 16:59:35.071023 23283 round_trippers.go:580] Cache-Control: no-cache, private I0912 16:59:35.108001 23283 request_token.go:477] unexpected error during system roots probe: x509: “ingress-operator@1661315083” certificate is not trusted I0912 16:59:35.108355 23283 round_trippers.go:466] curl -v -XGET -H "Accept: application/json, */*" -H "User-Agent: oc/4.11.0 (darwin/amd64) kubernetes/fcf512e" 'https://api.crc.testing:6443/api/v1/namespaces/openshift/configmaps/motd' I0912 16:59:35.112550 23283 round_trippers.go:553] GET https://api.crc.testing:6443/api/v1/namespaces/openshift/configmaps/motd 403 Forbidden in 4 milliseconds I0912 16:59:35.112575 23283 round_trippers.go:570] HTTP Statistics: GetConnection 0 ms ServerProcessing 4 ms Duration 4 ms I0912 16:59:35.112580 23283 round_trippers.go:577] Response Headers: I0912 16:59:35.112587 23283 round_trippers.go:580] X-Content-Type-Options: nosniff I0912 16:59:35.112591 23283 round_trippers.go:580] X-Kubernetes-Pf-Flowschema-Uid: 8e4390e2-7947-48f1-b7a5-aca69ea99dbc I0912 16:59:35.112595 23283 round_trippers.go:580] X-Kubernetes-Pf-Prioritylevel-Uid: bfdd957e-32ce-4b55-84aa-209ed9a9ef83 I0912 16:59:35.112599 23283 round_trippers.go:580] Content-Length: 303 I0912 16:59:35.112603 23283 round_trippers.go:580] Date: Mon, 12 Sep 2022 20:59:35 GMT I0912 16:59:35.112607 23283 round_trippers.go:580] Audit-Id: df94edd2-1cf5-411e-8195-b16505313f30 I0912 16:59:35.112611 23283 round_trippers.go:580] Cache-Control: no-cache, private I0912 16:59:35.112990 23283 round_trippers.go:580] Content-Type: application/json I0912 16:59:35.113146 23283 request.go:1073] Response Body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"configmaps \"motd\" is forbidden: User \"system:anonymous\" cannot get resource \"configmaps\" in API group \"\" in the namespace \"openshift\"","reason":"Forbidden","details":{"name":"motd","kind":"configmaps"},"code":403} error: x509: “ingress-operator@1661315083” certificate is not trusted ```

Logs

Before beginning, I ran crc delete -f; rm -rf ~/.crc/, and downloaded a fresh version of crc.

crc setup

linux$ crc setup --log-level 9 ``` Successfully configured consent-telemetry to yes Successfully configured pull-secret-file to /home/kevin/crc_pull_secret Changes to configuration property 'memory' are only applied when the CRC instance is started. If you already have a running CRC instance, then for this configuration change to take effect, stop the CRC instance with 'crc stop' and restart it with 'crc start'. Changes to configuration property 'disk-size' are only applied when the CRC instance is started. If you already have a running CRC instance, then for this configuration change to take effect, stop the CRC instance with 'crc stop' and restart it with 'crc start'. INFO Using bundle path /home/kevin/.crc/cache/crc_libvirt_4.11.1_amd64.crcbundle INFO Checking if running as non-root INFO Checking if running inside WSL2 INFO Checking if crc-admin-helper executable is cached INFO Caching crc-admin-helper executable INFO Using root access: Changing ownership of /home/kevin/.crc/bin/crc-admin-helper-linux [sudo] password for kevin: INFO Using root access: Setting suid for /home/kevin/.crc/bin/crc-admin-helper-linux INFO Checking for obsolete admin-helper executable INFO Checking if running on a supported CPU architecture INFO Checking minimum RAM requirements INFO Checking if crc executable symlink exists INFO Creating symlink for crc executable INFO Checking if Virtualization is enabled INFO Checking if KVM is enabled INFO Checking if libvirt is installed INFO Checking if user is part of libvirt group INFO Checking if active user/process is currently part of the libvirt group INFO Checking if libvirt daemon is running INFO Checking if a supported libvirt version is installed INFO Checking if crc-driver-libvirt is installed INFO Installing crc-driver-libvirt INFO Checking crc daemon systemd service INFO Checking crc daemon systemd socket units INFO Checking if systemd-networkd is running INFO Checking if NetworkManager is installed INFO Checking if NetworkManager service is running INFO Checking if dnsmasq configurations file exist for NetworkManager INFO Checking if the systemd-resolved service is running INFO Checking if /etc/NetworkManager/dispatcher.d/99-crc.sh exists INFO Checking if libvirt 'crc' network is available INFO Checking if libvirt 'crc' network is active INFO Checking if CRC bundle is extracted in '$HOME/.crc' INFO Checking if /home/kevin/.crc/cache/crc_libvirt_4.11.1_amd64.crcbundle exists INFO Getting bundle for the CRC executable INFO Downloading crc_libvirt_4.11.1_amd64.crcbundle (progress bar redacted) INFO Uncompressing /home/kevin/.crc/cache/crc_libvirt_4.11.1_amd64.crcbundle (progress bar redacted) Your system is correctly setup for using CRC. Use 'crc start' to start the instance ```

crc start

linux$ crc start --log-level=9 ``` level=info msg="Checking if running as non-root" level=info msg="Checking if running inside WSL2" level=info msg="Checking if crc-admin-helper executable is cached" level=info msg="Checking for obsolete admin-helper executable" level=info msg="Checking if running on a supported CPU architecture" level=info msg="Checking minimum RAM requirements" level=info msg="Checking if crc executable symlink exists" level=info msg="Checking if Virtualization is enabled" level=info msg="Checking if KVM is enabled" level=info msg="Checking if libvirt is installed" level=info msg="Checking if user is part of libvirt group" level=info msg="Checking if active user/process is currently part of the libvirt group" level=info msg="Checking if libvirt daemon is running" level=info msg="Checking if a supported libvirt version is installed" level=info msg="Checking if crc-driver-libvirt is installed" level=info msg="Checking crc daemon systemd socket units" level=info msg="Checking if systemd-networkd is running" level=info msg="Checking if NetworkManager is installed" level=info msg="Checking if NetworkManager service is running" level=info msg="Checking if dnsmasq configurations file exist for NetworkManager" level=info msg="Checking if the systemd-resolved service is running" level=info msg="Checking if /etc/NetworkManager/dispatcher.d/99-crc.sh exists" level=info msg="Checking if libvirt 'crc' network is available" level=info msg="Checking if libvirt 'crc' network is active" level=info msg="Loading bundle: crc_libvirt_4.11.1_amd64..." level=info msg="Creating CRC VM for openshift 4.11.1..." level=info msg="Generating new SSH key pair..." level=info msg="Generating new password for the kubeadmin user" level=info msg="Starting CRC VM for openshift 4.11.1..." level=info msg="CRC instance is running with IP 192.168.130.11" level=info msg="CRC VM is running" level=info msg="Updating authorized keys..." level=info msg="Resizing /dev/vda4 filesystem" level=info msg="Configuring shared directories" level=info msg="Check internal and public DNS query..." level=info msg="Check DNS query from host..." level=info msg="Verifying validity of the kubelet certificates..." level=info msg="Starting kubelet service" level=info msg="Waiting for kube-apiserver availability... [takes around 2min]" level=info msg="Adding user's pull secret to the cluster..." level=info msg="Updating SSH key to machine config resource..." level=info msg="Waiting for user's pull secret part of instance disk..." level=info msg="Changing the password for the kubeadmin user" level=info msg="Updating cluster ID..." level=info msg="Updating root CA cert to admin-kubeconfig-client-ca configmap..." level=info msg="Starting openshift instance... [waiting for the cluster to stabilize]" level=info msg="3 operators are progressing: image-registry, network, openshift-controller-manager" level=info msg="2 operators are progressing: image-registry, openshift-controller-manager" level=info msg="All operators are available. Ensuring stability..." level=info msg="2 operators are progressing: kube-apiserver, openshift-controller-manager" level=info msg="Operator kube-apiserver is progressing" level=info msg="Operator kube-apiserver is progressing" level=info msg="Operator authentication is not yet available" level=info msg="Operator authentication is not yet available" level=info msg="Operator authentication is not yet available" level=info msg="Operator authentication is not yet available" level=error msg="Cluster is not ready: cluster operators are still not stable after 10m1.49983431s" level=info msg="Adding crc-admin and crc-developer contexts to kubeconfig..." Started the OpenShift cluster. The server is accessible via web console at: https://console-openshift-console.apps-crc.testing Log in as administrator: Username: kubeadmin Password: REDACTED Log in as user: Username: developer Password: developer Use the 'oc' command line interface: $ eval (crc oc-env) $ oc login -u developer https://api.crc.testing:6443 ```
KevinMGranger commented 2 years ago

I'm happy to talk in Google Chat if that's easier. I'd love to help solve this for others if they're experiencing it too.

cfergeau commented 2 years ago

This is fixed by https://github.com/code-ready/snc/pull/578 The fix was too late for the 2.8.0 release, but should be in the 2.9.0 one.

KevinMGranger commented 2 years ago

I just tried it with 2.9.0 directly on macOS and the issue is still present. I'll try to change my writeup, it was just a lot of work collecting that the first time. I wish I had automated it 😅

Interesting that the issue was with the client though. I guess that's a good workaround.

I know it's not strictly relevant to CRC, but if you know what certs I'd need to export from the cluster, I'd be happy to manually import and trust them until there's a fix.

KevinMGranger commented 2 years ago

Oh, it also looks like the fix wasn't in the release, or isn't working?

$ uname -a; crc version
Darwin m1a1 21.6.0 Darwin Kernel Version 21.6.0: Wed Aug 10 14:28:23 PDT 2022; root:xnu-8020.141.5~2/RELEASE_ARM64_T6000 arm64
CRC version: 2.9.0+589ab2cd
OpenShift version: 4.11.3
Podman version: 4.2.0
cfergeau commented 2 years ago

Can you check ~/.crc/bin/oc/oc version?

KevinMGranger commented 2 years ago

Ah! I thought https://github.com/code-ready/snc/pull/578 was talking about downgrading the cluster, not the client.

The dev env setup we have for our project automatically downloads the latest. I knew it was 4.11, and was about to manually downgrade. I'll use the version that came with CRC now. Maybe I'll even add a "I'm using CRC" option to our setup script.

Thank you for helping with this, this has been bothering me for a long time.

This should definitely be in the "Known Issues" section for CRC, since many folks won't read the openshift release notes in addition to them. But it looks like the release notes / docs haven't been updated for a few releases?

KevinMGranger commented 2 years ago

Looks like anyone using crc properly won't hit this, so it's already fixed :)