search

crc-org / crc

CRC is a tool to help you run containers. It manages a local OpenShift 4.x cluster, Microshift or a Podman VM optimized for testing and development purposes
https://crc.dev
Apache License 2.0
1.27k stars 243 forks source link

Nested virtualization causes startup delays and inconsitent deployments; Hyper-V from a Windows VM #3922

Closed eliassal closed 1 year ago

eliassal commented 1 year ago

General information

CRC version

# CRC version: 2.29.0+da5f55

CRC status

PS C:\Users\salam> CRC status
CRC VM:          Running
OpenShift:       Unreachable (v4.14.1)
RAM Usage:       632.5MB of 9.361GB
Disk Usage:      18.35GB of 32.68GB (Inside the CRC VM)
Cache Usage:     25.11GB
Cache Directory: C:\Users\salam\.crc\cache

CRC config

# - consent-telemetry                     : yes

Host Operating System

Host Name:                 DEMOS
OS Name:                   Microsoft Windows Server 2022 Standard
OS Version:                10.0.20348 N/A Build 20348
OS Manufacturer:           Microsoft Corporation
OS Configuration:          Member Server
OS Build Type:             Multiprocessor Free
Registered Owner:          Windows User
Registered Organization:
Product ID:                00453-60520-76521-AA838
Original Install Date:     6/9/2022, 6:30:50 AM
System Boot Time:          11/19/2023, 9:09:10 AM
System Manufacturer:       Microsoft Corporation
System Model:              Virtual Machine
System Type:               x64-based PC
Processor(s):              1 Processor(s) Installed.
                           [01]: Intel64 Family 6 Model 94 Stepping 3 GenuineIntel ~3408 Mhz
BIOS Version:              Microsoft Corporation Hyper-V UEFI Release v4.0, 11/1/2019
Windows Directory:         C:\Windows
System Directory:          C:\Windows\system32
Boot Device:               \Device\HarddiskVolume1
System Locale:             en-us;English (United States)
Input Locale:              en-us;English (United States)
Time Zone:                 (UTC-08:00) Pacific Time (US & Canada)
Total Physical Memory:     14,335 MB
Available Physical Memory: 2,665 MB
Virtual Memory: Max Size:  16,511 MB
Virtual Memory: Available: 4,825 MB
Virtual Memory: In Use:    11,686 MB
Page File Location(s):     C:\pagefile.sys
Domain:                    salam.net
Logon Server:              \\SALAMMAIN
Hotfix(s):                 3 Hotfix(s) Installed.
                           [01]: KB5015733
                           [02]: KB5016627
                           [03]: KB5015897
Network Card(s):           2 NIC(s) Installed.
                           [01]: Microsoft Hyper-V Network Adapter
                                 Connection Name: Ethernet
                                 DHCP Enabled:    Yes
                                 DHCP Server:     N/A
                                 IP address(es)
                           [02]: Hyper-V Virtual Ethernet Adapter
                                 Connection Name: vEthernet (Microsoft Hyper-V Network Adapter - Virtual Switch)
                                 DHCP Enabled:    No
                                 IP address(es)
                                 [01]: 192.168.1.56
                                 [02]: fe80::34cc:19ac:b477:b682
Hyper-V Requirements:      A hypervisor has been detected. Features required for Hyper-V will not be displayed.

Steps to reproduce

  1. crs setup
  2. crc start
  3. I get

Started the OpenShift cluster.

The server is accessible via web console at: https://console-openshift-console.apps-crc.testing

Log in as administrator: Username: kubeadmin Password: XXXX-XXXX-WUyKM-XXXXX

Log in as user: Username: developer Password: developer

Use the 'oc' command line interface: PS> & crc oc-env | Invoke-Expression PS> oc login -u developer https://api.crc.testing:6443 DEBU Running '(Get-ItemProperty -Path "HKLM:\SOFTWARE\Microsoft\Windows NT\CurrentVersion" -Name ReleaseId).ReleaseId' DEBU Running '(Get-ItemProperty -Path "HKLM:\SOFTWARE\Microsoft\Windows NT\CurrentVersion").EditionID'

Expected

Be able to browse https://console-openshift-console.apps-crc.testing Be able to run oc login -u developer https://api.crc.testing:6443

Actual

For first link console-openshift-console.apps-crc.testing, I get

image

for 2nd link oc login -u developer https://api.crc.testing:6443 I get

oc login -u developer https://api.crc.testing:6443 error: read tcp 127.0.0.1:49802->127.0.0.1:6443: wsarecv: An existing connection was forcibly closed by the remote host. - verify you have provided the correct host and port and that the server is currently running.

Logs

Before gather the logs try following if that fix your issue I TRIED so many times 

PS C:\Users\salam> crc delete -f
Deleted the instance

 crc cleanup
INFO Removing the daemon task
INFO Removing the daemon task powershell script
INFO Stop daemon process if it is running
INFO Removing crc's virtual machine
INFO Removing CRC Specific entries from user's known_hosts file
INFO Removing hosts file records added by CRC
INFO Removing pull secret from the keyring
INFO Removing older logs
Cleanup finished

crc setup
INFO Using bundle path C:\Users\salam\.crc\cache\crc_hyperv_4.14.1_amd64.crcbundle
INFO Checking minimum RAM requirements
INFO Checking if current user is in crc-users and Hyper-V admins group
INFO Checking if CRC bundle is extracted in '$HOME/.crc'
INFO Checking if C:\Users\salam\.crc\cache\crc_hyperv_4.14.1_amd64.crcbundle exists
INFO Checking if the daemon task powershell script is present
INFO Creating the daemon task powershell script
INFO Checking if the daemon task is installed
INFO Installing the daemon task
INFO Checking if the daemon task is running
INFO Running the daemon task
INFO Checking admin helper service is running
Your system is correctly setup for using CRC. Use 'crc start' to start the instance

**After following steps I started getting the following**

PS C:\Users\salam> crc start --log-level debug
DEBU CRC version: 2.29.0+da5f55
DEBU OpenShift version: 4.14.1
DEBU Podman version: 4.4.4
DEBU Running 'crc start'
DEBU Total memory of system is 15031246848 bytes
DEBU No new version available. The latest version is 2.29.0
DEBU Checking file: C:\Users\salam\.crc\machines\crc\.crc-exist
INFO Using bundle path C:\Users\salam\.crc\cache\crc_hyperv_4.14.1_amd64.crcbundle
INFO Checking minimum RAM requirements
DEBU Total memory of system is 15031246848 bytes
INFO Checking if running in a shell with administrator rights
DEBU Running '$currentPrincipal = New-Object Security.Principal.WindowsPrincipal([Security.Principal.WindowsIdentity]::GetCurrent());$currentPrincipal.IsInRole([Security.Principal.WindowsBuiltInRole]::Administrator)'
INFO Checking Windows release
DEBU Running '(Get-ItemProperty -Path "HKLM:\SOFTWARE\Microsoft\Windows NT\CurrentVersion" -Name ReleaseId).ReleaseId'
INFO Checking Windows edition
DEBU Running '(Get-ItemProperty -Path "HKLM:\SOFTWARE\Microsoft\Windows NT\CurrentVersion").EditionID'
DEBU Running on Windows ServerStandard edition
INFO Checking if Hyper-V is installed and operational
DEBU Running '@(Get-Wmiobject Win32_ComputerSystem).HypervisorPresent'
DEBU Running '@(Get-Service vmms).Status'
INFO Checking if Hyper-V service is enabled
DEBU Running '@(Get-Service vmms).Status'
INFO Checking if crc-users group exists
DEBU Running 'Get-LocalGroup -Name crc-users'
INFO Checking if current user is in crc-users and Hyper-V admins group
DEBU Running '(Get-LocalGroupMember -Group 'crc-users').Name'
DEBU Checking current user is in the 'crc-user' group
DEBU group members: SALAM\salam
DEBU Running '(Get-LocalGroupMember -SID 'S-1-5-32-578').Name'
DEBU Checking current user is in the 'Hyper-v Administrators' group
DEBU group members: SALAM\,SALAM\salam
INFO Checking if vsock is correctly configured
DEBU Running 'Get-Item -Path "HKLM:\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Virtualization\GuestCommunicationServices\00000400-FACB-11E6-BD58-64006A7986D3"'
INFO Checking if the daemon task powershell script is present
INFO Checking if the daemon task is installed
DEBU Running 'Get-ScheduledTask -TaskName crcDaemon'
DEBU Running '(Get-ScheduledTask -TaskName "crcDaemon").Version'
INFO Checking if the daemon task is running
DEBU Running '(Get-ScheduledTask -TaskName "crcDaemon").State'
INFO Checking admin helper service is running
DEBU Running '(Get-Service crcAdminHelper).Status'
DEBU Checking file: C:\Users\salam\.crc\machines\crc\.crc-exist
DEBU Copying 'C:\Users\salam\.crc\cache\crc_hyperv_4.14.1_amd64\oc.exe' to 'C:\Users\salam\.crc\bin\oc\oc.exe'
DEBU Copying 'C:\Users\salam\.crc\cache\crc_hyperv_4.14.1_amd64\podman.exe' to 'C:\Users\salam\.crc\bin\oc\podman.exe'
INFO Loading bundle: crc_hyperv_4.14.1_amd64...
DEBU Cannot load secret from configuration: empty path
DEBU Cannot load secret from keyring: secret not found in keyring
CRC requires a pull secret to download content from Red Hat.
You can copy it from the Pull Secret section of https://console.redhat.com/openshift/create/local.
? Please enter the pull secret *****************************************************************************************INFO Creating CRC VM for OpenShift 4.14.1...
DEBU Running pre-create checks...
DEBU Running '@(Get-Module -ListAvailable hyper-v).Name | Get-Unique'
DEBU Running '@([Security.Principal.WindowsPrincipal][Security.Principal.WindowsIdentity]::GetCurrent()).IsInRole(([System.Security.Principal.SecurityIdentifier]::new("S-1-5-32-578")))'
DEBU Creating machine...
DEBU Creating VM...
DEBU Running 'Hyper-V\New-VM crc -Path 'C:\Users\salam\.crc\machines\crc' -MemoryStartupBytes 9216MB'
DEBU Running 'Hyper-V\Remove-VMNetworkAdapter -VMName crc'
DEBU Running 'Hyper-V\Set-VMMemory -VMName crc -DynamicMemoryEnabled $false'
DEBU Running 'Hyper-V\Set-VMProcessor crc -Count 4'
DEBU Running 'Hyper-V\Set-VM -VMName crc -AutomaticStartAction Nothing -AutomaticStopAction ShutDown -CheckpointType Disabled'
DEBU Running 'Hyper-V\Add-VMHardDiskDrive -VMName crc -Path 'C:\Users\salam\.crc\machines\crc\crc.vhdx''
DEBU Running '@(Get-VHD -Path 'C:\Users\salam\.crc\machines\crc\crc.vhdx').Size'
DEBU C:\Users\salam\.crc\machines\crc\crc.vhdx is already 33285996544 bytes
INFO Generating new SSH key pair...
INFO Generating new password for the kubeadmin user
DEBU Created C:\Users\salam\.crc\machines\crc\.crc-exist
DEBU Machine successfully created
DEBU Checking file: C:\Users\salam\.crc\machines\crc\.crc-exist
DEBU Running 'Hyper-V\Get-VM crc | Select-Object -ExpandProperty State'
DEBU Copying 'C:\Users\salam\.crc\cache\crc_hyperv_4.14.1_amd64\oc.exe' to 'C:\Users\salam\.crc\bin\oc\oc.exe'
DEBU Copying 'C:\Users\salam\.crc\cache\crc_hyperv_4.14.1_amd64\podman.exe' to 'C:\Users\salam\.crc\bin\oc\podman.exe'
INFO Starting CRC VM for openshift 4.14.1...
DEBU Updating CRC VM configuration
DEBU Running 'Hyper-V\Start-VM crc'
DEBU Waiting for machine to be running, this may take a few minutes...
DEBU retry loop: attempt 0
DEBU Running 'Hyper-V\Get-VM crc | Select-Object -ExpandProperty State'
DEBU Machine is up and running!
DEBU Running 'Hyper-V\Get-VM crc | Select-Object -ExpandProperty State'
INFO CRC instance is running with IP 127.0.0.1
DEBU Waiting until ssh is available
DEBU retry loop: attempt 0
DEBU Running SSH command: exit 0
DEBU Using ssh private keys: [C:\Users\salam\.crc\machines\crc\id_ecdsa C:\Users\salam\.crc\cache\crc_hyperv_4.14.1_amd64\id_ecdsa_crc]
DEBU SSH command results: err: ssh: handshake failed: read tcp 127.0.0.1:49955->127.0.0.1:2222: wsarecv: An existing connection was forcibly closed by the remote host., output:
DEBU error: Temporary error: ssh command error:
command : exit 0
err     : ssh: handshake failed: read tcp 127.0.0.1:49955->127.0.0.1:2222: wsarecv: An existing connection was forcibly closed by the remote host.
 - sleeping 1s
DEBU retry loop: attempt 1
DEBU Running SSH command: exit 0
DEBU Using ssh private keys: [C:\Users\salam\.crc\machines\crc\id_ecdsa C:\Users\salam\.crc\cache\crc_hyperv_4.14.1_amd64\id_ecdsa_crc]
DEBU SSH command results: err: ssh: handshake failed: read tcp 127.0.0.1:49956->127.0.0.1:2222: wsarecv: An existing connection was forcibly closed by the remote host., output:
DEBU error: Temporary error: ssh command error:
command : exit 0
err     : ssh: handshake failed: read tcp 127.0.0.1:49956->127.0.0.1:2222: wsarecv: An existing connection was forcibly closed by the remote host.
 - sleeping 1s
DEBU retry loop: attempt 2
DEBU Running SSH command: exit 0
DEBU Using ssh private keys: [C:\Users\salam\.crc\machines\crc\id_ecdsa C:\Users\salam\.crc\cache\crc_hyperv_4.14.1_amd64\id_ecdsa_crc]
DEBU SSH command results: err: ssh: handshake failed: read tcp 127.0.0.1:49957->127.0.0.1:2222: wsarecv: An existing connection was forcibly closed by the remote host., output:
DEBU error: Temporary error: ssh command error:
command : exit 0
err     : ssh: handshake failed: read tcp 127.0.0.1:49957->127.0.0.1:2222: wsarecv: An existing connection was forcibly closed by the remote host.
 - sleeping 1s
DEBU retry loop: attempt 3
DEBU Running SSH command: exit 0
DEBU Using ssh private keys: [C:\Users\salam\.crc\machines\crc\id_ecdsa C:\Users\salam\.crc\cache\crc_hyperv_4.14.1_amd64\id_ecdsa_crc]
DEBU SSH command results: err: ssh: handshake failed: read tcp 127.0.0.1:49958->127.0.0.1:2222: wsarecv: An existing connection was forcibly closed by the remote host., output:
DEBU error: Temporary error: ssh command error:
command : exit 0
err     : ssh: handshake failed: read tcp 127.0.0.1:49958->127.0.0.1:2222: wsarecv: An existing connection was forcibly closed by the remote host.
 - sleeping 1s

output of crc start --log-level debug at https://gist.github.com/eliassal/7de92c774cf396d5cd8042360ec97498

eliassal commented 1 year ago

I reran crc start --log-level debug

now crc goes through but not oc command which is in bold hereunder(I choose Y or N for Use insecure connections I get the same response) nor browsing console-openshift-console.apps-crc.testing

DEBU CRC version: 2.29.0+da5f55 DEBU OpenShift version: 4.14.1 DEBU Podman version: 4.4.4 DEBU Running 'crc start' DEBU Total memory of system is 15031246848 bytes DEBU No new version available. The latest version is 2.29.0 DEBU Checking file: C:\Users\salam.crc\machines\crc.crc-exist DEBU Running 'Hyper-V\Get-VM crc | Select-Object -ExpandProperty State' DEBU Checking file: C:\Users\salam.crc\machines\crc.crc-exist DEBU Copying 'C:\Users\salam.crc\cache\crc_hyperv_4.14.1_amd64\oc.exe' to 'C:\Users\salam.crc\bin\oc\oc.exe' DEBU Copying 'C:\Users\salam.crc\cache\crc_hyperv_4.14.1_amd64\podman.exe' to 'C:\Users\salam.crc\bin\oc\podman.exe' INFO Loading bundle: crc_hyperv_4.14.1_amd64... DEBU Checking file: C:\Users\salam.crc\machines\crc.crc-exist DEBU Running 'Hyper-V\Get-VM crc | Select-Object -ExpandProperty State' INFO A CRC VM for OpenShift 4.14.1 is already running Started the OpenShift cluster.

The server is accessible via web console at: https://console-openshift-console.apps-crc.testing

Log in as administrator: Username: kubeadmin Password: 3UqwP-uUz2z-4QvCz-3MqsN

Log in as user: Username: developer Password: developer

Use the 'oc' command line interface: PS> & crc oc-env | Invoke-Expression

PS> oc login -u developer https://api.crc.testing:6443 DEBU Running '(Get-ItemProperty -Path "HKLM:\SOFTWARE\Microsoft\Windows NT\CurrentVersion" -Name ReleaseId).ReleaseId' DEBU Running '(Get-ItemProperty -Path "HKLM:\SOFTWARE\Microsoft\Windows NT\CurrentVersion").EditionID' PS C:\Users\salam> oc login -u developer https://api.crc.testing:6443 The server uses a certificate signed by an unknown authority. You can bypass the certificate check, but any data you send to the server could be intercepted by others. Use insecure connections? (y/n): y

WARNING: Using insecure TLS client config. Setting this option is not supported!

Unable to connect to the server: read tcp 127.0.0.1:50592->127.0.0.1:443: wsarecv: An existing connection was forcibly closed by the remote host. PS C:\Users\salam> oc login -u developer https://api.crc.testing:6443 The server uses a certificate signed by an unknown authority. You can bypass the certificate check, but any data you send to the server could be intercepted by others. Use insecure connections? (y/n): n**

error: The server uses a certificate signed by unknown authority. You may need to use the --certificate-authority flag to provide the path to a certificate file for the certificate authority, or --insecure-skip-tls-verify to bypass the certificate check and use insecure connections. PS C:\Users\salam>**

eliassal commented 1 year ago

When I try to connect using firefox I get

image

eliassal commented 1 year ago

And here using MS Edge

image

eliassal commented 1 year ago

After waiting for 60 minutes, MS Edge displayed the login page but once I enter I get the following error

image

praveenkumar commented 1 year ago

This looks like duplicate of https://github.com/crc-org/crc/issues/3898 one.

eliassal commented 1 year ago

OK but what is the solution or fix? as in #3898 no solution only description of the issue as this one

eliassal commented 1 year ago

Can you please le us know how to proceed and fix this problem in order to be able to progress in our setup? Thanks

gbraad commented 1 year ago

Running CRC on: VM

We do not support running CRC from a VM. So, can you clarify if this runs as a VM on Hyper-V or as a VM inside a VM on Hyper-V ?

I can answer this from the info:

System Model:              Virtual Machine
System Type:               x64-based PC

It seems Windows is running as a Virtual Machine and therefore this is a nested virtualization which do not support, due to resource overhead and possible timing issues with different hypervisors. Closing as this can not be resolved.

gbraad commented 1 year ago

Can you please le us know how to proceed and fix this problem in order to be able to progress in our setup?

As described in the documentation: https://crc.dev/crc/getting_started/getting_started/installing/#_minimum_system_requirements "physical CPU cores" are required. Please run this on your native hypervisor, without a nested setup.

eliassal commented 1 year ago

yes it is a nested VM. OK I will. run another setup directly on hyper-V but I dont want the CRC VM be created in the C drive, how can this be done? I mean can let crc setup create the VM on another drive?

gbraad commented 1 year ago

the VM has to exist on the WINDRIVE, that is a hard requirement from the drivers we are using.

eliassal commented 1 year ago

So if I change the CRC install drive location it will create the VM on the C drive? is my understanding is correct?

gbraad commented 1 year ago

You might be able to move the harddisk file AFTER the start to another drive, but initially it will have to create this as part of the USERPROFILE location .crc/machines/crc/.

Note: moving these files around is not a setup we will support or fix issues for.

eliassal commented 1 year ago

OK, so many thanks got it. Anyway, some good news, I followed install instructions directly on a physical Hyper-V and was able to access https://console-openshift-console.apps-crc.testing/add/all-namespaces locally but when I try to access remotely, after updating my hosts file to point to the IP of the machine as follows 192.168.10.14 5console-openshift-console.apps-crc.testing I get the alerte in Firefox, I accept the risk, then I get

image

So is there any config to be able to access the we site remotely? Thanks again

gbraad commented 1 year ago

Please read the documentation.