Closed tquigly1287 closed 4 years ago
Been having the same issue (Fedora 31, libvirt, CRC v1.3.0 Openshift v4.2.10). The retry loop goes on but basically this is the error:
(crc) Calling .GetSSHHostname
(crc) DBG | GetIP called for crc
(crc) DBG | Getting current state...
(crc) DBG | IP address: 192.168.130.11
(crc) Calling .GetSSHPort
(crc) Calling .GetSSHKeyPath
(crc) Calling .GetSSHKeyPath
(crc) Calling .GetSSHUsername
Using SSH client type: external
Using SSH private key: /home/hsharon/.crc/machines/crc/id_rsa (-rw-------)
&{[-F /dev/null -o ConnectionAttempts=3 -o ConnectTimeout=10 -o ControlMaster=no -o ControlPath=none -o LogLevel=quiet -o PasswordAuthentication=no -o ServerAliveInterval=60 -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null core@192.168.130.11 -o IdentitiesOnly=yes -i /home/hsharon/.crc/machines/crc/id_rsa -p 22] /usr/bin/ssh <nil>}
About to run SSH command:
exit 0
SSH cmd err, output: exit status 255:
DEBU error: Temporary Error: ssh command error:
command : exit 0
err : exit status 255
output : - sleeping 1s
What gives?
Same issues here on Windows 10
crc setup
before starting it (Yes/No)? YES, the following:
.\crc.exe stop
.\crc.exe delete -f
.\crc.exe setup
.\crc.exe start -n 8.8.8.8 --log-level debug
crc version: 1.3.0+918756b
OpenShift version: 4.2.10 (embedded in binary)
PS C:\OpenShiftCodeReady-1.3.0> .\crc.exe status
ERRO Unable to connect to the server: dial tcp 31.199.53.9:6443: connectex: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.
- exit status 1
command returns nothing
PS C:\OpenShiftCodeReady-1.3.0> .\crc.exe config view PS C:\OpenShiftCodeReady-1.3.0>
PS C:\OpenShiftCodeReady-1.3.0> systeminfo
Host Name: DINO-PC
OS Name: Microsoft Windows 10 Pro
OS Version: 10.0.19041 N/A Build 19041
OS Manufacturer: Microsoft Corporation
OS Configuration: Standalone Workstation
OS Build Type: Multiprocessor Free
Registered Owner: N/A
Registered Organization: N/A
Product ID: 00330-80000-00000-AA444
Original Install Date: 23/12/2019, 08:28:48
System Boot Time: 26/12/2019, 12:09:34
System Manufacturer: MSI
System Model: MS-7922
System Type: x64-based PC
Processor(s): 1 Processor(s) Installed.
[01]: Intel64 Family 6 Model 60 Stepping 3 GenuineIntel ~4000 Mhz
BIOS Version: American Megatrends Inc. V10.6, 13/08/2015
Windows Directory: C:\WINDOWS
System Directory: C:\WINDOWS\system32
Boot Device: \Device\HarddiskVolume2
System Locale: en-us;English (United States)
Input Locale: en-us;English (United States)
Time Zone: (UTC+01:00) Amsterdam, Berlin, Bern, Rome, Stockholm, Vienna
Total Physical Memory: 32.720 MB
Available Physical Memory: 4.189 MB
Virtual Memory: Max Size: 39.668 MB
Virtual Memory: Available: 7.076 MB
Virtual Memory: In Use: 32.592 MB
Page File Location(s): C:\pagefile.sys
Domain: WORKGROUP
Logon Server: \\DINO-PC
Hotfix(s): N/A
Network Card(s): 4 NIC(s) Installed.
[01]: Xbox Wireless Adapter for Windows
Connection Name: Ethernet 2
DHCP Enabled: No
IP address(es)
[02]: Realtek PCIe GbE Family Controller
Connection Name: Ethernet
DHCP Enabled: Yes
DHCP Server: 192.168.178.1
IP address(es)
[01]: 192.168.178.87
[02]: fe80::3087:8222:22e0:17e0
[03]: Hyper-V Virtual Ethernet Adapter
Connection Name: vEthernet (Default Switch)
DHCP Enabled: No
IP address(es)
[01]: 172.18.240.1
[02]: fe80::950c:b540:c83a:17c1
[04]: Hyper-V Virtual Ethernet Adapter
Connection Name: vEthernet (WSL)
DHCP Enabled: No
IP address(es)
[01]: 172.18.128.1
[02]: fe80::5417:a718:e576:314a
Hyper-V Requirements: A hypervisor has been detected. Features required for Hyper-V will not be displayed.
The system must connect to the correct VM ip address":
172.18.251.67
while it seems it is connecting to:
31.199.53.9:6443
Stuck after trying to pull secret:
INFO Adding user's pull secret ...
(crc) Calling .GetSSHHostname
(crc) DBG | [executing ==>] : C:\WINDOWS\System32\WindowsPowerShell\v1.0\powershell.exe -NoProfile -NonInteractive ( Hyper-V\Get-VM crc ).state
(crc) DBG | [stdout =====>] : Running
(crc) DBG |
(crc) DBG | [stderr =====>] :
(crc) DBG | [executing ==>] : C:\WINDOWS\System32\WindowsPowerShell\v1.0\powershell.exe -NoProfile -NonInteractive (( Hyper-V\Get-VM crc ).networkadapters[0]).ipaddresses[0]
(crc) DBG | [stdout =====>] : 172.18.251.67
(crc) DBG |
(crc) DBG | [stderr =====>] :
(crc) Calling .GetSSHPort
(crc) Calling .GetSSHKeyPath
(crc) Calling .GetSSHKeyPath
(crc) Calling .GetSSHUsername
Using SSH client type: native
&{{{<nil> 0 [] [] []} core [0x7b6f70] 0x7b6f40 <nil> [] 0s} 172.18.251.67 22 <nil> <nil>}
DEBU About to run SSH command with hidden output
DEBU Unable to connect to the server: dial tcp 31.199.53.9:6443: connectex: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.
DEBU error: Temporary Error: exit status 1 - sleeping 1s
DEBU retry loop 1
DEBU Unable to connect to the server: dial tcp 31.199.53.9:6443: connectex: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.
DEBU error: Temporary Error: exit status 1 - sleeping 1s
DEBU retry loop 2
This loop continues indefinitely until I cancel the start process with crtl + c.
https://gist.github.com/dinolupo/28b72b111a6e4782613807f8a0b654fe
@tquigly1287 @hadarsharon @dinolupo By any chance you folks are using crc behind the vpn?
@praveenkumar Not in my case.
Alright after looking closely all the users logs looks like all of you have different issue.
In case of @tquigly1287 , he didn't waited till the timeout happen due to ssh connection and manually used ctrl + C
and then again started it without ./crc delete
. @tquigly1287 Can you please do crc delete
and then crc start --log-level debug
and wait till it either works or failed with timeout, please update the new logs.
In case of @dinolupo crc start to create the VM but then dns doesn't work properly, following snip from the logs show it is using different IP. 172.18.251.67 should be the VM IP but then it is querying to 31.199.53.9 when check with the connection.
time="2019-12-27T16:34:24+01:00" level=debug msg="About to run SSH command:\nhost -R 3 foo.apps-crc.testing"
time="2019-12-27T16:34:25+01:00" level=debug msg="SSH cmd err, output: <nil>: foo.apps-crc.testing has address 172.18.251.67"
...
time="2019-12-27T16:36:38+01:00" level=debug msg="Unable to connect to the server: dial tcp 31.199.53.9:6443: connectex: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond."
time="2019-12-27T16:36:38+01:00" level=debug msg="error: Temporary Error: exit status 1 - sleeping 1s"
@hadarsharon In your case the logs which you uploaded looks like you didn't started a fresh crc container but either stopped or manually stopped the process. Can you please try to do crc delete
and then crc start --log-level debug
and then share the logs again?
Hi @praveenkumar,
Yes, you are correct I was using a VPN. I have tried the crc start once more with the VPN turned off and it started the cluster. Just to be sure I re-tested the crc with the VPN on and it hangs like before.
For reference - I was following this blog post:
If you still need logs - to help fix the VPN problem, I can re-run asupply these. Let me know.
Thanks for your help,
Tim
@praveenkumar Actually you might be on to something since every time I do crc delete and crc start I get the following error, after which I have to manually change permissions on the file using chmod, then it continues to the error I had originally posted. See log for more info:
(crc) DBG | Verifying executable bit set on /home/hsharon/.crc/machines/crc
(crc) DBG | Setting executable bit set on /home/hsharon/.crc/machines/crc
(crc) DBG | Verifying executable bit set on /home/hsharon/.crc/machines
(crc) DBG | Verifying executable bit set on /home/hsharon/.crc
(crc) DBG | Verifying executable bit set on /home/hsharon
(crc) DBG | Verifying executable bit set on /home
(crc) DBG | Defining VM...
(crc) DBG | Adding the file: /home/hsharon/.crc/machines/crc/.crc-exist
(crc) DBG | Starting VM crc
(crc) Failed to start: virError(Code=38, Domain=18, Message='Cannot access storage file '/home/hsharon/.crc/machines/crc/crc' (as uid:107, gid:107): Permission denied')
ERRO Error creating host: Error creating the VM: Error creating machine: Error in driver during machine creation: virError(Code=38, Domain=18, Message='Cannot access storage file '/home/hsharon/.crc/machines/crc/crc' (as uid:107, gid:107): Permission denied')
Making call to close driver server
(crc) Calling .Close
(crc) DBG | Closing plugin on server side
Successfully made call to close driver server
Making call to close connection to plugin binary
My user has sudo permissions, is in the list of sudoers etc. so I don't see why the permission denied error is happening but obviously something is off.
@hadarsharon This looks like the libvirt setup might be broken for you, what is the output of ls -lZ ~/.crc/machines/crc/crc
in my case it is following
$ ls -lZ ~/.crc/machines/crc/crc
-rw-------. 1 qemu qemu system_u:object_r:svirt_image_t:s0:c510,c529 14574878720 Jan 9 15:30 /home/prkumar/.crc/machines/crc/crc
@praveenkumar
-rw------- 1 hsharon hsharon ? 11323703296 Jan 9 11:30 /home/hsharon/.crc/machines/crc/crc
@hadarsharon did you use ls -lZ
I don't see any SELinux context here?
@praveenkumar Yes.
`[hsharon@localhost ~]$ ls -lZ ~/.crc/machines/crc/crc
-rw------- 1 hsharon hsharon ? 11323703296 Jan 9 11:30 /home/hsharon/.crc/machines/crc/crc `
I have startup "hang" also.
% sw_vers
ProductName: Mac OS X
ProductVersion: 10.15.2
% ./crc version
crc version: 1.3.0+918756b
OpenShift version: 4.2.10 (embedded in binary)
Seems to be same scenario as reported by @tquigly1287. I had Cisco AnyConnect VPN in play. After shutdown of AnyConnect and retry 'start', I now seem to have successful startup.
INFO Starting OpenShift cluster ... [waiting 3m]
...
Started the OpenShift cluster
VM IP but then it is querying to 31.199.53.9 when check with the connection.
Hi @praveenkumar I am not using any VPN. I am stuck, let me know if I can help in any way, I am available also for a shared debugging session if you wish :D
What software are you using besides CRC? What can you find in the system/menu bar's notification area?
Also, which timezone are you in?
On Fri, Jan 10, 2020 at 3:50 PM dino lupo notifications@github.com wrote:
VM IP but then it is querying to 31.199.53.9 when check with the connection.
Hi @praveenkumar https://github.com/praveenkumar I am not using any VPN. I am stuck, let me know if I can help in any way, I am available also for a shared debugging session if you wish :D
— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/code-ready/crc/issues/908?email_source=notifications&email_token=AAAAOZVKS4GS7GWOFYPAGZDQ5CDKZA5CNFSM4J6CL5UKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEIUEO3A#issuecomment-573065068, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAAAOZXKQSORH4SXEVFR7H3Q5CDKZANCNFSM4J6CL5UA .
--
Gerard Braad | http://gbraad.nl [ Doing Open Source Matters ]
@gbraad @praveenkumar
Regards, Dino.
it is not a difficult timezone (was there just last week), but even then we could set up a meeting. Sent me an email and will see what we can do.
The other software might be services or drivers even, like a virus scanner (mcafee), etc.
Ok, I wrote an email to your address. To summarize:
Hi, I tried the 1.4.0 on this new Windows 10 machine, I think the problem is with the Check Point Endpoint Security VPN, even if it is not enabled and the network adapter is disabled.
OS: Windows 10
Hypervisor: Hyper-V
Did you run crc setup before starting it: Yes
.\crc.exe stop
.\crc.exe delete -f
.\crc.exe setup
.\crc.exe start -n 8.8.8.8 --log-level debug
crc version: 1.3.0+918756b
OpenShift version: 4.2.10 (not embedded in binary)
Machine 'crc' does not exist. Use 'crc start' to create it
command returns nothing
PS C:\crc-windows-1.4.0-amd64> crc config view PS C:\crc-windows-1.4.0-amd64>
Host Operating System
PS C:\crc-windows-1.4.0-amd64> systeminfo
Host Name: LAPTOP-409RM56B
OS Name: Microsoft Windows 10 Enterprise
OS Version: 10.0.17763 N/A Build 17763
OS Manufacturer: Microsoft Corporation
OS Configuration: Standalone Workstation
OS Build Type: Multiprocessor Free
Registered Owner: Windows User
Registered Organization:
Product ID: 00330-80000-00000-AA340
Original Install Date: 12/04/2019, 19:21:55
System Boot Time: 14/01/2020, 17:21:15
System Manufacturer: LENOVO
System Model: 20LAS6V41K
System Type: x64-based PC
Processor(s): 1 Processor(s) Installed.
[01]: Intel64 Family 6 Model 142 Stepping 10 GenuineIntel ~1910 Mhz
BIOS Version: LENOVO N27ET32W (1.18 ), 08/11/2018
Windows Directory: C:\Windows
System Directory: C:\Windows\system32
Boot Device: \Device\HarddiskVolume2
System Locale: en-us;English (United States)
Input Locale: en-us;English (United States)
Time Zone: (UTC+01:00) Amsterdam, Berlin, Bern, Rome, Stockholm, Vienna
Total Physical Memory: 16.231 MB
Available Physical Memory: 6.510 MB
Virtual Memory: Max Size: 21.607 MB
Virtual Memory: Available: 8.581 MB
Virtual Memory: In Use: 13.026 MB
Page File Location(s): C:\pagefile.sys
Domain: AtosC2-DWP
Logon Server: \\LAPTOP-409RM56B
Hotfix(s): 15 Hotfix(s) Installed.
[01]: KB4533001
[02]: KB4465065
[03]: KB4465477
[04]: KB4480056
[05]: KB4486153
[06]: KB4493510
[07]: KB4499728
[08]: KB4504369
[09]: KB4509095
[10]: KB4512577
[11]: KB4512937
[12]: KB4516115
[13]: KB4521862
[14]: KB4523204
[15]: KB4530715
Network Card(s): 2 NIC(s) Installed.
[01]: Intel(R) Ethernet Connection (4) I219-LM
Connection Name: Ethernet
DHCP Enabled: Yes
DHCP Server: 172.31.17.252
IP address(es)
[01]: 172.31.17.113
[02]: fe80::9c:ca60:8e69:3567
[02]: Hyper-V Virtual Ethernet Adapter
Connection Name: vEthernet (Default Switch)
DHCP Enabled: No
IP address(es)
[01]: 172.18.110.81
[02]: fe80::dd9e:25b9:ca44:7123
Hyper-V Requirements: A hypervisor has been detected. Features required for Hyper-V will not be displayed.
correct installation
error during start
Logs
time="2020-01-15T14:55:06+01:00" level=debug msg="No new version available. The latest version is 1.4.0"
time="2020-01-15T14:55:06+01:00" level=info msg="Checking if oc binary is cached"
time="2020-01-15T14:55:06+01:00" level=debug msg="oc binary already cached"
time="2020-01-15T14:55:06+01:00" level=info msg="Checking if running as normal user"
time="2020-01-15T14:55:07+01:00" level=info msg="Checking Windows 10 release"
time="2020-01-15T14:55:08+01:00" level=info msg="Checking if Hyper-V is installed and operational"
time="2020-01-15T14:55:11+01:00" level=info msg="Checking if user is a member of the Hyper-V Administrators group"
time="2020-01-15T14:55:12+01:00" level=info msg="Checking if Hyper-V service is enabled"
time="2020-01-15T14:55:14+01:00" level=info msg="Checking if the Hyper-V virtual switch exist"
time="2020-01-15T14:55:19+01:00" level=info msg="Found Virtual Switch to use: Default Switch"
time="2020-01-15T14:55:19+01:00" level=info msg="Loading bundle: crc_hyperv_4.2.13.crcbundle ..."
time="2020-01-15T14:55:19+01:00" level=info msg="Creating CodeReady Containers VM for OpenShift 4.2.13..."
time="2020-01-15T14:55:34+01:00" level=error msg="Error creating host: Error creating new host: Error attempting to get plugin server address for RPC: Failed to dial the plugin server in 10s"
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
@hadarsharon This looks like the libvirt setup might be broken for you, what is the output of
ls -lZ ~/.crc/machines/crc/crc
in my case it is following
EDIT: got it fixed (see below)..
Hi @praveenkumar , I got exactly the same issue (with crc file ownership , beeing owned somehow by my own user, not qemu), maybe we should have a separate issue for this one!?
INFO Creating CodeReady Containers VM for OpenShift 4.4.3...
ERRO Error creating host: Error creating the VM: Error creating machine: Error in driver during machine creation: virError(Code=38, Domain=18, Message='Cannot access storage file '/home/lhoss/.crc/machines/crc/crc' (as uid:107, gid:107): Permission denied')
here the permissions:
[lhoss@localhost .crc]$ ls -lZ ~/.crc/machines/crc/crc
-rw-------. 1 lhoss lhoss unconfined_u:object_r:unlabeled_t:s0 9769582592 May 29 12:40 /home/lhoss/.crc/machines/crc/crc
# In my case I had to move my .crc dir to a larger partition, and use a symlink. Here ls -Z:
[lhoss@localhost .crc]$ ls -lZart ~/.crc /mnt/xfs_200g
lrwxrwxrwx. 1 lhoss lhoss unconfined_u:object_r:user_home_t:s0 18 May 29 12:27 /home/lhoss/.crc -> /mnt/xfs_200g/.crc
/mnt/xfs_200g:
total 0
drwx------. 3 lhoss lhoss system_u:object_r:unlabeled_t:s0 18 May 29 12:26 .
drwxr-xr-x. 1 root root system_u:object_r:mnt_t:s0 52 May 29 12:28 ..
drwxrwxr-x. 6 lhoss lhoss unconfined_u:object_r:unlabeled_t:s0 104 May 29 12:44 .crc
Infos on my OS and libvirt: Fedora 31, with latest updates and kernel (updated 3h ago)
UPDATE: Got it fixed :)
My mountpoint /mnt/xfs_200g
had to restrictive permissions, so no wonder one get's permission denied issue for non lhoss user. Fixed via chmod 755 ....
General Information
CRC Version
crc version: 1.3.0+918756b OpenShift version: 4.2.10 (embedded in binary)
CRC Status
CRC VM: Stopped OpenShift: Stopped Disk Usage: 0B of 0B (Inside the CRC VM) Cache Usage: 13.89GB Cache Directory: /Users/tq@domain.com/.crc/cache
Issue
When I run "crc start" this initially runs ok, but then hangs.
Running "crc start --log-level debug" shows that the start up hangs here (i did "crc delete -f", then "crc setup" before running with the debug flag, so as to recreate from the beginning):
Using API Version 1 () Calling .SetConfigRaw () Calling .GetMachineName DEBU Waiting until ssh is available (crc) Calling .GetSSHHostname (crc) Calling .GetSSHPort (crc) Calling .GetSSHKeyPath (crc) Calling .GetSSHKeyPath (crc) Calling .GetSSHUsername Using SSH client type: external Using SSH private key: /Users/tq@domain.com/.crc/cache/crc_hyperkit_4.2.10/id_rsa_crc (-r--------) &{[-F /dev/null -o ConnectionAttempts=3 -o ConnectTimeout=10 -o ControlMaster=no -o ControlPath=none -o LogLevel=quiet -o PasswordAuthentication=no -o ServerAliveInterval=60 -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null core@192.168.64.7 -o IdentitiesOnly=yes -i /Users/tq@domain.com/.crc/cache/crc_hyperkit_4.2.10/id_rsa_crc -p 22] /usr/bin/ssh <nil>} DEBU About to run SSH command: exit 0 DEBU SSH cmd err, output: exit status 255: DEBU error: Temporary Error: ssh command error: command : exit 0 err : exit status 255 output : - sleeping 1s DEBU retry loop 1 (crc) Calling .GetSSHHostname (crc) Calling .GetSSHPort (crc) Calling .GetSSHKeyPath (crc) Calling .GetSSHKeyPath (crc) Calling .GetSSHUsername Using SSH client type: external Using SSH private key: /Users/tq@domain.com/.crc/cache/crc_hyperkit_4.2.10/id_rsa_crc (-r--------) &{[-F /dev/null -o ConnectionAttempts=3 -o ConnectTimeout=10 -o ControlMaster=no -o ControlPath=none -o LogLevel=quiet -o PasswordAuthentication=no -o ServerAliveInterval=60 -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null core@192.168.64.7 -o IdentitiesOnly=yes -i /Users/tq@domain.com/.crc/cache/crc_hyperkit_4.2.10/id_rsa_crc -p 22] /usr/bin/ssh <nil>} DEBU About to run SSH command: exit 0 DEBU SSH cmd err, output: exit status 255: DEBU error: Temporary Error: ssh command error: command : exit 0 err : exit status 255 output : - sleeping 1s
This loop continues indefinitely until I cancel the start process with crtl + c.
Up until this point the only changes I had to make was adding my user to the 'wheel' group and giving wheel group rw permissions on /etc/host.
Any help appreciated. Many thanks.
Attached crc.log file. crc.log