crc start failing with "Failed to add podman host dns entry"

[Sanne]

General information

• OS: Linux , Fedora 37 with all system updates.
• Hypervisor: I'm not sure - the default? All of KVM, qemu, libvirt, virsh, etc.. are available.
• Did you run crc setup before starting it? Yes
• Running CRC on: Baremetal-Workstation

CRC version

CRC version: 2.13.1+3b466f8
OpenShift version: 4.12.0
Podman version: 4.3.1

(upgraded today to crc-linux-2.13.1-amd64 - had the same issue with crc-linux-2.12.0-amd64)

CRC status

DEBU CRC version: 2.13.1+3b466f8                  
DEBU OpenShift version: 4.12.0                    
DEBU Podman version: 4.3.1                        
DEBU Running 'crc status'                         
Machine does not exist. Use 'crc start' to create it

CRC config

- consent-telemetry                     : no
- preset                                : podman

Host Operating System

NAME="Fedora Linux"
VERSION="37 (Workstation Edition)"
ID=fedora
VERSION_ID=37
VERSION_CODENAME=""
PLATFORM_ID="platform:f37"
PRETTY_NAME="Fedora Linux 37 (Workstation Edition)"
ANSI_COLOR="0;38;2;60;110;180"
LOGO=fedora-logo-icon
CPE_NAME="cpe:/o:fedoraproject:fedora:37"
DEFAULT_HOSTNAME="fedora"
HOME_URL="https://fedoraproject.org/"
DOCUMENTATION_URL="https://docs.fedoraproject.org/en-US/fedora/f37/system-administrators-guide/"
SUPPORT_URL="https://ask.fedoraproject.org/"
BUG_REPORT_URL="https://bugzilla.redhat.com/"
REDHAT_BUGZILLA_PRODUCT="Fedora"
REDHAT_BUGZILLA_PRODUCT_VERSION=37
REDHAT_SUPPORT_PRODUCT="Fedora"
REDHAT_SUPPORT_PRODUCT_VERSION=37
SUPPORT_END=2023-11-14
VARIANT="Workstation Edition"
VARIANT_ID=workstation

Steps to reproduce

1. Download crc-linux-amd64.tar.xz , unpack, add it on path... followed the instructions on redhat.com product documentation.
2. crc setup
3. crc start

Expected

Some kind of successfull start.

Actual

It fails to start, last message is:

Failed to add podman host dns entry: exit status 1

Logs

Uploaded: https://gist.githubusercontent.com/Sanne/6df940bae0e3d09ec34d865a4f2e709a/raw/e46f3b75e50d31668da84343caeb1872396512d1/crc%2520startup%2520failure

It was previously suggested to me to check the permissions of /etc.hosts ;

Running ls -la /etc/hosts :

-rw-r--r-- 1 root root 182 May  5  2022 /etc/hosts

Thanks!

[praveenkumar]

@Sanne looking into following logs looks like crc-admin-helper-linux file permission is not correct

DEBU Using root access: make root Podman socket accessible 
DEBU Running SSH command: sudo chmod 777 /run/podman/ /run/podman/podman.sock 
DEBU SSH command results: err: <nil>, output:     
DEBU Running '/home/sanne/.crc/bin/crc-admin-helper-linux rm podman.crc.testing' 

Can you check if crc-admin-helper-linux permission is same as this? if not can you try crc setup --log-level debug and share the logs?

ls -l $HOME/.crc/bin/
-r-s--x---. 1 root    prkumar  5212680 Dec 24 10:01 crc-admin-helper-linux
-r-x------. 1 prkumar prkumar 12606528 Aug  4 15:58 crc-driver-libvirt

[Sanne]

It would seem I have the same permissions:

-r-s--x--- 1 root  sanne  5212680 Jan 26 00:00 crc-admin-helper-linux
-r-x------ 1 sanne sanne 12606528 Jan 26 00:00 crc-driver-libvirt

[cfergeau]

Can you try running ~/.crc/bin/crc-admin-helper-linux add 127.0.0.127 issue3490.crc.testing manually and see if this adds an entry to /etc/hosts? Could it be selinux blocking the changes to /etc/hosts? If the previous test fails, can you try again after sudo setenforce 0?

[Sanne]

I've run ~/.crc/bin/crc-admin-helper-linux add 127.0.0.127 issue3490.crc.testing and got:

Error: host file not writable, try running with elevated privileges host file not writable, try running with elevated privileges

No difference even after sudo setenforce 0.

Then tried sudo ~/.crc/bin/crc-admin-helper-linux add 127.0.0.127 issue3490.crc.testing ... that worked, but I understand it shouldn't be necessary?

[cfergeau]

~/.crc/bin/crc-admin-helper-linux is suid root, so sudo should not be making a difference :-/ Maybe the mount options of ~/.crc/bin disable suid?

[Sanne]

Maybe the mount options of ~/.crc/bin disable suid?

Nailed it!

mount | grep home : /dev/nvme1n1p1 on /home type ext4 (rw,nosuid,nodev,relatime)

Thanks - not sure how this option got there.

[cfergeau]

Thanks - not sure how this option got there.

Just tried a fresh f37 install and I get:

/dev/vda3 on /home type btrfs (rw,relatime,seclabel,compress=zstd:1,space_cache=v2,subvolid=256,subvol=/home)

so fedora does not seem to disable this by default.

[Sanne]

Interesting - I might have inherited it from an older Fedora version, I've been doing upgrades for some years.

Bottomline, that might have been my own fault, but I do wonder if it's worth patching the crc installation process to be more robus about this; I wouldn't consider it out of this world for more people to disallow suid binaries they download from "the internet"...

What do you prefer, should we mark this issue resolved, or would you rather keep it open to potentially improve the sanity checks and/or error messages?

[praveenkumar]

@Sanne keep it open we can add it as part of preflight check.

[gbraad]

Issue https://github.com/crc-org/crc/issues/2119 also refers to the use of nosuid.