panic: error relabeling required files

dirien commented 3 years ago

Is this a BUG REPORT or FEATURE REQUEST?:

Uncomment only one, leave it on its own line:

/kind bug /kind enhancement

What happened:

Followed the install instructions under https://kubevirt.io/quickstart_cloud/ on a STACKIT SKE 1.19 Kubernetes.

Every other pod is running fine, except the virt handler

It panics with following error:

➜  ~ k logs virt-handler-lxxjz -n kubevirt
{"component":"virt-handler","hostname":"shoot--p1t9r9jxg1--kubevirt-worker-tlx5n-z1-5df6d-qmtzl","level":"info","pos":"virt-handler.go:183","timestamp":"2021-05-28T19:31:02.128104Z"}
W0528 19:31:02.131403   63743 client_config.go:614] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
{"component":"virt-handler","level":"info","msg":"set verbosity to 2","pos":"virt-handler.go:427","timestamp":"2021-05-28T19:31:02.153896Z"}
{"component":"virt-handler","level":"info","msg":"Starting collector: node name=shoot--p1t9r9jxg1--kubevirt-worker-tlx5n-z1-5df6d-qmtzl","pos":"prometheus.go:556","timestamp":"2021-05-28T19:31:02.153941Z"}
{"component":"virt-handler","level":"info","msg":"STARTING informer CRDInformer","pos":"virtinformers.go:261","timestamp":"2021-05-28T19:31:02.154020Z"}
{"component":"virt-handler","level":"info","msg":"STARTING informer kubeVirtInformer","pos":"virtinformers.go:261","timestamp":"2021-05-28T19:31:02.154067Z"}
{"component":"virt-handler","level":"info","msg":"STARTING informer vmiInformer-sources","pos":"virtinformers.go:261","timestamp":"2021-05-28T19:31:02.154087Z"}
{"component":"virt-handler","level":"info","msg":"STARTING informer vmiInformer-targets","pos":"virtinformers.go:261","timestamp":"2021-05-28T19:31:02.154114Z"}
{"component":"virt-handler","level":"info","msg":"STARTING informer extensionsKubeVirtCAConfigMapInformer","pos":"virtinformers.go:261","timestamp":"2021-05-28T19:31:02.154133Z"}
{"component":"virt-handler","level":"info","msg":"STARTING informer configMapInformer","pos":"virtinformers.go:261","timestamp":"2021-05-28T19:31:02.154147Z"}
{"component":"virt-handler","level":"info","msg":"metrics: max concurrent requests=3","pos":"virt-handler.go:436","timestamp":"2021-05-28T19:31:02.154221Z"}
{"component":"virt-handler","level":"info","msg":"certificate with common name 'kubevirt.io:system:client:virt-handler' retrieved.","pos":"cert-manager.go:198","timestamp":"2021-05-28T19:31:02.154506Z"}
{"component":"virt-handler","level":"info","msg":"certificate with common name 'kubevirt.io:system:node:virt-handler' retrieved.","pos":"cert-manager.go:198","timestamp":"2021-05-28T19:31:02.154517Z"}
{"component":"virt-handler","level":"info","msg":"SELinux is reported as 'permissive'","pos":"virt-handler.go:335","timestamp":"2021-05-28T19:31:02.160496Z"}
{"component":"virt-handler","level":"info","msg":"Updating cluster config from KubeVirt to resource version '9394'","pos":"config-map.go:516","timestamp":"2021-05-28T19:31:02.161825Z"}
{"component":"virt-handler","level":"info","msg":"set verbosity to 2","pos":"virt-handler.go:427","timestamp":"2021-05-28T19:31:02.161856Z"}
{"component":"virt-handler","level":"warning","msg":"Permissive mode, ignoring 'semodule' failure: out: \"libsemanage.semanage_create_store: Could not access module store at /var/lib/selinux/mcs, or it is not a directory. (Read-only file system).\\nlibsemanage.semanage_direct_connect: could not establish direct connection (Read-only file system).\\n/sbin/semodule:  Could not connect to policy handler\\n\", error: exit status 1","pos":"labels.go:102","timestamp":"2021-05-28T19:31:02.168777Z"}
panic: error relabeling required files: error relabeling file /dev/net/tun with label system_u:object_r:container_file_t:s0. Reason: exit status 1

goroutine 1 [running]:
main.(*virtHandlerApp).Run(0xc000236300)
    cmd/virt-handler/virt-handler.go:346 +0x2c94
main.main()
    cmd/virt-handler/virt-handler.go:546 +0x6e

What you expected to happen: That everything is up and running:

By default KubeVirt will deploy 7 pods, 3 services, 1 daemonset, 3 deployment apps, 3 replica sets.

How to reproduce it (as minimally and precisely as possible):

Anything else we need to know?:

Environment:

KubeVirt version (use virtctl version): v0.41.0

Kubernetes version (use kubectl version):

Client Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.2", GitCommit:"f5743093fd1c663cb0cbc89748f730662345d44d", GitTreeState:"clean", BuildDate:"2020-09-16T13:41:02Z", GoVersion:"go1.15", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.2", GitCommit:"f5743093fd1c663cb0cbc89748f730662345d44d", GitTreeState:"clean", BuildDate:"2020-09-16T13:32:58Z", GoVersion:"go1.15", Compiler:"gc", Platform:"linux/amd64"}

Cloud provider or hardware configuration: STACKIT (Gardener on Opentstack)
OS (e.g. from /etc/os-release): Flatcar Linux

dirien commented 3 years ago

Link exising issue to this https://github.com/kubevirt/kubevirt/issues/5298

dirien commented 3 years ago

So is using Flarcar not possible for kubevirt? Do i need to ask my kubernetes provider to change something?

rmohr commented 3 years ago

It is interesting that that we can't change this file:

panic: error relabeling required files: error relabeling file /dev/net/tun with label system_u:object_r:container_file_t:s0. Reason: exit status 1

This line

{"component":"virt-handler","level":"warning","msg":"Permissive mode, ignoring 'semodule' failure: out: \"libsemanage.semanage_create_store: Could not access module store at /var/lib/selinux/mcs, or it is not a directory. (Read-only file system).\\nlibsemanage.semanage_direct_connect: could not establish direct connection (Read-only file system).\\n/sbin/semodule:  Could not connect to policy handler\\n\", error: exit status 1","pos":"labels.go:102","timestamp":"2021-05-28T19:31:02.168777Z"}

indicates that selinux may be set up in a non-standard way. Can you influence that somehow on STACKIT?

dirien commented 3 years ago

@rmohr, we use a standart non customise Flatcar image. I just verified it, starting Flatcar in Openstack and spawn RKE on it.

kubevirt-bot commented 3 years ago

Issues go stale after 90d of inactivity. Mark the issue as fresh with /remove-lifecycle stale. Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

/lifecycle stale

dirien commented 3 years ago

I close this isse, i can't solve it on my own. :(

rmohr commented 3 years ago

I close this isse, i can't solve it on my own. :(

Sorry I missed your response.

We try to install a selinux policy by default. The issue is probably that the directories are read-only on the filesystem to avoid that someone can tamper with selinux. We would have to change our permissive mode to still continue if we can't install the policies.

rmohr commented 3 years ago

Looks like this may make selinux writeable on the nodes: https://kinvolk.io/docs/flatcar-container-linux/latest/setup/security/selinux/#check-a-containers-compatibility-with-selinux-policy

rmohr commented 3 years ago

@dirien the daily developer build contains #6377 now. If you are still interested you can give it a try: https://kubevirt.io/user-guide/operations/installation/#installing-the-daily-developer-builds.

kubevirt / kubevirt

panic: error relabeling required files #5736