Monitor reboot after create storageclass

[lincate]

Is this a bug report or feature request?

• Bug Report

Deviation from expected behavior: After create Storageclass rook-ceph-block, provisioner: rook-ceph.rbd.csi.ceph.com, monitor will rebooting again and again. Then, the PVC which reference rook-ceph-block status is pending.

Expected behavior: Storageclass can be created, PVC can be bound.

How to reproduce it (minimal and precise): kubectl create -f crds.yaml -f commons.yaml -f operator.yaml kubectl create -f cluster.yaml after that, ceph-cluster status is HEALTH_OK then kubectl create -f storageclass.yaml

File(s) to submit:

• Cluster CR (custom resource), typically called cluster.yaml, if necessary

Logs to submit: Info: Checking mon quorum and ceph health details HEALTH_OK

after create storageclass: one of monitor log is: debug 2022-09-29T09:10:21.318+0000 7f937291a700 0 log_channel(cluster) log [DBG] : fsmap debug 2022-09-29T09:10:21.318+0000 7f937291a700 0 log_channel(cluster) log [DBG] : osdmap e25: 3 total, 3 up, 3 in debug 2022-09-29T09:10:21.319+0000 7f937291a700 0 log_channel(cluster) log [DBG] : mgrmap e30: no daemons active (since 18s) debug 2022-09-29T09:10:21.320+0000 7f937291a700 0 log_channel(cluster) log [WRN] : Health check update: 1/3 mons down, quorum a,c (MON_DOWN) debug 2022-09-29T09:10:59.737+0000 7f9371117700 1 mon.a@0(leader) e3 handle_auth_request failed to assign global_id debug 2022-09-29T09:11:15.942+0000 7f9370916700 1 mon.a@0(leader) e3 handle_auth_request failed to assign global_id debug 2022-09-29T09:11:25.176+0000 7f9378125700 -1 received signal: Terminated from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() ) UID: 0 debug 2022-09-29T09:11:25.176+0000 7f9378125700 -1 mon.a@0(leader) e3 Got Signal Terminated debug 2022-09-29T09:11:25.176+0000 7f9378125700 1 mon.a@0(leader) e3 shutdown

• Operator's logs, if necessary

• Crashing pod(s) logs, if necessary

  To get logs, use kubectl -n <namespace> logs <pod name> When pasting logs, always surround them with backticks or use the insert code button from the Github UI. Read GitHub documentation if you need help.

Cluster Status to submit:

• Output of krew commands, if necessary

  To get the health of the cluster, use kubectl rook-ceph health To get the status of the cluster, use kubectl rook-ceph ceph status For more details, see the Rook Krew Plugin

Environment:

• OS (e.g. from /etc/os-release): Centos 7
• Kernel (e.g. uname -a): Linux 4.18.0-147.5.1.6.h579.x86_64
• Cloud provider or hardware configuration: on-prem
• Rook version (use rook version inside of a Rook Pod): 1.10.1 & 1.10.0
• Storage backend version (e.g. for ceph do ceph -v): 17.2.3
• Kubernetes version (use kubectl version): 1.25.0 & 1.24.2
• Kubernetes cluster type (e.g. Tectonic, GKE, OpenShift): bare metal
• Storage backend status (e.g. for Ceph use ceph health in the Rook Ceph toolbox):

[travisn]

Some questions:

• This is a default install, or did you modify the cluster.yaml? If you added resource limits on the mons, they may need to be increased
• The liveness probes may be failing for the mons. You could try disabling the liveness probes to see if it helps.

[shangjin92]

It maybe similar to https://github.com/rook/rook/issues/10110

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in a week if no further activity occurs. Thank you for your contributions.

[github-actions[bot]]

This issue has been automatically closed due to inactivity. Please re-open if this still requires investigation.