OSD init container fails on minikube

[jmdots]

Is this a request for help?: Yes

Is this a BUG REPORT or FEATURE REQUEST? (choose one): Bug?

Version of Helm and Kubernetes:

Client Version: version.Info{Major:"1", Minor:"7", GitVersion:"v1.7.5", GitCommit:"17d7182a7ccbb167074be7a87f0a68bd00d58d97", GitTreeState:"clean", BuildDate:"2017-08-31T09:14:02Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"8", GitVersion:"v1.8.0", GitCommit:"0b9efaeb34a2fc51ff8e4d34ad9bc6375459c4a4", GitTreeState:"clean", BuildDate:"2017-11-29T22:43:34Z", GoVersion:"go1.9.1", Compiler:"gc", Platform:"linux/amd64"}

Which chart:

ceph

What happened:

The OSD container fails on this line:

https://github.com/ceph/ceph-helm/blob/fc3cddac3b34ea9d97a77fe80ed77cd2280b42f2/ceph/ceph/templates/bin/_osd_disk_prepare.sh.tpl#L19

What you expected to happen:

Success

How to reproduce it (as minimally and precisely as possible):

• Start minikube in virtualbox mode. Attach a disk which becomes /dev/sdb. Deploy ceph-helm.

Anything else we need to know:

MON is also failing, but it appears to be failing due to a lack of storage.

$  kubectl logs -n ceph ceph-osd-sdb-jhfd8 -c osd-prepare-pod
+ export LC_ALL=C
+ LC_ALL=C
+ source variables_entrypoint.sh
++ ALL_SCENARIOS='osd osd_directory osd_directory_single osd_ceph_disk osd_ceph_disk_prepare osd_ceph_disk_activate osd_ceph_activate_journal mgr'
++ : ceph
++ : ceph-config/ceph
++ :
++ : osd_ceph_disk_prepare
++ : 1
++ : minikube
++ : minikube
++ : /etc/ceph/monmap-ceph
++ : /var/lib/ceph/mon/ceph-minikube
++ : 0
++ : 0
++ : mds-minikube
++ : 0
++ : 100
++ : 0
++ : 0
+++ uuidgen
++ : 9c0528e0-da2b-4801-a920-4e476b4c6a69
+++ uuidgen
++ : 19e4efd2-405f-4c70-b9fd-0c18d937fbdb
++ : root=default host=minikube
++ : 0
++ : cephfs
++ : cephfs_data
++ : 8
++ : cephfs_metadata
++ : 8
++ : minikube
++ :
++ :
++ : 8080
++ : 0
++ : 9000
++ : 0.0.0.0
++ : cephnfs
++ : minikube
++ : 0.0.0.0
++ CLI_OPTS='--cluster ceph'
++ DAEMON_OPTS='--cluster ceph --setuser ceph --setgroup ceph -d'
++ MOUNT_OPTS='-t xfs -o noatime,inode64'
++ MDS_KEYRING=/var/lib/ceph/mds/ceph-mds-minikube/keyring
++ ADMIN_KEYRING=/etc/ceph/ceph.client.admin.keyring
++ MON_KEYRING=/etc/ceph/ceph.mon.keyring
++ RGW_KEYRING=/var/lib/ceph/radosgw/minikube/keyring
++ MGR_KEYRING=/var/lib/ceph/mgr/ceph-minikube/keyring
++ MDS_BOOTSTRAP_KEYRING=/var/lib/ceph/bootstrap-mds/ceph.keyring
++ RGW_BOOTSTRAP_KEYRING=/var/lib/ceph/bootstrap-rgw/ceph.keyring
++ OSD_BOOTSTRAP_KEYRING=/var/lib/ceph/bootstrap-osd/ceph.keyring
++ OSD_PATH_BASE=/var/lib/ceph/osd/ceph
+ source common_functions.sh
++ set -ex
+ is_available rpm
+ command -v rpm
+ is_available dpkg
+ command -v dpkg
+ OS_VENDOR=ubuntu
+ source /etc/default/ceph
++ TCMALLOC_MAX_TOTAL_THREAD_CACHE_BYTES=134217728
+ case "$CEPH_DAEMON" in
+ OSD_TYPE=prepare
+ start_osd
+ [[ ! -e /etc/ceph/ceph.conf ]]
+ '[' 1 -eq 1 ']'
+ [[ ! -e /etc/ceph/ceph.client.admin.keyring ]]
+ case "$OSD_TYPE" in
+ source osd_disk_prepare.sh
++ set -ex
+ osd_disk_prepare
+ [[ -z /dev/sdb ]]
+ [[ ! -e /dev/sdb ]]
+ '[' '!' -e /var/lib/ceph/bootstrap-osd/ceph.keyring ']'
+ timeout 10 ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring health
+ exit 1

[XericZephyr]

Facing same issue. No idea how to debug.

[rootfs]

Need to see what is going on with ceph mon container, do you have logs to share?

[rootfs]

@jmdots the above osd bootstrap failure was due to mon failure. As observed, osd tried to determine mon health, but at that time, mon also failed, so that caused the trouble.

[whmzsu]

Same issue. Not enough logs to debug from the kubectl logs

[githubfoam]

I am running minikube on ubuntu.Problem is that ceph is trying to find those directories which are obviously missing.I tried to create manually and did not work.Minikube dashboard says it all anyway.

/etc/ceph /var/lib

++ MDS_KEYRING=/var/lib/ceph/mds/ceph-mds-minikube/keyring ++ ADMIN_KEYRING=/etc/ceph/ceph.client.admin.keyring ++ MON_KEYRING=/etc/ceph/ceph.mon.keyring ++ RGW_KEYRING=/var/lib/ceph/radosgw/minikube/keyring ++ MGR_KEYRING=/var/lib/ceph/mgr/ceph-minikube/keyring ++ MDS_BOOTSTRAP_KEYRING=/var/lib/ceph/bootstrap-mds/ceph.keyring ++ RGW_BOOTSTRAP_KEYRING=/var/lib/ceph/bootstrap-rgw/ceph.keyring ++ OSD_BOOTSTRAP_KEYRING=/var/lib/ceph/bootstrap-osd/ceph.keyring ++ OSD_PATH_BASE=/var/lib/ceph/osd/ceph

Is it possible to create those missing keyrings with cephx and copy them to those missind directories? Cephx Config Reference

How can I run ceph-authtool with minikube?

2.Add the keys to the related keyrings. First for client.admin, then for monitors, and then other related services, such as OSD, RADOS Gateway, or MDS

4.Create a new keyring and import keys from the admin and monitors' keyrings there. Then use them to start the monitors: ceph-authtool --create-keyring /tmp/tmpuuhxm3/keyring \ --import-keyring /var/lib/ceph/bootstrap-mon/ceph-osceph-03.keyring ceph-authtool /tmp/tmpuuhxm3/keyring \ --import-keyring /etc/ceph/ceph.client.admin.keyring Example Procedure of Manual Ceph Installation

kubectl get po -n ceph

NAME READY STATUS RESTARTS AGE ceph-mds-696bd98bdb-gh2zf 0/1 Pending 0 41m ceph-mds-keyring-generator-z6lmk 0/1 Completed 0 41m ceph-mgr-56f45bb99c-55kvl 0/1 CrashLoopBackOff 11 41m ceph-mgr-keyring-generator-rvlz4 0/1 Completed 0 41m ceph-mon-check-74d98c5b95-72dq6 1/1 Running 0 41m ceph-mon-keyring-generator-tzmtt 0/1 Completed 0 41m ceph-mon-rgkvd 2/3 CrashLoopBackOff 11 41m ceph-namespace-client-key-generator-f7tln 0/1 Completed 0 41m ceph-osd-dev-sdd-pzqck 0/1 Init:CrashLoopBackOff 12 41m ceph-osd-dev-sde-9vlsm 0/1 Init:CrashLoopBackOff 12 41m ceph-osd-keyring-generator-4m7mj 0/1 Completed 0 41m ceph-rbd-provisioner-b58659dc9-f6kgq 1/1 Running 0 41m ceph-rbd-provisioner-b58659dc9-vhsrb 1/1 Running 0 41m ceph-rgw-5bd9dd66c5-kzxcj 0/1 Pending 0 41m ceph-rgw-keyring-generator-wtl9j 0/1 Completed 0 41m ceph-storage-keys-generator-k6z5v 0/1 Completed 0 41m

kubectl describe -n ceph po ceph-osd-dev-sdd-pzqck

Mounts: /common_functions.sh from ceph-bin (ro) /dev from devices (rw) /etc/ceph/ceph.client.admin.keyring from ceph-client-admin-keyring (ro) /etc/ceph/ceph.conf from ceph-etc (ro) /etc/ceph/ceph.mon.keyring from ceph-mon-keyring (ro) /osd_activate_journal.sh from ceph-bin (ro) /osd_disk_activate.sh from ceph-bin (ro) /osd_disk_prepare.sh from ceph-bin (ro) /osd_disks.sh from ceph-bin (ro) /run from pod-run (rw) /run/udev from run-udev (rw) /start_osd.sh from ceph-bin (ro) /var/lib/ceph from pod-var-lib-ceph (rw) /var/lib/ceph/bootstrap-mds/ceph.keyring from ceph-bootstrap-mds-keyring (ro) /var/lib/ceph/bootstrap-osd/ceph.keyring from ceph-bootstrap-osd-keyring (ro) /var/lib/ceph/bootstrap-rgw/ceph.keyring from ceph-bootstrap-rgw-keyring (ro) /var/run/secrets/kubernetes.io/serviceaccount from default-token-jcxvx (ro) /variables_entrypoint.sh from ceph-bin (ro)

Warning FailedMount 41m (x3 over 41m) kubelet, minikube MountVolume.SetUp failed for volume "ceph-mon-keyring" : secrets "ceph-mon-keyring" not found Warning FailedMount 41m (x3 over 41m) kubelet, minikube MountVolume.SetUp failed for volume "ceph-bootstrap-mds-keyring" : secrets "ceph-bootstrap-mds-keyring" not found Warning FailedMount 41m (x4 over 41m) kubelet, minikube MountVolume.SetUp failed for volume "ceph-client-admin-keyring" : secrets "ceph-client-admin-keyring" not found Warning FailedMount 41m (x4 over 41m) kubelet, minikube MountVolume.SetUp failed for volume "ceph-bootstrap-osd-keyring" : secrets "ceph-bootstrap-osd-keyring" not found Warning FailedMount 41m (x4 over 41m) kubelet, minikube MountVolume.SetUp failed for volume "ceph-bootstrap-rgw-keyring" : secrets "ceph-bootstrap-rgw-keyring" not found Normal Pulled 36m (x5 over 38m) kubelet, minikube Container image "docker.io/ceph/daemon:tag-build-master-luminous-ubuntu-16.04" already present on machine Warning BackOff 1m (x165 over 38m) kubelet, minikube Back-off restarting failed container

minikube dashboard-> select namespace ceph->pods

MountVolume.SetUp failed for volume "ceph-bootstrap-rgw-keyring" : secrets "ceph-bootstrap-rgw-keyring" not found Back-off restarting failed container

minikube dashboard-> select namespace ceph->pods->actions-> view/edit YAML

{ "name": "ceph-mon-keyring", "readOnly": true, "mountPath": "/etc/ceph/ceph.mon.keyring", "subPath": "ceph.mon.keyring" },

{ "name": "ceph-bootstrap-osd-keyring", "readOnly": true, "mountPath": "/var/lib/ceph/bootstrap-osd/ceph.keyring", "subPath": "ceph.keyring" },

`kubectl get secrets -n ceph NAME TYPE DATA AGE ceph-bootstrap-mds-keyring Opaque 1 1h ceph-bootstrap-mgr-keyring Opaque 1 1h ceph-bootstrap-osd-keyring Opaque 1 1h ceph-bootstrap-rgw-keyring Opaque 1 1h ceph-client-admin-keyring Opaque 1 1h ceph-keystone-user-rgw Opaque 7 1h ceph-mon-keyring Opaque 1 1h default-token-jcxvx kubernetes.io/service-account-token 3 1h pvc-ceph-client-key kubernetes.io/rbd 1 1h pvc-ceph-conf-combined-storageclass kubernetes.io/rbd 1 1h

kubectl -n ceph edit secrets/ceph-bootstrap-osd-keyring