search

ceph / ceph-helm

Curated applications for Kubernetes
Apache License 2.0
108 stars 36 forks source link

[secrets "ceph-bootstrap-osd-keyring" not found] ceph-osd fail on k8s 1.10.1 install by kubeadm #55

Open foxundermoon opened 6 years ago

foxundermoon commented 6 years ago 

kubectl get po -n ceph
NAME                                            READY     STATUS                  RESTARTS   AGE
ceph-mds-696bd98bdb-bnvpg                       0/1       Pending                 0          18m
ceph-mds-keyring-generator-q679r                0/1       Completed               0          18m
ceph-mgr-6d5f86d9c4-nr76h                       1/1       Running                 1          18m
ceph-mgr-keyring-generator-v825z                0/1       Completed               0          18m
ceph-mon-86lth                                  1/1       Running                 0          18m
ceph-mon-check-74d98c5b95-wf9tm                 1/1       Running                 0          18m
ceph-mon-keyring-generator-rfg8j                0/1       Completed               0          18m
ceph-mon-pp5hc                                  1/1       Running                 0          18m
ceph-namespace-client-key-cleaner-g9dri-sjmqd   0/1       Completed               0          1h
ceph-namespace-client-key-cleaner-qwkee-pdkh6   0/1       Completed               0          21m
ceph-namespace-client-key-cleaner-t25ui-5gkb7   0/1       Completed               0          2d
ceph-namespace-client-key-generator-xk4w6       0/1       Completed               0          18m
ceph-osd-dev-sda-6jbgd                          0/1       Init:CrashLoopBackOff   8          18m
ceph-osd-dev-sda-khfhw                          0/1       Init:CrashLoopBackOff   8          18m
ceph-osd-dev-sda-krkjf                          0/1       Init:CrashLoopBackOff   8          18m
ceph-osd-keyring-generator-mvktj                0/1       Completed               0          18m
ceph-rbd-provisioner-b58659dc9-nhx2q            1/1       Running                 0          18m
ceph-rbd-provisioner-b58659dc9-nnlh2            1/1       Running                 0          18m
ceph-rgw-5bd9dd66c5-gh946                       0/1       Pending                 0          18m
ceph-rgw-keyring-generator-dz9kd                0/1       Completed               0          18m
ceph-storage-admin-key-cleaner-1as0t-fq589      0/1       Completed               0          1h
ceph-storage-admin-key-cleaner-oayjp-fglzr      0/1       Completed               0          2d
ceph-storage-admin-key-cleaner-zemvx-jxn7c      0/1       Completed               0          21m
ceph-storage-keys-generator-szps9               0/1       Completed               0          18m

Version of Helm and Kubernetes:

Which chart:

commit 70681c8218e75bb10acc7dc210b791b69545ce0d
Merge: a4fa8a1 6cb2e1e
Author: Huamin Chen <rootfs@users.noreply.github.com>
Date:   Thu Apr 26 12:11:43 2018 -0400
kubectl describe -n ceph po  ceph-osd-dev-sda-6jbgd
Name:           ceph-osd-dev-sda-6jbgd
Namespace:      ceph
Node:           k8s-2/192.168.16.40
Start Time:     Sat, 28 Apr 2018 15:54:08 +0800
Labels:         application=ceph
                component=osd
                controller-revision-hash=1450926272
                pod-template-generation=1
                release_group=ceph
Annotations:    <none>
Status:         Pending
IP:             192.168.16.40
Controlled By:  DaemonSet/ceph-osd-dev-sda
Init Containers:
  init:
    Container ID:  docker://8c9536cc4c5d811f57ba6349c87245121651841f52db682f858ae0ac70555856
    Image:         docker.io/kolla/ubuntu-source-kubernetes-entrypoint:4.0.0
    Image ID:      docker-pullable://kolla/ubuntu-source-kubernetes-entrypoint@sha256:75116ab2f9f65c5fc078e68ce7facd66c1c57496947f37b7209b32f94925e53b
    Port:          <none>
    Host Port:     <none>
    Command:
      kubernetes-entrypoint
    State:          Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Sat, 28 Apr 2018 15:54:34 +0800
      Finished:     Sat, 28 Apr 2018 15:54:36 +0800
    Ready:          True
    Restart Count:  0
    Environment:
      POD_NAME:              ceph-osd-dev-sda-6jbgd (v1:metadata.name)
      NAMESPACE:             ceph (v1:metadata.namespace)
      INTERFACE_NAME:        eth0
      DEPENDENCY_SERVICE:    ceph-mon
      DEPENDENCY_JOBS:
      DEPENDENCY_DAEMONSET:
      DEPENDENCY_CONTAINER:
      COMMAND:               echo done
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-z5m75 (ro)
  ceph-init-dirs:
    Container ID:  docker://1562879ebbc52c47cfd9fb292339e548d26450207846ff6eeb38594569d5ec5f
    Image:         docker.io/ceph/daemon:tag-build-master-luminous-ubuntu-16.04
    Image ID:      docker-pullable://ceph/daemon@sha256:687056228e899ecbfd311854e3864db0b46dd4a9a6d4eb4b47c815ca413f25ee
    Port:          <none>
    Host Port:     <none>
    Command:
      /tmp/init_dirs.sh
    State:          Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Sat, 28 Apr 2018 15:54:38 +0800
      Finished:     Sat, 28 Apr 2018 15:54:39 +0800
    Ready:          True
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /run from pod-run (rw)
      /tmp/init_dirs.sh from ceph-bin (ro)
      /var/lib/ceph from pod-var-lib-ceph (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-z5m75 (ro)
      /variables_entrypoint.sh from ceph-bin (ro)
  osd-prepare-pod:
    Container ID:  docker://2b5bed33de8f35533eb72ef3208010153b904a8ed34c527a4916b88f549d5f6b
    Image:         docker.io/ceph/daemon:tag-build-master-luminous-ubuntu-16.04
    Image ID:      docker-pullable://ceph/daemon@sha256:687056228e899ecbfd311854e3864db0b46dd4a9a6d4eb4b47c815ca413f25ee
    Port:          <none>
    Host Port:     <none>
    Command:
      /start_osd.sh
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       Error
      Exit Code:    1
      Started:      Sat, 28 Apr 2018 16:11:06 +0800
      Finished:     Sat, 28 Apr 2018 16:11:07 +0800
    Ready:          False
    Restart Count:  8
    Environment:
      CEPH_DAEMON:         osd_ceph_disk_prepare
      KV_TYPE:             k8s
      CLUSTER:             ceph
      CEPH_GET_ADMIN_KEY:  1
      OSD_DEVICE:          /dev/mapper/centos-root
      HOSTNAME:             (v1:spec.nodeName)
    Mounts:
      /common_functions.sh from ceph-bin (ro)
      /dev from devices (rw)
      /etc/ceph/ceph.client.admin.keyring from ceph-client-admin-keyring (ro)
      /etc/ceph/ceph.conf from ceph-etc (ro)
      /etc/ceph/ceph.mon.keyring from ceph-mon-keyring (ro)
      /osd_activate_journal.sh from ceph-bin (ro)
      /osd_disk_activate.sh from ceph-bin (ro)
      /osd_disk_prepare.sh from ceph-bin (ro)
      /osd_disks.sh from ceph-bin (ro)
      /run from pod-run (rw)
      /start_osd.sh from ceph-bin (ro)
      /var/lib/ceph from pod-var-lib-ceph (rw)
      /var/lib/ceph/bootstrap-mds/ceph.keyring from ceph-bootstrap-mds-keyring (ro)
      /var/lib/ceph/bootstrap-osd/ceph.keyring from ceph-bootstrap-osd-keyring (ro)
      /var/lib/ceph/bootstrap-rgw/ceph.keyring from ceph-bootstrap-rgw-keyring (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-z5m75 (ro)
      /variables_entrypoint.sh from ceph-bin (ro)
Containers:
  osd-activate-pod:
    Container ID:
    Image:         docker.io/ceph/daemon:tag-build-master-luminous-ubuntu-16.04
    Image ID:
    Port:          <none>
    Host Port:     <none>
    Command:
      /start_osd.sh
    State:          Waiting
      Reason:       PodInitializing
    Ready:          False
    Restart Count:  0
    Liveness:       tcp-socket :6800 delay=60s timeout=5s period=10s #success=1 #failure=3
    Readiness:      tcp-socket :6800 delay=0s timeout=5s period=10s #success=1 #failure=3
    Environment:
      CEPH_DAEMON:         osd_ceph_disk_activate
      KV_TYPE:             k8s
      CLUSTER:             ceph
      CEPH_GET_ADMIN_KEY:  1
      OSD_DEVICE:          /dev/mapper/centos-root
      HOSTNAME:             (v1:spec.nodeName)
    Mounts:
      /common_functions.sh from ceph-bin (ro)
      /dev from devices (rw)
      /etc/ceph/ceph.client.admin.keyring from ceph-client-admin-keyring (ro)
      /etc/ceph/ceph.conf from ceph-etc (ro)
      /etc/ceph/ceph.mon.keyring from ceph-mon-keyring (ro)
      /osd_activate_journal.sh from ceph-bin (ro)
      /osd_disk_activate.sh from ceph-bin (ro)
      /osd_disk_prepare.sh from ceph-bin (ro)
      /osd_disks.sh from ceph-bin (ro)
      /run from pod-run (rw)
      /start_osd.sh from ceph-bin (ro)
      /var/lib/ceph from pod-var-lib-ceph (rw)
      /var/lib/ceph/bootstrap-mds/ceph.keyring from ceph-bootstrap-mds-keyring (ro)
      /var/lib/ceph/bootstrap-osd/ceph.keyring from ceph-bootstrap-osd-keyring (ro)
      /var/lib/ceph/bootstrap-rgw/ceph.keyring from ceph-bootstrap-rgw-keyring (ro)
      /var/log/ceph from pod-var-log-ceph (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-z5m75 (ro)
      /variables_entrypoint.sh from ceph-bin (ro)
Conditions:
  Type           Status
  Initialized    False
  Ready          False
  PodScheduled   True
Volumes:
  devices:
    Type:          HostPath (bare host directory volume)
    Path:          /dev
    HostPathType:
  pod-var-lib-ceph:
    Type:    EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
  pod-var-log-ceph:
    Type:          HostPath (bare host directory volume)
    Path:          /var/log/ceph/osd
    HostPathType:
  pod-run:
    Type:    EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:  Memory
  ceph-bin:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      ceph-bin
    Optional:  false
  ceph-etc:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      ceph-etc
    Optional:  false
  ceph-client-admin-keyring:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  ceph-client-admin-keyring
    Optional:    false
  ceph-mon-keyring:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  ceph-mon-keyring
    Optional:    false
  ceph-bootstrap-osd-keyring:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  ceph-bootstrap-osd-keyring
    Optional:    false
  ceph-bootstrap-mds-keyring:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  ceph-bootstrap-mds-keyring
    Optional:    false
  ceph-bootstrap-rgw-keyring:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  ceph-bootstrap-rgw-keyring
    Optional:    false
  default-token-z5m75:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-z5m75
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  ceph-osd=enabled
                 ceph-osd-device-dev-sda=enabled
Tolerations:     node.kubernetes.io/disk-pressure:NoSchedule
                 node.kubernetes.io/memory-pressure:NoSchedule
                 node.kubernetes.io/not-ready:NoExecute
                 node.kubernetes.io/unreachable:NoExecute
Events:
  Type     Reason                 Age                From            Message
  ----     ------                 ----               ----            -------
  Normal   SuccessfulMountVolume  19m                kubelet, k8s-2  MountVolume.SetUp succeeded for volume "pod-var-log-ceph"
  Normal   SuccessfulMountVolume  19m                kubelet, k8s-2  MountVolume.SetUp succeeded for volume "devices"
  Normal   SuccessfulMountVolume  19m                kubelet, k8s-2  MountVolume.SetUp succeeded for volume "pod-var-lib-ceph"
  Normal   SuccessfulMountVolume  19m                kubelet, k8s-2  MountVolume.SetUp succeeded for volume "pod-run"
  Normal   SuccessfulMountVolume  19m                kubelet, k8s-2  MountVolume.SetUp succeeded for volume "ceph-bin"
  Normal   SuccessfulMountVolume  19m                kubelet, k8s-2  MountVolume.SetUp succeeded for volume "ceph-etc"
  Normal   SuccessfulMountVolume  19m                kubelet, k8s-2  MountVolume.SetUp succeeded for volume "default-token-z5m75"
  Warning  FailedMount            19m (x2 over 19m)  kubelet, k8s-2  MountVolume.SetUp failed for volume "ceph-bootstrap-osd-keyring" : secrets "ceph-bootstrap-osd-keyring" not found
  Warning  FailedMount            19m (x2 over 19m)  kubelet, k8s-2  MountVolume.SetUp failed for volume "ceph-bootstrap-rgw-keyring" : secrets "ceph-bootstrap-rgw-keyring" not found
  Warning  FailedMount            19m (x2 over 19m)  kubelet, k8s-2  MountVolume.SetUp failed for volume "ceph-client-admin-keyring" : secrets "ceph-client-admin-keyring" not found
  Normal   SuccessfulMountVolume  19m                kubelet, k8s-2  MountVolume.SetUp succeeded for volume "ceph-client-admin-keyring"
  Normal   SuccessfulMountVolume  19m                kubelet, k8s-2  MountVolume.SetUp succeeded for volume "ceph-bootstrap-osd-keyring"
  Warning  FailedMount            19m (x4 over 19m)  kubelet, k8s-2  MountVolume.SetUp failed for volume "ceph-mon-keyring" : secrets "ceph-mon-keyring" not found
  Warning  FailedMount            19m (x4 over 19m)  kubelet, k8s-2  MountVolume.SetUp failed for volume "ceph-bootstrap-mds-keyring" : secrets "ceph-bootstrap-mds-keyring" not found
  Normal   SuccessfulMountVolume  19m (x2 over 19m)  kubelet, k8s-2  (combined from similar events): MountVolume.SetUp succeeded for volume "ceph-mon-keyring"
  Warning  BackOff                4m (x65 over 18m)  kubelet, k8s-2  Back-off restarting failed container

kubectl  logs  -n ceph   ceph-osd-dev-sda-6jbgd
Error from server (BadRequest): container "osd-activate-pod" in pod "ceph-osd-dev-sda-6jbgd" is waiting to start: PodInitializing
foxundermoon commented 6 years ago 
[tiller] 2018/04/28 07:34:20 preparing install for ceph
[storage] 2018/04/28 07:34:20 getting release history for "ceph"
[tiller] 2018/04/28 07:34:20 rendering ceph chart using values
2018/04/28 07:34:20 info: manifest "ceph/templates/configmap-bin-ks.yaml" is empty. Skipping.
2018/04/28 07:34:20 info: manifest "ceph/templates/job-ks-service.yaml" is empty. Skipping.
2018/04/28 07:34:20 info: manifest "ceph/templates/secret-keystone.yaml" is empty. Skipping.
2018/04/28 07:34:20 info: manifest "ceph/templates/job-bootstrap.yaml" is empty. Skipping.
2018/04/28 07:34:20 info: manifest "ceph/templates/daemonset-osd.yaml" is empty. Skipping.
2018/04/28 07:34:20 info: manifest "ceph/templates/job-ks-endpoints.yaml" is empty. Skipping.
2018/04/28 07:34:20 info: manifest "ceph/templates/job-ceph-commands.yaml" is empty. Skipping.
2018/04/28 07:34:20 info: manifest "ceph/templates/job-ks-user.yaml" is empty. Skipping.
[tiller] 2018/04/28 07:34:20 performing install for ceph
[tiller] 2018/04/28 07:34:20 executing 2 pre-install hooks for ceph
[tiller] 2018/04/28 07:34:20 hooks complete for pre-install ceph
[storage] 2018/04/28 07:34:20 getting release history for "ceph"
[storage] 2018/04/28 07:34:20 creating release "ceph.v1"
[kube] 2018/04/28 07:34:20 building resources from manifest
[kube] 2018/04/28 07:34:20 creating 22 resource(s)
[tiller] 2018/04/28 07:34:20 executing 2 post-install hooks for ceph
[tiller] 2018/04/28 07:34:20 hooks complete for post-install ceph
[storage] 2018/04/28 07:34:20 updating release "ceph.v1"
[storage] 2018/04/28 07:34:21 getting last revision of "ceph"
[storage] 2018/04/28 07:34:21 getting release history for "ceph"
[kube] 2018/04/28 07:34:21 Doing get for Secret: "ceph-keystone-user-rgw"
[kube] 2018/04/28 07:34:21 get relation pod of object: ceph/Secret/ceph-keystone-user-rgw
[kube] 2018/04/28 07:34:21 Doing get for ConfigMap: "ceph-bin-clients"
[kube] 2018/04/28 07:34:21 get relation pod of object: ceph/ConfigMap/ceph-bin-clients
[kube] 2018/04/28 07:34:21 Doing get for ConfigMap: "ceph-bin"
[kube] 2018/04/28 07:34:21 get relation pod of object: ceph/ConfigMap/ceph-bin
[kube] 2018/04/28 07:34:21 Doing get for ConfigMap: "ceph-etc"
[kube] 2018/04/28 07:34:21 get relation pod of object: ceph/ConfigMap/ceph-etc
[kube] 2018/04/28 07:34:21 Doing get for ConfigMap: "ceph-templates"
[kube] 2018/04/28 07:34:21 get relation pod of object: ceph/ConfigMap/ceph-templates
[kube] 2018/04/28 07:34:21 Doing get for StorageClass: "ceph-rbd"
[kube] 2018/04/28 07:34:21 get relation pod of object: /StorageClass/ceph-rbd
[kube] 2018/04/28 07:34:21 Doing get for Service: "ceph-mon"
[kube] 2018/04/28 07:34:21 get relation pod of object: ceph/Service/ceph-mon
[kube] 2018/04/28 07:34:21 Doing get for Service: "ceph-rgw"
[kube] 2018/04/28 07:34:21 get relation pod of object: ceph/Service/ceph-rgw
[kube] 2018/04/28 07:34:21 Doing get for DaemonSet: "ceph-mon"
[kube] 2018/04/28 07:34:21 get relation pod of object: ceph/DaemonSet/ceph-mon
[kube] 2018/04/28 07:34:21 Doing get for DaemonSet: "ceph-osd-dev-sda"
[kube] 2018/04/28 07:34:21 get relation pod of object: ceph/DaemonSet/ceph-osd-dev-sda
[kube] 2018/04/28 07:34:21 Doing get for Deployment: "ceph-mds"
[kube] 2018/04/28 07:34:21 get relation pod of object: ceph/Deployment/ceph-mds
[kube] 2018/04/28 07:34:21 Doing get for Deployment: "ceph-mgr"
[kube] 2018/04/28 07:34:21 get relation pod of object: ceph/Deployment/ceph-mgr
[kube] 2018/04/28 07:34:21 Doing get for Deployment: "ceph-mon-check"
[kube] 2018/04/28 07:34:21 get relation pod of object: ceph/Deployment/ceph-mon-check
[kube] 2018/04/28 07:34:21 Doing get for Deployment: "ceph-rbd-provisioner"
[kube] 2018/04/28 07:34:21 get relation pod of object: ceph/Deployment/ceph-rbd-provisioner
[kube] 2018/04/28 07:34:21 Doing get for Deployment: "ceph-rgw"
[kube] 2018/04/28 07:34:21 get relation pod of object: ceph/Deployment/ceph-rgw
[kube] 2018/04/28 07:34:21 Doing get for Job: "ceph-mds-keyring-generator"
[kube] 2018/04/28 07:34:21 get relation pod of object: ceph/Job/ceph-mds-keyring-generator
[kube] 2018/04/28 07:34:21 Doing get for Job: "ceph-mgr-keyring-generator"
[kube] 2018/04/28 07:34:21 get relation pod of object: ceph/Job/ceph-mgr-keyring-generator
[kube] 2018/04/28 07:34:21 Doing get for Job: "ceph-mon-keyring-generator"
[kube] 2018/04/28 07:34:21 get relation pod of object: ceph/Job/ceph-mon-keyring-generator
[kube] 2018/04/28 07:34:21 Doing get for Job: "ceph-rgw-keyring-generator"
[kube] 2018/04/28 07:34:21 get relation pod of object: ceph/Job/ceph-rgw-keyring-generator
[kube] 2018/04/28 07:34:21 Doing get for Job: "ceph-osd-keyring-generator"
[kube] 2018/04/28 07:34:21 get relation pod of object: ceph/Job/ceph-osd-keyring-generator
[kube] 2018/04/28 07:34:21 Doing get for Job: "ceph-namespace-client-key-generator"
[kube] 2018/04/28 07:34:21 get relation pod of object: ceph/Job/ceph-namespace-client-key-generator
[kube] 2018/04/28 07:34:21 Doing get for Job: "ceph-storage-keys-generator"
[kube] 2018/04/28 07:34:21 get relation pod of object: ceph/Job/ceph-storage-keys-generator
[storage] 2018/04/28 07:50:39 getting release history for "ceph"
[tiller] 2018/04/28 07:50:39 uninstall: Deleting ceph
[tiller] 2018/04/28 07:50:39 executing 2 pre-delete hooks for ceph
[kube] 2018/04/28 07:50:39 building resources from manifest
[kube] 2018/04/28 07:50:39 creating 1 resource(s)
[kube] 2018/04/28 07:50:39 Watching for changes to Job ceph-namespace-client-key-cleaner-qwkee with timeout of 5m0s
[kube] 2018/04/28 07:50:39 Add/Modify event for ceph-namespace-client-key-cleaner-qwkee: ADDED
[kube] 2018/04/28 07:50:39 ceph-namespace-client-key-cleaner-qwkee: Jobs active: 0, jobs failed: 0, jobs succeeded: 0
[kube] 2018/04/28 07:50:40 Add/Modify event for ceph-namespace-client-key-cleaner-qwkee: MODIFIED
[kube] 2018/04/28 07:50:40 ceph-namespace-client-key-cleaner-qwkee: Jobs active: 1, jobs failed: 0, jobs succeeded: 0
[kube] 2018/04/28 07:50:42 Add/Modify event for ceph-namespace-client-key-cleaner-qwkee: MODIFIED
[kube] 2018/04/28 07:50:42 building resources from manifest
[kube] 2018/04/28 07:50:42 creating 1 resource(s)
[kube] 2018/04/28 07:50:42 Watching for changes to Job ceph-storage-admin-key-cleaner-zemvx with timeout of 5m0s
[kube] 2018/04/28 07:50:42 Add/Modify event for ceph-storage-admin-key-cleaner-zemvx: ADDED
[kube] 2018/04/28 07:50:42 ceph-storage-admin-key-cleaner-zemvx: Jobs active: 0, jobs failed: 0, jobs succeeded: 0
[kube] 2018/04/28 07:50:42 Add/Modify event for ceph-storage-admin-key-cleaner-zemvx: MODIFIED
[kube] 2018/04/28 07:50:42 ceph-storage-admin-key-cleaner-zemvx: Jobs active: 1, jobs failed: 0, jobs succeeded: 0
[kube] 2018/04/28 07:50:45 Add/Modify event for ceph-storage-admin-key-cleaner-zemvx: MODIFIED
[tiller] 2018/04/28 07:50:45 hooks complete for pre-delete ceph
[storage] 2018/04/28 07:50:45 updating release "ceph.v1"
[kube] 2018/04/28 07:50:45 Starting delete for "ceph-mon" Service
[kube] 2018/04/28 07:50:45 Starting delete for "ceph-rgw" Service
[kube] 2018/04/28 07:50:45 Starting delete for "ceph-mds-keyring-generator" Job
[kube] 2018/04/28 07:50:45 Using reaper for deleting "ceph-mds-keyring-generator"
[kube] 2018/04/28 07:50:47 Starting delete for "ceph-mgr-keyring-generator" Job
[kube] 2018/04/28 07:50:47 Using reaper for deleting "ceph-mgr-keyring-generator"
[kube] 2018/04/28 07:50:49 Starting delete for "ceph-mon-keyring-generator" Job
[kube] 2018/04/28 07:50:49 Using reaper for deleting "ceph-mon-keyring-generator"
[kube] 2018/04/28 07:50:52 Starting delete for "ceph-rgw-keyring-generator" Job
[kube] 2018/04/28 07:50:52 Using reaper for deleting "ceph-rgw-keyring-generator"
[kube] 2018/04/28 07:50:54 Starting delete for "ceph-osd-keyring-generator" Job
[kube] 2018/04/28 07:50:54 Using reaper for deleting "ceph-osd-keyring-generator"
[kube] 2018/04/28 07:50:56 Starting delete for "ceph-namespace-client-key-generator" Job
[kube] 2018/04/28 07:50:56 Using reaper for deleting "ceph-namespace-client-key-generator"
[kube] 2018/04/28 07:50:58 Starting delete for "ceph-storage-keys-generator" Job
[kube] 2018/04/28 07:50:58 Using reaper for deleting "ceph-storage-keys-generator"
[kube] 2018/04/28 07:51:00 Starting delete for "ceph-mds" Deployment
[kube] 2018/04/28 07:51:00 Using reaper for deleting "ceph-mds"
[kube] 2018/04/28 07:51:03 Starting delete for "ceph-mgr" Deployment
[kube] 2018/04/28 07:51:03 Using reaper for deleting "ceph-mgr"
[kube] 2018/04/28 07:51:06 Starting delete for "ceph-mon-check" Deployment
[kube] 2018/04/28 07:51:06 Using reaper for deleting "ceph-mon-check"
[kube] 2018/04/28 07:51:09 Starting delete for "ceph-rbd-provisioner" Deployment
[kube] 2018/04/28 07:51:09 Using reaper for deleting "ceph-rbd-provisioner"
[kube] 2018/04/28 07:51:12 Starting delete for "ceph-rgw" Deployment
[kube] 2018/04/28 07:51:12 Using reaper for deleting "ceph-rgw"
[kube] 2018/04/28 07:51:15 Starting delete for "ceph-mon" DaemonSet
[kube] 2018/04/28 07:51:15 Using reaper for deleting "ceph-mon"
[kube] 2018/04/28 07:51:58 Starting delete for "ceph-osd-dev-sda" DaemonSet
[kube] 2018/04/28 07:51:58 Using reaper for deleting "ceph-osd-dev-sda"
[kube] 2018/04/28 07:52:58 Starting delete for "ceph-rbd" StorageClass
[kube] 2018/04/28 07:52:59 Starting delete for "ceph-bin-clients" ConfigMap
[kube] 2018/04/28 07:52:59 Starting delete for "ceph-bin" ConfigMap
[kube] 2018/04/28 07:52:59 Starting delete for "ceph-etc" ConfigMap
[kube] 2018/04/28 07:52:59 Starting delete for "ceph-templates" ConfigMap
[kube] 2018/04/28 07:52:59 Starting delete for "ceph-keystone-user-rgw" Secret
2018/04/28 07:52:59 uninstall: Failed deletion of "ceph": no objects visited
[tiller] 2018/04/28 07:52:59 error: object not found, skipping delete
[tiller] 2018/04/28 07:52:59 executing 2 post-delete hooks for ceph
[tiller] 2018/04/28 07:52:59 hooks complete for post-delete ceph
[tiller] 2018/04/28 07:52:59 purge requested for ceph
[storage] 2018/04/28 07:52:59 deleting release "ceph.v1"
[tiller] 2018/04/28 07:54:07 preparing install for ceph
[storage] 2018/04/28 07:54:07 getting release history for "ceph"
[tiller] 2018/04/28 07:54:07 rendering ceph chart using values
2018/04/28 07:54:07 info: manifest "ceph/templates/job-ks-endpoints.yaml" is empty. Skipping.
2018/04/28 07:54:07 info: manifest "ceph/templates/configmap-bin-ks.yaml" is empty. Skipping.
2018/04/28 07:54:07 info: manifest "ceph/templates/job-bootstrap.yaml" is empty. Skipping.
2018/04/28 07:54:07 info: manifest "ceph/templates/secret-keystone.yaml" is empty. Skipping.
2018/04/28 07:54:07 info: manifest "ceph/templates/job-ceph-commands.yaml" is empty. Skipping.
2018/04/28 07:54:07 info: manifest "ceph/templates/daemonset-osd.yaml" is empty. Skipping.
2018/04/28 07:54:07 info: manifest "ceph/templates/job-ks-service.yaml" is empty. Skipping.
2018/04/28 07:54:07 info: manifest "ceph/templates/job-ks-user.yaml" is empty. Skipping.
[tiller] 2018/04/28 07:54:07 performing install for ceph
[tiller] 2018/04/28 07:54:07 executing 2 pre-install hooks for ceph
[tiller] 2018/04/28 07:54:07 hooks complete for pre-install ceph
[storage] 2018/04/28 07:54:07 getting release history for "ceph"
[storage] 2018/04/28 07:54:07 creating release "ceph.v1"
[kube] 2018/04/28 07:54:07 building resources from manifest
[kube] 2018/04/28 07:54:07 creating 22 resource(s)
[tiller] 2018/04/28 07:54:09 executing 2 post-install hooks for ceph
[tiller] 2018/04/28 07:54:09 hooks complete for post-install ceph
[storage] 2018/04/28 07:54:09 updating release "ceph.v1"
[storage] 2018/04/28 07:54:09 getting last revision of "ceph"
[storage] 2018/04/28 07:54:09 getting release history for "ceph"
[kube] 2018/04/28 07:54:09 Doing get for Secret: "ceph-keystone-user-rgw"
[kube] 2018/04/28 07:54:09 get relation pod of object: ceph/Secret/ceph-keystone-user-rgw
[kube] 2018/04/28 07:54:09 Doing get for ConfigMap: "ceph-bin-clients"
[kube] 2018/04/28 07:54:09 get relation pod of object: ceph/ConfigMap/ceph-bin-clients
[kube] 2018/04/28 07:54:09 Doing get for ConfigMap: "ceph-bin"
[kube] 2018/04/28 07:54:09 get relation pod of object: ceph/ConfigMap/ceph-bin
[kube] 2018/04/28 07:54:09 Doing get for ConfigMap: "ceph-etc"
[kube] 2018/04/28 07:54:09 get relation pod of object: ceph/ConfigMap/ceph-etc
[kube] 2018/04/28 07:54:09 Doing get for ConfigMap: "ceph-templates"
[kube] 2018/04/28 07:54:09 get relation pod of object: ceph/ConfigMap/ceph-templates
[kube] 2018/04/28 07:54:09 Doing get for StorageClass: "ceph-rbd"
[kube] 2018/04/28 07:54:09 get relation pod of object: /StorageClass/ceph-rbd
[kube] 2018/04/28 07:54:09 Doing get for Service: "ceph-mon"
[kube] 2018/04/28 07:54:09 get relation pod of object: ceph/Service/ceph-mon
[kube] 2018/04/28 07:54:09 Doing get for Service: "ceph-rgw"
[kube] 2018/04/28 07:54:09 get relation pod of object: ceph/Service/ceph-rgw
[kube] 2018/04/28 07:54:09 Doing get for DaemonSet: "ceph-mon"
[kube] 2018/04/28 07:54:09 get relation pod of object: ceph/DaemonSet/ceph-mon
[kube] 2018/04/28 07:54:09 Doing get for DaemonSet: "ceph-osd-dev-sda"
[kube] 2018/04/28 07:54:09 get relation pod of object: ceph/DaemonSet/ceph-osd-dev-sda
[kube] 2018/04/28 07:54:09 Doing get for Deployment: "ceph-mds"
[kube] 2018/04/28 07:54:09 get relation pod of object: ceph/Deployment/ceph-mds
[kube] 2018/04/28 07:54:09 Doing get for Deployment: "ceph-mgr"
[kube] 2018/04/28 07:54:09 get relation pod of object: ceph/Deployment/ceph-mgr
[kube] 2018/04/28 07:54:09 Doing get for Deployment: "ceph-mon-check"
[kube] 2018/04/28 07:54:09 get relation pod of object: ceph/Deployment/ceph-mon-check
[kube] 2018/04/28 07:54:09 Doing get for Deployment: "ceph-rbd-provisioner"
[kube] 2018/04/28 07:54:09 get relation pod of object: ceph/Deployment/ceph-rbd-provisioner
[kube] 2018/04/28 07:54:09 Doing get for Deployment: "ceph-rgw"
[kube] 2018/04/28 07:54:10 get relation pod of object: ceph/Deployment/ceph-rgw
[kube] 2018/04/28 07:54:10 Doing get for Job: "ceph-osd-keyring-generator"
[kube] 2018/04/28 07:54:10 get relation pod of object: ceph/Job/ceph-osd-keyring-generator
[kube] 2018/04/28 07:54:10 Doing get for Job: "ceph-mds-keyring-generator"
[kube] 2018/04/28 07:54:10 get relation pod of object: ceph/Job/ceph-mds-keyring-generator
[kube] 2018/04/28 07:54:10 Doing get for Job: "ceph-mon-keyring-generator"
[kube] 2018/04/28 07:54:10 get relation pod of object: ceph/Job/ceph-mon-keyring-generator
[kube] 2018/04/28 07:54:10 Doing get for Job: "ceph-rgw-keyring-generator"
[kube] 2018/04/28 07:54:10 get relation pod of object: ceph/Job/ceph-rgw-keyring-generator
[kube] 2018/04/28 07:54:10 Doing get for Job: "ceph-mgr-keyring-generator"
[kube] 2018/04/28 07:54:10 get relation pod of object: ceph/Job/ceph-mgr-keyring-generator
[kube] 2018/04/28 07:54:10 Doing get for Job: "ceph-namespace-client-key-generator"
[kube] 2018/04/28 07:54:10 get relation pod of object: ceph/Job/ceph-namespace-client-key-generator
[kube] 2018/04/28 07:54:10 Doing get for Job: "ceph-storage-keys-generator"
[kube] 2018/04/28 07:54:10 get relation pod of object: ceph/Job/ceph-storage-keys-generator
foxundermoon commented 6 years ago

system centos CentOS Linux release 7.4.1708 (Core) install by vmware

sudo fdisk -l
[sudo] password for k8s:

Disk /dev/sda: 53.7 GB, 53687091200 bytes, 104857600 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos
Disk identifier: 0x000aae00

   Device Boot      Start         End      Blocks   Id  System
/dev/sda1   *        2048     2099199     1048576   83  Linux
/dev/sda2         2099200   104857599    51379200   8e  Linux LVM

Disk /dev/mapper/centos-root: 50.5 GB, 50457477120 bytes, 98549760 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes

Disk /dev/mapper/centos-swap: 2147 MB, 2147483648 bytes, 4194304 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
cat /etc/fstab

#
# /etc/fstab
# Created by anaconda on Thu Apr 19 03:44:34 2018
#
# Accessible filesystems, by reference, are maintained under '/dev/disk'
# See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info
#
/dev/mapper/centos-root /                       xfs     defaults        0 0
UUID=fdf34de5-5810-4b0e-a7fe-4b92c03cf582 /boot                   xfs     defaults        0 0
#/dev/mapper/centos-swap swap                    swap    defaults        0 0

You can configure your mgr modules

below. Each module has its own set

of key/value. Refer to the doc

above for more info. For example:

ceph_mgr_modules_config: dashboard: port: 7000 localpool: failure_domain: host subtree: rack pg_num: "128" num_rep: "3" min_size: "2" storageclass: provision_storage_class: true provisioner: ceph.com/rbd name: ceph-rbd monitors: null pool: rbd admin_id: admin admin_secret_name: pvc-ceph-conf-combined-storageclass admin_secret_namespace: ceph user_id: admin user_secret_name: pvc-ceph-client-key image_format: "2" image_features: layering

whmzsu commented 6 years ago

I met that error before when I install it after uninstall a "failed helm install ceph before PR #54" . So I doubt that there maybe some files were not cleaned by the "helm delete --purge ". So I did a clean installation started from all new VMs and kubernetes cluster, finally got a successful installation of ceph.

whmzsu commented 6 years ago

BTW: It seems that you config the OSD device to use the OS device ??? I think you should add one or more bland disk for OSD device, NOT the OS root device

foxundermoon commented 6 years ago

@whmzsu thank you. i deleted the /var/log/ceph /var/lib/ceph-helm also clean the k8s all source under ceph namespace.

whmzsu commented 6 years ago

@foxundermoon I try to install the ceph again after uninstall the success installation, delete the ceph-helm dir , and got the same error like your. So I think there must be something not cleared by the helm while uninstalling.

praparn commented 6 years ago

We also found this issue on Kubernetes 1.9.2. As we try all is clear all helm environment/reinital k8s farm and try more than 3 times still issue this

foxundermoon commented 6 years ago

@whmzsu still issue. i am reinstall on other 3 servers.
still meet the error 

  Normal   SuccessfulMountVolume  54m                 kubelet, bj-h03-11-cm.getui  MountVolume.SetUp succeeded for volume "default-token-xp4fc"
  Warning  FailedMount            54m (x3 over 54m)   kubelet, bj-h03-11-cm.getui  MountVolume.SetUp failed for volume "ceph-mon-keyring" : secrets "ceph-mon-keyring" not found
  Warning  FailedMount            54m (x4 over 54m)   kubelet, bj-h03-11-cm.getui  MountVolume.SetUp failed for volume "ceph-bootstrap-rgw-keyring" : secrets "ceph-bootstrap-rgw-keyring" not found
  Warning  FailedMount            54m (x4 over 54m)   kubelet, bj-h03-11-cm.getui  MountVolume.SetUp failed for volume "ceph-bootstrap-osd-keyring" : secrets "ceph-bootstrap-osd-keyring" not found
  Warning  FailedMount            54m (x4 over 54m)   kubelet, bj-h03-11-cm.getui  MountVolume.SetUp failed for volume "ceph-bootstrap-mds-keyring" : secrets "ceph-bootstrap-mds-keyring" not found
  Warning  FailedMount            54m (x4 over 54m)   kubelet, bj-h03-11-cm.getui  MountVolume.SetUp failed for volume "ceph-client-admin-keyring" : secrets "ceph-client-admin-keyring" not found
  Normal   Pulled                 48m                 kubelet, bj-h03-11-cm.getui  Successfully pulled image "docker.io/kolla/ubuntu-source-kubernetes-entrypoint:4.0.0"
githubfoam commented 6 years ago

I am getting same problem.Problem is "/etc/ceph/ceph.mon.keyring" I do not have such a path "/etc/ceph" or "var/lib/ceph". How can I all do this with a VM ? I do not want to run a host with minikube.

Installation (Kubernetes + Helm)

minikube version: v0.27.0

helm version Client: &version.Version{SemVer:"v2.9.1", GitCommit:"20adb27c7c5868466912eebdf6664e7390ebe710", GitTreeState:"clean"}

Server: &version.Version{SemVer:"v2.9.1", GitCommit:"20adb27c7c5868466912eebdf6664e7390ebe710", GitTreeState:"clean"}

kubectl version Client Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.3", GitCommit:"2bba0127d85d5a46ab4b778548be28623b32d0b0", GitTreeState:"clean", BuildDate:"2018-05-21T09:17:39Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.0", GitCommit:"fc32d2f3698e36b93322a3465f63a14e9f0eaead", GitTreeState:"clean", BuildDate:"2018-03-26T16:44:10Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}

stat: can't stat '/var/log/ceph/ceph.audit.log': No such file or directory container "osd-activate-pod" in pod "ceph-osd-dev-sdd-pzqck" is waiting to start: PodInitializing MountVolume.SetUp failed for volume "ceph-mon-keyring" : secrets "ceph-mon-keyring" not found Back-off restarting failed container

{ "name": "ceph-mon-keyring", "readOnly": true, "mountPath": "/etc/ceph/ceph.mon.keyring", "subPath": "ceph.mon.keyring" },

pasikon commented 6 years ago

@githubfoam I have exactly same issue

githubfoam commented 6 years ago

@pasikon #48 r u running minikube ?

pasikon commented 6 years ago

@githubfoam yes, 27.0

githubfoam commented 6 years ago

not merged yet I guess. Fix keyring does not exist

ksingh7 commented 6 years ago

Encountered with the same issue in discussion

Events:
  Type     Reason       Age              From               Message
  ----     ------       ----             ----               -------
  Warning  FailedMount  1m (x3 over 1m)  kubelet, server-6  MountVolume.SetUp failed for volume "ceph-bootstrap-rgw-keyring" : secrets "ceph-bootstrap-rgw-keyring" not found
  Warning  FailedMount  1m (x4 over 1m)  kubelet, server-6  MountVolume.SetUp failed for volume "ceph-bootstrap-osd-keyring" : secrets "ceph-bootstrap-osd-keyring" not found
  Warning  FailedMount  1m (x5 over 1m)  kubelet, server-6  MountVolume.SetUp failed for volume "ceph-mon-keyring" : secrets "ceph-mon-keyring" not found
  Warning  FailedMount  1m (x6 over 1m)  kubelet, server-6  MountVolume.SetUp failed for volume "ceph-client-admin-keyring" : secrets "ceph-client-admin-keyring" not found
  Warning  FailedMount  1m (x6 over 1m)  kubelet, server-6  MountVolume.SetUp failed for volume "ceph-bootstrap-mds-keyring" : secrets "ceph-bootstrap-mds-keyring" not found
  Normal   Pulled       50s              kubelet, server-6  Container image "docker.io/kolla/ubuntu-source-kubernetes-entrypoint:4.0.0" already present on machine

When i tried for the first time everything worked fine. As soon as i deleted the cluster using helm delete --purge ceph --timeout 1000 --debug and retried helm install --name=ceph local/ceph --namespace=ceph -f ~/ceph-overrides.yaml i started getting these errors.

Also i do not have directories like /etc/ceph /var/lib/ceph /var/log/ceph /var/lib/ceph-helm.

Need help to get find A) Clean way to delete the cluster B) Fix this issue.

@rootfs any suggestions ?

ksingh7 commented 6 years ago

@githubfoam i am running the fix you mentioned in your last comment.

ksingh7 commented 6 years ago

@foxundermoon @githubfoam did you ever managed to get this working. Whats the clean way to purge the cluster ?

jasine commented 6 years ago

@ksingh7 did you find any workaround?

ksingh7 commented 6 years ago

@jasine unfortunately not

sunx commented 6 years ago

You must remove the partitions on target devices before installing a new ceph cluster.

xforze commented 6 years ago

Hi All,

I have exactly the same issue with Kubernetes 1.11.2. Is there still no fix or workaround for this ? @sunx The osd discs/partitions are untouched as the osd Pod are not coming up. Cheers!

mamoit commented 6 years ago

Just use fdisk to create a new partition table.

xforze commented 6 years ago

The OSD Pods still not coming up, a "kubectl describe pod xxxxxx" shows:

  Type     Reason       Age                From                Message
  ----     ------       ----               ----                -------
  Warning  FailedMount  11m (x5 over 11m)  kubelet, k8s-ceph3  MountVolume.SetUp failed for volume "ceph-bootstrap-mds-keyring" : secrets "ceph-bootstrap-mds-keyring" not found
  Warning  FailedMount  11m (x5 over 11m)  kubelet, k8s-ceph3  MountVolume.SetUp failed for volume "ceph-client-admin-keyring" : secrets "ceph-client-admin-keyring" not found
  Warning  FailedMount  11m (x5 over 11m)  kubelet, k8s-ceph3  MountVolume.SetUp failed for volume "ceph-bootstrap-osd-keyring" : secrets "ceph-bootstrap-osd-keyring" not found
  Warning  FailedMount  11m (x5 over 11m)  kubelet, k8s-ceph3  MountVolume.SetUp failed for volume "ceph-bootstrap-rgw-keyring" : secrets "ceph-bootstrap-rgw-keyring" not found
  Warning  FailedMount  11m (x5 over 11m)  kubelet, k8s-ceph3  MountVolume.SetUp failed for volume "ceph-mon-keyring" : secrets "ceph-mon-keyring" not found
  Warning  BackOff      1m (x41 over 10m)  kubelet, k8s-ceph3  Back-off restarting failed container

But all of these Secrets are existing in the same namespace.

Amos-85 commented 6 years ago

same issue for me in K8S v1.10.5 / RHEL7.5 / KERNEL 3.10.0. osd-prepare-pod with mounting event exceptions like @xforze .

any ideas?

elvinasp commented 6 years ago

same issue for me in K8S v1.10.5 / RHEL7.5 / KERNEL 3.10.0. osd-prepare-pod with mounting event exceptions like @xforze .

Same issue occured on K8S v1.11.2 / Ubuntu 18.04. However after I deleted pods, they mounted secrets successfully.

After that I have stumbled upon other error related to reinstall. At fault was storage disks with old information from previous install. @mamoit incorrectly suggested to use fdisk, where you should use gdisk to wipe/recreate partitions! Cephs expects GPT partition table and fdisk does not properly handle that.

To verify, check logs from init container: kubectl -n ceph logs ceph-osd-dev-sdc-XXXX -c osd-prepare-pod

I was getting the following error:

create_partition: Creating journal partition num 2 size 5120 on /dev/sdc
command_check_call: Running command: /sbin/sgdisk --new=2:0:+5120M --change-name=2:ceph journal --partition-guid=2:5341d5ed-b6eb-4bcd-8e35-1620d85ef776 --typecode=2:45b0969e-9b03-4f30-b4c6-b4b80ceff106 --mbrtogpt -- /dev/sdc
Caution: invalid main GPT header, but valid backup; regenerating main header
from backup!

Invalid partition data!
/sbin/sgdisk --new=2:0:+5120M --change-name=2:ceph journal --partition-guid=2:5341d5ed-b6eb-4bcd-8e35-1620d85ef776 --typecode=2:45b0969e-9b03-4f30-b4c6-b4b80ceff106 --mbrtogpt -- /dev/sdc' failed with status code 2

After I have recreated GPT partition table (option 2) with gdisk /dev/sdc, pod successfully initialized.

Amos-85 commented 6 years ago

cool @elvinasp ! I should give a try for that. I moved to rook.io project to deploy ceph.

danielxolsson commented 6 years ago

I had the exact same issue. The last piece in the cleanup puzzle was to delete dangling docker volumes on all nodes:

docker volume rm $(docker volume ls -qf dangling=true)

neuhalje commented 5 years ago

Purge Ceph

The following will remove all remains of Ceph (after that I could install). This assumes that the nodename is reachable via ssh.

helm del --purge ceph
kubectl delete namespaces ceph

while read host; do
echo $host; ssh -n $host -l root -- rm -rf /var/lib/ceph-helm /etc/ceph
done < <(kubectl get nodes -o name|sed 's!^node/!!')

I also rebooted all nodes (not sure if this is a must).

Verify that kubernetes is gone:

kubectl get all --all-namespaces -o name|grep ceph

EDIT: Still fails :-(

Deleting the OSD pods (works!)

After deleting all OSD pods the cluster came up:

kubectl delete pod -n ceph ceph-osd-dev-vdb-m28wr
...
robin-2016 commented 5 years ago

the same problem. when I run make: find: ‘secrets’: No such file or directory

xinghui-hou commented 5 years ago

I have same problem. ceph-rgw-keyring-generator job create pod ，alway run failed 。 This causes other created pods to fail all the time image

I've tried purge twice, but it's the same problem

tx19980520 commented 5 years ago

I have same problem. ceph-rgw-keyring-generator job create pod ，alway run failed 。 This causes other created pods to fail all the time image

I've tried purge twice, but it's the same problem

the same, @xinghui-hou, did you find the solution?

alter commented 5 years ago

Hello, kubernetes 1.15.5 and the same things here... Does this project(I mean k8s chart) is still alive?

sudo-undefined commented 4 years ago

Last commit was on 26:th June 2019. ceph-helm is not deployable on 1.16+ and as we can see have problems in earlier versions as well I'm going for rook-ceph instead

githubfoam commented 4 years ago

@ksingh7

@githubfoam i am running the fix you mentioned in your last comment.

I did not try but only way possible is to git clone/code/compile the solution and run it locally I guess.The solution seems merged.You can see the details of root cause analysis in the link I refer to. Manually typing "auth" in the "ceph.conf" could be last resort before pulling the source.