jtaleric
commented
5 years ago Open jtaleric opened 5 years ago
jtaleric
commented
5 years ago jtaleric
commented
5 years ago sh-4.2# ceph -s
cluster:
id: dcf2163a-2430-4136-91b8-a2c731bf966c
health: HEALTH_ERR
1 filesystem is offline
1 filesystem is online with fewer MDS than max_mds
Reduced data availability: 24 pgs inactive
too few PGs per OSD (4 < min 30)
services:
mon: 3 daemons, quorum a,b,c (age 5h)
mgr: a(active, since 5h)
mds: mycluster-cephfilesystem:0
osd: 12 osds: 12 up (since 5h), 12 in (since 5h)
data:
pools: 5 pools, 40 pgs
objects: 0 objects, 0 B
usage: 12 GiB used, 17 TiB / 17 TiB avail
pgs: 60.000% pgs unknown
24 unknown
16 active+clean
e-minguez
commented
5 years ago In my environment (with autoscale on out of the box because I didn't enabled it...)
$ oc get pods -n openshift-storage
NAME READY STATUS RESTARTS AGE
csi-cephfsplugin-9stz5 3/3 Running 0 24h
csi-cephfsplugin-b8zqw 3/3 Running 0 24h
csi-cephfsplugin-n9kh8 3/3 Running 0 24h
csi-cephfsplugin-provisioner-678db5bf66-h6kq9 4/4 Running 2 24h
csi-cephfsplugin-provisioner-678db5bf66-lxt6l 4/4 Running 0 24h
csi-rbdplugin-provisioner-7bf4556986-f4v9r 5/5 Running 0 24h
csi-rbdplugin-provisioner-7bf4556986-r4sbn 5/5 Running 0 24h
csi-rbdplugin-tg77k 3/3 Running 0 24h
csi-rbdplugin-vklpx 3/3 Running 0 24h
csi-rbdplugin-wkmht 3/3 Running 0 24h
local-storage-operator-bcfd5765f-gxj5v 1/1 Running 0 24h
noobaa-core-0 2/2 Running 0 24h
noobaa-operator-6f96d69f89-6rfdv 1/1 Running 0 24h
ocs-operator-7878485678-nbhwf 1/1 Running 0 23h
osd-disks-local-diskmaker-9wfm6 1/1 Running 0 24h
osd-disks-local-diskmaker-hcwpg 1/1 Running 0 24h
osd-disks-local-diskmaker-vrplc 1/1 Running 0 24h
osd-disks-local-provisioner-4ftqr 1/1 Running 0 24h
osd-disks-local-provisioner-b5vwq 1/1 Running 0 24h
osd-disks-local-provisioner-w95rb 1/1 Running 0 24h
rook-ceph-mds-openshift-storage-cephfilesystem-a-56d6bd4d849j5c 1/1 Running 0 23h
rook-ceph-mds-openshift-storage-cephfilesystem-b-8cfd6f79-nhzkm 1/1 Running 0 23h
rook-ceph-mgr-a-74f685f55c-skhtw 1/1 Running 0 23h
rook-ceph-mon-a-c8cd8875c-vc4p8 1/1 Running 0 24h
rook-ceph-mon-b-79449bd57d-v6sn7 1/1 Running 0 24h
rook-ceph-mon-c-64df8f4c5-dxqg4 1/1 Running 0 24h
rook-ceph-operator-685695b58f-9pmhz 1/1 Running 0 23h
rook-ceph-osd-0-b54548558-szczk 1/1 Running 0 23h
rook-ceph-osd-1-779955999d-k55d4 1/1 Running 0 23h
rook-ceph-osd-2-7889cd7d8-qk6c4 1/1 Running 0 23h
rook-ceph-osd-3-5fcf74b6b7-lgxn4 1/1 Running 0 23h
rook-ceph-osd-4-67c56b4555-bjmqf 1/1 Running 0 23h
rook-ceph-osd-5-6945d7c764-jrbqq 1/1 Running 0 23h
rook-ceph-osd-6-65d468c79f-g6r7d 1/1 Running 0 23h
rook-ceph-osd-7-5b496db4d7-bfxxv 1/1 Running 0 23h
rook-ceph-osd-8-54995f85f9-wvh2n 1/1 Running 0 23h
rook-ceph-osd-prepare-osd-0-9dtlp-v4frm 0/1 Completed 0 23h
rook-ceph-osd-prepare-osd-1-n74tf-hghtl 0/1 Completed 0 23h
rook-ceph-osd-prepare-osd-2-nttzf-lw8gr 0/1 Completed 0 23h
rook-ceph-osd-prepare-osd-3-24tp9-fpwh2 0/1 Completed 0 23h
rook-ceph-osd-prepare-osd-4-xjvs7-tfbfl 0/1 Completed 0 23h
rook-ceph-osd-prepare-osd-5-fjtb7-zxs8h 0/1 Completed 0 23h
rook-ceph-osd-prepare-osd-6-29nc4-njkk4 0/1 Completed 0 23h
rook-ceph-osd-prepare-osd-7-f2s5g-n2w5n 0/1 Completed 0 23h
rook-ceph-osd-prepare-osd-8-fbb97-cnmrp 0/1 Completed 0 23h
rook-ceph-tools-5f5dc75fd5-fwwq7 1/1 Running 0 22h$ oc exec $(oc get pods -n openshift-storage -l app=rook-ceph-tools -o jsonpath="{.items[*].metadata.name}") -n openshift-storage -- ceph -s
cluster:
id: 0b23199a-52a3-4e0c-9e14-9769055e6026
health: HEALTH_WARN
too few PGs per OSD (10 < min 30)
services:
mon: 3 daemons, quorum a,b,c (age 24h)
mgr: a(active, since 23h)
mds: openshift-storage-cephfilesystem:1 {0=openshift-storage-cephfilesystem-a=up:active} 1 up:standby-replay
osd: 9 osds: 9 up (since 23h), 9 in (since 23h)
data:
pools: 4 pools, 32 pgs
objects: 10.71k objects, 41 GiB
usage: 126 GiB used, 20 TiB / 20 TiB avail
pgs: 32 active+clean
io:
client: 853 B/s rd, 116 KiB/s wr, 1 op/s rd, 2 op/s wr$ oc exec $(oc get pods -n openshift-storage -l app=rook-ceph-tools -o jsonpath="{.items[*].metadata.name}") -n openshift-storage -- ceph osd pool autoscale-status
POOL SIZE TARGET SIZE RATE RAW CAPACITY RATIO TARGET RATIO BIAS PG_NUM NEW PG_NUM AUTOSCALE
openshift-storage-cephblockpool 116.9G 3.0 20106G 0.0175 1.0 8 on
openshift-storage-cephfilesystem-metadata 1536k 3.0 20106G 0.0000 1.0 8 on
openshift-storage-cephfilesystem-data0 0 3.0 20106G 0.0000 1.0 8 on
.rgw.root 0 3.0 20106G 0.0000 1.0 8 on@e-minguez yeah, it looks to be hard coded to be on in the OCS Operator.
Additionally @e-minguez @karmab the Operator seems to default to HostNetwork: False -- just a heads up.