search

rook / rook

Storage Orchestration for Kubernetes
https://rook.io
Apache License 2.0
12.37k stars 2.69k forks source link

RGW pools appearing without objectstore present #1950

Closed benceszikora closed 2 years ago

benceszikora commented 6 years ago

I have created a new cluster from scratch using v0.8.0. I have created an objectstore call rook-objectstore When I do ceph osd pool ls, I can see these pools:

.rgw.root                       
default.rgw.control             
default.rgw.meta                
default.rgw.log                 
rook-objectstore.rgw.control    
replicapool                     
rook-objectstore.rgw.meta       
rook-objectstore.rgw.log        
rook-objectstore.rgw.buckets.index                               
rook-objectstore.rgw.buckets.data

Where do the default. pools come from? Can I safely delete them?

travisn commented 6 years ago

@benceszikora the default. pools are not expected. Ceph creates them automatically when the rgw is started if it is not configured correctly for the desired object store. This is why rook explicitly creates all the necessary pools before starting the rgw.

I am not able to repro seeing these default. pools when I create a new object store. Do you have more repro steps?

stale[bot] commented 6 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

stale[bot] commented 6 years ago

This issue has been automatically closed due to inactivity. Please re-open if this still requires investigation.

huanggx-sea commented 2 years ago

Encounter the same problem after upgrading from rook v1.7.3 to rook v1.8.9.

These pools are automatically created.

default.rgw.control             
default.rgw.meta                
default.rgw.log
travisn commented 2 years ago

@huanggx-sea What is your configuration for the object store(s)? Do you have one object store, or multiple? I'm definitely curious on the repro steps since this is not expected, if you have any more details than upgrading.

microyahoo commented 2 years ago

@travisn I have encounter the same problem like follows. The default.rgw.log pool's pg are all in unknown state.

➜ /root/go/src/rook/rook ☞ git:(notes) ✗ ceph -s            
  cluster:
    id:     a1135755-be6c-42b2-a99b-b06250959014
    health: HEALTH_WARN
            clock skew detected on mon.b
            Reduced data availability: 32 pgs inactive

  services:
    mon: 3 daemons, quorum a,b,c (age 21h)
    mgr: a(active, since 21h)
    osd: 6 osds: 6 up (since 20h), 6 in (since 20h)
    rgw: 3 daemons active (3 hosts, 1 zones)

  data:
    pools:   9 pools, 209 pgs
    objects: 387 objects, 30 KiB
    usage:   554 MiB used, 299 GiB / 300 GiB avail
    pgs:     15.311% pgs unknown
             177 active+clean
             32  unknown

  progress:
    Global Recovery Event (33m)
      [=======================.....] (remaining: 5m)

🍺 /root/go/src/rook/rook ☞ git:(notes) ✗rados lspools
device_health_metrics
os-test3.rgw.buckets.index
os-test3.rgw.log
.rgw.root
os-test3.rgw.control
os-test3.rgw.meta
os-test3.rgw.buckets.non-ec
default.rgw.log
os-test3.rgw.buckets.data
🍺 /root/go/src/deeproute.ai/smd ☞ git:(optimize_crush_root) ✗ ceph pg ls-by-pool default.rgw.log           
PG     OBJECTS  DEGRADED  MISPLACED  UNFOUND  BYTES  OMAP_BYTES*  OMAP_KEYS*  LOG  STATE    SINCE  VERSION  REPORTED  UP     ACTING  SCRUB_STAMP                      DEEP_SCRUB_STAMP               
38.0         0         0          0        0      0            0           0    0  unknown    22m      0'0       0:0  []p-1   []p-1  2022-06-09T03:21:34.422619+0000  2022-06-09T03:21:34.422619+0000
38.1         0         0          0        0      0            0           0    0  unknown    22m      0'0       0:0  []p-1   []p-1  2022-06-09T03:21:34.422619+0000  2022-06-09T03:21:34.422619+0000
38.2         0         0          0        0      0            0           0    0  unknown    22m      0'0       0:0  []p-1   []p-1  2022-06-09T03:21:34.422619+0000  2022-06-09T03:21:34.422619+0000
38.3         0         0          0        0      0            0           0    0  unknown    22m      0'0       0:0  []p-1   []p-1  2022-06-09T03:21:34.422619+0000  2022-06-09T03:21:34.422619+0000
38.4         0         0          0        0      0            0           0    0  unknown    22m      0'0       0:0  []p-1   []p-1  2022-06-09T03:21:34.422619+0000  2022-06-09T03:21:34.422619+0000
38.5         0         0          0        0      0            0           0    0  unknown    22m      0'0       0:0  []p-1   []p-1  2022-06-09T03:21:34.422619+0000  2022-06-09T03:21:34.422619+0000
38.6         0         0          0        0      0            0           0    0  unknown    22m      0'0       0:0  []p-1   []p-1  2022-06-09T03:21:34.422619+0000  2022-06-09T03:21:34.422619+0000
38.7         0         0          0        0      0            0           0    0  unknown    22m      0'0       0:0  []p-1   []p-1  2022-06-09T03:21:34.422619+0000  2022-06-09T03:21:34.422619+0000
38.8         0         0          0        0      0            0           0    0  unknown    22m      0'0       0:0  []p-1   []p-1  2022-06-09T03:21:34.422619+0000  2022-06-09T03:21:34.422619+0000
38.9         0         0          0        0      0            0           0    0  unknown    22m      0'0       0:0  []p-1   []p-1  2022-06-09T03:21:34.422619+0000  2022-06-09T03:21:34.422619+0000
38.a         0         0          0        0      0            0           0    0  unknown    22m      0'0       0:0  []p-1   []p-1  2022-06-09T03:21:34.422619+0000  2022-06-09T03:21:34.422619+0000
38.b         0         0          0        0      0            0           0    0  unknown    22m      0'0       0:0  []p-1   []p-1  2022-06-09T03:21:34.422619+0000  2022-06-09T03:21:34.422619+0000
38.c         0         0          0        0      0            0           0    0  unknown    22m      0'0       0:0  []p-1   []p-1  2022-06-09T03:21:34.422619+0000  2022-06-09T03:21:34.422619+0000
38.d         0         0          0        0      0            0           0    0  unknown    22m      0'0       0:0  []p-1   []p-1  2022-06-09T03:21:34.422619+0000  2022-06-09T03:21:34.422619+0000
38.e         0         0          0        0      0            0           0    0  unknown    22m      0'0       0:0  []p-1   []p-1  2022-06-09T03:21:34.422619+0000  2022-06-09T03:21:34.422619+0000
38.f         0         0          0        0      0            0           0    0  unknown    22m      0'0       0:0  []p-1   []p-1  2022-06-09T03:21:34.422619+0000  2022-06-09T03:21:34.422619+0000
38.10        0         0          0        0      0            0           0    0  unknown    22m      0'0       0:0  []p-1   []p-1  2022-06-09T03:21:34.422619+0000  2022-06-09T03:21:34.422619+0000
38.11        0         0          0        0      0            0           0    0  unknown    22m      0'0       0:0  []p-1   []p-1  2022-06-09T03:21:34.422619+0000  2022-06-09T03:21:34.422619+0000
38.12        0         0          0        0      0            0           0    0  unknown    22m      0'0       0:0  []p-1   []p-1  2022-06-09T03:21:34.422619+0000  2022-06-09T03:21:34.422619+0000
38.13        0         0          0        0      0            0           0    0  unknown    22m      0'0       0:0  []p-1   []p-1  2022-06-09T03:21:34.422619+0000  2022-06-09T03:21:34.422619+0000
38.14        0         0          0        0      0            0           0    0  unknown    22m      0'0       0:0  []p-1   []p-1  2022-06-09T03:21:34.422619+0000  2022-06-09T03:21:34.422619+0000
38.15        0         0          0        0      0            0           0    0  unknown    22m      0'0       0:0  []p-1   []p-1  2022-06-09T03:21:34.422619+0000  2022-06-09T03:21:34.422619+0000
38.16        0         0          0        0      0            0           0    0  unknown    22m      0'0       0:0  []p-1   []p-1  2022-06-09T03:21:34.422619+0000  2022-06-09T03:21:34.422619+0000
38.17        0         0          0        0      0            0           0    0  unknown    22m      0'0       0:0  []p-1   []p-1  2022-06-09T03:21:34.422619+0000  2022-06-09T03:21:34.422619+0000
38.18        0         0          0        0      0            0           0    0  unknown    22m      0'0       0:0  []p-1   []p-1  2022-06-09T03:21:34.422619+0000  2022-06-09T03:21:34.422619+0000
38.19        0         0          0        0      0            0           0    0  unknown    22m      0'0       0:0  []p-1   []p-1  2022-06-09T03:21:34.422619+0000  2022-06-09T03:21:34.422619+0000
38.1a        0         0          0        0      0            0           0    0  unknown    22m      0'0       0:0  []p-1   []p-1  2022-06-09T03:21:34.422619+0000  2022-06-09T03:21:34.422619+0000
38.1b        0         0          0        0      0            0           0    0  unknown    22m      0'0       0:0  []p-1   []p-1  2022-06-09T03:21:34.422619+0000  2022-06-09T03:21:34.422619+0000
38.1c        0         0          0        0      0            0           0    0  unknown    22m      0'0       0:0  []p-1   []p-1  2022-06-09T03:21:34.422619+0000  2022-06-09T03:21:34.422619+0000
38.1d        0         0          0        0      0            0           0    0  unknown    22m      0'0       0:0  []p-1   []p-1  2022-06-09T03:21:34.422619+0000  2022-06-09T03:21:34.422619+0000
38.1e        0         0          0        0      0            0           0    0  unknown    22m      0'0       0:0  []p-1   []p-1  2022-06-09T03:21:34.422619+0000  2022-06-09T03:21:34.422619+0000
38.1f        0         0          0        0      0            0           0    0  unknown    22m      0'0       0:0  []p-1   []p-1  2022-06-09T03:21:34.422619+0000  2022-06-09T03:21:34.422619+0000

* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details.

The corresponding osds tree info

🍺 /root/go/src/rook/rook ☞ git:(notes) ✗ceph osd tree      
ID   CLASS  WEIGHT   TYPE NAME                                         STATUS  REWEIGHT  PRI-AFF
-17         0.14639  root root2                                                                 
-19         0.04880      host k1-08552839-9e35-46b6-8f49-48e867dac355                           
  1    hdd  0.04880          osd.1                                         up   1.00000  1.00000
-20         0.04880      host k2-f68e6f6f-05f4-48c0-ac65-cfd1a4354adc                           
  0    hdd  0.04880          osd.0                                         up   1.00000  1.00000
-18         0.04880      host k3-935608a4-6baa-4389-b391-9161f5cb0dc9                           
  5    hdd  0.04880          osd.5                                         up   1.00000  1.00000
 -9         0.14639  root root1                                                                 
-10         0.04880      host k1-06317ccd-eaa1-4c4c-a380-8d04dae60a03                           
  4    hdd  0.04880          osd.4                                         up   1.00000  1.00000
-11         0.04880      host k2-a484b847-03b9-492a-895a-1279d24d1d8a                           
  3    hdd  0.04880          osd.3                                         up   1.00000  1.00000
-12         0.04880      host k3-7eabbe97-feed-4a43-af75-26cfa2b15e25                           
  2    hdd  0.04880          osd.2                                         up   1.00000  1.00000
 -1               0  root default                                                               
 -5               0      host k1                                                                
 -3               0      host k2                                                                
 -7               0      host k3

object store info:

🍺 /root/go/src/deeproute.ai/smd ☞ git:(optimize_crush_root) ✗ kubectl describe -n rook-ceph cephobjectstores.ceph.rook.io os-test3
Name:         os-test3
Namespace:    rook-ceph
Labels:       <none>
Annotations:  <none>
API Version:  ceph.rook.io/v1
Kind:         CephObjectStore
Metadata:
  Creation Timestamp:  2022-06-09T03:21:12Z
  Finalizers:
    cephobjectstore.ceph.rook.io
  Generation:  1
  Managed Fields:
    API Version:  ceph.rook.io/v1
    Fields Type:  FieldsV1
    fieldsV1:
      f:metadata:
        f:finalizers:
          .:
          v:"cephobjectstore.ceph.rook.io":
    Manager:      rook
    Operation:    Update
    Time:         2022-06-09T03:21:12Z
    API Version:  ceph.rook.io/v1
    Fields Type:  FieldsV1
    fieldsV1:
      f:spec:
        .:
        f:dataPool:
          .:
          f:crushRoot:
          f:erasureCoded:
            .:
            f:codingChunks:
            f:dataChunks:
          f:failureDomain:
          f:mirroring:
          f:parameters:
            .:
            f:compression_mode:
            f:target_size_ratio:
          f:quotas:
          f:replicated:
            .:
            f:size:
          f:statusCheck:
            .:
            f:mirror:
        f:gateway:
          .:
          f:instances:
          f:placement:
          f:port:
          f:resources:
        f:healthCheck:
          .:
          f:bucket:
            .:
            f:interval:
          f:livenessProbe:
          f:readinessProbe:
        f:metadataPool:
          .:
          f:crushRoot:
          f:erasureCoded:
            .:
            f:codingChunks:
            f:dataChunks:
          f:failureDomain:
          f:mirroring:
          f:parameters:
            .:
            f:compression_mode:
            f:target_size_ratio:
          f:quotas:
          f:replicated:
            .:
            f:requireSafeReplicaSize:
            f:size:
          f:statusCheck:
            .:
            f:mirror:
        f:zone:
          .:
          f:name:
    Manager:      smd
    Operation:    Update
    Time:         2022-06-09T03:21:12Z
    API Version:  ceph.rook.io/v1
    Fields Type:  FieldsV1
    fieldsV1:
      f:status:
        .:
        f:bucketStatus:
          .:
          f:health:
          f:lastChanged:
          f:lastChecked:
        f:info:
          .:
          f:endpoint:
        f:phase:
    Manager:         rook
    Operation:       Update
    Subresource:     status
    Time:            2022-06-09T03:22:49Z
  Resource Version:  201744
  UID:               b58f19c7-bdb9-45ca-b689-95290875ac91
Spec:
  Data Pool:
    Crush Root:  root1
    Erasure Coded:
      Coding Chunks:  1
      Data Chunks:    2
    Failure Domain:   host
    Mirroring:
    Parameters:
      compression_mode:   none
      target_size_ratio:  .5
    Quotas:
    Replicated:
      Size:  0
    Status Check:
      Mirror:
  Gateway:
    Instances:  3
    Placement:
    Port:  80
    Resources:
  Health Check:
    Bucket:
      Interval:  1m0s
    Liveness Probe:
    Readiness Probe:
  Metadata Pool:
    Crush Root:  root2
    Erasure Coded:
      Coding Chunks:  0
      Data Chunks:    0
    Failure Domain:   host
    Mirroring:
    Parameters:
      compression_mode:   none
      target_size_ratio:  .5
    Quotas:
    Replicated:
      Require Safe Replica Size:  true
      Size:                       3
    Status Check:
      Mirror:
  Zone:
    Name:  
Status:
  Bucket Status:
    Health:        Connected
    Last Changed:  2022-06-09T03:22:49Z
    Last Checked:  2022-06-09T03:58:53Z
  Info:
    Endpoint:  http://rook-ceph-rgw-os-test3.rook-ceph.svc:80
  Phase:       Connected
Events:
  Type    Reason              Age   From                         Message
  ----    ------              ----  ----                         -------
  Normal  ReconcileSucceeded  37m   rook-ceph-object-controller  successfully configured CephObjectStore "rook-ceph/os-test3"

the version of rook and ceph:

🍺 /root/go/src/deeproute.ai/smd ☞ git:(optimize_crush_root) ✗ crictl images | egrep "rook|ceph"
docker.io/rook/ceph                                   v1.8.5              336629ee5ebe9       445MB
quay.io/ceph/ceph                                     <none>              1d0ab362ee4cc       454MB
quay.io/ceph/ceph                                     v16.2.7             463ec4b1fdc01       458MB
quay.io/cephcsi/cephcsi                               v3.5.1              a68e15f8572a2       451MB
microyahoo commented 2 years ago

It cannot be reproduced 100%, about once every ten times when creating object store.

microyahoo commented 2 years ago

/reopen

travisn commented 2 years ago

Interesting that you are just seeing this issue after it was first reported four years ago. It does look like the exact symptoms, so will reopen it. I am guessing the rgw daemon is causing those default pools to be created even though it's not configured to create them.

default.rgw.control             
default.rgw.meta                
default.rgw.log

@thotz Do you know what might be creating these pools in this rare case?

thotz commented 2 years ago

AFAIR radosgw-admin cli creates these pools if you especially the zone/zonegroup related commands to point default zone/zonegroup, I don't exactly remember how/when these creations get triggers, but my guess is if there is a zone/zone group but master zone/zonegroup is not configured then rgw sets those to default or period commit performed without setting master zone/zonegroup

microyahoo commented 2 years ago

I found another strange phenomenon, which maybe same root cause. rgw.log , rgw.meta and .rgw.root pools are recreated after the corresponding object store has been deleted from ceph cluster like follows. @thotz prior to delete object store:

➜ /root/go/src/rook/rook ☞ git:(notes) ✗ ceph osd lspools             
1 device_health_metrics               
2 os-test.rgw.buckets.non-ec                                          
3 .rgw.root                                                           
4 os-test.rgw.meta                                                    
5 os-test.rgw.buckets.index                                           
6 os-test.rgw.log                    
7 os-test.rgw.control                                                 
8 os-test.rgw.buckets.data

after delete object store:

🍺 /root/go/src/rook/rook ☞ git:(notes) ✗ kubectl get -n rook-ceph cephobjectstores.ceph.rook.io -o yaml
apiVersion: v1
items: []
kind: List
metadata:
  resourceVersion: ""
  selfLink: ""

🍺 /root/go/src/rook/rook ☞ git:(notes) ✗ ceph osd lspools
1 device_health_metrics
9 os-test.rgw.log
10 os-test.rgw.meta
11 .rgw.root

new created pools are all inactive.

🍺 /root/go/src/rook/rook ☞ git:(notes) ✗ceph -s                           
  cluster:
    id:     1260d8f3-acce-473d-8904-d8ec8e9caccf
    health: HEALTH_WARN
            clock skew detected on mon.c
            Reduced data availability: 96 pgs inactive

  services:
    mon: 3 daemons, quorum a,b,c (age 5h)
    mgr: b(active, since 5h), standbys: a
    osd: 6 osds: 6 up (since 5h), 6 in (since 5h)

  data:
    pools:   4 pools, 97 pgs
    objects: 0 objects, 0 B
    usage:   245 MiB used, 300 GiB / 300 GiB avail
    pgs:     98.969% pgs unknown
             96 unknown
             1  active+clean
gauravsitlani commented 2 years ago

Hi @microyahoo are you still seeing this issue with the latest master ?

microyahoo commented 2 years ago

Hi @microyahoo are you still seeing this issue with the latest master ?

I didn't try to use latest master, and I'll try it later to see if it can reproduce. @gauravsitlani

github-actions[bot] commented 2 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in a week if no further activity occurs. Thank you for your contributions.

github-actions[bot] commented 2 years ago

This issue has been automatically closed due to inactivity. Please re-open if this still requires investigation.

zhucan commented 1 year ago

if no objectstore was created, any "radosgw-cmd" was called will create the default pool of rgw. https://github.com/ceph/ceph/blob/a2bc2ab72d4a746f33f081eefcee1b326d07e46c/src/rgw/rgw_admin.cc#L5870