search

ceph / ceph-ansible

Ansible playbooks to deploy Ceph, the distributed filesystem.
Apache License 2.0
1.68k stars 1.01k forks source link

`create dashboard admin user` hangs and `ceph -s` has PGs pending on creation with slow ops. #6531

Closed karasjoh000 closed 3 years ago

karasjoh000 commented 3 years ago

Bug Report

What happened:
Once in a while, the installation hangs on create dashboard admin user and takes 40min+ to complete that particular task (ceph-dashboard-no-grafana is a duplicate of ceph-dashboard with grafana server installation removed) : 

TASK [ceph-dashboard-no-grafana : config the current dashboard backend] *************************************************
Friday 14 May 2021  23:40:56 +0000 (0:00:00.098)       0:21:21.884 ************                                         
ok: [ceph01 -> 10.165.223.111]

TASK [ceph-dashboard-no-grafana : disable mgr dashboard module (restart)] ***********************************************
Friday 14 May 2021  23:40:57 +0000 (0:00:00.989)       0:21:22.873 ************                                         
changed: [ceph01 -> 10.165.223.111]

TASK [ceph-dashboard-no-grafana : enable mgr dashboard module (restart)] ************************************************
Friday 14 May 2021  23:40:59 +0000 (0:00:01.786)       0:21:24.660 ************                                         
changed: [ceph01 -> 10.165.223.111]

TASK [ceph-dashboard-no-grafana : create dashboard admin user] **********************************************************
Friday 14 May 2021  23:41:01 +0000 (0:00:02.019)       0:21:26.679 ************

ceph -s

  cluster:
    id:     <fsid>
    health: HEALTH_WARN
            mons are allowing insecure global_id reclaim
            1 MDSs report slow metadata IOs
            25484 PGs pending on creation
            Reduced data availability: 50 pgs inactive
            20 slow ops, oldest one blocked for 1609 sec, daemons [osd.36,osd.37,osd.46] have slow ops.

  services:
    mon: 3 daemons, quorum ceph-node-2,ceph-node-0,ceph-node-1 (age 38m)
    mgr: ceph-node-1(active, since 21m), standbys: ceph-node-2, ceph-node-0
    mds: 1/1 daemons up, 2 standby
    osd: 66 osds: 66 up (since 27m), 66 in (since 28m)

  data:
    volumes: 1/1 healthy
    pools:   13 pools, 2164 pgs
    objects: 20 objects, 2.3 KiB
    usage:   4.0 GiB used, 6.4 TiB / 6.4 TiB avail
    pgs:     0.832% pgs unknown
             1.571% pgs not active
             2112 active+clean
             32   creating+activating
             18   unknown
             2    peering

What you expected to happen:
ceph mgr and rados-gw commands should not hang for over 40min each.

How to reproduce it (minimal and precise):
install ceph and once in a while, it will hang on that task. Roughly once every three installs that happens to me. Not sure if it is specific to the env or is an installation issue. Running 3 VMs on top of openstack with lvm mounts and installing ceph on those vms and volumes.

Share your group_vars files, inventory and full ceph-ansibe log

# grep -rEv '^ *#|^\s*$|^\s*dummy:' . --include="*.yml"
./clients.yml:---
./mdss.yml:---
./mgrs.yml:---
./mons.yml:---
./osds.yml:---
./osds.yml:devices:
./osds.yml:  - /dev/vdd
./osds.yml:  - /dev/vde
./osds.yml:  - /dev/vdf
./osds.yml:  - /dev/vdg
./osds.yml:  - /dev/vdh
./osds.yml:  - /dev/vdi
./osds.yml:  - /dev/vdj
./osds.yml:  - /dev/vdk
./osds.yml:  - /dev/vdl
./osds.yml:  - /dev/vdm
./osds.yml:  - /dev/vdn
./osds.yml:  - /dev/vdo
./osds.yml:  - /dev/vdp
./osds.yml:  - /dev/vdq
./osds.yml:  - /dev/vdr
./osds.yml:  - /dev/vds
./osds.yml:  - /dev/vdt
./osds.yml:  - /dev/vdu
./osds.yml:  - /dev/vdv
./osds.yml:  - /dev/vdw
./osds.yml:  - /dev/vdx
./osds.yml:  - /dev/vdy
./osds.yml:bluestore_wal_devices:
./osds.yml: - /dev/vdb
./osds.yml: - /dev/vdc
./osds.yml:osd_auto_discovery: false
./rgws.yml:---
./rgws.yml:rgw_create_pools:
./rgws.yml:  "{{ rgw_zone }}.rgw.buckets.data":
./rgws.yml:    pg_num: 64
./rgws.yml:    type: replicated
./rgws.yml:    size: 3
./rgws.yml:  "{{ rgw_zone }}.rgw.buckets.index":
./rgws.yml:    pg_num: 16
./rgws.yml:    size: 3
./rgws.yml:    type: replicated
./rgws.yml:  "{{ rgw_zone }}.rgw.meta":
./rgws.yml:    pg_num: 8
./rgws.yml:    size: 3
./rgws.yml:    type: replicated
./rgws.yml:  "{{ rgw_zone }}.rgw.log":
./rgws.yml:    pg_num: 8
./rgws.yml:    size: 3
./rgws.yml:    type: replicated
./rgws.yml:  "{{ rgw_zone }}.rgw.control":
./rgws.yml:    pg_num: 8
./rgws.yml:    size: 3
./rgws.yml:    type: replicated
./zone.yml:rgw_zone: us-west-1
./zone.yml:rgw_zonemaster: true
./zone.yml:rgw_zonesecondary: false
./zone.yml:rgw_zonegroup: us-west
./zone.yml:rgw_zonegroupmaster: true
./all/all.yml:---
./all/all.yml:ceph_release_num:
./all/all.yml:  octopus: 15
./all/all.yml:ntp_service_enabled: false
./all/all.yml:ceph_origin: repository
./all/all.yml:ceph_repository: community
./all/all.yml:ceph_stable_release: octopus
./all/all.yml:monitor_interface: ens3
./all/all.yml:journal_size: 5120 # OSD journal size in MB
./all/all.yml:public_network: 10.250.1.0/24
./all/all.yml:radosgw_interface: ens3
./all/all.yml:containerized_deployment: true
./all/all.yml:openstack_config: true
./all/all.yml:openstack_glance_pool:
./all/all.yml: name: "images"
./all/all.yml: application: "rbd"
./all/all.yml: pg_autoscale_mode: true
./all/all.yml:openstack_cinder_pool:
./all/all.yml: name: "volumes"
./all/all.yml: application: "rbd"
./all/all.yml: pg_autoscale_mode: true
./all/all.yml:openstack_nova_pool:
./all/all.yml: name: "vms"
./all/all.yml: application: "rbd"
./all/all.yml: pg_autoscale_mode: true
./all/all.yml:openstack_cinder_backup_pool:
./all/all.yml: name: "backups"
./all/all.yml: application: "rbd"
./all/all.yml: pg_autoscale_mode: true
./all/all.yml:openstack_pools:
./all/all.yml: - "{{ openstack_glance_pool }}"
./all/all.yml: - "{{ openstack_cinder_pool }}"
./all/all.yml: - "{{ openstack_nova_pool }}"
./all/all.yml: - "{{ openstack_cinder_backup_pool }}"
./all/all.yml:openstack_keys:
./all/all.yml: - { name: client.glance, caps: { mon: "profile rbd", osd: "profile rbd pool={{ openstack_cinder_pool.name }}, profile rbd pool={{ openstack_glance_pool.name }}"}, mode: "0600" }
./all/all.yml: - { name: client.cinder, caps: { mon: "profile rbd", osd: "profile rbd pool={{ openstack_cinder_pool.name }}, profile rbd pool={{ openstack_nova_pool.name }}, profile rbd pool={{ openstack_glance_pool.name }}"}, mode: "0600" }
./all/all.yml: - { name: client.cinder-backup, caps: { mon: "profile rbd", osd: "profile rbd pool={{ openstack_cinder_backup_pool.name }}"}, mode: "0600" }
./all/all.yml: - { name: client.openstack, caps: { mon: "profile rbd", osd: "profile rbd pool={{ openstack_glance_pool.name }}, profile rbd pool={{ openstack_nova_pool.name }}, profile rbd pool={{ openstack_cinder_pool.name }}, profile rbd pool={{ openstack_cinder_backup_pool.name }}"}, mode: "0600" }
./all/all.yml:dashboard_enabled: True
./all/all.yml:dashboard_protocol: http
./all/all.yml:dashboard_admin_user: admin
./all/all.yml:dashboard_admin_password: password
./all/all.yml:grafana_admin_user: admin
./all/all.yml:grafana_admin_password: password

Environment:

guits commented 3 years ago

@karasjoh000 that's probably because of this :

20 slow ops, oldest one blocked for 1609 sec, daemons [osd.36,osd.37,osd.46] have slow ops.

could you check the output of :

ceph pg dump
ceph osd dump
github-actions[bot] commented 3 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in a week if no further activity occurs. Thank you for your contributions.

HashFail commented 3 years ago

I encountered the same error.

group_vars/all.yml

ceph_origin: repository
ceph_repository: community
ceph_stable_release: pacific
public_network: "10.0.0.0/24"
radosgw_interface: eth0
monitor_interface: eth0
dashboard_enabled: true
containerized_deployment: false
devices:
  - '/dev/vdb'
  - '/dev/vdc'
  - '/dev/vdd'
  - '/dev/vde'
monitoring_group_name: monitoring
dashboard_admin_password: xxx
grafana_admin_password: xxx

/etc/os-release

NAME="CentOS Linux"
VERSION="8"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="8"
PLATFORM_ID="platform:el8"
PRETTY_NAME="CentOS Linux 8"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:8"
HOME_URL="https://centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"
CENTOS_MANTISBT_PROJECT="CentOS-8"
CENTOS_MANTISBT_PROJECT_VERSION="8"

Full log: https://textbin.net/raw/1owjr4ovud

The solution was to do sudo ceph osd pool set (pool name) size 2 for each pool, doing this again each time a new pool was created. It seems ceph did not want to acknowledge the write until the required replication factor was decreased.

However, my ceph -s now looks like this:

sudo ceph -s
  cluster:
    id:     127318bb-031f-42c5-acc3-bc43e37ce230
    health: HEALTH_WARN
            mon is allowing insecure global_id reclaim
            Degraded data redundancy: 192/384 objects degraded (50.000%), 46 pgs degraded, 161 pgs undersized

  services:
    mon: 1 daemons, quorum ceph-mon0 (age 59m)
    mgr: ceph-mgr0(active, since 20m)
    osd: 4 osds: 4 up (since 56m), 4 in (since 56m)
    rgw: 1 daemon active (1 hosts, 1 zones)

  data:
    pools:   6 pools, 161 pgs
    objects: 192 objects, 6.9 KiB
    usage:   28 MiB used, 400 GiB / 400 GiB avail
    pgs:     192/384 objects degraded (50.000%)
             115 active+undersized
             46  active+undersized+degraded

  io:
    client:   1.2 KiB/s rd, 1 op/s rd, 0 op/s wr

Since I have 4 OSDs, shouldn't the writes be able to go through at the default size of 3? Is this a bug or am I misunderstanding something?

guits commented 3 years ago

could you share the output of ceph osd dump ?

HashFail commented 3 years ago 
epoch 77
fsid 127318bb-031f-42c5-acc3-bc43e37ce230
created 2021-06-03T20:19:45.629002+0000
modified 2021-06-04T00:03:12.171511+0000
flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit
crush_version 9
full_ratio 0.95
backfillfull_ratio 0.9
nearfull_ratio 0.85
require_min_compat_client luminous
min_compat_client jewel
require_osd_release pacific
stretch_mode_enabled false
pool 1 'device_health_metrics' replicated size 2 min_size 1 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode on last_change 77 flags hashpspool stripe_width 0 pg_num_min 1 application mgr_devicehealth
pool 2 '.rgw.root' replicated size 2 min_size 1 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 50 flags hashpspool stripe_width 0 application rgw
pool 3 'rbd' replicated size 2 min_size 1 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 67 flags hashpspool stripe_width 0 application rbd
pool 4 'default.rgw.log' replicated size 2 min_size 1 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 43 flags hashpspool stripe_width 0 application rgw
pool 5 'default.rgw.control' replicated size 2 min_size 1 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 51 flags hashpspool stripe_width 0 application rgw
pool 6 'default.rgw.meta' replicated size 2 min_size 1 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 pg_num_target 8 pgp_num_target 8 autoscale_mode on last_change 60 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 8 application rgw
max_osd 4
osd.0 up   in  weight 1 up_from 18 up_thru 67 down_at 0 last_clean_interval [0,0) [v2:10.0.0.47:6800/17411,v1:10.0.0.47:6801/17411] [v2:10.0.0.47:6802/17411,v1:10.0.0.47:6803/17411] exists,up ba392249-5421-4de0-971a-2233b17da392
osd.1 up   in  weight 1 up_from 18 up_thru 67 down_at 0 last_clean_interval [0,0) [v2:10.0.0.47:6808/18825,v1:10.0.0.47:6809/18825] [v2:10.0.0.47:6810/18825,v1:10.0.0.47:6811/18825] exists,up e476f828-cd73-40b7-a926-393c93a43558
osd.2 up   in  weight 1 up_from 18 up_thru 67 down_at 0 last_clean_interval [0,0) [v2:10.0.0.47:6816/20248,v1:10.0.0.47:6817/20248] [v2:10.0.0.47:6818/20248,v1:10.0.0.47:6819/20248] exists,up 1d7a1700-91f9-4822-9546-a54a26f908de
osd.3 up   in  weight 1 up_from 18 up_thru 68 down_at 0 last_clean_interval [0,0) [v2:10.0.0.47:6824/21663,v1:10.0.0.47:6825/21663] [v2:10.0.0.47:6826/21663,v1:10.0.0.47:6827/21663] exists,up 7f125387-6bae-4a4c-a188-511b0f0ba082
blocklist 10.0.0.9:0/678357123 expires 2021-06-04T21:53:06.125600+0000
blocklist 10.0.0.9:0/2528750619 expires 2021-06-04T21:53:06.125600+0000
blocklist 10.0.0.9:0/3826862751 expires 2021-06-04T21:52:34.408413+0000
blocklist 10.0.0.9:0/558637544 expires 2021-06-04T20:58:55.166815+0000
blocklist 10.0.0.9:0/3912413307 expires 2021-06-04T21:52:34.408413+0000
blocklist 10.0.0.9:0/137647918 expires 2021-06-04T20:21:26.797963+0000
blocklist 10.0.0.9:6800/15427 expires 2021-06-04T20:21:26.797963+0000
blocklist 10.0.0.9:6801/15427 expires 2021-06-04T20:21:26.797963+0000
blocklist 10.0.0.9:0/128005271 expires 2021-06-04T20:21:26.797963+0000
blocklist 10.0.0.9:0/908547403 expires 2021-06-04T20:28:16.266869+0000
blocklist 10.0.0.9:0/2663080188 expires 2021-06-04T20:58:55.166815+0000
blocklist 10.0.0.9:0/807399896 expires 2021-06-04T20:28:16.266869+0000
karasjoh000 commented 3 years ago

@HashFail does your ceph-ansible branch correspond with the ceph_stable_release: pacific config? See https://docs.ceph.com/projects/ceph-ansible/en/latest/index.html#releases.

HashFail commented 3 years ago

@karasjoh000 yes, the commit was the head of stable-6.0 at the time.

I think I might have figured out what is behind this: as you can see from my ceph osd dump, all my OSDs are on the same host. Adding a second host seems to fix the issue.

Perhaps this is the expected behavior? Maybe when the pool size is greater than 2, ceph wants at least one of the OSDs to be on a different host?

guits commented 3 years ago

@karasjoh000 yes, the commit was the head of stable-6.0 at the time.

I think I might have figured out what is behind this: as you can see from my ceph osd dump, all my OSDs are on the same host. Adding a second host seems to fix the issue.

Perhaps this is the expected behavior? Maybe when the pool size is greater than 2, ceph wants at least one of the OSDs to be on a different host?

yes, this is expected behavior, your pool are all size 2 and you only have 1 osd node. Hence the 161 pgs undersized