search

ceph / ceph-container

Docker files and images to run Ceph in containers
Apache License 2.0
1.32k stars 522 forks source link

Add new OSD generate a Permission denied #2234

Closed boussoufiane closed 1 month ago

boussoufiane commented 1 month ago

Hello ,

I added a 3 OSD on a kubernetes cluster . In fact i added partition without mount on the workers and Rook Ceph detect them automatically and create OSD . 2 of the OSD are created without no problem . The Third OSD generate an error on the Job POD "rook-ceph-osd-prepare" : 

[2024-08-21 15:48:49,445][ceph_volume.process][INFO  ] Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2/keyring
[2024-08-21 15:48:49,449][ceph_volume.process][INFO  ] Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2/
[2024-08-21 15:48:49,453][ceph_volume.process][INFO  ] Running command: /usr/bin/ceph-osd --cluster ceph --osd-objectstore bluestore --mkfs -i 2 --monmap /var/lib/ceph/osd/ceph-2/activate.monmap --keyfile - --osd-data /var/lib/ceph/osd/ceph-2/ --osd-uuid 7786a8e2-87b1-404f-8ccd-99a0640da43c --setuser ceph --setgroup ceph
[2024-08-21 15:48:49,636][ceph_volume.process][INFO  ] stderr unable to read/decode monmap from /var/lib/ceph/osd/ceph-2/activate.monmap: (13) Permission denied
[2024-08-21 15:48:49,636][ceph_volume.process][INFO  ] stderr 2024-08-21T15:48:49.634+0000 7f11b6ab4200 -1 monclient: get_monmap_and_config cannot identify monitors to contact
[2024-08-21 15:48:49,636][ceph_volume.process][INFO  ] stderr failed to fetch mon config (--no-mon-config to skip)
[2024-08-21 15:48:49,637][ceph_volume.devices.raw.prepare][ERROR ] raw prepare was unable to complete

Full logs : 

2024-08-21 15:48:45,743][ceph_volume.process][INFO  ] stdout SCSI_TPGS=0
[2024-08-21 15:48:45,743][ceph_volume.process][INFO  ] stdout SCSI_TYPE=disk
[2024-08-21 15:48:45,743][ceph_volume.process][INFO  ] stdout SCSI_VENDOR=VMware
[2024-08-21 15:48:45,743][ceph_volume.process][INFO  ] stdout SCSI_VENDOR_ENC=VMware\x20\x20
[2024-08-21 15:48:45,743][ceph_volume.process][INFO  ] stdout SUBSYSTEM=block
[2024-08-21 15:48:45,743][ceph_volume.process][INFO  ] stdout TAGS=:systemd:
[2024-08-21 15:48:45,743][ceph_volume.process][INFO  ] stdout USEC_INITIALIZED=4596386
[2024-08-21 15:48:45,744][ceph_volume.process][INFO  ] Running command: /usr/bin/ceph-authtool --gen-print-key
[2024-08-21 15:48:45,843][ceph_volume.process][INFO  ] stdout AQBdDMZm0U0pMhAAcRKEyrA3YJ3pTHl9vyU+SA==
[2024-08-21 15:48:45,844][ceph_volume.process][INFO  ] Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 7786a8e2-87b1-404f-8ccd-99a0640da43c
[2024-08-21 15:48:47,446][ceph_volume.process][INFO  ] stdout 2
[2024-08-21 15:48:47,446][ceph_volume.process][INFO  ] Running command: /usr/bin/ceph-authtool --gen-print-key
[2024-08-21 15:48:47,548][ceph_volume.process][INFO  ] stdout AQBfDMZmvHeVIBAAgvudWD8J9dP1xcA2RZRsrg==
[2024-08-21 15:48:47,549][ceph_volume.process][INFO  ] Running command: /usr/bin/mount -t tmpfs tmpfs /var/lib/ceph/osd/ceph-2
[2024-08-21 15:48:47,554][ceph_volume.util.system][INFO  ] CEPH_VOLUME_SKIP_RESTORECON environ is set, will not call restorecon
[2024-08-21 15:48:47,555][ceph_volume.process][INFO  ] Running command: /usr/bin/chown -R ceph:ceph /dev/sde
[2024-08-21 15:48:47,634][ceph_volume.process][INFO  ] Running command: /usr/bin/ln -s /dev/sde /var/lib/ceph/osd/ceph-2/block
[2024-08-21 15:48:47,640][ceph_volume.process][INFO  ] Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring mon getmap -o /var/lib/ceph/osd/ceph-2/activate.monmap
[2024-08-21 15:48:49,251][ceph_volume.process][INFO  ] stderr got monmap epoch 7
[2024-08-21 15:48:49,343][ceph_volume.process][INFO  ] Running command: /usr/bin/ceph-authtool /var/lib/ceph/osd/ceph-2/keyring --create-keyring --name osd.2 --add-key AQBdDMZm0U0pMhAAcRKEyrA3YJ3pTHl9vyU+SA==
[2024-08-21 15:48:49,443][ceph_volume.process][INFO  ] stdout creating /var/lib/ceph/osd/ceph-2/keyring
added entity osd.2 auth(key=AQBdDMZm0U0pMhAAcRKEyrA3YJ3pTHl9vyU+SA==)
[2024-08-21 15:48:49,445][ceph_volume.process][INFO  ] Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2/keyring
[2024-08-21 15:48:49,449][ceph_volume.process][INFO  ] Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2/
[2024-08-21 15:48:49,453][ceph_volume.process][INFO  ] Running command: /usr/bin/ceph-osd --cluster ceph --osd-objectstore bluestore --mkfs -i 2 --monmap /var/lib/ceph/osd/ceph-2/activate.monmap --keyfile - --osd-data /var/lib/ceph/osd/ceph-2/ --osd-uuid 7786a8e2-87b1-404f-8ccd-99a0640da43c --setuser ceph --setgroup ceph
[2024-08-21 15:48:49,636][ceph_volume.process][INFO  ] stderr unable to read/decode monmap from /var/lib/ceph/osd/ceph-2/activate.monmap: (13) Permission denied
[2024-08-21 15:48:49,636][ceph_volume.process][INFO  ] stderr 2024-08-21T15:48:49.634+0000 7f11b6ab4200 -1 monclient: get_monmap_and_config cannot identify monitors to contact
[2024-08-21 15:48:49,636][ceph_volume.process][INFO  ] stderr failed to fetch mon config (--no-mon-config to skip)
[2024-08-21 15:48:49,637][ceph_volume.devices.raw.prepare][ERROR ] raw prepare was unable to complete
Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/ceph_volume/devices/raw/prepare.py", line 91, in safe_prepare
    self.prepare()
  File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root
    return func(*a, **kw)
  File "/usr/lib/python3.6/site-packages/ceph_volume/devices/raw/prepare.py", line 134, in prepare
    tmpfs,
  File "/usr/lib/python3.6/site-packages/ceph_volume/devices/raw/prepare.py", line 68, in prepare_bluestore
    db=db
  File "/usr/lib/python3.6/site-packages/ceph_volume/util/prepare.py", line 481, in osd_mkfs_bluestore
    raise RuntimeError('Command failed with exit code %s: %s' % (returncode, ' '.join(command)))
RuntimeError: Command failed with exit code 1: /usr/bin/ceph-osd --cluster ceph --osd-objectstore bluestore --mkfs -i 2 --monmap /var/lib/ceph/osd/ceph-2/activate.monmap --keyfile - --osd-data /var/lib/ceph/osd/ceph-2/ --osd-uuid 7786a8e2-87b1-404f-8ccd-99a0640da43c --setuser ceph --setgroup ceph
[2024-08-21 15:48:49,637][ceph_volume.devices.raw.prepare][INFO  ] will rollback OSD ID creation
[2024-08-21 15:48:49,638][ceph_volume.process][INFO  ] Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.2 --yes-i-really-mean-it
[2024-08-21 15:48:51,335][ceph_volume.process][INFO  ] stderr purged osd.2
[2024-08-21 15:48:51,356][ceph_volume][ERROR ] exception caught by decorator
Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc
    return f(*a, **kw)
  File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 152, in main
    terminal.dispatch(self.mapper, subcommand_args)
  File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch
    instance.main()
  File "/usr/lib/python3.6/site-packages/ceph_volume/devices/raw/main.py", line 32, in main
    terminal.dispatch(self.mapper, self.argv)
  File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch
    instance.main()
  File "/usr/lib/python3.6/site-packages/ceph_volume/devices/raw/prepare.py", line 169, in main
    self.safe_prepare(self.args)
  File "/usr/lib/python3.6/site-packages/ceph_volume/devices/raw/prepare.py", line 91, in safe_prepare
    self.prepare()
  File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root
    return func(*a, **kw)
  File "/usr/lib/python3.6/site-packages/ceph_volume/devices/raw/prepare.py", line 134, in prepare
    tmpfs,
  File "/usr/lib/python3.6/site-packages/ceph_volume/devices/raw/prepare.py", line 68, in prepare_bluestore
    db=db
  File "/usr/lib/python3.6/site-packages/ceph_volume/util/prepare.py", line 481, in osd_mkfs_bluestore
    raise RuntimeError('Command failed with exit code %s: %s' % (returncode, ' '.join(command)))
RuntimeError: Command failed with exit code 1: /usr/bin/ceph-osd --cluster ceph --osd-objectstore bluestore --mkfs -i 2 --monmap /var/lib/ceph/osd/ceph-2/activate.monmap --keyfile - --osd-data /var/lib/ceph/osd/ceph-2/ --osd-uuid 7786a8e2-87b1-404f-8ccd-99a0640da43c --setuser ceph --setgroup ceph
[2024-08-21 20:04:24,253][ceph_volume.main][INFO  ] Running command: ceph-volume  inventory --format json /dev/sde

Environment:

Is someont has any idea from where it come ?

guits commented 1 month ago

I don't think you are reporting this at the right place, this project is about building a Ceph container image. As I understand, you are deploying with rook, so I think you should report your issue at https://github.com/rook/rook if you suspect there's an issue with the deployment tool.