OSD :: add custom file/path as osd

[adriansev]

Issue report

What version of MicroCeph are you using ?

snap list | grep ceph
microceph  18.2.0+snap21acc74fff  948    reef/stable    canonical**  held

What are the steps to reproduce this issue ?

1. try to add a generated file as OSD

What happens (observed behaviour) ?

microceph disk add /home/mceph-disks/osd1.disk

+-----------------------------+---------+
|            PATH             | STATUS  |
+-----------------------------+---------+
| /home/mceph-disks/osd1.disk | Failure |
+-----------------------------+---------+
Error: failed to set stable disk path: invalid disk path: /home/mceph-disks/osd1.disk

What were you expecting to happen ?

OSD to be added

Relevant logs, error output, etc.

Additional comments.

It would be really useful to have full flexibility of what is added as OSD .. the current usage of loop device brings the problems of filling the /var or host partition with the loop disks added With possibility to add custom files (hosted on various storages) as OSDs would allow better management of both available space and access in micro-deployments (the intended target of microceph)

[UtkarshBhatthere]

@adriansev accepting arbitrarily placed files as OSDs is not supported by microceph since it is a confined snap. However, you can create a loop device backed by the file and pass it as a device to be enrolled as an OSD.

[adriansev]

I apologize, i do not get what is an confined snap ... for the loop devices that microceph defines i see in the /var/snap/microceph/common/data/osd/ceph-1 a block symlink the have a full path to the backing file:

[Friday 29.03.24 19:45] root@sev : /var/snap/microceph/common/data/osd/ceph-1 $ 
ls -la block 
lrwxrwxrwx 1 root root 58 Mar 29 00:28 block -> /var/snap/microceph/common/data/osd/ceph-1/osd-backing.img

I did not tried yet to move the backing image and change the block symlink but i was thinking that maybe microceph cand do this from the start?

[UtkarshBhatthere]

Unlike a classic snap, a confined snap is isolated from it's host and does not have rw permissions for arbitrary paths. This is why we create the backing file on the permitted path /var/snap/microceph/....

Reference Strict Confinement

[adriansev]

so, i just moved the backing image and repoint the block symlink to it and ceph status, ceph health and ceph osd status show ok status .. so it works :) Given this, would it be possible to add to microceph disk add loop,4G,3 command an option that for would set the full path for a given singular OSD, or a base dir where the backing images would be placed if N>1 in the above command?

L.E.

snap info --verbose microceph | grep 'confinement\|devmode'
  confinement:       strict
  devmode:           false

snap list
Name       Version                Rev    Tracking       Publisher   Notes
core22     20240111               1122   latest/stable  canonical✓  base
microceph  18.2.0+snap21acc74fff  948    reef/stable    canonical✓  held
snapd      2.61.2                 21184  latest/stable  canonical✓  snapd

[UtkarshBhatthere]

That's a nice trick 🚀 However, I am afraid we cannot do that from within the MicroCeph snap due to the said confinements.

[UtkarshBhatthere]

Thus for arbitrarily placed backing files, the suggested way is to create loop devices backed by those files and enroll them as OSD: e.g. sudo microceph disk add /dev/loop7

[adriansev]

Got it, thanks a lot!