search

astro / skyflake

NixOS Hyperconverged Infrastructure on Nomad/NixOS
https://astro.github.io/skyflake/
MIT License
192 stars 6 forks source link

[Feature request] Use `ceph-volume` to manage OSDs #6

Open spacekitteh opened 1 year ago

spacekitteh commented 1 year ago

So, I'm converting my existing Ceph cluster to use Skyflake. One issue that is stopping me from using it to deploy OSDs with, however, is that I require full disk encryption for each OSD.

I deployed my OSDs via ceph-volume create, with the --dmcrypt flag. This sets up each OSD to be encrypted via LUKS. At boot, ceph-volume activate grabs the LUKS keys from a Ceph monitor, passes them to dm-crypt, and then proceeds as normal in activating the OSDs.

Existing OSDs can be easily converted to use ceph-volume; so is it possible to change the OSD management logic to use ceph-volume instead? I suspect it would greatly simplify the implementation.

astro commented 1 year ago

IIRC I didn't want that because it forced an extra LVM layer, duplicating what I was doing myself already.

Caution: don't put production data into Ceph on NixOS before there are good and safe ways to upgrade that cluster to future Ceph versions.

spacekitteh commented 1 year ago

ceph-volume has two modes; one which uses LVM, the other doesn't :)

Caution: don't put production data into Ceph on NixOS before there are good and safe ways to upgrade that cluster to future Ceph versions.

I've already done an upgrade cycle with it! From quincy to reef. It was relatively painless; just upgraded one monitor at a time, then the OSDs :)

spacekitteh commented 1 year ago

Also, recall that there is a services.ceph.<daemon>.package option, thanks to @SuperSandro2000 :)