A lot of Ceph issues I've been having with NixOS stem from the services generated by nixpkgs. Some things which I think will help massively are:
Use template service files, particularly for the OSD daemons. It's rather annoying at the moment to manually modify each service of a given type when you have a large number of them in your cluster; I don't want to hardcode names, and this is a perfect use-case for templated services here.
Create a service for activating ceph-volume OSDs. If you use ceph-volume to provision OSDs, then at each boot, you have to run ceph-volume lvm activate --all --no-systemd /after/ monitors become available, but /before/ any OSDs provisioned by ceph-volume are started. Currently, this is what I use:
Ensure the daemons have the proper paths. E.g. the mgr daemons should have pkgs.ceph and pkgs.smartmontools in their path (but smartctl still doesn't work on the dashboard; I'm still investigating).
Each daemon should have its state folder created automatically in /var/lib/ceph/, by user ceph. Currently, you have to manually create that folder.
If I think of more improvements that could be made, I'll add them here.
A lot of Ceph issues I've been having with NixOS stem from the services generated by nixpkgs. Some things which I think will help massively are:
ceph-volume
OSDs. If you useceph-volume
to provision OSDs, then at each boot, you have to runceph-volume lvm activate --all --no-systemd
/after/ monitors become available, but /before/ any OSDs provisioned byceph-volume
are started. Currently, this is what I use:but I'm still tinkering.
mgr
daemons should havepkgs.ceph
andpkgs.smartmontools
in their path (butsmartctl
still doesn't work on the dashboard; I'm still investigating)./var/lib/ceph/
, by userceph
. Currently, you have to manually create that folder.If I think of more improvements that could be made, I'll add them here.