search

astro / skyflake

NixOS Hyperconverged Infrastructure on Nomad/NixOS
https://astro.github.io/skyflake/
MIT License
192 stars 6 forks source link

Exchanging Ceph for SeaweedFS #9

Open c4lliope opened 9 months ago

c4lliope commented 9 months ago

Hello @astro ! Also cc @spacekitteh

I am a member of HacDC, an american hackerspace inspired by the CCC. I had seen @astro's NixCon talk regarding microvm.nix, and i am searching for how to run the skyflake code in my company's lab - also in the hackerspace soon if possible.

As the talk ends, @astro says 'One does not simply do a major-version upgrade of Ceph'. This is similar to my experience, and Ceph seems to be a large and cumbersome dependency for this code.

Since i had no luck launching Ceph locally on my 3 nodes, i am choosing to use SeaweedFS; the commands to run a HA cluster are super simple and i accomplished them in less than an hour. My seaweed systemd config is here: https://base.bingo/code/build/src/branch/mesh/nixos/process.nix#L31

Can you help me remake skyflake to run on the much simpler seaweedFS, and drop ceph as a dependency?

I looked around the nix modules, and my guess is the host nodes only need shared volumes for the nix store and the garbage collection roots. Are these volumes also shared inside the microVMs, or only used during the original vm builds? There should also be persistent VM storage, and that is a secondary concern.

I plan to spend some more days on this though I'm in a hurry to be back to application programming; i could use as much help as possible. Your approach seems much more dependable than kubernetes and i think reducing your dependency on Ceph can make your program much more accessible.

Glad for your help! Callie

astro commented 9 months ago

Cool, I haven't had SeaweedFS on my radar! If you don't do the PR yourself, I would like to steal from your nix config.

Does SeaweedFS perform well for you? Does anything hang when one node disappears?

My intention behind this project's directory hierarchy was that there is space for more than one network filesystem in nixos-modules/storage/. Everything else should not require Ceph.

c4lliope commented 8 months ago

@astro - I lag a small number of hours behind you on our globe; can you and I manage a call tomorrow during your early afternoon?

astro commented 8 months ago

I don't have Discord as there is no NixOS module for Discord servers yet. Are you on Matrix? microvm.nix has its own room: https://matrix.to/#/#microvm.nix:envs.net

astro commented 5 months ago

There is also: