search

BastilleBSD / bastille

Bastille is an open-source system for automating deployment and management of containerized applications on FreeBSD.
https://bastillebsd.org
BSD 3-Clause "New" or "Revised" License
860 stars 140 forks source link

[ENHANCEMENT] Migrating jails between Bastille servers #679

Open VulcanRidr opened 9 months ago

VulcanRidr commented 9 months ago

I am in the process of migrating off of iocage, bastille is awesome so far. One of the very cool features is that you can "hot" export jails.

I have two mini-servers (Lenovo Thinkcentre m710qs) that are my jail servers. A really nice feature would be for some type of telemetry between the bastille processes to talk to each other, and be able to hot swap jails "on the fly".

Initially, it would be nice to extend bastille export functionality you could do something like "bastille export --host ...Then through the telemetry link, shut down the jail on the old host.

I wrote a set of scripts for iocage where I could iocage export and then copy the files over to the other host, and shut the jail down on the first host that I would be happy to share.

SnapshotCiTy commented 8 months ago

@VulcanRidr I'd be thankful to get a look at your iocage migration scripts if you'd accept to share.

About Bastille migration of jails between hosts, I stumbled upon this article, let us know if that works for you and solves it. But yes, I upvote that - in-the-tool migration of jails between hosts would be great, even for the purpose of getting a host ready to take over in case of failure of the first one.

https://hackacad.net/post/2023-04-28-migrating-multiple-bastille-jails-new-host/

Slightly out of topic: I'm looking for info/ressources to be more HA with jails - if you have any info, articles, I'm very interested. Would be great to do HA jails with Bastille.

VulcanRidr commented 8 months ago

Sure! You may get them from https://github.com/VulcanRidr/iocage_jail_mgmt. Be warned they work, but they are really ugly and kludgey.

I will take a look at that hackacad article.

Now that I have things more fully stood up, I can give you some details on my setup, and other thoughts on how I am using it...I have two jail hosts, erebus and equinox, running Bastille. I have mounted the backups directory on both to my TrueNAS box, so all backups are shared.

I am converting all of my jails from thin to thick jails, since they are long-term servers, and since I am having problems upgrading jails from 13.2 to 14.0. So I usually run all of the jails on one host or the other, but especially while I am converting the jails, I have all of the 13.2 ones on equinox, and then recreating them on 14.0 on erebus. But I can see other reasons why I would need to split jails across hosts.

So these are some of my thoughts. I know you can use the bastille_list directive to tell bastille which jails to start, but what about

VulcanRidr commented 8 months ago

You know, as I was mulling this over, another thought occurred to me. Since, unlike iocage, bastille takes advantage of thesnapshot feature of ZFS, instead of going down the route of exporting and importing jails, you could easily migrate jails by doing a zfs send /usr/bastille/ and any associated datasets (root, pgsql, mysql, etc). I always create separate pgsql and mysql datasets, since the ideal recordsizes are 8k and 16k respectively, though it appears I am going to have to fix my database datasets for the jails since they are set at 1M...Sigh.

In any case, I believe a mechanism that allows you to migrate jails through the zfs send | zfs recv would be faster than exporting and importing...

SnapshotCiTy commented 8 months ago

Thanks for the info you provide - this puts me on the trail to try to setup bastille (i want to move from Iocage to Bastille) and to set-up the migration or quick switch-over (in case of the main host going offline) of bastille-managed-jails from one host to another - through zfs send/rcv...

VulcanRidr commented 8 months ago

Cedric, just for the record, I have changed the way that I have my backups set up. I have started NFS mounting my /usr/local/bastille/backups from my NAS box on both jail servers. Then I run something similar to the following to back up the jails:

bastille list | grep -v Hostname | awk '{ print $3 }' | while read x ; do bastille export --xz $x ; done

which makes the jail backups immediately available on both servers.

VulcanRidr commented 8 months ago

Cedric, just for the record, I have changed the way that I have my backups set up. I have started NFS mounting my /usr/local/bastille/backups from my NAS box on both jail servers. Then I run something similar to the following to back up the jails:

bastille list | grep -v Hostname | awk '{ print $3 }' | while read x ; do bastille export --xz $x ; done

which makes the jail backups immediately available on both servers. I could write a wrapper script around this, but am considering the possibility/feasability of doing jail management through ansible.

bmac2 commented 3 days ago

@VulcanRidr is there an issue to do on this ticket? Seems you found a solution that works with your script.

VulcanRidr commented 2 days ago

@bmac2, what I was looking for was something where you could instrument bastille to be able to "push" jails between two bastille hosts, instead of backing up on one host, then when complete, going to the other host, turning it off (and possibly deleting it) before you turn on the jail on the new host...I already share /usr/local/bastille/backups between all bastille hosts (I technically have 3 since I have one on my desktop for spinning up boxes before they go onto the servers).

Instead, I was thinking, as it is a ZFS dataset, is it safe to zfs send | zfs recv /usr/local/bastille/jails/ from one host to the other, or does more need to be migrated?