[FEAT]: enable failover, hot/hot, hot/wam, hot/cold

[FlyingToto]

Feature Description

As a user I would like to have the ability to replicate my comos-cloud setup to a secondary host so I can failover in case of failure...

Proposed Approach:

1. document manual steps in the documentation... :-)

• theoritically, comsos itself would have to be stetup on the secondary node ideally by cloning configs, db and certs.
• for each app installed, the mounted folders should get replicated to the secondary node this could be done using syncthing running within cosmos-cloud itself(not 100%sure) .
• provide high level instructions on how one can setup load balancers failovers, and give a step-by-step example for 1 specific front-end (like opnsense FW) then let the community provide more step-by-step recipes.
• provide steps for handling hot/hot vs hot/warm vs hot/cold(-> require container to not start automatically)
• provide steps for failing back (ie: ensuring primary is up, and replication from dr to prod completed before failing back)
• provide high level explanations on failover limits (stateless services can be Hot/Hot, vs stateful db which might have to be hot/warm or hot/cold)

2. automate setting up secondary nodes

• I would assume that all steps above could be scripted except maybe the fact that cosmos-server itself can't really assess if app X bound folder Y requires live replica or not, nor which parameters might be hosts specifics (although most containers should remain agnostics any that require hosts network might end up with a host specific dependancy for instance...)
• this might require applications to provide additional custom settings to enable full automation like (ie: which specific HA strategy works, which folders need replication, which limitations apply, etc... )
• for hosts specific services, there might be a need to use a VIP and migrate it as part of the failover between the 2 nodes...

[azukaar]

While this is a certainly a well written ticket (thanks) I doubt I would implement this. I could implement it for a "business" level feature-set that would not be accessible for free, but that's it, this is WAY beyond the scope of consumer level home selfhosting. This is why:

• The motivation behind this is high availability. Meaning that this feature would need to be very meticulously tested and implemented, to ensure maximum level of stability. This is a lot of pressure that I'm not ready to take on (yet) as a single dev
• This requires you to have AT LEAST 3 servers (the LB, on server, and one failover). Justifying that this is way beyond the scope of simple selfhosting at home (also because 2 of the servers need to be remote for this to be effective)
• Most of the apps people self-host with Cosmos cannot be replicated safely. For example, if you try to run two instances at the same time of Nextcloud (assuming this does not have specific configs for this scenario), it will not work and you won't be able to switch over with 0 downtime (the instance B would need to booted AFTER instance A failed, adding a downtime, then the config would lag behind because of delays in synchronisation). Meaning it's only relevant in specific apps (usually much more pro-level application)