Come up with "production deployment" setup

[remiscarlet]

Still using hacky 'load dev env as prod' crap. Start actually thinking of a proper deployment scheme under the minecraft user.

More generally this gives us the benefit of being able to take live traffic. Wouldn't want to do that with an env that's being dev'd on.

[remiscarlet]

• New user on YC host
• SSH key stored as GH secrets
• GHA workflow to run commands to pull latest on deploy.
  • How to handle restarting server if needed? Default to just pulling new code with no env restarts probably. Yukkuricraft/Yukkuricraft updates generally won't affect player-facing things on the minecraft server itself so the need to restart should be rare and thus reasonable to manually restart through YakumoDash if required.

In the future:

• Do we want to auto rotate SSH keys? GHA has an API for updating secrets.
  • AWS SM? GCP KMS?
  • Manual at beginning but as we mature our CaC/IaC might be worth looking into.
• Cron to update GHA secret for values like IP/username/etc.

[remiscarlet]

YC user setup is finished.

Thoughts on deployments:

• Bootstrapping process might be manual, but we will build a full "update manager" lib and expose invocations through our api.
• Token required to invoke endpoints
• Create Github secrets wrapper and create/update our auth token as needed.
• Invoked from Github Webhooks.
• Because of use of webhooks, we can configure each individual repo that needs to be updated and pull down updates as they happen individually.

Further Questions:

• What else needs to be done on deployments?
  • Will we ever need to restart containers due to code updates? Should we keep those manual?
  • Naively would think no because docker is its own system and should be stable from code updates here.
  • Counterargument would be anything that somehow affect plugin configs may require a full mc server restart to take effect
  • Others?

Ultimately this is neat because we keep all business logic within our py api and don't rely on github actions for deployment.

[remiscarlet]

Random thoughts:

• Unless we want to look into running multiple docker daemons on a single host, should probably set up the dev server hardware again so we can keep YC host only running "production" containers. Reduces chance of crossing streams due to a shared docker daemon.

[remiscarlet]

At this point we have:

• Dedicated hardware operating as a reverse proxy serving requests to separate prod and dev hosts.
  • No more "reverse proxy running on dev host" headaches
• Dynamic configs based on whether running on dev or prod host
• Pseudo-deployment from main branch to prod host.
  • Currently only does a git pull. Will need to decide whether api/yakumo build/restarts should be automated or manual.

Mostly there at this point in terms of infra.

• Last major task would be to implement an automated backup system.
  • This should include:
  • Backing up the worlds to backup disk
  • Moving periodic backups from backup disk to NAS
  • Moving monthly/quarterly backups from NAS to GCP/AWS/etc
• And of course actually "refactoring" monohost YC to containerized YC

[remiscarlet]

Use https://hub.docker.com/r/databack/mysql-backup for mysql data backups. This will likely require a slight refactor to fs structure.

MC backups may be non-trivial. How to separate out backup plugins from prod vs dev envs? Want to backup prod, not dev, but ideally shared (or possible to be shared) configs.

Could have a bindmount for backup location on prod, and maybe ephem volume for dev? Should clean itself up on docker compose down.

[remiscarlet]

MC data

• idfk
• https://github.com/itzg/docker-mc-backup

Mysql data

First layer backups (Local host):

• Backing up YC API mysql
• Backing up MC's prod env mysql
• Optionally configured backups of non-prod env MC mysql data

Second layer backups (NAS):

• Move daily backups to NAS
• 6 month retention? Use backup cleaner script?

Third layer (Cloud):

• Monthly
• Need to look into cheaper options - GCP backups are getting pricey.

General

• Use restic for everything
  • docker-mc-backup natively supports restic
  • Because we're using a sidecar, "enable for prod, disable for dev" is handled by just adding or not adding the docker-compose template based on env type.
  • mysql-backup will be the backup generation portion and restic will be the backup management portion.
• One liner for init'ing restic repo docker run --hostname $(hostname) -e RESTIC_PASSWORD=foobar -e RESTIC_REPOSITORY=/backup -v /media/backups-primary/restic:/backup restic/restic init

[remiscarlet]

https://github.com/Yukkuricraft/Yukkuricraft/commit/af3759393f652cb2dcecc2d203695eb6e5679002 implements MC backups using itzg/mc-backup with restic as the backend.

For mysql data, may want to create a new image using databack/mysql-backup as a base, install restic, and utilize /scripts.d/post-backup to move the mysql backups to the restic repo accordingly.

[remiscarlet]

https://github.com/Yukkuricraft/Yukkuricraft/commit/eab7fdf0f79bba17ec742223d3cd8148dbe67449 implements MySQL backups using databack/mysql-backup along with restic

Both MC and MySQL now use a custom image that chown's the restic directory first before continuing with the original entrypoints.

Remaining:

• Only local backups are implemented. Need to still implement NAS and cloud backups
• Implement/test automated restoring of backups.

[remiscarlet]

Not super important for now but should implement a sidecar to run restic check https://github.com/Yukkuricraft/Yukkuricraft/issues/33

[remiscarlet]

Consider: Eventually have every worldgroup backed up at the same time. Will probably have to reimplement some portion of itzg/mc-backup to accomplish this.

https://github.com/Yukkuricraft/Yukkuricraft/issues/34