Closed jaxjexjox closed 4 months ago
I will have to write new documentation regarding backups.
The easiest method to create a backup is to stop all containers, in which case, if the image's implementation and the configuration used to start the container are correct, the directories used as volumes can simply be copied. If the directory structure is librenms/web/[logs|rrd|...]
for the web container and librenms/db
for MariaDB, simply stop both containers and copy the librenms
parent directory. You can verify the backup by creating a separate LibreNMS instance using the backed up directories and comparing the contents of the existing and separate installation.
Creating a backup at runtime (no docker stop
) usually entails knowledge of the inner workings of an application or container. This is because, at the time you issue cp
or a similar command, it may not be guaranteed that the data being copied remains in a logically consistent state if it is concurrently updated by the application during the copying process. For instance, MariaDB or MySQL have to be backed up using dedicated utilities such as mysqldump --single-transaction
. Furthermore, if multiple containers share state, logical consistency must be guaranteed collectively, not merely for each individual container.
A binary backup of MariaDB is probably less safe than an SQL backup, but faster to create and restore. If you are using mariadb:latest
instead of fixed version numbers, a sudden major jump in versions could lead to problems, since the MariaDB version must be compatible with the binary backup. This is not a problem in many cases, since MariaDB can upgrade old storage formats, but it may not be true for all storage engines, configurations and versions.
Creating an SQL backup and compressing it to gzip on the fly
MariaDB's port must be published for this using docker run -p 3306:3306
, or you have to figure out (using docker inspect
) the internal IP of the container: mysqldump --host=127.0.0.1 --port=3306 --user=mysqluser -p --single-transaction librenms | gzip -1 > librenms.gz
The password prompt can be avoided by either passing the password via command line, by using the implicit ~/.my.cnf
, or by passing an additional configuration file using defaults-extra-file=/some/path/.my.cnf
. This file would be located on the host, or wherever the backup is run from.
This is less secure than the other options: mysqldump --host=127.0.0.1 --port=3306 --user=mysqluser -pmysqlpass --single-transaction librenms | gzip -1 > librenms.gz
Create a .my.cnf
file in the home directory of the user running the backup, or specify mysqldump --defaults-extra-file=/some/path/.my.cnf
. Make sure the file is protected using chmod 600
or similar commands.
[mysqldump]
user=mysqluser
password=mysqlpass
I will have to write new documentation regarding backups.
Offline backups
The easiest method to create a backup is to stop all containers, in which case, if the image's implementation and the configuration used to start the container are correct, the directories used as volumes can simply be copied. If the directory structure is
librenms/web/[logs|rrd|...]
for the web container andlibrenms/db
for MariaDB, simply stop both containers and copy thelibrenms
parent directory. You can verify the backup by creating a separate LibreNMS instance using the backed up directories and comparing the contents of the existing and separate installation.Online backups
Creating a backup at runtime (no
docker stop
) usually entails knowledge of the inner workings of an application or container. This is because, at the time you issuecp
or a similar command, it may not be guaranteed that the data being copied remains in a logically consistent state if it is concurrently updated by the application during the copying process. For instance, MariaDB or MySQL have to be backed up using dedicated utilities such asmysqldump --single-transaction
. Furthermore, if multiple containers share state, logical consistency must be guaranteed collectively, not merely for each individual container.MariaDB backups
A binary backup of MariaDB is probably less safe than an SQL backup, but faster to create and restore. If you are using
mariadb:latest
instead of fixed version numbers, a sudden major jump in versions could lead to problems, since the MariaDB version must be compatible with the binary backup. This is not a problem in many cases, since MariaDB can upgrade old storage formats, but it may not be true for all storage engines, configurations and versions.Creating an SQL backup and compressing it to gzip on the fly MariaDB's port must be published for this using
docker run -p 3306:3306
, or you have to figure out (usingdocker inspect
) the internal IP of the container:mysqldump --host=127.0.0.1 --port=3306 --user=mysqluser -p --single-transaction librenms | gzip -1 > librenms.gz
The password prompt can be avoided by either passing the password via command line, by using the implicit
~/.my.cnf
, or by passing an additional configuration file usingdefaults-extra-file=/some/path/.my.cnf
. This file would be located on the host, or wherever the backup is run from.Passing the password via command line
This is less secure than the other options:
mysqldump --host=127.0.0.1 --port=3306 --user=mysqluser -pmysqlpass --single-transaction librenms | gzip -1 > librenms.gz
Passing the password via .my.cnf
Create a
.my.cnf
file in the home directory of the user running the backup, or specifymysqldump --defaults-extra-file=/some/path/.my.cnf
. Make sure the file is protected usingchmod 600
or similar commands.[mysqldump] user=mysqluser password=mysqlpass
I'll be honest with you, I'm actually terrified to STOP the containers. I could be wrong, I've been playing with your docker build of librenms for a few years on and off, perhaps I know better now, but when I did last time, I had difficulty spinning it back up, to the same state.
As I said in my op, my 'dumber' dockers, I just stop, pull new, re-start and off we go.
With libre, I feel like the API key or something may have impacted it starting again properly. Regardless it's time to play and learn! So thank you.
For what it's worth, I'm more than comfortable, with running multiple varieties of backups, storage is cheap and since I don't entirely know what I'm doing, I am happy to take multiple types.
Here's what I put together, before I realised you replied:
sudo mkdir /dockerconfigs/mariadbOther/backup
sudo docker exec -i mariadbOther mariabackup --backup --target-dir=/var/lib/mysql/backup --user=root --password=XXXXXXXXX
sudo docker exec -i mariadbOther mariabackup --prepare --target-dir=/var/lib/mysql/backup
sudo mkdir /dockerconfigs/mariadbOther/backupdump
sudo docker container exec mariadbOther bash -c "mysqldump -u root -pzxasqw12 --all-databases > "/var/lib/mysql/backupdump/all.sql""
sudo mkdir /backups/mariadbOther
sudo mkdir /backups/mariadbOther/backupdump
sudo mkdir /backups/mariadbOther/backup
sudo mv /dockerconfigs/mariadbOther/backupdump /backups/mariadbOther/
sudo mv /dockerconfigs/mariadbOther/backup /backups/mariadbOther/
I would actually prefer your method, take a cold, offline copy of the entire DB directory, so I will try that too shortly, thank you.
@jarischaefer
Thanks for your help, it's always appreciated! I'm really happy to admit, I was incorrect. I did manage to stop librenms, then mariadb (assuming that's correct order?)
I then backed up the fully offline data folders. Pulled a new container for libre (thank you) Re-ran them and they worked, did need to run something different to your instructions due to this error. "Your database is out of date" "lnms migrate" This did not fix the problem: https://github.com/jarischaefer/docker-librenms#12-database-update "docker exec librenms setup_database."
None the less, it's back up and running, with a new version of librenms and I feel a little safer stopping and starting.
I'll be honest here, I've been playing with your container on and off for so long, that it might have been back before I know to map content out, with the -v command, not sure.
So to be clear, my method was as follows
docker stop libre docker stop maria docker rm libre docker rm maria docker pull libre:specifictag docker run maria (same command / variables I used months ago) docker run libre (same command / variables I used months ago, except :specifictag for version) docker exec -i libre /bin/bash ./lnms migrate
That was about it.
I have only 2 questions remaining (sorry)
1, does that method seem correct to you?
2, at what point should I be considering updating the mariadb image for my container, if ever?
(I assume at some point in time, it's the smart thing to do for security, features or performance?)
Thank you!
There should be no need to run migrations. Was this message already present on the previous instance, or did it only appear after copying? Perhaps there is an issue with MariaDB's volumes. If you are unsure about losing data, there is no need to remove the containers, merely stopping them is sufficient. I assume all volumes are explicit via -v
and not implicit (see docker volume ls
).
Unless you have a good reason, just stick to a version that still receives updates: Docker Hub. New versions might introduce security features (algorithms, authentication schemes) that require new versions of the CLI and PHP libraries. The opposite, having a recent client, but a very old server, may also cause problems.
@jaxjexjox Some docs are available now: https://github.com/jarischaefer/docker-librenms#backups
Well, hello @jarischaefer I successfully upgraded my basic setup at home several months ago.
Sadly, I've attempted to upgrade both librenms and mariadb at my test work install, with no luck unfortunately. The long and the short of it is I am indeed being asked again, to migrate the database and this time it fails.
Here's what I've got now:
Version 23.1.0 - Wed Jan 25 2023 03:43:30 GMT+1100 Database Schema 2022_09_03_091314_update_ports_adsl_table_with_defaults (248) Web Server nginx/1.18.0 PHP 8.1.14 Python 3.10.6 Database MariaDB 10.10.3-MariaDB-1:10.10.3+maria~ubu2204 Laravel 8.83.27 RRDtool 1.7.2
I can upgrade my mariadb within the same chain as you outlined above, this works ok. MariaDB 10.10.7-MariaDB-1:10.10.7+maria~ubu2204 (I have reverted this to try and fix the problem)
Unfortunately when I upgrade my librenms that's where the problems begin (regardless of mariadb version, including unchanged)
Error from the web:
Database inconsistencies found during a database error, please fix to continue. Your database is out of date! ./lnms migrate
Then this:
USER@SERVERNAME:/dockerconfigs$ sudo docker exec -i librenmsOther /bin/bash su librenms cd /opt cd librenms ./lnms migrate
APPLICATION IN PRODUCTION.
WARN Command cancelled.
I've googled a few solutions and not getting much traction. Any thoughts? Setup database doesn't work (I have backups so I'm happy to try random things at this point)
and after all that and at least 90 minutes messing around, I managed to get it working.
sudo docker exec librenmsOther setup_database
Seems to work better than
sudo docker exec -i librenmsOther /bin/bash
then manually running the command? I'm not sure why this is, I thought I was starting to grasp some of this stuff, none the less re-setting up the database made migrating to a new version work fine!
Confused as to why this is but none the less I successfully upgraded again. I'm working and thanks for your hard work.
@jarischaefer I'm finally closing this off, you've been more than helpful and I should have closed it earlier.
One final thing, if you get a chance, can you merge / pull any changes from the official for a new build when you get a chance (I've no idea what criteria you use for new versions?) but I successfully got them to add a new feature very easily, I was quite impressed.
https://github.com/librenms/librenms/pull/15985
Thanks for your hard work!
@jaxjexjox I only build release versions, it should be in 24.5.0.
Hi @jarischaefer
Apologies, big apologies for this, I've been using docker for years but this is my first 'multi-docker' system with 2 dockers talking to one and other.
How do I absolutely, safely make a full backup of my configuration, so that I can comfortably update it, knowing it can be rescued? Again, I'm very sorry, I know this is probably elementary stuff but all my docker containers I've "backed up" the -v /data content and that's it, I've never had to backup a database before.
Here's how she was spun up.
I'm happy to follow the upgrade guide but now that I've finally got a copy working I like, it's become useful and I'd hate to lose the work put into it. Thank you, a lot for your time, really