search

inverse-inc / packetfence

PacketFence is a fully supported, trusted, Free and Open Source network access control (NAC) solution. Boasting an impressive feature set including a captive-portal for registration and remediation, centralized wired and wireless management, powerful BYOD management options, 802.1X support, layer-2 isolation of problematic devices; PacketFence can be used to effectively secure networks small to very large heterogeneous networks.
https://packetfence.org
GNU General Public License v2.0
1.38k stars 289 forks source link

Export/Import from 13.x to 14.x not working (MariaDBBackup Version VS. MariaServer Version) #8340

Closed MarioSpenc closed 1 month ago

MarioSpenc commented 1 month ago

According to Upgrade Guide, the Upgrade from 13.x (Debian 11) to 14.x (Debian 12) is only possible with new OS installation based on Import/Export Packefence Config.

For Export, MariaDBBackup Version must be equal (10.11.6).

According to your Upgrade Guide (which is also BUGGY: only Link to application is changed, NOT application itself), mariadbbackup should be downloaded and executed:

wget https://archive.mariadb.org/mariadb-10.11.6/repo/debian/pool/main/m/mariadb/mariadb-backup_10.11.6%2Bmaria~deb11_amd64.deb -O /root/mariadb-backup_10.11.6%2Bmaria~deb11_amd64.deb
dpkg-deb -xv /root/mariadb-backup_10.11.6%2Bmaria~deb11_amd64.deb /root/mariadb-backup
mv /root/mariadb-backup/usr/bin/mariabackup /usr/bin/mariabackup #### BUGGY ????####
mv /root/mariadb-backup/usr/bin/mbstream /usr/bin/mbstream

This works basically when I copy the application file from dpkg directory to /usr/bin/:

root@xxx:~# /usr/bin/mariabackup --version
/usr/bin/mariabackup based on MariaDB server 10.11.6-MariaDB debian-linux-gnu (x86_64)

But when running "backup-and-maintenance.sh", following error occures:

[00] 2024-10-08 15:13:13 Connecting to MariaDB server host: localhost, user: pf, password: set, port: not set, socket: /var/lib/mysql/mysql.sock
mariabackup: Warning: Charset id '33' csname 'utf8' trying to replace existing csname 'utf8mb3'
mariabackup: Warning: Charset id '83' csname 'utf8' trying to replace existing csname 'utf8mb3'
[00] 2024-10-08 15:13:13 Error: Unsupported server version: '10.5.26-MariaDB-0+deb11u2'.

No DB Backup possible.

JeGoi commented 1 month ago

Related to https://github.com/inverse-inc/packetfence/issues/8257

JeGoi commented 1 month ago

Did you update your pf version first? What is your current PF version?

MarioSpenc commented 1 month ago

PF Version is 13.1, all upgrades done (8 Oct.2024 15:00)

JeGoi commented 1 month ago

For now, if it is urgent, you can follow the el8 to deb12. https://www.packetfence.org/doc/14.0.0/PacketFence_Upgrade_Guide.html#_on_redhat_el8_from_packetfence_version_11_1_to_13_2

JeGoi commented 1 month ago

Could you do that please:

root@deb-cluster03:/usr/local/pf# mysql
Welcome to the MariaDB monitor.  Commands end with ; or \g.
Your MariaDB connection id is 4509
Server version: 10.5.24-MariaDB-1:10.5.24+maria~deb11-log mariadb.org binary distribution

Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

MariaDB [(none)]> show variables like 'char%'; show variables like 'collation%';

And copy paste the return? Thanks

JeGoi commented 1 month ago 
root@deb-cluster02:/usr/local/pf# mysql
Welcome to the MariaDB monitor.  Commands end with ; or \g.
Your MariaDB connection id is 4618
Server version: 10.5.23-MariaDB-0+deb11u1-log Debian 11

Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

MariaDB [(none)]> show variables like 'char%'; show variables like 'collation%';
+--------------------------+----------------------------+
| Variable_name            | Value                      |
+--------------------------+----------------------------+
| character_set_client     | utf8                       |
| character_set_connection | utf8                       |
| character_set_database   | utf8mb4                    |
| character_set_filesystem | binary                     |
| character_set_results    | utf8                       |
| character_set_server     | utf8mb4                    |
| character_set_system     | utf8                       |
| character_sets_dir       | /usr/share/mysql/charsets/ |
+--------------------------+----------------------------+
8 rows in set (0.001 sec)

+----------------------+-----------------+
| Variable_name        | Value           |
+----------------------+-----------------+
| collation_connection | utf8_general_ci |
| collation_database   | utf8mb4_bin     |
| collation_server     | utf8mb4_bin     |
+----------------------+-----------------+
3 rows in set (0.001 sec)
JeGoi commented 1 month ago

Hum, I think I am able to reproduce it on 13.0. I am testing 13.1 and 13.2 again. @MarioSpenc thanks I will search.

robbel1 commented 1 month ago

I am getting the same issue issue on 13.2, also had to install libpmem1 dependency for that version of mariadb backup.

root@packetfence:/usr/local/pf# mysql
Welcome to the MariaDB monitor.  Commands end with ; or \g.
Your MariaDB connection id is 4440
Server version: 10.5.26-MariaDB-0+deb11u2 Debian 11

Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

MariaDB [(none)]> show variables like 'char%'; show variables like 'collation%';
+--------------------------+----------------------------+
| Variable_name            | Value                      |
+--------------------------+----------------------------+
| character_set_client     | utf8                       |
| character_set_connection | utf8                       |
| character_set_database   | utf8mb4                    |
| character_set_filesystem | binary                     |
| character_set_results    | utf8                       |
| character_set_server     | utf8mb4                    |
| character_set_system     | utf8                       |
| character_sets_dir       | /usr/share/mysql/charsets/ |
+--------------------------+----------------------------+
8 rows in set (0.001 sec)

+----------------------+-----------------+
| Variable_name        | Value           |
+----------------------+-----------------+
| collation_connection | utf8_general_ci |
| collation_database   | utf8mb4_bin     |
| collation_server     | utf8mb4_bin     |
+----------------------+-----------------+
3 rows in set (0.001 sec)
root@packetfence:/usr/local/pf# lsb_release -a
No LSB modules are available.
Distributor ID:    Debian
Description:    Debian GNU/Linux 11 (bullseye)
Release:    11
Codename:    bullseye
root@packetfence:/usr/local/pf# mariabackup --version
mariabackup based on MariaDB server 10.11.6-MariaDB debian-linux-gnu (x86_64)
root@packetfence:/usr/local/pf/bin# ./pfcmd version
PacketFence 13.2.0
MarioSpenc commented 1 month ago 
Welcome to the MariaDB monitor.  Commands end with ; or \g.
Your MariaDB connection id is 23
Server version: 10.5.26-MariaDB-0+deb11u2 Debian 11

Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

MariaDB [(none)]> show variables like 'char%'; show variables like 'collation%';
+--------------------------+----------------------------+
| Variable_name            | Value                      |
+--------------------------+----------------------------+
| character_set_client     | utf8                       |
| character_set_connection | utf8                       |
| character_set_database   | utf8mb4                    |
| character_set_filesystem | binary                     |
| character_set_results    | utf8                       |
| character_set_server     | utf8mb4                    |
| character_set_system     | utf8                       |
| character_sets_dir       | /usr/share/mysql/charsets/ |
+--------------------------+----------------------------+
8 rows in set (0.002 sec)

+----------------------+-----------------+
| Variable_name        | Value           |
+----------------------+-----------------+
| collation_connection | utf8_general_ci |
| collation_database   | utf8mb4_bin     |
| collation_server     | utf8mb4_bin     |
+----------------------+-----------------+
3 rows in set (0.001 sec)

libpmem1 was also missing for me.

/Mario

MarioSpenc commented 1 month ago

Hum, I think I am able to reproduce it on 13.0. I am testing 13.1 and 13.2 again. @MarioSpenc thanks I will search.

My guess is that the problem is a version mismatch on MariaDB side, as PF only uses MariaDB Apps to export/import data.

I also tried to export without mariaDB-backup, so that we get a "versionless" SQL file to import. That ends up in a SQL error:

/tmp/tmp.rVVFP9lVKo ~
=================================================================================
Extracting archive...
Found the following content in the archive:
total 375560
-rw-r--r-- 1 root root       256 Oct  9 08:18 add_files.txt
-rw-r--r-- 1 root root      1100 Oct  9 08:18 grants.sql
-rw-r--r-- 1 root root  16360700 Oct  9 06:00 packetfence-db-dump-2024-10-09_06h00.sql.gz
-rw-r--r-- 1 root root 176592171 Oct  9 06:00 packetfence-files-dump-2024-10-09_06h00.tgz
-rw-r--r-- 1 root root      1630 Oct  9 08:18 stored_config_files.txt
drwxr-xr-x 3 root root      4096 Oct  9 08:18 usr
=================================================================================
Found files dump 'packetfence-files-dump-2024-10-09_06h00.tgz'
Extracting files dump
=================================================================================
Found compressed database dump 'packetfence-db-dump-2024-10-09_06h00.sql.gz'
Uncompressed database dump 'packetfence-db-dump-2024-10-09_06h00.sql'
=================================================================================
Get PF version in PacketFence export
13.1.0
=================================================================================
Stopping PacketFence services
Synchronizing state of monit.service with SysV service script with /lib/systemd/systemd-sysv-install.
Executing: /lib/systemd/systemd-sysv-install disable monit
=================================================================================
The database dump uses mysqldump
---------------------------------------------------------------------------------
Recreating database pf
---------------------------------------------------------------------------------
Dump file was made without triggers and procedures
Importing bare schema for 13.1
---------------------------------------------------------------------------------
Replacing create statements from the dump and removing drop statements
---------------------------------------------------------------------------------
Importing packetfence-db-dump-2024-10-09_06h00.sql into pf
ERROR at line 1: Unknown command '\-'.
Cleaning temporary directory

It seems that mariadb 10.11 is not able to handel first line of SQL export file:

/*M!999999\- enable the sandbox mode */

I have tried to delete first row of SQL file and run the import again: it seems to work, only on the last end of the skript I got a error:

Applying fixpermissions
Fixed permissions.
---------------------------------------------------------------------------------
Restarting packetfence-redis-cache
---------------------------------------------------------------------------------
Restarting packetfence-config
---------------------------------------------------------------------------------
Reloading configuration
Sereal: Error: Bad Sereal header: Not a valid Sereal document. at offset 1 of input at srl_decoder.c line 619 at /usr/local/pf/lib/pfconfig/util.pm line 84.
Failed to configreload once. Will wait a few seconds and try again
=================================================================================
Completed import of the database and the configuration! Complete any necessary adjustments and restart PacketFence
Synchronizing state of monit.service with SysV service script with /lib/systemd/systemd-sysv-install.
Executing: /lib/systemd/systemd-sysv-install enable monit
Created symlink /etc/systemd/system/packetfence.target.wants/monit.service → /etc/systemd/system/monit.service.
Cleaning temporary directory

PF seems to start normally within 14.0!!! Great!! Further tests pending .... ;-)

JeGoi commented 1 month ago

Ok thanks for your feedback.

Yes, there is no issue without mariadb-backup and I think we add a fix for the extra line at the beginning.

The latest error (Bad Sereal header) is something "normal" it will not have impact and need to be fixed since a long time now.

So, I should have keep it simple between EL8 and Debian11. My bad.

I am testing the current EL8 way on a Debian11. I will update the documentation accordingly.

JeGoi commented 1 month ago

@MarioSpenc

libpmem1 was also missing for me.

Yes it looks like my lab was messed up too. I add to install it too after revert a snapshot to an old 13.0.

JeGoi commented 1 month ago

I updated the doc for maintenance/14.0 and devel. https://github.com/inverse-inc/packetfence/commit/788b389d42616d09de9beff27376caae4000e701 https://github.com/inverse-inc/packetfence/commit/5af41aeea42848ed7ae97a7a23519728cc30861b I will update PacketFence docs as soon as the ci has prepared files.

JeGoi commented 1 month ago

Documentation has been updated on server website. Need to wait until Akamai cache updated or use a ?blablablabla at the end of the url with something like that https://www.packetfence.org/doc/PacketFence_Upgrade_Guide.html?blablablabla