All lists missing after move + upgrade

[ArchangeGabriel]

Version

6.2.70

Installation method

debian packages

Expected behavior

Previous lists should be available

Actual behavior

None of them are. The web interface show 0 lists, and all emails are rejected.

Additional information

I did a migration/upgrade following https://www.sympa.community/manual/upgrade/move.html.

Note that the database did not need to be moved, since it is on a remote server (so only connection parameters were copied over).

[ArchangeGabriel]

I don’t know where to start debugging this, I’ve tried to found some cli interface that would show database or existing lists but to no avail.

[ArchangeGabriel]

OK, it appears that most of the SQL tables went empty. I do have a backup of before the move/upgrade. What should be the correct procedure to fix this? Do I just need to import the SQL backup and it should be good? Or is there some part of the upgrade that should be redone? I can still trash the new install and copy over the old data again if required.

[ArchangeGabriel]

(I should precise that the previous version of SYMPA in use was 6.2.24)

[ikedas]

Hi @ArchangeGabriel ,

OK, it appears that most of the SQL tables went empty. I do have a backup of before the move/upgrade.

When and what tables went empty?

[ArchangeGabriel]

So after sleeping a bit, I’ve checked table by table and it appears most tables were already empty before, I guess that’s normal. The two that actually went empty are list_table and session_table (but I expect that last one to be OK?).

When

I can’t know for sure, sql seems to be logging only errors. What I know for sure is that the table was OK at the backup before upgrade (less than 24h ago), and that it is now empty.

Should I redo the whole upgrade process by restarting with the backup, or is there a way to only do the sql upgrade part (I don’t know exactly what is involved in the upgrade outside the sql tables)?

[ikedas]

Of the tables, only subscriber_table, admin_table, and user_table are absolutely necessary for Sympa operation, the others are optional and are needed depending on usage.

Since it is unlikely that the contents of the tables would be emptied on its own, it is likely that there has been some kind of mishandling. The best course of action is to start from a backup and redo the upgrade process, according to the documentation.

[ArchangeGabriel]

OK. I’ve done so now, the list_table is not empty anymore… but I still can’t see any list in the listmaster web interface.

Also, /etc/mail/sympa/aliases does not get populated, but maybe I should copy over the old one too?

[ArchangeGabriel]

P.S.: I can be available over IRC/Matrix/Discord if that could help debugging my setup.

[ikedas]

I am not good at English speaking. If you want to get direct support from me, please learn Japanese first 😃

[ArchangeGabriel]

Is there no command line instruction to dump the list of lists, or a way to see check Sympa is loading them? I’m trying to figure why Sympa does not see any lists while I have 90 of them…

[ikedas]

The manual says:

When you decide to move the existing service to the new server:

1. Stop all sympa processes on both servers.
2. Transfer all the data described in above from the old server.
3. Check "Upgrading notes" and make appropriate manual adjustments as necessity.
4. Edit the data_structure.version file on the new server to change the version value to reflect the old number.
5. Run "sympa.pl --upgrade". It will upgrade the database structure and several configuration files according to the hop you do.

Have you checked "Upgrading notes"? The transition from 6.2.24 to 6.2.70 requires quite a lot of adjustment.

[ArchangeGabriel]

Yes I did. It made me remove a lot of configuration knob, here are the old (first) and new (second) sympa.conf files (no other files were configured differently between the old setup and a fresh install excepted for topics.conf):

home    /var/lib/sympa
etc /etc/sympa
umask   027
spool   /var/spool/sympa
queue   /var/spool/sympa/msg
queuebounce /var/spool/sympa/bounce
queueautomatic  /var/spool/sympa/automatic
queuedigest /var/spool/sympa/digest
queuemod    /var/spool/sympa/moderation
queuetopic  /var/spool/sympa/topic
queueauth   /var/spool/sympa/auth
queueoutgoing   /var/spool/sympa/outgoing
queuetask   /var/spool/sympa/task
queuesubscribe  /var/spool/sympa/subscribe
http_host   http://listes.prepas.org
static_content_path /var/lib/sympa/static_content
static_content_url  /static-sympa
syslog `cat /etc/sympa/facility`
log_socket_type unix
log_level   0
log_smtp    off
logs_expiration_period  1
domain  listes.prepas.org
listmaster  <redacted>
email   sympa
create_list public_listmaster
edit_list   owner
cache_list_config   none
sympa_priority  1
default_list_priority   5
default_ttl 3600
default_distribution_ttl    300
sympa_packet_priority   5
request_priority    0
owner_priority  9
bulk_fork_threshold 1
bulk_max_count  3
bulk_lazytime   600
bulk_wait_to_fork   10
bulk_sleep  1
cookie  `/usr/bin/head -n1 /etc/sympa/cookie`
legacy_character_support_feature    off
max_size    5242880
use_blacklist   send,create_list
rfc2369_header_fields   help,subscribe,unsubscribe,post,owner,archive
remove_headers  X-Sympa-To,X-Family-To,Return-Receipt-To,Precedence,X-Sequence,Disposition-Notification-To
bounce_warn_rate    30
bounce_halt_rate    50
lang    fr
supported_lang  fr,en_US
sendmail    /usr/sbin/sendmail.wrapper
nrcpt   20
avg 10
maxsmtp 80
ssl_cert_dir    /var/lib/sympa/X509-user-certs
db_type mysql
db_name sympa
db_host <redacted>
db_user <redacted>
db_passwd   <redacted>
wwsympa_url http://listes.prepas.org/wws
antispam_tag_header_name    X-Spam-Status
antispam_tag_header_spam_regexp ^\s*Yes
antispam_tag_header_ham_regexp  ^\s*No
use_fast_cgi    1
arc_path    /var/lib/sympa/wwsarchive
archive_default_index   thrd
bounce_path /var/spool/sympa/wwsbounce
cookie_domain   localhost
cookie_expire   0
cookie_refresh  60
default_home    home
ldap_force_canonical_email  1
log_facility    `cat /etc/sympa/facility`
mhonarc /usr/bin/mhonarc
password_case   insensitive
review_page_size    25
title   Mailing lists service
use_html_editor 0
viewlogs_page_size  25
dmarc_protection_mode dmarc_reject

domain  listes.prepas.org
aliases_program postalias
listmaster  <redacted>
db_type mysql
db_host <redacted>
db_name sympa
db_user <redacted>
db_passwd   <redacted>
lang    fr
wwsympa_url https://listes.prepas.org/wws
static_content_path /usr/share/sympa/static_content
css_path    /var/lib/sympa/css
css_url /css-sympa
pictures_path   /var/lib/sympa/pictures
pictures_url    /pictures-sympa
password_hash   bcrypt
dmarc_protection.mode dmarc_reject,dkim_signature

[ArchangeGabriel]

OK… I think I’ve figured it out. It was home. I’ve overlooked the fact that it was not the default value.

[ArchangeGabriel]

OK, after copying over /etc/mail/sympa/aliases{,.db} from the old servers to the new one, everything seems in order now. Thank you for your premium rubber duck debugging.