search

mjl- / mox

modern full-featured open source secure mail server for low-maintenance self-hosted email
https://www.xmox.nl
MIT License
3.71k stars 113 forks source link

Error while upgrading to v0.0.7 + bugfix! #71

Closed naturalethic closed 7 months ago

naturalethic commented 1 year ago

Following the upgrade steps, when I get here:

mox verifydata data/tmp/tesetupgrade

I'm getting an error

error: data/tmp/tesetupgrade/accounts/joshua/index.db: checking message size: data/tmp/tesetupgrade/accounts/joshua/msg/f/47874: message size is 10586, should be 0 (length of MsgPrefix 0 + file size 0), see "mox fixmsgsize"
errors were found

Running fixmsgsize then doing the backup again, results in the same. Not sure how to proceed.

mjl- commented 1 year ago

Running fixmsgsize then doing the backup again, results in the same. Not sure how to proceed.

A few questions to zoom in the behaviour: Did "mox fixmsgsize" print that it is fixing that message (and/or other messages)? And did you have to create replacement empty files following the upgrade instructions? And are you going from v0.0.6 to v0.0.7?

naturalethic commented 1 year ago

I followed the steps and I replaced some empty files and everything seemed fine. Only when I went to then check the backup did it produce the errors. I am going from 0.0.6 to 0.0.7

mjl- commented 1 year ago

Just to be sure: The "mox-v0.0.6 fixmsgsize" after creating the empty files did not find any files with wrong size, right? Also not if you run it again?

Could you copy your account's index.db (e.g. data/accounts/joshua/index.db, of the active mox instance, not of the backup) to a temporary place, and build github.com/mjl-/bstore/cmd/bstore (go install github.com/mjl-/bstore/cmd/bstore@latest), or download it (https://beta.gobuilds.org/github.com/mjl-/bstore@v0.0.2/cmd/bstore/linux-amd64-go1.21.1-stripped/0vYhmpN2-JLTb-i_pncAISujF0dA/), and run bstore exportcsv copied-index.db Message >messages.csv, open it and look for the row with ID 47874, and check the column "Size" and "MsgPrefix"? Given the output from verifydata, it will probably say it has a Size of 10586 and empty MsgPrefix.

If possible, could you share the whole row of that message with any personal/identifyable information left out? You could also send it to me at mechiel@ueber.net, and I'll see if there's anything special about it and if I can reproduce the behaviour (I'm not yet seeing what could be causing this). We could also do an interactive attempt to fix this on irc/slack/matrix if you want to give that a try.

naturalethic commented 1 year ago

Sent to your email. Thanks for looking into it!

mjl- commented 1 year ago

We found the bug with testing from @naturalethic, thanks! The attached patch against v0.0.6 fixes the issue. The bug: "mox fixmsgsize" (mentioned in the release notes) would stop after the first batch of 10k messages. So if any problematic message was beyond the first 10k, it wouldn't be fixed. But it would be detected during the backup+verifydata pre-upgrade check. By applying this patch against v0.0.6, restarting mox with the new build, and running "mox fixmsgsize" again, the problem should be fixed.

fixmsgsize.patch.txt

To compile, you need a Go toolchain installed. Try "go version". It should say a version >= go1.20. If missing or older, get the latest from https://go.dev/dl/. The example below compiles for the same operating system/architecture you run on. Quite possibly a linux/amd64 machine. It's easy to cross-compile too (I can give instructions if needed).

$ go version
go version go1.21.1 linux/amd64

$ git clone https://github.com/mjl-/mox
Cloning into 'mox'...
remote: Enumerating objects: 4956, done.
remote: Counting objects: 100% (1092/1092), done.
remote: Compressing objects: 100% (367/367), done.
remote: Total 4956 (delta 757), reused 884 (delta 712), pack-reused 3864
Receiving objects: 100% (4956/4956), 7.02 MiB | 2.18 MiB/s, done.
Resolving deltas: 100% (2861/2861), done.

$ cd mox

$ git checkout v0.0.6
Note: switching to 'v0.0.6'.

You are in 'detached HEAD' state. You can look around, make experimental
[...]

$ patch -p1 < fixmsgsize.patch.txt

$ CGO_ENABLED=0 go build

$ ls -l mox
-rwxr-x--- 1 mjl mjl 26375707 Oct  2 16:38 mox

I'll update the release notes with a link to this issue, and keep this issue open at least until the next release.

mjl- commented 7 months ago

We kept this issue open for a while so people could more easily find it in case they ran into this, or when looking for details during upgrades. It has been a while so seems safe to close.