database timeout drops user connections

[k1ttt-remote]

Description

when the database(or maybe log file) gets locked for doing a server backup user connections may be dropped if it doesn't get unlocked soon enough, maybe 15-30 seconds is all it seems to take. Running mumble server 1.4.230 on windoze server 2022 that is very lightly loaded, but at 2am backup time users often get dropped as the backup process is freezing the database while it makes it's copy. at that time there are normally only a couple users connected but they get dropped and reconnect automatically in a minute or two. these users are not doing anything special, usually just listening to a channel or maybe doing some text chatting. on my old server i found that moving the sqlite and log files to a different disk away from the busy c: drive helped quite a bit so i did that on this new server, but the problem has returned, it just doesn't seem to last as long before users can reconnect.

Steps to reproduce

a scheduled server backup was starting

Mumble version

1.4.230

Mumble component

Both

OS

Windows

Reproducible?

Yes

Additional information

No response

Relevant log output

this is my connection (K1TTT) timing out 41 seconds after the server backup started
<W>2023-09-20 06:00:41.054 1 => <3:K1TTT(23)> Timeout
<W>2023-09-20 06:00:41.085 1 => <3:K1TTT(23)> Connection closed:  [-1]
<W>2023-09-20 06:01:15.714 1 => <0:(-1)> New connection: 192.168.0.18:56935
<W>2023-09-20 06:01:15.854 1 => <0:(-1)> Client version 1.4.230 (Windows Windows 10 Pro 2009 22621.1992 [x64]: 1.4.230)
<W>2023-09-20 06:01:15.917 1 => <2:K1TTT(23)> Authenticated

Screenshots

No response

[Krzmbrzl]

Well... you just can't freeze the database while the servervis running. The server needs access to the DB and thus freezing the DB freezes the server.

[k1ttt-remote]

Well, but, that is a normal function of windows servers to freeze database accesses during a backup. Normally database applications have some setting for how to wait for the database access to time out, and/or how many time to retry, before they give up and complain. Normally these periods on my server are only a matter of a couple of minutes at most.. Can't the mumble server live on cached data or retry a write for a few minutes before it dumps the user connections?

[Krzmbrzl]

that is a normal function of windows servers to freeze database accesses during a backup. 

Then it seems that Mumble is incompatible with that.

Normally database applications have some setting for how to wait for the database access to time out, and/or how many time to retry, before they give up and complain.

Perhaps it could be done. But I don't see it being terribly useful in general. So I don't expect this to be implemented by us - we have a lot more important construction sites that we don't have the manpower to handle.

Can't the mumble server live on cached data or retry a write for a few minutes before it dumps the user connections?

Not in general. Perhaps for certain contexts, but this would require careful investigation...

[github-actions[bot]]

As there has been no activity on this issue for a couple of days, we assume that your issue has been fixed in the meantime. Should this not be the case, please let us know.

If no further activity happens, this issue will be closed within 3 days.

[k1ttt-remote]

nothing has been fixed, i did find a windows server service that when disabled it speeds up the backup process enough that i don't see the problem every night now, but that isn't a guarantee that when the server gets busier it won't come back.

[Krzmbrzl]

We discussed this internally and came to the conclusion that we don't consider this to be an issue with the Mumble side of things. Freezing the database is not a valid use case for the Mumble server. Besides, backing up a SQLite database is effectively only copying a single file that (under normal circumstances) should have a couple of MB at most. Thus, copying it should happen effectively instantly.

I recommend using a proper backup tool for the job instead of using whatever Windows does with the building solution as this is doing something that we deem unacceptable for a live server instance (freezing the entire hard drive).

[davidebeatrici]

To be clear: running a Mumble server strictly in system memory should be feasible, but we consider it a feature request rather than something to fix.

Investigating why connections are dropped could be valuable, but it may very well be out of our scope (e.g. Qt or even the OS itself).