search

status-im / status-desktop

Status Desktop client made in Nim & QML
https://status.app
Mozilla Public License 2.0
290 stars 78 forks source link

Check for available disk space before DB migration/re-encryption #11501

Open IvanBelyakoff opened 1 year ago

IvanBelyakoff commented 1 year ago

Starting 0.13 for the first time caused "re-encryption" to happen, which failed at the end due to insufficient disk space.

Description

We should pre-calculate the available disk space to be sufficient before starting DB migration/re-encryption and not proceed until enough space. Status_DiskFull

alexjba commented 1 year ago

We should also check for available disk space on password change. It's the same process.

anastasiyaig commented 1 year ago

@jrainville it it something for your team?

alexjba commented 1 year ago

IMO we could improve the error message here to inform the user of how much disk space the app needs to complete the re-encryption.

We currently have the following re-encryption flows that could fail:

  1. Login - when we migrate the password from upper-case to lower-case
  2. Login - when we migrate the sqlciper to V4
  3. Password change
jrainville commented 1 year ago

@jrainville it it something for your team?

Could be us or wallet. Or Alex since he has experience with DB re-encryption

igor-sirotin commented 6 months ago

cc @jrainville @alexjba

I made some tests. Found an old shared-account, which was v3-encrypted.

TLDR; Conclusions

  1. The database was never corrupted. Even after 2 failing re-encryptions I was able to free space and successfully re-encrypt the database.
  2. The error message is distinctive enough. Yes, it's a bit developer-vibe, since it's coming from go, but it contains enough details for the user.
  3. For me it took not too much time to get the error (~30 seconds).

I think it's not worth now to fix this issue.

  1. There's no simple precise way to calculate required disk space. It should be enough to 2*(<file.db>.size(), but that's just an estimation.
  2. We could simply check if the disk has "roughly enough" space, but that doesn't block the user from occupying the disk space while the re-encryption is in progress.
  3. We can't acquire the disk space before running the re-encryption. This is done by sqlite.
  4. Would be nice the make the error message better. E.g. show how much disk space is needed. But that's not super-important now and also requires design. And also check clause (1).

The worst case that's possible here is that we successfully re-encrypt the main DB, but fail to re-encrypt the wallet DB. Then the DBs will end up in conflict state. But:

  1. There might be a different error, not just the disk full.
  2. Disk full error is unlikely in this case, because main DB is usually larger than the wallet DB.

Testing details

Input:

Some commands I used for testing on MacOS:

# Create an file with 8 millions of blocks with 1k bytes block size, i.e. ~8Gb.
# There's a simpler command `mkfile`, but it doesn't occupy any actual space on disk.
> dd if=/dev/zero of=temp_1m bs=1k count=8m seek=0 

# Check free space on disk
> df -H

# Check file space occupied on disk
du -sh <filename>

Attempt 1 - 76Mb free space (extremely low)

> df -H
Filesystem        Size    Used   Avail Capacity iused ifree %iused  Mounted on
/dev/disk3s1s1    995G     15G     76M   100%    387k  739k   34%   /
  1. Enter the password
  2. Get almost immediate error NO SPACE LEFT ON DEVICE.
One or more databases not created
ERR 2024-04-04 13:04:36.435+01:00 error:                                     topics="accounts-service" tid=1526368 file=service.nim:656 procName=verifyDatabasePassword errDesription="One or more databases not created"
DBG 2024-04-04 13:04:36.437+01:00 Account logged in                          topics="accounts-service" tid=1526368 file=service.nim:676
failed to open db failed to set `journal_mode` pragma: disk I/O error: no space left on device
ERROR[04-04|13:04:36.868|github.com/status-im/status-go/mobile/status.go:236] failed to update node config fleet       key-uid=0x6c1dbb3634e25a69cea733f9c9051a9aeda6019a432b1031cd326a7b9fed2a4e error="Failed to migrate db file: Failed to migrate v3 db to v4: failed to set `journal_mode` pragma: disk I/O error: no space left on device"
ERR 2024-04-04 13:04:36.869+01:00 node.login                                 topics="signals-manager" tid=1526368 file=signals_manager.nim:54 error="Failed to migrate db file: Failed to migrate v3 db to v4: failed to set `journal_mode` pragma: disk I/O error: no space left on device"
ERR 2024-04-04 13:04:36.869+01:00 login error                                topics="app-controller" tid=1526368 file=module.nim:458 methodName=onNodeLogin errDesription="Failed to migrate db file: Failed to migrate v3 db to v4: failed to set `journal_mode` pragma: disk I/O error: no space left on device"
Screenshot 2024-04-04 at 13 04 45

Attempt 2 - 2Gb free space (half of DB size)

> dd if=/dev/zero of=temp_1m bs=1k count=8m seek=0
8388608+0 records in
8388608+0 records out
8589934592 bytes transferred in 63.340697 secs (135614778 bytes/sec)

> df -H
Filesystem        Size    Used   Avail Capacity iused ifree %iused  Mounted on
/dev/disk3s1s1    995G     15G   2081M    88%    387k   20M    2%   /
  1. Input the password
  2. After 30 seconds get error DATABASE OR DISK FULL
  3. There's still ~2Gb free space after this operation.
DBG 2024-04-04 13:08:36.577+01:00 primary_action                             topics="app-controller" tid=1526368 file=module.nim:207 currFlow=AppLogin currState=LoginKeycardEnterPassword
One or more databases not created
ERR 2024-04-04 13:08:36.585+01:00 error:                                     topics="accounts-service" tid=1526368 file=service.nim:656 procName=verifyDatabasePassword errDesription="One or more databases not created"
DBG 2024-04-04 13:08:36.585+01:00 Account logged in                          topics="accounts-service" tid=1526368 file=service.nim:676
ERROR[04-04|13:09:07.985|github.com/status-im/status-go/mobile/status.go:236] failed to update node config fleet       key-uid=0x6c1dbb3634e25a69cea733f9c9051a9aeda6019a432b1031cd326a7b9fed2a4e error="Failed to migrate db file: Failed to migrate v3 db to v4: database or disk is full"
ERR 2024-04-04 13:09:07.997+01:00 node.login                                 topics="signals-manager" tid=1526368 file=signals_manager.nim:54 error="Failed to migrate db file: Failed to migrate v3 db to v4: database or disk is full"
ERR 2024-04-04 13:09:07.998+01:00 login error                                topics="app-controller" tid=1526368 file=module.nim:458 methodName=onNodeLogin errDesription="Failed to migrate db file: Failed to migrate v3 db to v4: database or disk is full"
Screenshot 2024-04-04 at 13 22 30

Attempt 3 - 11Gb free space (enough for re-encryption)

Free up space.

> df -H
Filesystem        Size    Used   Avail Capacity iused ifree %iused  Mounted on
/dev/disk3s1s1    995G     15G     11G    59%    387k  104M    0%   /
  1. Input the password
  2. After 52 seconds: successful login
alexjba commented 6 months ago

Nice investigation!

I think it's not worth now to fix this issue.

I agree with you! It's ok since the error message is saying there's no disk space (If the user is not intimidated by the first part of the error). Maybe we can work in that direction and improve the message.

BTW, we have the same flows on password change. That might happen more often than the migration, but we should see a similar error there.

jrainville commented 6 months ago

I moved this to 2.30 and reduced the priority, since the problem is only visual and does not corrupt the DBs