Previously Functional Chat Backups Failing

maxtheaxe commented 1 year ago

[x] I have searched open and closed issues for duplicates
[x] I am submitting a bug report for existing functionality that does not work as intended
[x] I have read https://github.com/signalapp/Signal-Android/wiki/Submitting-useful-bug-reports
[x] This isn't a feature request or a discussion topic

Bug description

Backups that were previously working are failing on the verification step, with the attached (screenshotted) notification. There is far more than enough space for it to work. I am on the beta version from the Play Store, I would guess that this has been happening for about a month.

In case it's helpful, the backups are fairly large—I've been using Signal for at least five years, so, even without any particularly large individual files, my backups are about 4 GB apiece.

Steps to reproduce

Run create backup (either manually from the settings, or scheduled)

Actual result: Backup "finishes," at least percentage-wise, then verification fails and thus backup fails. No file is saved. Expected result: Backup should finish and succeed.

Screenshots

signal-2022-11-22-223258_002

Device info

Device: Samsung SM-A125F Android version: 12 Signal version: 6.3.2

Link to debug log

Multiple instances of the debug log (seemingly with different failure reasons, from a layman's examination) have previously been emailed to Signal support. I would prefer to email them to devs directly if support logs are inaccessible, if that's possible, rather than posting them on the issue tracker. Attached are the last handful of (seemingly relevant) lines, from the most recent failure:

signal_log_excerpt.txt

cody-signal commented 1 year ago

Can you send me the full log? cody@signal.org

maxtheaxe commented 1 year ago

Just sent—let me know if there's anything else you need.

cody-signal commented 1 year ago

Did you recently install any new sticker packs? Can you check for anything busted in stickers when viewing them in the keyboard or the overall sticker configuration screen?

maxtheaxe commented 1 year ago

I don't believe I've installed any sticker packs recently. I also just scrolled through all that I have installed and am not seeing any issues (though I'm unclear what exactly one would look like). Would it be helpful to uninstall all of them and then try running the backup again?

cody-signal commented 1 year ago

If you're okay when uninstalling them, then yes please, try that.

maxtheaxe commented 1 year ago

I deleted all the sticker packs and attempted a backup, with no luck. I've just sent you the latest log via email.

cody-signal commented 1 year ago

Thanks, I think I'll need to add more logs to know what's happening now. I'll get something into 6.4.

It's likely there is so malformed media creeping in that's impacting the backup creation and causing problems.

maxtheaxe commented 1 year ago

Understood. A couple (possibly unrelated) things worth noting:

This has happened in the past (in August, on version 5.46.1), then resolved itself upon update
In the past, when trying to free up space, I have noticed that some (presumably previously-functional, though I can't say for certain) media was corrupted(?), or at least unplayable, when I went back to check what was taking up space via the storage usage screen

cody-signal commented 1 year ago

Please try again when 6.4 comes out for you. It'll likely hit beta today or soon. Then send me another log please and thank you.

maxtheaxe commented 1 year ago

Just got the 6.4.0 update and tried it again—the debug logs are in your email.

cody-signal commented 1 year ago

Thanks for the updated logs. It appears the verify is correct in failing and that the backup file that was created is indeed broken. I'll take a look based on the logs and see if I can figure out what's causing the issue, but it's likely related to some attachment being in a weird state.

maxtheaxe commented 1 year ago

Understood. Is there a good way that you know of for tracking down problematic attachments and/or avoiding this problem in the future?

shawn1thompson commented 1 year ago

I'm having what appears to be the same issue, i.e., previously working backups now failing at verification. Would it be OK if I email @cody-signal my debug log?

alex-signal commented 1 year ago

@shawn1thompson You can certainly DM it to one of us if you don't feel comfortable posting it here.

cody-signal commented 1 year ago

@maxtheaxe @shawn1thompson

Still at a lost for what exactly is happening. I've added a bit more additional information to 6.5.3 that should land today/tomorrow. When it's available to you, can you do another backup attempt and send me the log.

Uploading and sending the link is much easier for me personally to process over the zip if you don't mind, but either way is better than no way.

maxtheaxe commented 1 year ago

Just uploaded and sent you the latest link via email.

cody-signal commented 1 year ago

Thanks for the updated log, sadly didn't help me identify what's going on.

And now for the ramblings of someone slowly going crazy:

It's like the backup file becomes unreadable as we are verifying it and causes the fail. What is wild it's the same attachment each time that it bails on. It's either the system taking the file away (but highly unlikely given the deterministic-ness) or creating the backup isn't doing a full write and is breaking at some point but the process continuing on as if it doesn't fail. I'm really at a lost at this point on what to try next. I'll get back to you.

maxtheaxe commented 1 year ago

As a relative layman, I don't want to throw you off the trail by giving you extraneous (potentially incorrect?) information. However, I looked through the new logs and noticed a specific attachment size. Out of the many, many thousands of attachments I have, I only have two (one, sent to two different chats on the same day) that seems to match the correct file size your new error message prints. That attachment was only sent December 10th, long after I started having this problem, which makes me think there's a deeper issue at hand. That being said, I did try running the backup again just now, and it seems to fail on the same attachment.

I'd like to help in any way I can—if you give me suggestions for further things to look at, I can do so. I'd also be happy to get on a call and give you the opportunity to walk through my app directly, if you think that would be helpful. Let me know.

cody-signal commented 1 year ago

Working with @maxtheaxe we've likely identified the issue as their backup being larger than 4gb and their filesystem being fat32 which doesn't support files that large.

@shawn1thompson can you see if the location for your backup is formatted as fat32? Based on the log you sent me it seems like it's likely.

shawn1thompson commented 1 year ago

I'm not sure how to check without putting the disk into a laptop which I can't do currently. Is there a way to check within Android, TWRP, or via adb maybe?

cody-signal commented 1 year ago

adb shell mount will output all things that are mounted. Which is a lot but may be able to see in there. I'd have to google around to see if there's a way to do it via an app/util.

shawn1thompson commented 1 year ago

Here's what I got:

gandalf@greyhavens:~$ adb shell mount | grep 6ABE-536C /dev/block/vold/public:179,1 on /mnt/media_rw/6ABE-536C type vfat (rw,dirsync,nosuid,nodev,noexec,noatime,gid=1023,fmask=0007,dmask=0007,allow_utime=0020,codepage=437,iocharset=iso8859-1,shortname=mixed,utf8,errors=remount-ro) /mnt/media_rw/6ABE-536C on /mnt/runtime/default/6ABE-536C type sdcardfs (rw,nosuid,nodev,noexec,noatime,fsuid=1023,fsgid=1023,gid=1015,mask=6) /mnt/media_rw/6ABE-536C on /mnt/runtime/read/6ABE-536C type sdcardfs (rw,nosuid,nodev,noexec,noatime,fsuid=1023,fsgid=1023,gid=9997,mask=18) /mnt/media_rw/6ABE-536C on /mnt/runtime/write/6ABE-536C type sdcardfs (rw,nosuid,nodev,noexec,noatime,fsuid=1023,fsgid=1023,gid=9997,mask=18) /mnt/media_rw/6ABE-536C on /mnt/runtime/full/6ABE-536C type sdcardfs (rw,nosuid,nodev,noexec,noatime,fsuid=1023,fsgid=1023,gid=9997,mask=7) /dev/fuse on /mnt/user/0/6ABE-536C type fuse (rw,lazytime,nosuid,nodev,noexec,noatime,user_id=0,group_id=0,allow_other) /dev/fuse on /mnt/installer/0/6ABE-536C type fuse (rw,lazytime,nosuid,nodev,noexec,noatime,user_id=0,group_id=0,allow_other) /dev/fuse on /mnt/androidwritable/0/6ABE-536C type fuse (rw,lazytime,nosuid,nodev,noexec,noatime,user_id=0,group_id=0,allow_other) /dev/fuse on /storage/6ABE-536C type fuse (rw,lazytime,nosuid,nodev,noexec,noatime,user_id=0,group_id=0,allow_other) /mnt/media_rw/6ABE-536C on /mnt/pass_through/0/6ABE-536C type sdcardfs (rw,nosuid,nodev,noexec,noatime,fsuid=1023,fsgid=1023,gid=9997,mask=7) Looks like vfat

cody-signal commented 1 year ago

Okay, seems like a similar situation where your backup is too large for fat formatted media. You can try reformatting as exfat or some other format that can handle larger than 4gb files. We are planning to incorporate some community and our work to bring chunked backups in during the upcoming SIgnal quarter.

shawn1thompson commented 1 year ago

That seems to have resolved it for me. Thank you.

greyson-signal commented 1 year ago

A lot of different things have been brought up here, but my read has been that at least some of the issues were pinned down to Fat32 filesystem issues. As cody mentioned, we do have plans to try to address this, but we probably won't be tracking that work here. I'm going to close this issue, and if people are having issues that they thing are not a result of filesystem format issues, then please open a new one :)

signalapp / Signal-Android