signalapp / Signal-iOS

A private messenger for iOS.
https://signal.org
GNU Affero General Public License v3.0
10.74k stars 3k forks source link

Signal for iOS/iPadOS is taking up too much space even though all media is deleted #4916

Open b3hni4 opened 3 years ago

b3hni4 commented 3 years ago

Bug description

I have only had a single group chat which I also deleted all media files. But the app still uses close to a gigabyte of storage on iPhone and the other linked device (iPad). Only thing that helped was to delete entire message history on the linked device. But It does not make sense for only text chats to take up such space, and I don’t want to loose the chat history in my case.

Steps to reproduce

Device info

Device: iPhone XS, iPad Pro (11 inch)

iOS version: 14.4.1

Signal version: 5.7.0.19

Link to debug log

https://debuglogs.org/3897732fbea5289b0b108c64f5657c71f09267a58d92c7d1596b97fc38cd9574.zip

HanSeoulOh commented 2 years ago

I have deleted all chats and I am still having this issue. This should have higher priority than it is currently given as it suggests that Signal app does not actually delete conversations or messages when the user thinks they have.

ablakey commented 2 years ago

The storage details API was not properly utilized for iOS. So unlike other apps, you cannot see a breakdown of usage or clear that usage from the iOS settings.

I have 8GB of storage Signal is holding on to, even after I deleted every chat with the "Clear Chat History" option.

Please let me know if I can help with this issue somehow because it seems to have been around for years and is pretty severe in my opinion.

hyiltiz commented 2 years ago

As @ablakey pointed out, on iOS you do not have the option to "manage storate" while on Android you do. As a result, you cannot clear large photos or videos sent for one-time viewing by friends. This is a critical feature mismatch across platforms.

jonathancross commented 2 years ago

I'm supporting iOS users which are having this same issue. Example: 11gb used space is not being reduced after clearing out the conversations which contain large media files, etc. There is clearly an issue with freeing storage.

Note: there have been several bugs before this one which were ignored and automatically closed as "stale", despite being serious user facing issues.

stale[bot] commented 2 years ago

Is this still relevant? If so, what is blocking it? Is there anything you can do to help move it forward?

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.

ablakey commented 2 years ago

This is not stale. It’s just not being addressed in a timely manner. This is a very debilitating problem: eating gigs of space and crippling security guarantees by not actually deleting message contents from local storage.

ablakey commented 2 years ago

That's rather embarrassing, Signal. Is this problem too complicated for you?

Edit: I regret this comment. It's rude and unhelpful. My apologies.

HanSeoulOh commented 2 years ago

I've stopped using signal and recommended everyone to not use signal because of this issue. I suspect the consequences of this problem are a lot more sinister than what you might conclude at a first glance.

EDIT: I do not mean sinister as in the devs' have sinister motives, but as in the consequences of this issue are sinister as in potentially harmful. It is a valid flag for a security concern.

b3hni4 commented 2 years ago

@Imperiopolis-Signal How can we reopen this issue?

javajosh commented 2 years ago

Signal maintainers, you ought to reopen this issue, or at least merge it with an active issue. This issue is serious and real, and it does you no favors to ignore it. And I say this as one who gives you money.

nathantypanski commented 2 years ago

This is likely a consequence of the encrypted SQLite message database. Claims that this is a security issue rather than simply a resource waste issue should be viewed with suspicion until substantiated.

  1. Signal uses SQLCipher to store messages, encrypted, on disk.
  2. This is a known problem with SQLCipher. It can probably be resolved by vacuuming SQLCipher, or enabling autovacuum.
joshaustintech commented 2 years ago

Hi Signal maintainers (@michelle-signal, @jrose-signal @Imperiopolis-Signal, someone?), I'm sure you all are very busy, but when you have a chance can this be reopened and prioritized, please? As an iOS user and donor I'm very concerned that this hasn't been prioritized for a year now.

HanSeoulOh commented 2 years ago

This is likely a consequence of the encrypted SQLite message database. Claims that this is a security issue rather than simply a resource waste issue should be viewed with suspicion until substantiated.

  1. Signal uses SQLCipher to store messages, encrypted, on disk.
  2. This is a known problem with SQLCipher. It can probably be resolved by vacuuming SQLCipher, or enabling autovacuum.

I certainly hope this is true, and here's hoping for a simple fix. I am a long time user of Signal and was brought to it because of its better security intentions. Having this issue for a while now, I check back frequently and am still surprised that nearly 3 years later it still has not received the attention it deserves. I would be much more comfortable and confident that there is not much to worry about if a dev chimed in and assured us there is no problem and that it is low priority. But they have not made any comments yet, which made me increasingly concerned.

Reading into your suggestions it does seem indeed that your explanation is the most likely cause of the issue, however I definitely would feel better hearing it from the devs themselves that they have investigated the anomaly and came to the same conclusion.

pitis commented 2 years ago

this is unacceptable. I've been using signal for a few years now but only had a few contacts and signal is eating 11gbs of my storage and although I deleted the chats, it still uses 11gb...

This looks like a security issue as well as a performance issue. Just can't tell people to change to Signal anymore

sgreadly commented 2 years ago

I too wonder how this can be fixed. My iPhone Signal app is 16GB large and I can't figure out how to reduce its size.

jonathancross commented 2 years ago

@EvanHahn-Signal or @max-signal Any chance one you can reopen this issue which was closed incorrectly by the stalebot?

mkerollos commented 1 year ago

@harry-signal @sashaweiss-signal @igor-signal @Imperiopolis-Signal sorry for the spam (you're the most recent contributors) but it looks this one was incorrectly closed as stale but still impacts iOS (iOS 15.6.1 with Signal 5.51.0.14). This has a bigger impact the longer a user is on signal, as media accumulates to tens of gigabytes and will be a significant portion of storage for those on older/lower storage phones.

sgreadly commented 1 year ago

I must say, I'm finding very difficult to keep the family and friends on Signal for this very bug and reason. They're not tech savvy, and aren't bothered to do workarounds (not that I know of any), and if the only workaround is to delete and reinstall the app, no one will bother reinstalling it. And when they run out of space.. guess which app will be deleted and ridden with.

derekmorr commented 1 year ago

I looked at the code last night and confirmed that the database is never vacuumed and auto-vacuum is not used. Those are the likely culprits.

Enabling vacuum shouldn't be difficult, but you need to strike a balance between running it often enough to reclaim space vs to often which adds overhead (and blocks other DB operations). You could use a heuristic and look at the free page pragma to see if there is enough unused space to warrant vacuum. Or if an attachment has been deleted you could look at byteCount. If those are over a threshold, you could run vacuum.

Its also worth noting that the Android and desktop apps do not call VACUUM or run auto-vacuum either.

sgreadly commented 1 year ago

Thanks @derekmorr ,

Is that vacuum instruction aimed at us end users? or at dev? If us, do you have any instructions on how I can do that on iOS? I don't mind doing it even if just once to significantly reduce the current size.

Ta.

derekmorr commented 1 year ago

It was aimed at developers. The app should vacuum its own database -- end-users should not have to do it.

sgreadly commented 1 year ago

Ah ok :) Thanks for the clarification.

jonathancross commented 1 year ago

Its also worth noting that the Android and desktop apps do not call VACUUM or run auto-vacuum either.

Given that there is not a problem reclaiming space on Android... it suggests that this is not the main culprit. Right?

jonathancross commented 1 year ago

@derekmorr What would be very helpful would be to simply submit a PR that demonstrates the issue for devs. It seems you have the skills for this, correct?

From there, we might get a discussion going and build a case that this is important for Signal devs to address.

EvanHahn-Signal commented 1 year ago

We have tried to reproduce this issue but cannot. Thanks to folks here for trying to help us figure it out.

sgreadly commented 1 year ago

@EvanHahn-Signal What can we provide from our end, that can help you troubleshoot or debug what is happening? Any debug version of Signal we can install that gives more info? Or logs?

Mine is currently showing as App size: 110.8 MB Documents & Data: 18.32 GB

I manually deleted ~1,500 videos/images from 2 of my main groups. That 18.32 did not go down at all.

iPhone XS Max, iOS 16 (but seen in previous iOSs too since ~14 at least)

hyiltiz commented 1 year ago

The feature mismatch described here https://github.com/signalapp/Signal-iOS/issues/4916#issuecomment-986330092 requires no steps to "reproduce", as it has been the case for the apps across the platforms since Signal was released.

sgreadly commented 1 year ago

So again I ask the developers, what else can be done, or what do you need form us, to solve this serious issue?

sgreadly commented 1 year ago

@harry-signal @sashaweiss-signal @igor-signal @Imperiopolis-Signal

Really getting desperate here. Can we please get a developer to look into this?

We're losing faith with Signal, and I already have 2 users that switched away from it as a result.

What, if anything, can we provide so that you have the data you need to fix this?

1F4C6574-165F-4500-BEDF-C511D3D20FB7

jonathancross commented 1 year ago

FWIW: I tested this with 2 users and the bug did not show up. We carefully checked overall used space by Signal, deleted some media files and immediately saw the reduction is storage reported by iOS.

iOS v15.7 Signal v5.57.0

And...

iOS v15.5 Signal v5.57.0

So this clearly is not affecting ALL iOS users, which is good.

When I last saw the issue, it was on an old phone which was probably behind in software updates.

wahlfeld commented 1 year ago

FWIW: I tested this with 2 users and the bug did not show up. We carefully checked overall used space by Signal, deleted some media files and immediately saw the reduction is storage reported by iOS.

iOS v15.7 Signal v5.57.0

And...

iOS v15.5 Signal v5.57.0

So this clearly is not affecting ALL iOS users, which is good.

When I last saw the issue, it was on an old phone which was probably behind in software updates.

Happened on my phone and I had to factory reset to delete the data. iPhone 11 Pro iOS 15. 🤷

ablakey commented 1 year ago

How do I delete thousands of images/videos from a chat quickly? I can test it right now on my 12 Pro Max newest iOS.

jonathancross commented 1 year ago

How do I delete thousands of images/videos from a chat quickly? I can test it right now on my 12 Pro Max newest iOS.

Just make it simple and delete some videos / audio / pictures, then check.

ablakey commented 1 year ago

How do I delete thousands of images/videos from a chat quickly? I can test it right now on my 12 Pro Max newest iOS.

Just make it simple and delete some videos / audio / pictures, then check.

I suppose that does provide some useful info. I deleted about fifty pieces of media. Didn’t seem to change the reported number.

Nevertheless, if I can’t bulk clear thousands of images without hours of tapping each item, it’s a moot point if this bug is fixed. 😅

sgreadly commented 1 year ago

@harry-signal @sashaweiss-signal @igor-signal @Imperiopolis-Signal

So just to prove a point here. I just ran another test.

I don't that many media left to delete, but

before1 before2 .

after1

after2 .

iPhone XS Max. iOS 16.1.1 (but this has been happening on many earlier versions over the past 2+ years)

Exact same issue with my wife's iPhone X, same versions, same problem.

Also same observation with Signal on MacOS - the attachments.noindex directory never decreases when I delete media from Signal app.

Screenshot 2022-11-13 at 10 16 26 AM

Screenshot 2022-11-13 at 10 12 55 AM

So.. what is there that we can provide, for you to look into this? I understand for some reason you can't seem to reproduce this. But many of us can. What can we provide? What can we do?

At the very least is there any script you can give us to run for it to auto-scan all this used space, and delete/purge whatever doesn't need to be there ?

dm82m commented 1 year ago

This issue makes signal unusable and a lot of people will run into this issue as their signal database will increase by every media file they receive. Very problematic.

sgreadly commented 1 year ago

Checking for an update..

sgreadly commented 1 year ago

@harry-signal @sashaweiss-signal @igor-signal @Imperiopolis-Signal

EvanHahn-Signal commented 1 year ago

Please submit new debug logs after you update to the latest version. These logs will help us figure out whether data is being properly deleted.

I'm also adding some additional logging to help us diagnose this problem.

sgreadly commented 1 year ago

Thanks @EvanHahn-Signal ,

https://debuglogs.org/desktop/6.0.0/b6e20b93c18185578de968a8956b54caee8d3d882e56adb077eca972004fd21b.gz

v6.0.0. Ventura 13.0.1 (22A400) Let me know if there's anything else you need.

Ta.

EvanHahn-Signal commented 1 year ago

@sgreadly Could you open an issue on the Desktop repository for Desktop issues?

ablakey commented 1 year ago

Hi Evan. Thank you for taking the time to help address this issue. I’m happy to supply whatever user debugging support you need that I’m able to offer.

I got a new iPhone so I’m not at >10GB anymore. But the OS is reporting the same storage usage both before and after I tapped “Clear Chat History” just minutes ago.

I’m mindful of the possibility that iOS takes time to do some sort of vacuum or re-calculation so I’ll keep an eye on that.

5E23EEC6-521A-4E13-9A51-65F726E95F38

https://debuglogs.org/ios/6.2.3/11ea5b00cc91bdda92c9ad047d47333287bb45ec24647787060549b290e2bfcb.zip

sgreadly commented 1 year ago

@EvanHahn-Signal ,

Sure thing I can open a Desktop related issue. Apologies I thought you wanted the desktop debug logs for this as it was the only new version I saw today.

That said, the above issue for me is reproducible on both Desktop, and iOS apps.

Refer to my earlier comment as the latest example, https://github.com/signalapp/Signal-iOS/issues/4916#issuecomment-1312573591

I'm on v6.2.3.0 on iOS. Not sure if this is the latest version you're asking about regarding debug logs, but in case, here it is: https://debuglogs.org/ios/6.2.3/48546167aad48bdbfac126ea1176d7257677d2f0afb2c088886bc6f6a4c68479.zip

sgreadly commented 1 year ago

@EvanHahn-Signal ,

Another set of logs on the latest iOS Signal version as of date (6.6.0.14)

https://debuglogs.org/ios/6.6.0/f9b8cf860f1bf4760894da3dbfb92090f7413fdb4e5b5bc417472c76d722ab7b.zip

SwampyApple commented 1 year ago

Yesterday (2022-12-21) I deleted the only large group chat I'm in as the Signal app was taking over 17GB of storage space and there being a lack of a media-clear option. After deleting the chat and restarting Signal iOS still reported over 17GB of data use for a full day. Just now (2022-12-22) I opened up the iOS storage menu and it still reported 17GB of documents & data use so I looked up this issue. Going back to take a screenshot suddenly had it report 3GB (reasonable given the other chats).

I'm trying to submit the debug log through the iOS Signal app help setting but it enforces you to have the iOS mail app installed and configured. If it's not configured it will crash and delete your debug report. Very user unfriendly. Eventually I found the copy link option.

https://debuglogs.org/ios/6.6.0/b8dd2cd207eb89edf95eaf1aeebdd2a9a883a8d4f75d2a5b8ded27cd922e6ad7.zip

desmond132518 commented 1 year ago

[2023-01-03] It's been 2023 already guys haha! @SwampyApple I did the same thing as you did last night. Now I have 12.43GB of data storage consumed by Signal App, where 12.32GB is "Data and documents"

32be4e7f6cb74dbfc569c00fbb0950f09898e09c405f13b175e0b23ba61f8076

I hope this debug log is helpful to you @EvanHahn-Signal https://debuglogs.org/ios/6.6.0/788fcc5ca92524a712c3948b5b57f7ba6721cd0038bee2f7d2ea4963715a4eac.zip

JanKalkan commented 1 year ago

Same problem here! I am really a big supporter of Signal and heavily promoting it since 2014 + monthly supporter. I deleted all my chats and dis not manage to reduce the size of over ~17GB. Signal is the app that consumes most storage on my phone. Super annoying. Would really appreciate if this can be prioritized after auch a long time.

NiklasBr commented 1 year ago

Try backing up your data, followed by re-installing the app and… Oh right, that's verboten and user's are not allowed to access their own data in this manner. :(

If there ever was a need for yet another argument in the long list of reasons why at minimum we would be allowed a manual export/backup option of our own data, this would be one, maybe hidden deep in the settings or somewhere.

desmond132518 commented 1 year ago

@NiklasBr Agree, at least give us a quick fix to punch down the storage usage, that's really too much :( Largest app on my phone.

sgreadly commented 1 year ago

@EvanHahn-Signal, can we please have an update to this with the debug logs provided?

I finally upgraded my phone to a 14, and did not restore the whole ~19GB as it would take half the night. Did a clean install instead. I just tested now where the Documents & Data was 1.15GB. Deleted ~350 images, and it dropped to ~680MB.

So it finally does drop, but I'm concerned if it grows big enough that we'll end up in the same situation again, and several others also report the same issue, so we really need this to be looked into.

Ta.