search

damus-io / damus

iOS nostr client
GNU General Public License v3.0
2k stars 289 forks source link

Fix Clear cache #1472

Closed alltheseas closed 1 year ago

alltheseas commented 1 year ago

what happens Clear cache does not seem to free up space. Damus storage size in iOS settings remains same after clear cache button is tapped.

what I think should happen Clearing cache should drastically reduce damus size

Origin: Yegor

https://damus.io/note13acj7qkr9p5kjx0nqwznuje07guw3x99l6xfeuvsduw3ycplh99q5xgsal

alltheseas commented 1 year ago

Remains open

https://damus.io/note1lc3gerr9rxg7yxhm7dq6aug7tsnyvw7wxqxq859z5xart2fglwpqr85r94

jb55 commented 1 year ago

On Tue, Sep 19, 2023 at 12:07:59PM -0700, alltheseas wrote:

Remains open

https://damus.io/note1lc3gerr9rxg7yxhm7dq6aug7tsnyvw7wxqxq859z5xart2fglwpqr85r94

I downloaded a damus storage container from my phone and there are many things that are stored that are not yet cleared: apple link previews, etc.

Clear cache only clears kingfisher cache, so I'm guessing this is the issue.

If anyone wants to take a stab at fixing this I would appreciate it!

alltheseas commented 1 year ago

kingfisher cache is not deleting video cache, btw. I notesed it when I played with video deletion and nostr.build

@fishcakeday

I think that makes sense… video is not kingfisher

-Will

alltheseas commented 1 year ago

https://github.com/onevcat/Kingfisher/wiki/Cheat-Sheet#set-default-expiration-for-cache

@fishcakeday

I think there is also a problem with cache-control header not being respected, so it’s not possible to delete media if it’s cached

When I delete an image from the server, and invalidate all CDN caches, Damus still retains a copy. Clearing cache removes it, but otherwise even if I have cache-control: max-age=600, must-revalidate, etc, it will not abide by it. Same goes for video

And clearing cache doesn't clear video cache, so I still see it even if it's gone

Ex.: curl -I https://i.nostr.build/o0Mx.jpg : cache-control: public, max-age=21600, must-revalidate, proxy-revalidate, stale-while-revalidate=3600, s-maxage=2592000

Which ask to cache for 6h and revalidate (using Etag) each time. This is not the case with Damus, as far as I could tell.

jb55 commented 1 year ago

On Thu, Sep 28, 2023 at 10:39:34AM -0700, alltheseas wrote:

https://github.com/onevcat/Kingfisher/wiki/Cheat-Sheet#set-default-expiration-for-cache

didn't know it had this. we should set a disk expiry for videos and images, maybe no expiry for PFPs.

alltheseas commented 1 year ago

On Thu, Sep 28, 2023 at 10:39:34AM -0700, alltheseas wrote:

https://github.com/onevcat/Kingfisher/wiki/Cheat-Sheet#set-default-expiration-for-cache

didn't know it had this. we should set a disk expiry for videos and images, maybe no expiry for PFPs.

Agree

alltheseas commented 1 year ago

Added https://github.com/damus-io/damus/issues/1565

alltheseas commented 1 year ago

0) download snapshot 1) clear cache - 2) download snapshot 3) see in devices section in xCode what remains

danieldaquino commented 1 year ago

Issue repro experiment

Device: iPhone 13 mini iOS: 17.0.2 Damus: 736f51971c08d66393744d477c181e34cf907d25 (On top of my #1301 changes) Special remarks: Locally removed app entitlements to be able to locally sign a build to run on my physical device Steps:

  1. Follow some active users on Damus
  2. Scroll down on the feed for a couple of minutes (or until you have seen at least a few images, a few videos, and link previews)
  3. In Xcode, download a storage container (Window > Devices and Simulators > Select the device > Select Damus > click on (...) > Download container)
  4. Open the app data package using terminal
  5. Run du -h . | sort -hr
  6. Clear cache
  7. Download a new storage container
  8. Run du -h . | sort -hr on it
  9. Compare. There should be very little data left after cache reset. But on my end there is still tons of data

Results data

Before cache clear:

495M    ./AppData
495M    .
302M    ./AppData/Library
297M    ./AppData/Library/Caches
294M    ./AppData/Library/Caches/com.onevcat.Kingfisher.ImageCache.default
193M    ./AppData/Documents
3.1M    ./AppData/Library/SplashBoard/Snapshots
3.1M    ./AppData/Library/SplashBoard
2.9M    ./AppData/Library/Caches/com.daniel.damus
1.9M    ./AppData/Library/HTTPStorages/com.daniel.damus
1.9M    ./AppData/Library/HTTPStorages
1.8M    ./AppData/Library/SplashBoard/Snapshots/com.daniel.damus - {DEFAULT GROUP}
1.3M    ./AppData/Library/SplashBoard/Snapshots/sceneID:com.daniel.damus-4c7f7ed7-b31d-4b98-8bf8-e60ac9cb80ed
504K    ./AppData/Library/SplashBoard/Snapshots/sceneID:com.daniel.damus-4c7f7ed7-b31d-4b98-8bf8-e60ac9cb80ed/downscaled
324K    ./AppData/Library/Caches/com.daniel.damus/com.apple.metal
316K    ./AppData/Library/Caches/com.apple.WebKit.GPU/com.apple.metal
316K    ./AppData/Library/Caches/com.apple.WebKit.GPU
152K    ./AppData/Library/Caches/com.apple.dyld
 60K    ./AppData/Library/Caches/com.daniel.damus/com.apple.metal/archiveUsage.db
 12K    ./AppData/Library/Saved Application State/com.daniel.damus.savedState
 12K    ./AppData/Library/Saved Application State
8.0K    ./AppData/Library/Saved Application State/com.daniel.damus.savedState/4c7f7ed7-b31d-4b98-8bf8-e60ac9cb80ed
8.0K    ./AppData/Library/Preferences
4.0K    ./AppData/Library/Saved Application State/com.daniel.damus.savedState/KnownSceneSessions
4.0K    ./AppData/Library/Cookies
  0B    ./AppData/Library/Caches/RelayLogs

After cache clear:

201M    ./AppData
201M    .
193M    ./AppData/Documents
8.4M    ./AppData/Library
3.4M    ./AppData/Library/Caches
3.1M    ./AppData/Library/SplashBoard/Snapshots
3.1M    ./AppData/Library/SplashBoard
2.9M    ./AppData/Library/Caches/com.daniel.damus
1.9M    ./AppData/Library/HTTPStorages/com.daniel.damus
1.9M    ./AppData/Library/HTTPStorages
1.8M    ./AppData/Library/SplashBoard/Snapshots/com.daniel.damus - {DEFAULT GROUP}
1.3M    ./AppData/Library/SplashBoard/Snapshots/sceneID:com.daniel.damus-4c7f7ed7-b31d-4b98-8bf8-e60ac9cb80ed
504K    ./AppData/Library/SplashBoard/Snapshots/sceneID:com.daniel.damus-4c7f7ed7-b31d-4b98-8bf8-e60ac9cb80ed/downscaled
324K    ./AppData/Library/Caches/com.daniel.damus/com.apple.metal
316K    ./AppData/Library/Caches/com.apple.WebKit.GPU/com.apple.metal
316K    ./AppData/Library/Caches/com.apple.WebKit.GPU
152K    ./AppData/Library/Caches/com.apple.dyld
 60K    ./AppData/Library/Caches/com.daniel.damus/com.apple.metal/archiveUsage.db
 12K    ./AppData/Library/Saved Application State/com.daniel.damus.savedState
 12K    ./AppData/Library/Saved Application State
8.0K    ./AppData/Library/Saved Application State/com.daniel.damus.savedState/4c7f7ed7-b31d-4b98-8bf8-e60ac9cb80ed
8.0K    ./AppData/Library/Preferences
4.0K    ./AppData/Library/Saved Application State/com.daniel.damus.savedState/KnownSceneSessions
4.0K    ./AppData/Library/Cookies

It seems that kingfisher cache is completely removed, and the bulk of what remains is in Documents:

Documents % du -ah
8.0K    ./lock.mdb
193M    ./data.mdb
193M    .

data.mdb seems to be where NostrDB stores notes, which is further confirmed from this code:

static var db_path: String {
        let path = FileManager.default.urls(for: .documentDirectory, in: .userDomainMask).first?.absoluteString
        return remove_file_prefix(path!)
    }

Notes

@jb55, before I continue, I would like to ask:

danieldaquino commented 1 year ago

Are there any "easy" ways to reduce NostrDB size?

I will rephrase this for clarity:

alltheseas commented 1 year ago

What format is nostrDB profile @jb55 ? A lot of JSON?

danieldaquino commented 1 year ago

@alltheseas, I believe it would be in a binary format — Flatbuffer objects (which is sort of like a binary "JSON") stored in a modified LMDB, if I am not mistaken. @jb55 definitely knows more about it and probably has a more definite answer 😅

jb55 commented 1 year ago

On Mon, Oct 02, 2023 at 07:43:32PM -0700, Daniel D’Aquino wrote:

Are there any "easy" ways to reduce NostrDB size?

I will rephrase this for clarity:

  • Are there any existing and "approved" ways to shrink NostrDB? Don't worry if not, I can try to create one.

not yet, but it wouldn't be too difficult. The most space usage would be contact lists. We could add a method that purges old versions of those. We could also add a method to purge notes from people you don't follow (notes that were received from global, likes, etc)

The nuclear option would be to just purge the entire DB.

One thing I will look into is adding some methods for getting the sizes of different databases so we can start measuring the sizes of the data, indices, etc.

jb55 commented 1 year ago

On Mon, Oct 02, 2023 at 08:07:56PM -0700, Daniel D’Aquino wrote:

@alltheseas, I believe it would be in a binary format — Flatbuffer objects (which is sort of like a binary "JSON") stored in a modified LMDB, if I am not mistaken. @jb55 definitely knows more about it and probably has a more definite answer 😅

yes notes use a custom compact binary format that doesn't require serialization in/out. It's not flatbuffers, its more efficient than flatbuffers. Everything else uses flatbuffers because I didn't want to create a custom format for every data type.

I haven't documented it yet but I probably should.

danieldaquino commented 1 year ago

yes notes use a custom compact binary format that doesn't require serialization in/out. It's not flatbuffers, its more efficient than flatbuffers. Everything else uses flatbuffers because I didn't want to create a custom format for every data type. I haven't documented it yet but I probably should.

Interesting! I did not know about that detail yet. I will keep that in mind.

danieldaquino commented 1 year ago

@jb55, I believe that at some point you mentioned that we might solve most of this issue by just clearing the entire cache folder (beyond just clearing kingfisher).

When you mentioned this, was it a root cause + solution that you were already confident of, or was it more of a hypothesis?

Currently I am working on a patch that:

This based on my own (short) test. But if you happen to know that NostrDB won't constitute a big chunk of storage on the long-term through your experience, and is not worth purging, please let me know (the scope of the ticket could be reduced in that case).

danieldaquino commented 1 year ago

Wrote the changes I mentioned, and did some quick tests. Now I am doing final testing on this before I send it over (assuming everything goes ok).

Testing note: One thing I did notice is that containers are not downloaded instantaneously. Xcode make it seem so by placing the file where you selected, but I noticed that the file keeps growing and growing even after the Xcode UI makes you believe that it is done.

jb55 commented 1 year ago

On Wed, Oct 04, 2023 at 09:43:32AM -0700, Daniel D’Aquino wrote:

@jb55, I believe that at some point you mentioned that we might solve most of this issue by just clearing the entire cache folder (beyond just clearing kingfisher).

When you mentioned this, was it a root cause + solution that you were already confident of, or was it more of a hypothesis?

Fishcake was saying clearing cache wasn't clearing 50GB of his data, so I assumed it was some other cache that we was missing. I was only speculating, because when I cleared 10GB of cache it brought my storage down to just nostrdb stuff (couple hundred MB)

Currently I am working on a patch that: - deletes kingfisher (existing functionality) - deletes the entire cache folder - purges things on NostrDB

This based on my own (short) test. But if you happen to know that NostrDB won't constitute a big chunk of storage on the long-term through your experience, and is not worth purging, please let me know (the scope of the ticket could be reduced in that case).

nostrdb only stores text efficiently, storage might become a concern after some time but it shouldn't be a concern short term. It's not really in the scope of this PR.

danieldaquino commented 1 year ago

Finished testing and preparing the changes. I sent it via email (Can be visualized at: https://groups.google.com/a/damus.io/g/patches/c/Qv8Y91nC4Cs)

During my test I was able to get the app data from 1.68 GB down to 4.4 MB

@jb55, due to timing of the messages my patch ended up including code to clear NostrDB as well. I can optionally remove it, but it seems to work well, so we might also want to just leave it in. Please let me know!

danieldaquino commented 1 year ago

Sent v3 of the patch here: https://groups.google.com/a/damus.io/g/patches/c/-JS5BKGc6cA