keybase / kbfs

Keybase Filesystem (KBFS)
https://keybase.io/docs/kbfs
BSD 3-Clause "New" or "Revised" License
1.21k stars 135 forks source link

[Mac OS] Old block DBs not being cleaned up #1558

Open kklash opened 6 years ago

kklash commented 6 years ago
diskCacheBlocks.leveldb admin$ date
Fri 27 Apr 2018 18:36:24
diskCacheBlocks.leveldb admin$ pwd
/Users/admin/Library/Application Support/Keybase/kbfs_block_cache/v1/diskCacheBlocks.leveldb
diskCacheBlocks.leveldb admin$ ls -l
total 10540072
drwxr-xr-x  117 admin  staff   3.9K 27 Apr 15:36 .
drwxr-xr-x    5 admin  staff   170B 30 Sep  2017 ..
-rw-r--r--    1 admin  staff    50M 20 Feb 23:11 009400.ldb
-rw-r--r--    1 admin  staff    50M 20 Feb 23:11 009401.ldb
-rw-r--r--    1 admin  staff    50M 20 Feb 23:20 009459.ldb
-rw-r--r--    1 admin  staff    50M 20 Feb 23:20 009460.ldb
-rw-r--r--    1 admin  staff    50M 20 Feb 23:39 009608.ldb
-rw-r--r--    1 admin  staff    50M 20 Feb 23:39 009609.ldb
-rw-r--r--    1 admin  staff    50M 20 Feb 23:39 009610.ldb
-rw-r--r--    1 admin  staff    50M 20 Feb 23:39 009611.ldb
-rw-r--r--    1 admin  staff    50M 20 Feb 23:47 009668.ldb
-rw-r--r--    1 admin  staff    50M 20 Feb 23:47 009669.ldb
-rw-r--r--    1 admin  staff    50M 20 Feb 23:53 009726.ldb
-rw-r--r--    1 admin  staff    50M 20 Feb 23:55 009751.ldb
-rw-r--r--    1 admin  staff    50M 20 Feb 23:55 009752.ldb
-rw-r--r--    1 admin  staff    50M 21 Feb 00:00 009804.ldb
-rw-r--r--    1 admin  staff    50M 21 Feb 00:15 009981.ldb
-rw-r--r--    1 admin  staff    50M 21 Feb 00:15 009982.ldb
-rw-r--r--    1 admin  staff    50M 21 Feb 00:15 009983.ldb
-rw-r--r--    1 admin  staff    50M 21 Feb 00:15 009984.ldb
-rw-r--r--    1 admin  staff    50M 21 Feb 00:15 009985.ldb
-rw-r--r--    1 admin  staff    50M 21 Feb 00:20 010043.ldb
-rw-r--r--    1 admin  staff    50M 21 Feb 00:20 010044.ldb
-rw-r--r--    1 admin  staff    50M 21 Feb 00:25 010098.ldb
-rw-r--r--    1 admin  staff    50M 21 Feb 00:25 010099.ldb
-rw-r--r--    1 admin  staff    50M 21 Feb 00:25 010100.ldb
-rw-r--r--    1 admin  staff    17M 21 Feb 00:25 010101.ldb
-rw-r--r--    1 admin  staff    50M 24 Feb 10:27 010303.ldb
-rw-r--r--    1 admin  staff    50M 24 Feb 10:40 010451.ldb
-rw-r--r--    1 admin  staff    50M 24 Feb 10:40 010452.ldb
-rw-r--r--    1 admin  staff    50M 24 Feb 10:40 010453.ldb
-rw-r--r--    1 admin  staff    50M 24 Feb 10:40 010454.ldb
-rw-r--r--    1 admin  staff    50M 24 Feb 10:40 010455.ldb
-rw-r--r--    1 admin  staff    50M 24 Feb 10:40 010456.ldb
-rw-r--r--    1 admin  staff    50M 24 Feb 10:40 010457.ldb
-rw-r--r--    1 admin  staff    50M 24 Feb 10:40 010458.ldb
-rw-r--r--    1 admin  staff    50M 24 Feb 10:51 010510.ldb
-rw-r--r--    1 admin  staff    50M 24 Feb 11:16 010752.ldb
-rw-r--r--    1 admin  staff    50M 24 Feb 11:16 010753.ldb
-rw-r--r--    1 admin  staff    50M 24 Feb 11:16 010754.ldb
-rw-r--r--    1 admin  staff    50M 24 Feb 11:16 010755.ldb
-rw-r--r--    1 admin  staff    50M 24 Feb 11:16 010756.ldb
-rw-r--r--    1 admin  staff    50M 24 Feb 11:16 010757.ldb
-rw-r--r--    1 admin  staff    50M 24 Feb 11:16 010758.ldb
-rw-r--r--    1 admin  staff    50M 24 Feb 11:16 010759.ldb
-rw-r--r--    1 admin  staff    50M 24 Feb 11:16 010760.ldb
-rw-r--r--    1 admin  staff    50M 24 Feb 11:16 010761.ldb
-rw-r--r--    1 admin  staff    50M 24 Feb 11:35 010991.ldb
-rw-r--r--    1 admin  staff    50M 24 Feb 11:35 010992.ldb
-rw-r--r--    1 admin  staff    50M 24 Feb 11:35 010993.ldb
-rw-r--r--    1 admin  staff    50M 24 Feb 11:35 010994.ldb
-rw-r--r--    1 admin  staff    50M 24 Feb 11:35 010995.ldb
-rw-r--r--    1 admin  staff    50M 24 Feb 11:35 010996.ldb
-rw-r--r--    1 admin  staff    50M 24 Feb 11:35 010997.ldb
-rw-r--r--    1 admin  staff    50M 24 Feb 11:35 010998.ldb
-rw-r--r--    1 admin  staff    50M 24 Feb 11:35 010999.ldb
-rw-r--r--    1 admin  staff    50M 24 Feb 11:35 011000.ldb
-rw-r--r--    1 admin  staff    50M 24 Feb 11:44 011104.ldb
-rw-r--r--    1 admin  staff    50M 24 Feb 11:44 011105.ldb
-rw-r--r--    1 admin  staff    50M 24 Feb 11:44 011106.ldb
-rw-r--r--    1 admin  staff    50M 24 Feb 11:44 011107.ldb
-rw-r--r--    1 admin  staff    50M 24 Feb 11:47 011153.ldb
-rw-r--r--    1 admin  staff    50M 24 Feb 11:47 011154.ldb
-rw-r--r--    1 admin  staff    50M 24 Feb 11:47 011155.ldb
-rw-r--r--    1 admin  staff    50M 24 Feb 11:51 011213.ldb
-rw-r--r--    1 admin  staff    50M 24 Feb 11:51 011214.ldb
-rw-r--r--    1 admin  staff    50M 24 Feb 11:52 011251.ldb
-rw-r--r--    1 admin  staff    50M 24 Feb 11:55 011299.ldb
-rw-r--r--    1 admin  staff    50M 24 Feb 12:00 011421.ldb
-rw-r--r--    1 admin  staff    50M 24 Feb 12:00 011422.ldb
-rw-r--r--    1 admin  staff    50M 24 Feb 12:00 011423.ldb
-rw-r--r--    1 admin  staff    50M 24 Feb 12:00 011424.ldb
-rw-r--r--    1 admin  staff    50M 24 Feb 12:00 011425.ldb
-rw-r--r--    1 admin  staff    50M 24 Feb 12:00 011426.ldb
-rw-r--r--    1 admin  staff    50M 24 Feb 12:00 011427.ldb
-rw-r--r--    1 admin  staff    50M 24 Feb 12:04 011473.ldb
-rw-r--r--    1 admin  staff    50M 24 Feb 12:04 011474.ldb
-rw-r--r--    1 admin  staff    50M 24 Feb 12:04 011475.ldb
-rw-r--r--    1 admin  staff    50M 24 Feb 12:08 011533.ldb
-rw-r--r--    1 admin  staff    50M 24 Feb 12:08 011534.ldb
-rw-r--r--    1 admin  staff    50M 24 Feb 12:08 011535.ldb
-rw-r--r--    1 admin  staff    44M 24 Feb 12:08 011536.ldb
-rw-r--r--    1 admin  staff    50M 11 Mar 14:21 011632.ldb
-rw-r--r--    1 admin  staff    50M 11 Mar 14:21 011633.ldb
-rw-r--r--    1 admin  staff    50M 11 Mar 14:21 011634.ldb
-rw-r--r--    1 admin  staff    50M 26 Apr 10:29 011852.ldb
-rw-r--r--    1 admin  staff    50M 26 Apr 10:29 011853.ldb
-rw-r--r--    1 admin  staff    50M 26 Apr 10:29 011854.ldb
-rw-r--r--    1 admin  staff    50M 26 Apr 10:29 011855.ldb
-rw-r--r--    1 admin  staff    50M 26 Apr 10:29 011856.ldb
-rw-r--r--    1 admin  staff    10M 26 Apr 10:29 011857.ldb
-rw-r--r--    1 admin  staff    50M 26 Apr 10:29 011858.ldb
-rw-r--r--    1 admin  staff    50M 26 Apr 10:29 011859.ldb
-rw-r--r--    1 admin  staff    50M 26 Apr 10:29 011860.ldb
-rw-r--r--    1 admin  staff    50M 26 Apr 10:29 011861.ldb
-rw-r--r--    1 admin  staff    12M 26 Apr 10:29 011862.ldb
-rw-r--r--    1 admin  staff   7.5M 26 Apr 10:29 011863.ldb
-rw-r--r--    1 admin  staff    50M 26 Apr 10:30 011864.ldb
-rw-r--r--    1 admin  staff    50M 26 Apr 10:30 011865.ldb
-rw-r--r--    1 admin  staff    35M 26 Apr 10:30 011866.ldb
-rw-r--r--    1 admin  staff    50M 26 Apr 10:30 011867.ldb
-rw-r--r--    1 admin  staff    45M 26 Apr 10:30 011868.ldb
-rw-r--r--    1 admin  staff    50M 26 Apr 10:30 011869.ldb
-rw-r--r--    1 admin  staff    19M 26 Apr 10:30 011870.ldb
-rw-r--r--    1 admin  staff    47M 26 Apr 10:30 011871.ldb
-rw-r--r--    1 admin  staff    11M 26 Apr 10:30 011872.ldb
-rw-r--r--    1 admin  staff    50M 26 Apr 10:30 011873.ldb
-rw-r--r--    1 admin  staff    50M 26 Apr 10:30 011874.ldb
-rw-r--r--    1 admin  staff    43M 26 Apr 10:30 011875.ldb
-rw-r--r--    1 admin  staff     0B 27 Apr 09:30 011887.log
-rw-r--r--    1 admin  staff    32M 27 Apr 15:36 011889.ldb
-rw-r--r--    1 admin  staff    46K 27 Apr 15:36 011890.ldb
-rw-r--r--    1 admin  staff    16B 27 Apr 09:30 CURRENT
-rw-r--r--    1 admin  staff     0B 30 Sep  2017 LOCK
-rw-r--r--    1 admin  staff   368K 27 Apr 15:37 LOG
-rw-r--r--    1 admin  staff   1.1M 21 Feb 00:28 LOG.old
-rw-r--r--    1 admin  staff    10K 27 Apr 15:36 MANIFEST-011888

Old .ldb's not being cleaned up. I made a backup, removed the old ones, relaunched keybase and everything still running normally.

As a sidenote, why are these DBs so big when I only have 17mb of files in my KBFS? (currently, had more before)

If there's any extra debugging information I can provide, just let me know!

strib commented 6 years ago

@kklash: as explained here under the "Local disk usage policy" section, we keep up to 20 GB of blocks, depending on the free space of your drive. If you use more of the drive, we will shrink that cache automatically if KBFS is running.

We cache all blocks you've even seen, including from team and public folders, and past blocks from old versions of the files in some cases.

adi-dev commented 5 years ago

I think we should re-consider the block storage. I have Kebyase on my laptop, I uploaded some files and deleted them. Now I have only like 400MB file but kbfs_block_cache eats almost 6GB on my hard drive. I'll try @strib solution to clean is up manually but sure it should remove blocks from the files what don't exists anymore. Btw. it's Windows 10 machine.

strib commented 5 years ago

@adi-dev recently we did add something that cleans up the blocks from old files that have been deleted, and old versions of files. But I think it only works going forward I think, not for old blocks from before that version was released. So it might be the case that a single manual cleanup will fix things for you.