Open wallentx opened 2 years ago
I've since had to switch to btrfs, but I'd love to switch back to f2fs, since these drives quite buggy with btrfs. I've also had a coworker verify that on his system, with these particular drives (ST14000NM0007-2G), mounting them when they are formatted as f2fs does consume ~1GB of RAM per drive.
With none mounted:
With some of them mounted:
Hi,
May I ask to capture this?
Thanks,
On Wed, Sep 14, 2022 at 8:28 AM William Allen @.***> wrote:
I've since had to switch to btrfs, but I'd love to switch back to f2fs, since these drives quite buggy with btrfs. I've also had a coworker verify that on his system, with these particular drives (ST14000NM0007-2G), mounting them when they are formatted as f2fs does consume ~1GB of RAM per drive.
With none mounted: [image: 1714a881e301fc4f] https://user-images.githubusercontent.com/8990544/190085945-ad347e3d-6e8a-44c0-950e-1329849fa904.png
With some of them mounted: [image: 1714a88523bb4308] https://user-images.githubusercontent.com/8990544/190086906-8699b20c-c466-4dd1-8d9b-0cb0043ef3c2.png
— Reply to this email directly, view it on GitHub https://github.com/jaegeuk/f2fs-stable/issues/9#issuecomment-1246357485, or unsubscribe https://github.com/notifications/unsubscribe-auth/AE4EPAUTEWFJ4OPPA7LPL2DV6F5BPANCNFSM54SRBD3Q . You are receiving this because you are subscribed to this thread.Message ID: @.***>
Sorry for the late reply. I had to wait for a drive to become available.
=====[ partition info(sdl). #0, RW, CP: Good]=====
[SB: 1] [CP: 2] [SIT: 476] [NAT: 120] [SSA: 13098] [MAIN: 6662144(OverProv:83685 Resv:42644)]
Current Time Sec: 97552 / Mounted Time Sec: 97454
Utilization: 0% (2 valid blocks, 0 discard blocks)
- Node: 1 (Inode: 1, Other: 0)
- Data: 1
- Inline_xattr Inode: 0
- Inline_data Inode: 0
- Inline_dentry Inode: 0
- Compressed Inode: 0, Blocks: 0
- Orphan/Append/Update Inode: 0, 0, 0
Main area: 6662144 segs, 52048 secs 52048 zones
TYPE segno secno zoneno dirty_seg full_seg valid_blk
- COLD data: 640 5 5 0 0 0
- WARM data: 512 4 4 0 0 0
- HOT data: 384 3 3 1 0 1
- Dir dnode: 0 0 0 1 0 1
- File dnode: 128 1 1 0 0 0
- Indir nodes: 256 2 2 0 0 0
- Pinned file: -1 -1 -1
- ATGC data: -1 -1 -1
- Valid: 6
- Dirty: 0
- Prefree: 0
- Free: 6662138 (52042)
CP calls: 0 (BG: 0)
- cp blocks : 0
- sit blocks : 0
- nat blocks : 0
- ssa blocks : 0
CP merge (Queued: 0, Issued: 0, Total: 0, Cur time: 0(ms), Peak time: 0(ms))
GC calls: 0 (BG: 1)
- data segments : 0 (0)
- node segments : 0 (0)
- Reclaimed segs : Normal (0), Idle CB (0), Idle Greedy (0), Idle AT (0), Urgent High (0), Urgent Mid (0), Urgent Low (0)
Try to move 0 blocks (BG: 0)
- data blocks : 0 (0)
- node blocks : 0 (0)
BG skip : IO: 0, Other: 0
Extent Cache:
- Hit Count: L1-1:0 L1-2:0 L2:0
- Hit Ratio: 0% (0 / 0)
- Inner Struct Count: tree: 0(0), node: 0
Balancing F2FS Async:
- DIO (R: 0, W: 0)
- IO_R (Data: 0, Node: 0, Meta: 0
- IO_W (CP: 0, Data: 0, Flush: ( 0 0 1), Discard: ( 0 0)) cmd: 0 undiscard: 0
- atomic IO: 0 (Max. 0)
- compress: 0, hit: 0
- nodes: 0 in 1
- dents: 0 in dirs: 0 ( 0)
- datas: 0 in files: 0
- quota datas: 0 in quota files: 0
- meta: 0 in 121152
- imeta: 0
- fsync mark: 0
- NATs: 0/ 0
- SITs: 0/ 6662144
- free_nids: 3640/ 13977596
- alloc_nids: 0
Distribution of User Blocks: [ valid | invalid | free ]
[|-|-------------------------------------------------]
IPU: 0 blocks
SSR: 0 blocks in 0 segments
LFS: 0 blocks in 0 segments
BDF: 99, avg. vblocks: 0
Memory: 2056548 KB
- static: 1571843 KB
- cached: 93 KB
- paged : 484612 KB
I just had a btrfs formatted drive with some data corruption, so I formatted it as f2fs and here is the output of /sys/kernel/debug/f2fs/status after mounting it:
=====[ partition info(sdbn). #0, RW, CP: Good]===== [SB: 1] [CP: 2] [SIT: 476] [NAT: 120] [SSA: 13098] [MAIN: 6662144(OverProv:83685 Resv:42644)]
Current Time Sec: 5908 / Mounted Time Sec: 105
Utilization: 18% (638268078 valid blocks, 0 discard blocks)
- Node: 627031 (Inode: 26, Other: 627005)
- Data: 637641047
- Inline_xattr Inode: 0
- Inline_data Inode: 0
- Inline_dentry Inode: 0
- Compressed Inode: 0, Blocks: 0
- Orphan/Append/Update Inode: 0, 0, 0
Main area: 6662144 segs, 52048 secs 52048 zones TYPE segno secno zoneno dirty_seg full_seg valid_blk
COLD data: 767 5 5 1 127 65533
WARM data: 1250958 9773 9773 1 1193486 611065120
HOT data: 779714 6091 6091 8 51771 26510394
Dir dnode: 0 0 0 1 0 2
File dnode: 1252423 9784 9784 2 1222 626381
Indir nodes: 258 2 2 3 0 648
Pinned file: -1 -1 -1
ATGC data: -1 -1 -1
Valid: 1246612
Dirty: 10
Prefree: 0
Free: 5415522 (42305)
CP calls: 2 (BG: 24)
- cp blocks : 16
- sit blocks : 14
- nat blocks : 298
- ssa blocks : 256 CP merge (Queued: 0, Issued: 24, Total: 24, Cur time: 0(ms), Peak time: 1741(ms)) GC calls: 4 (BG: 23)
- data segments : 0 (0)
- node segments : 256 (256)
- Reclaimed segs : Normal (256), Idle CB (0), Idle Greedy (0), Idle AT (0), Urgent High (0), Urgent Mid (0), Urgent Low (0) Try to move 131059 blocks (BG: 131059)
- data blocks : 0 (0)
- node blocks : 131059 (131059) BG skip : IO: 0, Other: 0
Extent Cache:
- Hit Count: L1-1:0 L1-2:0 L2:0
- Hit Ratio: 0% (0 / 0)
- Inner Struct Count: tree: 0(0), node: 0
Balancing F2FS Async:
- DIO (R: 0, W: 0)
- IO_R (Data: 0, Node: 0, Meta: 0
- IO_W (CP: 0, Data: 0, Flush: ( 0 0 1), Discard: ( 0 0)) cmd: 0 undiscard: 0
- atomic IO: 0 (Max. 0)
- compress: 0, hit: 0
- nodes: 0 in 131059
- dents: 0 in dirs: 0 ( 0)
- datas: 0 in files: 0
- quota datas: 0 in quota files: 0
- meta: 0 in 121626
- imeta: 0
- fsync mark: 0
- NATs: 0/ 32539
- SITs: 0/ 6662144
- free_nids: 3640/ 13350566
- alloc_nids: 0
Distribution of User Blocks: [ valid | invalid | free ] [---------|-|----------------------------------------]
IPU: 0 blocks SSR: 0 blocks in 0 segments LFS: 131059 blocks in 256 segments
BDF: 99, avg. vblocks: 29940
Memory: 2583693 KB
- static: 1571843 KB
- cached: 1110 KB
- paged : 1010740 KB
=====[ partition info(sdck). #1, RW, CP: Good]===== [SB: 1] [CP: 2] [SIT: 476] [NAT: 120] [SSA: 13098] [MAIN: 6662144(OverProv:83685 Resv:42644)]
Current Time Sec: 5908 / Mounted Time Sec: 5870
Utilization: 0% (2 valid blocks, 0 discard blocks)
- Node: 1 (Inode: 1, Other: 0)
- Data: 1
- Inline_xattr Inode: 0
- Inline_data Inode: 0
- Inline_dentry Inode: 0
- Compressed Inode: 0, Blocks: 0
- Orphan/Append/Update Inode: 0, 0, 0
Main area: 6662144 segs, 52048 secs 52048 zones TYPE segno secno zoneno dirty_seg full_seg valid_blk
COLD data: 640 5 5 0 0 0
WARM data: 512 4 4 0 0 0
HOT data: 384 3 3 1 0 1
Dir dnode: 0 0 0 1 0 1
File dnode: 128 1 1 0 0 0
Indir nodes: 256 2 2 0 0 0
Pinned file: -1 -1 -1
ATGC data: -1 -1 -1
Valid: 6
Dirty: 0
Prefree: 0
Free: 6662138 (52042)
CP calls: 0 (BG: 0)
- cp blocks : 0
- sit blocks : 0
- nat blocks : 0
- ssa blocks : 0 CP merge (Queued: 0, Issued: 0, Total: 0, Cur time: 0(ms), Peak time: 0(ms)) GC calls: 0 (BG: 0)
- data segments : 0 (0)
- node segments : 0 (0)
- Reclaimed segs : Normal (0), Idle CB (0), Idle Greedy (0), Idle AT (0), Urgent High (0), Urgent Mid (0), Urgent Low (0) Try to move 0 blocks (BG: 0)
- data blocks : 0 (0)
- node blocks : 0 (0) BG skip : IO: 0, Other: 0
Extent Cache:
- Hit Count: L1-1:0 L1-2:0 L2:0
- Hit Ratio: 0% (0 / 0)
- Inner Struct Count: tree: 0(0), node: 0
Balancing F2FS Async:
- DIO (R: 0, W: 0)
- IO_R (Data: 0, Node: 0, Meta: 0
- IO_W (CP: 0, Data: 0, Flush: ( 0 0 1), Discard: ( 0 0)) cmd: 0 undiscard: 0
- atomic IO: 0 (Max. 0)
- compress: 0, hit: 0
- nodes: 0 in 1
- dents: 0 in dirs: 0 ( 0)
- datas: 0 in files: 0
- quota datas: 0 in quota files: 0
- meta: 0 in 121152
- imeta: 0
- fsync mark: 0
- NATs: 0/ 1
- SITs: 0/ 6662144
- free_nids: 451/ 13977596
- alloc_nids: 0
Distribution of User Blocks: [ valid | invalid | free ] [|-|-------------------------------------------------]
IPU: 0 blocks SSR: 0 blocks in 0 segments LFS: 0 blocks in 0 segments
BDF: 99, avg. vblocks: 0
Memory: 2056474 KB
- static: 1571843 KB
- cached: 18 KB
- paged : 484612 KB
@jaegeuk I've been trying to troubleshoot this further, and can only determine that the increased memory usage is being utilized by cache:
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
r b swpd free buff cache si so bi bo in cs us sy id wa st
3 0 0 3287 531 8520 0 0 367 234 0 1 1 0 98 1 0
total used free shared buff/cache available
Mem: 31Gi 19Gi 3.2Gi 438Mi 8.8Gi 11Gi
Swap: 0B 0B 0B
sudo mount /dev/sdcq
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
r b swpd free buff cache si so bi bo in cs us sy id wa st
4 1 0 1439 531 8846 0 0 367 234 0 1 1 0 98 1 0
total used free shared buff/cache available
Mem: 31Gi 20Gi 1.4Gi 438Mi 9.2Gi 9.7Gi
Swap: 0B 0B 0B
These aren't the default mount options, but this is one of many things I've been trying, in order to see if I can narrow down a particular culprit:
grep sdcq /proc/mounts
/dev/sdcq /mnt/jbd1/plots93
f2fs
rw
lazytime
noatime
background_gc=off
discard
no_heap
user_xattr
inline_xattr
acl
inline_data
inline_dentry
flush_merge
extent_cache
mode=lfs
active_logs=2
alloc_mode=default
checkpoint_merge
fsync_mode=posix
discard_unit=section
memory=low
0 0
With the above, only these additional processes are spawned when I mount the HM-SMR volume:
f2fs_ckpt
f2fs_flush
f2fs_discard
I also have a slabtop diff from before and after mounting: https://gist.github.com/wallentx/a5e98371f8cc2c42084b86bbd99253fc#file-slabtop-diff
I experience somewhat the same with a much smaller partition sizes, on a relatively small NAS with 512MB RAM. In my case memory=low didn't really helped, the 900GiB partition was needed 100MB of memory anyways. Is there a way to go down the line and understand if we can still decrease the memory requirement?
sudo tail -4 /sys/kernel/debug/f2fs/status
Memory: 99637 KB
- static: 98421 KB
- cached: 79 KB
- paged : 1136 KB
side story: https://archlinuxarm.org/forum/viewtopic.php?f=57&t=16534&p=71416#p71416
I've spent a few days trying to figure out where my memory consumption was going to, and I finally found that for each f2fs formatted zoned disk I mount, 1GB of RAM becomes "used", though I can't find any evidence as to what exactly this is attributed to. I've even mounted the drives as r/o to make sure it wasn't due to some write cache. Is this to be expected? I'm running 5.18.12-arch1-1, and have 42 zoned disks mounted that are formatted with f2fs.