Closed DeHackEd closed 11 years ago
behlendorf
commented
11 years ago @DeHackEd When you say it crashed do you mean the system paniced or became unstable in some way, or that we just exceeded that meta limit? I believe I see why that's happening now regarding the meta limit but I'd like to better understand the instability your referring too.
behlendorf
commented
11 years ago @DeHackEd Can you please try 6c5207088f732168569d1a0b29f5f949b91bb503 which I believe should address your issue. Basically it was possible under a very heavy meta data workload to move all the meta data just from the MRU/MFU on to the MRU/MFU ghost lists. It would then never be free'd from there if there was data on the ghost list which could be reclaimed instead. I'd be interested to hear if this change improves your performance as well.
DeHackEd
commented
11 years ago While it does seem to help, it hasn't completely eliminated the issue. At best it just delays the inevitable.
arp096's method of using drop_caches does seem to keep it stable though. Not a great workaround but still...
aarrpp
commented
11 years ago I tried the patch, but nothing has changed
cat /proc/spl/kstat/zfs/arcstats 4 1 0x01 77 3696 2403385908311841 2403855656867260 name type data hits 4 38524104 misses 4 368775 demand_data_hits 4 0 demand_data_misses 4 22 demand_metadata_hits 4 37250368 demand_metadata_misses 4 159192 prefetch_data_hits 4 0 prefetch_data_misses 4 0 prefetch_metadata_hits 4 1273736 prefetch_metadata_misses 4 209561 mru_hits 4 1358445 mru_ghost_hits 4 47602 mfu_hits 4 36023113 mfu_ghost_hits 4 19495 deleted 4 259553 recycle_miss 4 602959 mutex_miss 4 62668 evict_skip 4 564914143 evict_l2_cached 4 0 evict_l2_eligible 4 19083264 evict_l2_ineligible 4 4364881408 hash_elements 4 42650 hash_elements_max 4 42650 hash_collisions 4 15009 hash_chains 4 1629 hash_chain_max 4 4 p 4 144727296 c 4 1000000000 c_min 4 838860800 c_max 4 1000000000 size 4 4533766376 hdr_size 4 25180480 data_size 4 1830418432 other_size 4 2678167464 anon_size 4 8264704 anon_evict_data 4 0 anon_evict_metadata 4 0 mru_size 4 966695424 mru_evict_data 4 0 mru_evict_metadata 4 21205504 mru_ghost_size 4 57323008 mru_ghost_evict_data 4 118272 mru_ghost_evict_metadata 4 57204736 mfu_size 4 855458304 mfu_evict_data 4 0 mfu_evict_metadata 4 9123328 mfu_ghost_size 4 1480704 mfu_ghost_evict_data 4 304640 mfu_ghost_evict_metadata 4 1176064 l2_hits 4 0 l2_misses 4 0 l2_feeds 4 0 l2_rw_clash 4 0 l2_read_bytes 4 0 l2_write_bytes 4 0 l2_writes_sent 4 0 l2_writes_done 4 0 l2_writes_error 4 0 l2_writes_hdr_miss 4 0 l2_evict_lock_retry 4 0 l2_evict_reading 4 0 l2_free_on_write 4 0 l2_abort_lowmem 4 0 l2_cksum_bad 4 0 l2_io_error 4 0 l2_size 4 0 l2_hdr_size 4 0 memory_throttle_count 4 0 memory_direct_count 4 0 memory_indirect_count 4 0 arc_no_grow 4 0 arc_tempreserve 4 0 arc_loaned_bytes 4 0 arc_prune 4 79 arc_meta_used 4 4533766376 arc_meta_limit 4 500000000 arc_meta_max 4 4533770472
DeHackEd
commented
11 years ago Yet more output from arcstats, /proc/meminfo and some IO stats, immediately before and after doing a drop_caches after I started hitting the redline of memory usage.
DeHackEd
commented
11 years ago A workaround for issue #1101 seems to fix this issue as well under my typical workload.
aarrpp
commented
11 years ago Yes, this workaround works for me. But I set ZFS_OBJ_MTX_SZ to 10240. It works fine with ~500 concurrent processes. Thank You!
behlendorf
commented
11 years ago I'm closing this long long long issue because a fix was merged for the deadlock, see issue #1101. The remaining memory management concerns I'll open a new more succinct issue to track.
durandalTR
commented
11 years ago @behlendorf: Do you have a reference to that new memory management issue?
behlendorf
commented
11 years ago See #1132, it's largely a place holder.
While running a lot of rsync instances (~45 at once... it makes sense in context) I found my ARC expanding beyond the set limit of 2 gigabytes until ZFS deadlocked - almost all rsync processes hung in the D state and some kernel threads as well. A reboot was necessary. It only took about 5-10 minutes of this kind of pressure to break.
Not too long after all ZFS-involved processes froze up.
Stack dumps for some hung processes:
Machine specs: Single-socket quad-core Xeon 8 Gigs of RAM
SPL version: b29012b99994ece46019b664d67dace29e5c2586 ZFS version: 409dc1a570a836737b2a5bb43658cdde703c935e Kernel version: 3.0.28 vanilla custom build ZPool version 26 (originally built/run by zfs-fuse)
pool: tankname1 state: ONLINE status: The pool is formatted using an older on-disk format. The pool can still be used, but some features are unavailable. action: Upgrade the pool using 'zpool upgrade'. Once this is done, the pool will no longer be accessible on older software versions. scan: none requested config: NAME STATE READ WRITE CKSUM tankname1 ONLINE 0 0 0 sda ONLINE 0 0 0 errors: No known data errorsI've also tried using the module/zfs/arc.c from https://github.com/zfsonlinux/zfs/pull/669 for testing and reducing the ARC size. RAM usage still exceeds the limits set.
Nevertheless it's been running for a few hours now reliably.
(Edit: I also raised vm.min_free_kbytes from its default up to 262144 as part of a shotgun attempt to make this more stable.)