lvm2 writeback cache flush loops forever

sulakshm commented 5 years ago

lvmcache in writeback mode does not ever complete flush sometimes. Can I please get some clarity on what exactly is the rootcause and when this was resolved?

Even putting the lvmcache into cleaner policy did not make any difference. There is no kernel crash/traceback seen.

root@pdc4-sm35:~# lvs -o cache_dirty_blocks,cache_policy pwx0/pool
  CacheDirtyBlocks Cache Policy
               367 cleaner
root@pdc4-sm35:~#

root@pdc4-sm35:~# lvconvert --splitcache pwx0/pool
  367 blocks must still be flushed.
  367 blocks must still be flushed.
  367 blocks must still be flushed.
  367 blocks must still be flushed.
  367 blocks must still be flushed.
  367 blocks must still be flushed.
  367 blocks must still be flushed.
  367 blocks must still be flushed.
  367 blocks must still be flushed.
  367 blocks must still be flushed.
  367 blocks must still be flushed.
  367 blocks must still be flushed.
  367 blocks must still be flushed.
  367 blocks must still be flushed.
  367 blocks must still be flushed.
  367 blocks must still be flushed.
  367 blocks must still be flushed.
^C
root@pdc4-sm35:~#

root@pdc4-sm35:~# cat /etc/os-release
NAME="Ubuntu"
VERSION="16.04 LTS (Xenial Xerus)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 16.04 LTS"
VERSION_ID="16.04"
HOME_URL="http://www.ubuntu.com/"
SUPPORT_URL="http://help.ubuntu.com/"
BUG_REPORT_URL="http://bugs.launchpad.net/ubuntu/"
UBUNTU_CODENAME=xenial
root@pdc4-sm35:~# lvm version
  LVM version:     2.02.133(2) (2015-10-30)
  Library version: 1.02.110 (2015-10-30)
  Driver version:  4.34.0
root@pdc4-sm35:~# uname -r
4.4.0-21-generic
root@pdc4-sm35:~#

sulakshm commented 5 years ago

A reboot of the node, forced the dirty blocks to not show up.

root@pdc4-sm35:~# lvs -o cache_dirty_blocks,cache_policy pwx0
  CacheDirtyBlocks Cache Policy
                   cleaner
root@pdc4-sm35:~#

But if the whole vg is tried to be removed, then it results in strange errors as below now....

root@pdc4-sm35:~# vgremove -f pwx0
  /usr/sbin/cache_check: execvp failed: No such file or directory
  Check of pool pwx0/cache failed (status:2). Manual repair required!
  Failed to active cache locally pwx0/pool.
  Failed to uncache pwx0/pool.
root@pdc4-sm35:~#
root@pdc4-sm35:~# pvscan --cache
root@pdc4-sm35:~# vgremove -f pwx0
  /usr/sbin/cache_check: execvp failed: No such file or directory
  Check of pool pwx0/cache failed (status:2). Manual repair required!
  Failed to active cache locally pwx0/pool.
  Failed to uncache pwx0/pool.
root@pdc4-sm35:~# vgs
  VG   #PV #LV #SN Attr   VSize  VFree
  pwx0   2   1   0 wz--n- 30.85t    0
root@pdc4-sm35:~# lvs
  LV   VG   Attr       LSize  Pool    Origin       Data%  Meta%  Move Log Cpy%Sync Convert
  pool pwx0 Cwi---C--- 29.11t [cache] [pool_corig]
root@pdc4-sm35:~#

sulakshm commented 5 years ago

I installed now the thin-provisioning-tools package and cache_check is now made available. The dirty blocks for some reason did not show up earlier, now comes back again. The count suggests the IO is stuck.

root@pdc4-sm35:~# apt-file search cache_check
thin-provisioning-tools: /usr/sbin/cache_check
thin-provisioning-tools: /usr/share/man/man8/cache_check.8.gz
root@pdc4-sm35:~# apt-get install -y thin-provisioning-tools
Reading package lists... Done
...
Preparing to unpack .../thin-provisioning-tools_0.5.6-1ubuntu1_amd64.deb ...
Unpacking thin-provisioning-tools (0.5.6-1ubuntu1) ...
Processing triggers for man-db (2.7.5-1) ...
Setting up thin-provisioning-tools (0.5.6-1ubuntu1) ...
root@pdc4-sm35:~#
root@pdc4-sm35:~#
root@pdc4-sm35:~#
root@pdc4-sm35:~# lvremove -f pwx0/pool
  367 blocks must still be flushed.
  367 blocks must still be flushed.
  367 blocks must still be flushed.
  367 blocks must still be flushed.
  367 blocks must still be flushed.
  367 blocks must still be flushed.
^C
root@pdc4-sm35:~#

csonto commented 5 years ago

On 7/30/19 6:45 AM, Lakshmi Narasimhan Sundararajan wrote:

I installed now the thin-provisioning-tools package and cache_check is now made available. The dirty blocks for some reason did not show up earlier, now comes back again. The count suggests the IO is stuck.


root@pdc4-sm35:~# apt-file search cache_check
thin-provisioning-tools: /usr/sbin/cache_check
thin-provisioning-tools: /usr/share/man/man8/cache_check.8.gz
root@pdc4-sm35:~# apt-get install -y thin-provisioning-tools
Reading package lists... Done
...
Preparing to unpack .../thin-provisioning-tools_0.5.6-1ubuntu1_amd64.deb ...
Unpacking thin-provisioning-tools (0.5.6-1ubuntu1) ...
Processing triggers for man-db (2.7.5-1) ...
Setting up thin-provisioning-tools (0.5.6-1ubuntu1) ...
root@pdc4-sm35:~#
root@pdc4-sm35:~#
root@pdc4-sm35:~#
root@pdc4-sm35:~# lvremove -f pwx0/pool
   367 blocks must still be flushed.
   367 blocks must still be flushed.
   367 blocks must still be flushed.
   367 blocks must still be flushed.
   367 blocks must still be flushed.
   367 blocks must still be flushed.

migration_threshold needs to be increased. See the following bug for details:

https://bugzilla.redhat.com/show_bug.cgi?id=1668163#c3

Fixed in newer versions of LVM

-- Martian

^C root@pdc4-sm35:~#

lvmteam / lvm2

lvm2 writeback cache flush loops forever #22