infinit loop in resilver if checkpoint present

ikozhukhov commented 3 years ago

System information

Type	Version/Name
Distribution Name	DilOS
Distribution Version
Linux Kernel
Architecture
ZFS Version
SPL Version

Describe the problem you're observing

infinit loop resilver with checkpoint on pool

Describe how to reproduce the problem

root@lenovo:/ztest/resilver# cat checkpoint.sh 
#!/bin/sh

# The pool size (MB)
PSIZE=2048

# Size of data on the pool (MB)
FSIZE=$((PSIZE>>1))

# VDEVs for the pool
FILE1=/var/tmp/file01.data
FILE2=/var/tmp/file02.data

POOL=testpool

# Create files for the testpool
truncate -s ${PSIZE}M ${FILE1} ${FILE2}

# Create a test pool on the only file
zpool create -o ashift=12 -o cachefile=none -O compression=on -O atime=off ${POOL} ${FILE1}

# Create a test fs on the pool...
zfs create -o compression=on -o recordsize=1m -o atime=off ${POOL}/testfs

# ... and fill it by random data
dd if=/dev/urandom of=/${POOL}/testfs/testf.rnd bs=1M count=${FSIZE} status=progress

# convert the pool to a mirror by attaching a new VDEV
zpool attach ${POOL} ${FILE1} ${FILE2}

# and set a checkpoint until the resilver finished
zpool checkpoint ${POOL}

# add another data after the checkpoint
dd if=/dev/urandom of=/${POOL}/testfs/testf2.rnd bs=1M count=$((FSIZE>>1)) status=progress

# here we have an infinite resilver process
# watch zpool status -s ${POOL}

root@lenovo:/ztest/resilver# sh -x checkpoint.sh 
+ PSIZE=2048
+ FSIZE=1024
+ FILE1=/var/tmp/file01.data
+ FILE2=/var/tmp/file02.data
+ POOL=testpool
+ truncate -s 2048M /var/tmp/file01.data /var/tmp/file02.data
+ zpool create -o ashift=12 -o cachefile=none -O compression=on -O atime=off testpool /var/tmp/file01.data
+ zfs create -o compression=on -o recordsize=1m -o atime=off testpool/testfs
+ dd if=/dev/urandom of=/testpool/testfs/testf.rnd bs=1M count=1024 status=progress
1051721728 bytes (1.1 GB, 1003 MiB) copied, 37 s, 28.4 MB/s
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 37.5735 s, 28.6 MB/s
+ zpool attach testpool /var/tmp/file01.data /var/tmp/file02.data
+ zpool checkpoint testpool
+ dd if=/dev/urandom of=/testpool/testfs/testf2.rnd bs=1M count=512 status=progress
536870912 bytes (537 MB, 512 MiB) copied, 187 s, 2.9 MB/s
512+0 records in
512+0 records out
536870912 bytes (537 MB, 512 MiB) copied, 186.62 s, 2.9 MB/s

start warch zpool status -s testpool and monitor status of pool

Include any warning/errors/backtraces from the system logs

ikozhukhov commented 3 years ago

@behlendorf @ahrens please take a look

ahrens commented 3 years ago

I don't recall what the intended behavior was here. @sdimitro could you take a look?

@ikozhukhov Is there a thread executing in an infinite loop? If so could you post the stack trace? It sounds like there might not be, but rather the observed problem is that the scrub does not make progress, according to zpool status?

frebib commented 3 years ago

I experienced this with a live pool, where resilvers complete and immediately restart again when a pool checkpoint is present. Here is a rough order of important events:

Create pool checkpoint
Add raidz vdev to the pool (hence the checkpoint above)
One drive removed from the pool whilst the pool was active. Note the drive also showed write errors in zpool status before removal - might be important, I'm not sure
Re-insert the drive and attach it to the pool
Standard resilver occurs, and finishes successfully
Resilver restarts straight away, succeeds and loops forever

Here is a truncated and redacted log of zpool history showing the strange behaviour: https://gist.githubusercontent.com/frebib/ee4143a606ec5252e1ca3986b586a662/raw The issue cleared right up when I removed the checkpoint.

behlendorf commented 3 years ago

@frebib thanks for letting us know, could you please add which version of OpenZFS you're seeing this with and the platform / distribution you're using.

frebib commented 3 years ago

$ zfs version
zfs-2.0.3-8
zfs-kmod-2.0.1-3
$ uname -a
Linux localhost 5.10.0-3-amd64 #1 SMP Debian 5.10.12-1 (2021-01-30) x86_64 GNU/Linux
$ head -n1 /etc/os-release
PRETTY_NAME="Debian GNU/Linux 11 (bullseye)"

stale[bot] commented 2 years ago

This issue has been automatically marked as "stale" because it has not had any activity for a while. It will be closed in 90 days if no further activity occurs. Thank you for your contributions.

openzfs / zfs