arturpzol commented 1 year ago

I've noticed a performance drop (possibly related with sync) on kernels 5.10 - 5.19 with aufs support.

uname -a
Linux node-351ecb94 5.9.0-oe64 #1 SMP Thu Mar 9 11:11:28 UTC 2023 x86_64 GNU/Linux
dd if=/dev/zero of=/file bs=1M count=100
100+0 records in
100+0 records out
104857600 bytes (105 MB) copied, 0.0892645 s, 1.2 GB/s

uname -a
Linux node-351ecb94 5.10.140-oe64 #1 SMP Wed Mar 8 12:46:31 UTC 2023 x86_64 GNU/Linux
dd if=/dev/zero of=/file bs=1M count=100
100+0 records in
100+0 records out
104857600 bytes (105 MB) copied, 2.66332 s, 39.4 MB/s

uname -a
Linux node-351ecb94 6.0.0-oe64 #1 SMP PREEMPT_DYNAMIC Wed Mar 8 12:19:20 UTC 2023 x86_64 GNU/Linux
dd if=/dev/zero of=/file bs=1M count=100
100+0 records in
100+0 records out
104857600 bytes (105 MB) copied, 0.106082 s, 988 MB/s

I know the test sample is small, but the decrease in performance is evident even with large data.

Even though I'm using sync on the root / :

aufs on / type aufs (rw,relatime,sync,si=3cd2d3ec17894fc,udba=none,nowarn_perm)

up to kernel 5.9 and from kernel 6.0 the performance seems to be ok.

Git bisection showed that two commits are suspicious:

https://github.com/sfjro/aufs-standalone/commit/2254d10ddd118d7581ac8d401cd80b839a7b43ed

https://github.com/sfjro/aufs-standalone/commit/70f8bb837bce181cc2e11d60005b548013e96aa0

These two commits were committed to 5.10 - 5.19 which match the test results.

If possible please suggest any advices.

sfjro commented 1 year ago

Hello Artur,

Artur Piechocki:

I've noticed a performance drop (possibly related with sync) on kernels 5= .10 - 5.19 with aufs support. ::: If possible please suggest any advices.

I will dive into this issue, but I am busy for now. Gimme some time.

And is there only one RW branch in your aufs mount? If so, I'd suggest you to try

mount -o remount,sync /your/RW/branch

# dd of=/your/RW/branch/file ...

for various version of kernel.

J. R. Okajima

arturpzol commented 1 year ago

Result of tests after remount RW branch to sync:

Kernel 4.19:

uname -a
Linux node-6e28f6f7 4.19.190 #3 SMP Mon May 15 08:00:25 CEST 2023 x86_64 GNU/Linux
mount |grep -E "aufs|changes"
/dev/mapper/changes.img on /mnt/live type ext4 (rw,noatime,nodiratime,nodelalloc,errors=remount-ro)
aufs on / type aufs (rw,relatime,sync,si=ca459d0d991bcb36,udba=none,nowarn_perm)

dd if=/dev/zero of=/file bs=1M count=100
100+0 records in
100+0 records out
104857600 bytes (105 MB) copied, 0.13638 s, 769 MB/s

dd if=/dev/zero of=/mnt/live/changes/file bs=1M count=100
100+0 records in
100+0 records out
104857600 bytes (105 MB) copied, 0.145789 s, 719 MB/s

mount -o remount,sync /mnt/live/
mount |grep -E "aufs|changes"
/dev/mapper/changes.img on /mnt/live type ext4 (rw,noatime,nodiratime,sync,nodelalloc,errors=remount-ro)
aufs on / type aufs (rw,relatime,sync,si=ca459d0d991bcb36,udba=none,nowarn_perm)

dd if=/dev/zero of=/file bs=1M count=100
100+0 records in
100+0 records out
104857600 bytes (105 MB) copied, 1.0748 s, 97.6 MB/s

dd if=/dev/zero of=/mnt/live/changes/file bs=1M count=100
100+0 records in
100+0 records out
104857600 bytes (105 MB) copied, 1.04 s, 101 MB/s

Kernel 5.15:

uname -a
Linux node-6e28f6f7 5.15.74 #26 SMP Mon Jan 9 13:37:31 UTC 2023 x86_64 GNU/Linux

mount |grep -E "aufs|changes"
/dev/mapper/changes.img on /mnt/live type ext4 (rw,noatime,nodiratime,nodelalloc,errors=remount-ro)
aufs on / type aufs (rw,relatime,sync,si=bf4ca7cb8bb708da,udba=none,nowarn_perm)

dd if=/dev/zero of=/file bs=1M count=100
100+0 records in
100+0 records out
104857600 bytes (105 MB) copied, 1.53498 s, 68.3 MB/s

dd if=/dev/zero of=/mnt/live/changes/file bs=1M count=100
100+0 records in
100+0 records out
104857600 bytes (105 MB) copied, 0.131501 s, 797 MB/s

mount -o remount,sync /mnt/live/
mount |grep -E "aufs|changes"
/dev/mapper/changes.img on /mnt/live type ext4 (rw,noatime,nodiratime,sync,nodelalloc,errors=remount-ro)
aufs on / type aufs (rw,relatime,sync,si=bf4ca7cb8bb708da,udba=none,nowarn_perm)

dd if=/dev/zero of=/file bs=1M count=100
100+0 records in
100+0 records out
104857600 bytes (105 MB) copied, 1.49549 s, 70.1 MB/s

dd if=/dev/zero of=/mnt/live/changes/file bs=1M count=100
100+0 records in
100+0 records out
104857600 bytes (105 MB) copied, 1.45185 s, 72.2 MB/s

Thank you in advance

sfjro commented 1 year ago

Artur Piechocki:

Result of tests after remount RW branch to sync:

I could reproduce the problem.

o current aufs5.10 kernel

async ext4 163--164 kB/s
async ext4 + async aufs 117 kB/s
async ext4 + sync aufs 4.9--16.6 kB/s
sync ext4 10.3--16.2 kB/s
sync ext4 + sync aufs 3.2--18.2 kB/s

And I tried reverting the commit you mentiond 2254d10ddd11 2020-12-22 aufs: linux-v5.10-rc1, no more f_op->read() and ->write()

By the previous commit,
        51958525b3fd9 2020-12-22 for aufs: linux-v5.10-rc1, no more vfs_(read|write)f_t
f_op->read() and ->write() are less meaningful.
Now aufs totally removes them.

o reverted 2254d10ddd11 aufs5.10 kernel

async ext4 157--162 kB/s
async ext4 + async aufs 116--118 kB/s
async ext4 + sync aufs 115 kB/s
sync ext4 16.2 kB/s
sync ext4 + sync aufs 6.2--14.9 kB/s

Also I reverted the commit described in the commit log 51958525b3fd 2020-12-22 for aufs: linux-v5.10-rc1, no more vfs_(read|write)f_t

By the commit in v5.10-rc1,
        4d03e3cc59828 2020-09-08 fs: don't allow kernel reads and writes without iter ops
every f_op->read() and ->write() are less meaningful.
Now aufs stops using them.

o reverted both of 2254d10ddd11 and 51958525b3fd aufs5.10 kernel

async ext4 164--165 kB/s
async ext4 + async aufs 117--118 kB/s
async ext4 + sync aufs 118 kB/s
sync ext4 16.3 kB/s
sync ext4 + sync aufs 12.6--13.8 kB/s

Note: these benchmarks are tested a few times only.

Without 51958525b3fd, the figures in "sync aufs" are distributed widely, but I can see it can be very slow. And reverting the commit seems helped a lot.

The commit 51958525b3fd seems to be a little more helpful.

I am going to revert these two commits, but not the commit you mentioned, 70f8bb837bce 2020-12-22 aufs: linux-v5.10-rc1, no more set_fs() which cannot be reverted anymore.

But I'm afraid two commits will be necessary for around linux-v6.0 again due to the changes in mainline. I will investigate more.

J. R. Okajima

arturpzol commented 1 year ago

sfjro thank you for the deeper testing and analysis of this, so I'm waiting for these two commits to be reverted and will test the performance on my side.

Regarding to kernel 6.0 and above. I had no performance issues on this family of kernel versions.

sfjro commented 1 year ago

I am going to revert these two commits, but not the commit you mentioned, 70f8bb837bce 2020-12-22 aufs: linux-v5.10-rc1, no more set_fs() which cannot be reverted anymore.

It didn't work. The 'dd' case you mentioned worked well, but exec systemcall didn't (since it causes the read in kernel-space). I was investigating deeper, especially why aufs6.0 doesn't have this problem. During the work, my ssd drive died. I can read from my aufs partition, but cannot write anymore.

My aufs work is suspended. Hopefully it will restart in a few weeks.

J. R. Okajima

sfjro commented 1 year ago

@.***:

During the work, my ssd drive died. I can read from my aufs partition, but cannot write anymore.

My aufs work is suspended. Hopefully it will restart in a few weeks.

I've got a new pc, my aufs git repository got broken and I lost some git comits. I could restore some of them from my backup, but others were totally gone. I need more time.

By the way, the ssd was 10 years old Intel one. I made some partitions, all are ext4. And ext4 records "Lifetime writes" in its header. The value of my aufs partition's is 8TB! while the partition size is 16GB. Hmm, my ssd seems very long lived and worked very well.

J. R. Okajima

arturpzol commented 1 year ago

Thank you for all your efforts to identify the problem.

If you need any additional logs/tests to investigate the issue further, please let me know.

sfjro commented 1 year ago

Artur Piechocki:

Thank you for all your efforts to identify the problem.

If you need any additional logs/tests to investigate the issue further, please let me know.

Thank you for your offer. But I am still struggling with restoring my aufs environment after my ssd disaster. The partition where aufs5 is located was most heavily damaged. I was bisecting around aufs5.19..aufs5.x-rcN for v6.0-rc1 or rc2, where I was gussing the commit exists which made the performance recovered.

J. R. Okajima

sfjro commented 1 year ago

Hello,

Sorry for long delay. Now I'm getting back to aufs development.

@.***:

I am going to revert these two commits, but not the commit you mentioned, 70f8bb837bce 2020-12-22 aufs: linux-v5.10-rc1, no more set_fs() which cannot be reverted anymore.

It didn't work. The 'dd' case you mentioned worked well, but exec systemcall didn't (since it causes the read in kernel-space). I was investigating deeper, especially why aufs6.0 doesn't have this problem. During the work, my ssd drive died. I can read from my aufs partition, but cannot write anymore.

By git-bisect'ing, I've found the commit 91b94c5d6ae5 2022-06-10 iocb: delay evaluation of IS_SYNC(...) until we want to check IOCB_DSYNC is the key. It is from v6.0-rc1, and it chages these files.

block/fops.c fs/btrfs/file.c fs/direct-io.c fs/fuse/file.c fs/iomap/direct-io.c fs/zonefs/super.c include/linux/fs.h

For us and for now, some file are unnecessary obviously. So here is a reduced version of the commit. Try and report plz.

J. R. Okajima

diff --git a/fs/block_dev.c b/fs/block_dev.c index 9e84b1928b94..5f2a4f3b7366 100644 --- a/fs/block_dev.c +++ b/fs/block_dev.c @@ -201,7 +201,7 @@ static unsigned int dio_bio_write_op(struct kiocb *iocb) unsigned int op = REQ_OP_WRITE | REQ_SYNC | REQ_IDLE;

/* avoid the need for a I/O completion work item */

if (iocb->ki_flags & IOCB_DSYNC)
if (iocb_is_dsync(iocb)) op |= REQ_FUA; return op; } diff --git a/fs/direct-io.c b/fs/direct-io.c index d53fa92a1ab6..a7c204a00fcc 100644 --- a/fs/direct-io.c +++ b/fs/direct-io.c @@ -1220,7 +1220,7 @@ do_blockdev_direct_IO(struct kiocb iocb, struct inode inode, */ if (dio->is_async && iov_iter_rw(iter) == WRITE) { retval = 0;
if (iocb->ki_flags & IOCB_DSYNC)
if (iocb_is_dsync(iocb)) retval = dio_set_defer_completion(dio); else if (!dio->inode->i_sb->s_dio_done_wq) { / diff --git a/include/linux/fs.h b/include/linux/fs.h index 837dd6e727e4..f9981007afae 100644 --- a/include/linux/fs.h +++ b/include/linux/fs.h @@ -2732,6 +2732,12 @@ extern int vfs_fsync(struct file file, int datasync); extern int sync_file_range(struct file *file, loff_t offset, loff_t nbytes, unsigned int flags);

+static inline bool iocb_is_dsync(const struct kiocb *iocb) +{

return (iocb->ki_flags & IOCB_DSYNC) ||
IS_SYNC(iocb->ki_filp->f_mapping->host); +}
/*
- Sync the bytes written if this was a synchronous write. Expect ki_pos
- to already be updated for the write, and will return either the amount @@ -2739,7 +2745,7 @@ extern int sync_file_range(struct file file, loff_t offset, loff_t nbytes, / static inline ssize_t generic_write_sync(struct kiocb *iocb, ssize_t count) {
if (iocb->ki_flags & IOCB_DSYNC) {
if (iocb_is_dsync(iocb)) { int ret = vfs_fsync_range(iocb->ki_filp, iocb->ki_pos - count, iocb->ki_pos - 1, (iocb->ki_flags & IOCB_SYNC) ? 0 : 1); @@ -3251,7 +3257,7 @@ static inline int iocb_flags(struct file *file) res |= IOCB_APPEND; if (file->f_flags & O_DIRECT) res |= IOCB_DIRECT;
if ((file->f_flags & O_DSYNC) || IS_SYNC(file->f_mapping->host))
if (file->f_flags & O_DSYNC) res |= IOCB_DSYNC; if (file->f_flags & __O_SYNC) res |= IOCB_SYNC;

arturpzol commented 1 year ago

@sfjro thank you for the patch. It seems that the proposed fix restores performance to what it was in kernel 5.9 and earlier.

The only thing is that I had to omit the changes in fs/block_dev.c as I am using kernel version 5.15 and in this version such a file is not available.

Are you going to commit this to aufs-standalone aufs5.x branch?

sfjro commented 1 year ago

Artur Piechocki:

@sfjro thank you for the patch. It seems that the proposed fix restores performance to what it was in kernel 5.9 and earlier.

Thank you for the report.

The only thing is that I had to omit the changes in fs/block_dev.c as I am using kernel version 5.15 and in this version such a file is not available.

Yes. In v5.15-rc1, many parts of fs/block_dev.c were moved to block/fops.c. The patch for fs/block_dev.c I sent can be applied to block/fops.c in linux-v5.15.

Are you going to commit this to aufs-standalone aufs5.x branch?

No. Because the patch is for linux kernel itself and less related to aufs. Moreover I am going to stop maintaining aufs5 series.

J. R. Okajima

sfjro / aufs-standalone

Performance drop on kernel 5.10 - 5.19 #27

mount -o remount,sync /your/RW/branch