search

sfjro / aufs-standalone

27 stars 14 forks source link

aufs Operation not allowed on linux 6.6.4, 6.6.5, 6.6.6 #33

Open KrasNIX opened 9 months ago

KrasNIX commented 9 months ago

Simple test:

mkdir -p /au/{rw,ro,res}
touch /au/rw/file.in.rw
touch /au/ro/file.in.ro
mount -t aufs -o dirs=/au/rw=rw:/au/ro=ro none /au/res
ls -li /au/res

result: ls: unable to access '/au/res': Operation not allowed ? d????????? ? ? ? ? ? res 524291 drwxr-xr-x 2 root root 4096 dec 12 21:23 ro 524290 drwxr-xr-x 4 root root 4096 dec 12 21:26 rw

dmesg: [ 463.707906] RIP: 0010:vfs_getattr+0x4c/0x60 [ 463.707909] Code: 13 5b 5d 41 5c 41 5d 31 d2 31 c9 31 f6 31 ff c3 cc cc cc cc 89 d9 44 89 ea 5b 4c 89 e6 48 89 ef 5d 41 5c 41 5d e9 b4 fe ff ff <0f> 0b b8 ff ff ff ff 31 d2 31 c9 31 f6 31 ff c3 cc cc cc cc 90 90 [ 463.707912] RSP: 0018:ffff956d80c6bd88 EFLAGS: 00010286 [ 463.707915] RAX: ffff88c0d012e900 RBX: 0000000000000000 RCX: 0000000080000000 [ 463.707917] RDX: 000000000000035e RSI: ffff956d80c6be50 RDI: ffff956d80c6bda8 [ 463.707918] RBP: ffff956d80c6bde8 R08: 0000000000000000 R09: 0000000000000000 [ 463.707920] R10: 0000000000000000 R11: 0000000000000000 R12: ffff956d80c6be50 [ 463.707921] R13: ffff88c0c051c540 R14: ffff88c0d00f20d8 R15: ffff88c0c3bb2800 [ 463.707923] FS: 00007f2f200fc800(0000) GS:ffff88c1f7d40000(0000) knlGS:0000000000000000 [ 463.707925] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 463.707926] CR2: 000055c4900afd58 CR3: 0000000101eee000 CR4: 00000000000006e0 [ 463.707930] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 463.707931] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 463.707933] Call Trace: [ 463.707948] [ 463.707949] ? vfs_getattr+0x4c/0x60 [ 463.707954] ? warn+0x7d/0x130 [ 463.707969] ? vfs_getattr+0x4c/0x60 [ 463.707972] ? report_bug+0x19e/0x1d0 [ 463.707977] ? handle_bug+0x42/0x80 [ 463.707983] ? exc_invalid_op+0x13/0x70 [ 463.707987] ? asm_exc_invalid_op+0x16/0x20 [ 463.707996] ? vfs_getattr+0x4c/0x60 [ 463.707998] aufs_getattr+0x11f/0x210 [ 463.708004] vfs_statx+0xc2/0x180 [ 463.708008] do_statx+0x67/0xc0 [ 463.708013] x64_sys_statx+0x62/0x90 [ 463.708017] do_syscall_64+0x3a/0x90 [ 463.708020] entry_SYSCALL_64_after_hwframe+0x6e/0xd8 [ 463.708024] RIP: 0033:0x7f2f2029092a [ 463.708030] Code: 48 8b 05 d9 a4 0d 00 ba ff ff ff ff 64 c7 00 16 00 00 00 e9 a5 fd ff ff e8 e3 05 02 00 0f 1f 00 41 89 ca b8 4c 01 00 00 0f 05 <48> 3d 00 f0 ff ff 77 2e 89 c1 85 c0 74 0f 48 8b 05 a1 a4 0d 00 64 [ 463.708032] RSP: 002b:00007ffc778bf498 EFLAGS: 00000202 ORIG_RAX: 000000000000014c [ 463.708035] RAX: ffffffffffffffda RBX: 000055c4900aac28 RCX: 00007f2f2029092a [ 463.708036] RDX: 0000000000000900 RSI: 00007ffc778c072f RDI: 00000000ffffff9c [ 463.708038] RBP: 000000000000035e R08: 00007ffc778bf4a0 R09: 0000000000000002 [ 463.708039] R10: 000000000000035e R11: 0000000000000202 R12: 00007ffc778c072f [ 463.708040] R13: 0000000000000000 R14: 000055c4900aac10 R15: 0000000000000001 [ 463.708044] [ 463.708045] ---[ end trace 0000000000000000 ]---

KrasNIX commented 9 months ago

At linux kernel 6.6.1, 6.6.2, 6.6.3 all is good.

sfjro commented 9 months ago

Hello,

KrasNIX:

result: ls: unable to access '/au/res': Operation not allowed ? d????????? ? ? ? ? ? res 524291 drwxr-xr-x 2 root root 4096 dec 12 21:23 ro 524290 drwxr-xr-x 4 root root 4096 dec 12 21:26 rw

dmesg: [ 463.707906] RIP: 0010:vfs_getattr+0x4c/0x60 :::

Thanks for the report. It seems that you've met a problem which roadie2 found recently. https://github.com/sfjro/aufs-standalone/issues/32 Until aufs6.6.4 comes up, replace fs/stat.c from linux-v6.6.3.

J. R. Okajima

roadie2 commented 9 months ago

J.R.O, I've found that the boot problem Porteus and PorteuX users were having can be fixed by adding the "udba=none" option to the initrd mount code. The kernel versions that gave problems boot cleanly without replacing or editing fs/stat.c

I don't know if this fix helps Puppy Linux users who were having a kernel panic as I don't know how aufs is handled in the initrd, though it must use a mount operation as well.

This may also mean your patches don't need updating, hopefully that's the case. I'll add that "udba=reval" and "udba=notify" did not work.

peabee commented 9 months ago

As far as I can tell (I'm not an expert), Puppy has in it's initrd: mount -t aufs -o udba=reval,diropq=w,br=/mnt/tmpfs/pup_rw=rw,xino=/mnt/tmpfs/.aufs.xino unionfs /pup_new

I have been building with a reverted fs/stack.c while waiting for aufs-6.6.4 so have not yet been able to check if udba=none fixes the boot crash.

What is the implication of changing from udba=reval to udba=none???

peabee commented 9 months ago

Confirmed - changing init to udba=none does result in a successful boot to a working desktop.

However - such a change would not be retrospective meaning that 6.6.4 onwards kernels could never be used on systems built before the change... so not viable from my perspective.

sfjro commented 9 months ago

Hello,

roadie2:

I've found that the boot problem Porteus and PorteuX users were having can be fixed by adding the "udba=none" option to the initrd mount code. The kernel versions that gave problems boot cleanly without replacing or editing fs/stat.c

Yes, udba=none makes aufs to skip internal getattr() and gain some performance. Note that it is useful when your all branch fs-es are local and hidden from others. In other words, all your branches (layers) are accessed via aufs only. If you or someone else touches a file on a branch directly, eg. bypassing aufs, then it will make aufs confused and MAY cause some destructive result.

J. R. Okajima

sfjro commented 9 months ago

Waoh, You found the root cause!! Now I can confirm that the commit 8a924db2d7b5 2023-11-18 fs: Pass AT_GETATTR_NOSEC flag to getattr interface function in linux-v6.7-rc4 and in v6.6.4 (3fb0fa086419) made the change. Aufs has to follow that change. I will release aufs6.6.4 as soon as possible.

Unfortunately, I have no time to test aufs6.6.4, but I made a patch. Just untested at all. If you guys don't mind, please test it and report the result. Thanx in advance.

J. R. Okajima

diff --git a/fs/aufs/i_op.c b/fs/aufs/i_op.c index c3ff23f485d7..329d73968dc3 100644 --- a/fs/aufs/i_op.c +++ b/fs/aufs/i_op.c @@ -1288,9 +1288,13 @@ static int aufs_getattr(struct mnt_idmap idmap, const struct path path, goto out_fill; / pretending success /

positive = d_is_positive(h_path.dentry);
peabee commented 9 months ago

Many thanks - confirmed - k6.6.7 built with patched /fs/aufs/i_op.c boots correctly. Will use it as daily.

roadie2 commented 8 months ago

Confirmed, booting well with Porteus. Thanks very much

vladns commented 8 months ago

It’s not clear, does the latest patch (for i_op.c) fix the problem described in the first message? For some reason, the last messages from the system booted successfully.

sfjro commented 8 months ago

vladns:

It=E2=80=99s not clear, does the latest patch (for i_op.c) fix the proble= m described in the first message? For some reason, the last messages from= the system booted successfully.

I's not clear for me what you refer by the words the "first" and "last" messages are, but the patch (for fs/aufs/i_op.c) is definitely necessary for v6.6.4 and later. And I guess the report from KrasNIX (is this the one you call the first msg?) will be fixed.

J. R. Okajima

KrasNIX commented 8 months ago

Until aufs6.6.4 comes up, replace fs/stat.c from linux-v6.6.3.

this helps for me

vladns commented 8 months ago

The fact is that on my 6.6.X kernels, a kernel message (error) periodically appears:

divide error: 0000 [#1] PREEMPT SMP PTI
router kernel: CPU: 5 PID: 8900 Comm: NTCP Pumper Not tainted 6.6.8-arch1-1-aufs #1 11d8e1235f53d9ca3d
Hardware name: MSI MS-7851, BIOS V4.10 XX/XX/20XX
RIP: 0010:tcp_rcv_space_adjust+0xbe/0x160

After "Comm:", there may be other services. It doesn't depend on whether I replaced stat.c or patched i_op.c. I'm certainly not sure what this has to do with all of this. My other (second) system does not report such errors, although the systems are almost identical in hardware. But for example, in the first system on the 6.5.9 aufs kernel there are no problems either. Now I don't understand what the problem is.

sfjro commented 8 months ago

KrasNIX:

Until aufs6.6.4 comes up, replace fs/stat.c from linux-v6.6.3.

this helps for me

aufs6.6.4 doesn't appear yet, but I posted a patch. I'd suggest you to keep your fs/stat.c as vanilla kernel and apply a patch to fs/aufs/i_op.c.

Happy Holidays!

sfjro commented 8 months ago

vladns:

The fact is that on my 6.6.X kernels, a kernel message (error) periodically appears:

divide error: 0000 [#1] PREEMPT SMP PTI
router kernel: CPU: 5 PID: 8900 Comm: NTCP Pumper Not tainted 6.6.8-arch1-1-aufs #1 11d8e1235f53d9ca3d
Hardware name: MSI MS-7851, BIOS V4.10 XX/XX/20XX
RIP: 0010:tcp_rcv_space_adjust+0xbe/0x160

I guess this is a problem of a module related to network, and aufs is NOT related. If you can post your full stack trace.

J. R. Okajima

vladns commented 8 months ago 
dec 27 20:56:51 router kernel: CPU: 5 PID: 8900 Comm: NTCP Pumper Not tainted 6.6.8-arch1-1-aufs #1 11d8e1235f53d9ca3d
dec 27 20:56:51 router kernel: Hardware name: MSI MS-7851, BIOS V4.10 02/17/2016
dec 27 20:56:51 router kernel: RIP: 0010:tcp_rcv_space_adjust+0xbe/0x160
dec 27 20:56:51 router kernel: Code: f8 41 89 d0 29 d0 31 d2 49 0f af c2 49 f7 f0 45 8b 81 bc 04 00 00 44 0f b6 8b 10 06 00 00 31 d2 49 8d 04 42 48 98 48 c1 e0 08 <49> f7 f1 49 63 d0 48 98 48 39 d0 48 0f 47 c2 39 83 18 01 00 00 7c
dec 27 20:56:51 router kernel: RSP: 0018:ffffa8ce4b647bd0 EFLAGS: 00010206
dec 27 20:56:51 router kernel: RAX: 0000000001e04800 RBX: ffff9a436520af80 RCX: 000000007bb067a3
dec 27 20:56:51 router kernel: RDX: 0000000000000000 RSI: 0000000032692097 RDI: 00000000000051fd
dec 27 20:56:51 router kernel: RBP: ffff9a436520af80 R08: 0000000000600000 R09: 0000000000000000
dec 27 20:56:51 router kernel: R10: 000000000000fdfa R11: ffff9a4310380400 R12: 0000000000000000
dec 27 20:56:51 router kernel: R13: ffff9a436520b058 R14: 0000000000000000 R15: ffff9a436520b514
dec 27 20:56:51 router kernel: FS:  00007efd065ff6c0(0000) GS:ffff9a458f540000(0000) knlGS:0000000000000000
dec 27 20:56:51 router kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
dec 27 20:56:51 router kernel: CR2: 00007fe572cf5e10 CR3: 00000001e9fd4001 CR4: 00000000003706e0
dec 27 20:56:51 router kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
dec 27 20:56:51 router kernel: DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
dec 27 20:56:51 router kernel: Call Trace:
dec 27 20:56:51 router kernel:  <TASK>
dec 27 20:56:51 router kernel:  ? die+0x36/0x90
dec 27 20:56:51 router kernel:  ? do_trap+0xda/0x100
dec 27 20:56:51 router kernel:  ? tcp_rcv_space_adjust+0xbe/0x160
dec 27 20:56:51 router kernel:  ? do_error_trap+0x6a/0x90
dec 27 20:56:51 router kernel:  ? tcp_rcv_space_adjust+0xbe/0x160
dec 27 20:56:51 router kernel:  ? exc_divide_error+0x38/0x50
dec 27 20:56:51 router kernel:  ? tcp_rcv_space_adjust+0xbe/0x160
dec 27 20:56:51 router kernel:  ? asm_exc_divide_error+0x1a/0x20
dec 27 20:56:51 router kernel:  ? tcp_rcv_space_adjust+0xbe/0x160
dec 27 20:56:51 router kernel:  tcp_recvmsg_locked+0x2d4/0x960
dec 27 20:56:51 router kernel:  tcp_recvmsg+0x87/0x1f0
dec 27 20:56:51 router kernel:  inet6_recvmsg+0x56/0x130
dec 27 20:56:51 router kernel:  ? __pfx_bpf_lsm_socket_recvmsg+0x10/0x10
dec 27 20:56:51 router kernel:  ? security_socket_recvmsg+0x44/0x70
dec 27 20:56:51 router kernel:  sock_recvmsg+0x57/0xd0
dec 27 20:56:51 router kernel:  sock_read_iter+0x96/0x100
dec 27 20:56:51 router kernel:  vfs_read+0x303/0x350
dec 27 20:56:51 router kernel:  ksys_read+0xbb/0xf0
dec 27 20:56:51 router kernel:  do_syscall_64+0x60/0x90
dec 27 20:56:51 router kernel:  ? __x64_sys_epoll_ctl+0x70/0xa0
dec 27 20:56:51 router kernel:  ? syscall_exit_to_user_mode+0x2b/0x40
dec 27 20:56:51 router kernel:  ? do_syscall_64+0x6c/0x90
dec 27 20:56:51 router kernel:  ? do_syscall_64+0x6c/0x90
dec 27 20:56:51 router kernel:  ? syscall_exit_to_user_mode+0x2b/0x40
dec 27 20:56:51 router kernel:  ? do_syscall_64+0x6c/0x90
dec 27 20:56:51 router kernel:  entry_SYSCALL_64_after_hwframe+0x6e/0xd8
dec 27 20:56:51 router kernel: RIP: 0033:0x7efd7d86956c
dec 27 20:56:51 router kernel: Code: ec 28 48 89 54 24 18 48 89 74 24 10 89 7c 24 08 e8 19 58 f8 ff 48 8b 54 24 18 48 8b 74 24 10 41 89 c0 8b 7c 24 08 31 c0 0f 05 <48> 3d 00 f0 ff ff 77 34 44 89 c7 48 89 44 24 08 e8 6f 58 f8 ff 48
dec 27 20:56:51 router kernel: RSP: 002b:00007efd065fe630 EFLAGS: 00000246 ORIG_RAX: 0000000000000000
dec 27 20:56:51 router kernel: RAX: ffffffffffffffda RBX: 0000000000002000 RCX: 00007efd7d86956c
dec 27 20:56:51 router kernel: RDX: 0000000000002000 RSI: 00007efc4000b0e0 RDI: 00000000000000ec
dec 27 20:56:51 router kernel: RBP: 00007efd065fe680 R08: 0000000000000000 R09: 00000000f3694908
dec 27 20:56:51 router kernel: R10: 000000000000000c R11: 0000000000000246 R12: 00007efc54019e98
dec 27 20:56:51 router kernel: R13: 00007efc4000b0e0 R14: 00007efd065fe6c0 R15: 00007efc54019ae0
dec 27 20:56:51 router kernel:  </TASK>
dec 27 20:56:51 router kernel: Modules linked in: xt_nat ccm nf_conntrack_netlink xt_connbytes xt_NFQUEUE xt_mark 8021q garp mrp cls_route cls_u32 cls_fw sch_sfq cmac sch_htb nls_utf8 xt_TCPMSS xt_tcpmss nfnetlink_queue cifs cifs_arc4 nls_ucs2_utils rdma_cm iw_cm ib_cm ib_core cifs_md4 dns_resolver fscache netfs tun pppoe pppox ppp_generic slhc xt_comment ax88179_178a cdc_ether usbnet uas usb_storage mii ip6t_REJECT nf_reject_ipv6 ip6table_filter ip6table_mangle ip6table_raw ip6table_nat ip6_tables ts_kmp xt_multiport ipt_REJECT nf_reject_ipv4 xt_recent intel_rapl_msr xt_string intel_rapl_common xt_conntrack intel_uncore_frequency intel_uncore_frequency_common iptable_filter nct6775 xt_set x86_pkg_temp_thermal nct6775_core intel_powerclamp iptable_raw hwmon_vid xt_MASQUERADE coretemp snd_hda_codec_hdmi xt_REDIRECT kvm_intel xt_tcpudp iptable_nat nf_nat kvm iptable_mangle aufs ip_set_hash_ip irqbypass crct10dif_pclmul crc32_pclmul polyval_clmulni polyval_generic ath10k_pci i915 mt76x2u mt76x2_common gf128mul ghash_clmulni_intel
dec 27 20:56:51 router kernel:  mt76x02_usb ath10k_core mt76_usb snd_hda_intel sha512_ssse3 mt76x02_lib sha256_ssse3 snd_intel_dspcfg sha1_ssse3 snd_intel_sdw_acpi aesni_intel mt76 ip_set_hash_net drm_buddy ath snd_hda_codec i2c_algo_bit ip_set snd_hda_core crypto_simd r8169 ttm mei_pxp snd_hwdep cryptd iTCO_wdt mac80211 mei_hdcp ixgbe at24 drm_display_helper realtek snd_pcm rapl intel_pmc_bxt mxm_wmi intel_cstate cec snd_timer vfat mdio_devres iTCO_vendor_support libarc4 snd mei_me mdio intel_gtt i2c_i801 intel_uncore fat pcspkr i2c_smbus libphy dca mei soundcore lpc_ich video cfg80211 wmi acpi_pad mac_hid rfkill br_netfilter bridge stp llc nf_conntrack_ftp nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c fuse dm_mod loop nfnetlink zram ip_tables x_tables ext4 crc32c_generic crc16 mbcache jbd2 crc32c_intel xhci_pci xhci_pci_renesas
dec 27 20:56:51 router kernel: ---[ end trace 0000000000000000 ]---
dec 27 20:56:51 router kernel: RIP: 0010:tcp_rcv_space_adjust+0xbe/0x160
dec 27 20:56:51 router kernel: Code: f8 41 89 d0 29 d0 31 d2 49 0f af c2 49 f7 f0 45 8b 81 bc 04 00 00 44 0f b6 8b 10 06 00 00 31 d2 49 8d 04 42 48 98 48 c1 e0 08 <49> f7 f1 49 63 d0 48 98 48 39 d0 48 0f 47 c2 39 83 18 01 00 00 7c
dec 27 20:56:51 router kernel: RSP: 0018:ffffa8ce4b647bd0 EFLAGS: 00010206
dec 27 20:56:51 router kernel: RAX: 0000000001e04800 RBX: ffff9a436520af80 RCX: 000000007bb067a3
dec 27 20:56:51 router kernel: RDX: 0000000000000000 RSI: 0000000032692097 RDI: 00000000000051fd
dec 27 20:56:51 router kernel: RBP: ffff9a436520af80 R08: 0000000000600000 R09: 0000000000000000
dec 27 20:56:51 router kernel: R10: 000000000000fdfa R11: ffff9a4310380400 R12: 0000000000000000
dec 27 20:56:51 router kernel: R13: ffff9a436520b058 R14: 0000000000000000 R15: ffff9a436520b514
dec 27 20:56:51 router kernel: FS:  00007efd065ff6c0(0000) GS:ffff9a458f540000(0000) knlGS:0000000000000000
dec 27 20:56:51 router kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
dec 27 20:56:51 router kernel: CR2: 00007fe572cf5e10 CR3: 00000001e9fd4001 CR4: 00000000003706e0
dec 27 20:56:51 router kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
dec 27 20:56:51 router kernel: DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
sfjro commented 8 months ago

Hello vladns,

Your stacktrace shows that

which means aufs is unrelated and I can do nothing to help you.

Do you think there is a scenario like this?

J. R. Okajima

vladns commented 8 months ago

Thanks for the help. NTCP is part of the i2p(d) router. I don’t know what exactly is happening there. But this is not so important, because... instead of NTCP there may be another program, at least 2 or 3. This is the last one in the messages. But when I install kernel 6.5.9 with aufs support, these problems do not exist.

sempervictus commented 8 months ago

@sfjro have had the NOSEC fix in our tree for a bit but running into something new it appears in 6.6.8 when launching a 1:23 docker to run hello-world over AUFS:

aufs au_lkup_dentry:238:dockerd[496]: I/O Error, both of real entry and whiteout found, proc, err -5

have tried enabling SHWH to no avail - same effect with and without.

sfjro commented 8 months ago

Hello RageLtMan,

RageLtMan:

aufs au_lkup_dentry:238:dockerd[496]: I/O Error, both of real entry and whiteout found, proc, err -5

Hmm, it should not happen (but happened, I know).

The normal case:

The abnormal case (your case):

The possible scenario (which is just my guess) is

Did you run such command on RW directly (bypassing aufs)?

J. R. Okajima

sempervictus commented 8 months ago

@sfjro: thanks for pinging back. I ran no commands whatsoever, the invocation causing this is

# docker run --rm -ti hello-world
docker: Error response from daemon: stat /var/lib/docker/aufs/mnt/2c41877fe89b5a846606a2c86b3e032d55dcdfbfb384819c99e365aea6c30bf5-init/dev/pts: input/output error.
See 'docker run --help'.
[root@svl-arch00 vagrant]# dmesg|grep aufs
[   20.508179] aufs 6.6-20231106
[   20.931173] aufs au_opts_verify:763:dockerd[497]: dirperm1 breaks the protection by the permission bits on the lower branch
[   23.697143] aufs au_lkup_dentry:238:dockerd[493]: I/O Error, both of real entry and whiteout found, dev, err -5
[   23.697182] aufs au_lkup_dentry:238:dockerd[493]: I/O Error, both of real entry and whiteout found, dev, err -5
[   23.697196] aufs au_lkup_dentry:238:dockerd[493]: I/O Error, both of real entry and whiteout found, dev, err -5

The /var/lib/docker/aufs/mnt/2c41877fe89b5a846606a2c86b3e032d55dcdfbfb384819c99e365aea6c30bf5-init path remains, but empty so doesn't look like there was anything in there to be mounted over in the first place (nor did it exist prior to invocation).

This works in 6.1 by the way so figure its something with how 6.6 is stacking those fs'

sfjro commented 8 months ago

RageLtMan:

docker run --rm -ti hello-world

docker: Error response from daemon: stat /var/lib/docker/aufs/mnt/2c41877fe89b5a846606a2c86b3e032d55dcdfbfb384819c99e365aea6c30bf5-init/dev/pts: input/output error. See 'docker run --help'. @.*** vagrant]# dmesg grep aufs [ 20.508179] aufs 6.6-20231106 [ 20.931173] aufs au_opts_verify:763:dockerd[497]: dirperm1 breaks the protection by the permission bits on the lower branch [ 23.697143] aufs au_lkup_dentry:238:dockerd[493]: I/O Error, both of real entry and whiteout found, dev, err -5

Ok, Now I see something is changed in mainline kernel, and I have to check it. Unfortunately I am busy and I can't do it right now. I hope I will be able to return aufs in a few months.

J. R. Okajima

sempervictus commented 8 months ago

Roger, thanks. In the meantime, would you be able to point me in the general area where i should look for a solution? So far 62d53c4a1dfe3 looks interesting - looking into whether that changed any interface used by AUFS.

sfjro commented 8 months ago

RageLtMan:

Roger, thanks. In the meantime, would you be able to point me in the general area where i should look for a solution?

If I had enough time to investigate, I would try git-diff or git-bisect.

I don't force you to try these. It's a simplified instruction for myself.

J. R. Okajima

TurboBlaze commented 8 months ago

@sfjro I successfully compiled and run kernel 6.6.10 in Porteus 5.01 x86_64 with your /fs/aufs/i_op.c patch More info is here https://forum.porteus.org/viewtopic.php?p=97977#p97977 Thanks and Happy New Year!

sempervictus commented 8 months ago

@TurboBlaze - just vanilla 6.6.10? I'm still getting:

# docker run --rm -ti hello-world
docker: Error response from daemon: stat /var/lib/docker/aufs/mnt/a23767eccedee97aa675346307b0b2cc37582d31e319d06d70d2f1049ee1a85f-init/sys: input/output error.
See 'docker run --help'.
# dmesg | tail -200| grep aufs
[   67.592285] aufs au_opts_verify:763:dockerd[551]: dirperm1 breaks the protection by the permission bits on the lower branch
[   69.952540] aufs au_lkup_dentry:238:dockerd[554]: I/O Error, both of real entry and whiteout found, sys, err -5
[   69.954114] aufs au_lkup_dentry:238:dockerd[554]: I/O Error, both of real entry and whiteout found, sys, err -5
# uname -r
6.6.10

built in-tree with the following config options:

CONFIG_AUFS_FS=m
# CONFIG_AUFS_BRANCH_MAX_127 is not set
# CONFIG_AUFS_BRANCH_MAX_511 is not set
CONFIG_AUFS_BRANCH_MAX_1023=y
# CONFIG_AUFS_BRANCH_MAX_32767 is not set
CONFIG_AUFS_SBILIST=y
# CONFIG_AUFS_HNOTIFY is not set
CONFIG_AUFS_EXPORT=y
CONFIG_AUFS_INO_T_64=y
CONFIG_AUFS_XATTR=y
CONFIG_AUFS_FHSM=y
CONFIG_AUFS_RDU=y
# CONFIG_AUFS_DIRREN is not set
# CONFIG_AUFS_SHWH is not set
# CONFIG_AUFS_BR_RAMFS is not set
# CONFIG_AUFS_BR_FUSE is not set
CONFIG_AUFS_BR_HFSPLUS=y
# CONFIG_AUFS_BDEV_LOOP is not set
# CONFIG_AUFS_DEBUG is not set
TurboBlaze commented 8 months ago

@sempervictus I applied these patches for vanilla 6.6.10:

aufs6-base.patch
aufs6-kbuild.patch
aufs6-loopback.patch
aufs6-mmap.patch
aufs6-standalone-i-op.patch
aufs6-standalone.patch
lockdep-debug.patch
tmpfs-idr.patch
vfs-ino.patch

config is

CONFIG_AUFS_FS=y
# CONFIG_AUFS_BRANCH_MAX_127 is not set
CONFIG_AUFS_BRANCH_MAX_511=y
# CONFIG_AUFS_BRANCH_MAX_1023 is not set
# CONFIG_AUFS_BRANCH_MAX_32767 is not set
CONFIG_AUFS_SBILIST=y
CONFIG_AUFS_HNOTIFY=y
CONFIG_AUFS_HFSNOTIFY=y
CONFIG_AUFS_EXPORT=y
CONFIG_AUFS_INO_T_64=y
CONFIG_AUFS_XATTR=y
CONFIG_AUFS_FHSM=y
CONFIG_AUFS_RDU=y
CONFIG_AUFS_DIRREN=y
CONFIG_AUFS_SHWH=y
CONFIG_AUFS_BR_RAMFS=y
CONFIG_AUFS_BR_FUSE=y
CONFIG_AUFS_POLL=y
CONFIG_AUFS_BR_HFSPLUS=y
CONFIG_AUFS_BDEV_LOOP=y
# CONFIG_AUFS_DEBUG is not set