Open ilikejam opened 9 months ago
Switching to osxfs file sharing eliminates the problem.
Hi, thanks for reporting, we are investigating.
FWIW, I am experiencing the exact same issue (came here by googling for the stack trace): Docker for mac crashes in virtualization framework when doing overlay mounts in a linux container. This is on a M2 MacBook Pro running macOS 14.0.
For current version (4.27.0) google hits, the error is now:
ERRO[0004] error waiting for container: unexpected EOF
Trace looks like:
Fatal error reported: Linux kernel v6.6.12 crash on virtualization.framework
[ 3.739983] Internal error: Oops: 0000000096000006 [#1] SMP [ 3.740033] Modules linked in: selfowner(O) shiftfs(O) rosetta(O) grpcfuse(O) fakeowner(O) [ 3.740161] CPU: 2 PID: 1056 Comm: black Tainted: G O 6.6.12-linuxkit #1 [ 3.740242] pstate: 81400005 (Nzcv daif +PAN -UAO -TCO +DIT -SSBS BTYPE=--) [ 3.740310] pc : backing_file_open+0x6c/0xc0 [ 3.740360] lr : backing_file_open+0x54/0xc0 [ 3.740433] sp : ffff0000cb5279c0 [ 3.740478] x29: ffff0000cb5279c0 x28: ffff0000cb527c60 x27: 0000000000000000 [ 3.740546] x26: 0000000000000000 x25: ffff0000c0678000 x24: ffff800081ffc648 [ 3.740623] x23: ffff80007a2691d8 x22: ffff0000f8a7f098 x21: ffff0000f8a7f098 [ 3.740696] x20: ffff0000cb5279f8 x19: ffff0000f8a7e700 x18: 0000000000000000 [ 3.740778] x17: 0000000000000000 x16: 0000000000000000 x15: 0000000000000000 [ 3.740865] x14: 0000000000000000 x13: ffff0000c17e2020 x12: ffff0000cb527ca4 [ 3.740958] x11: 000000050556c1c0 x10: 0000000000000002 x9 : ffff800080301f8c [ 3.741027] x8 : ffff0000f8a7e800 x7 : 0000000000000000 x6 : 000000000000003f [ 3.741082] x5 : 0000000000000040 x4 : ffff0000cb5278f0 x3 : 0000000000000000 [ 3.741160] x2 : 0000000000000000 x1 : 0000000000000000 x0 : ffff0000f8a7e700 [ 3.741235] Call trace: [ 3.741268] backing_file_open+0x6c/0xc0 [ 3.741315] fakeowner_open+0x6c/0x190 [fakeowner] [ 3.741367] do_dentry_open+0x17c/0x528 [ 3.741406] backing_file_open+0x74/0xc0 [ 3.741436] ovl_open_realfile+0xdc/0xf0 [ 3.741481] ovl_open+0xc8/0x110 [ 3.741523] do_dentry_open+0x17c/0x528 [ 3.741567] vfs_open+0x34/0x40 [ 3.741604] path_openat+0xb04/0xdd8 [ 3.741641] do_filp_open+0xa4/0x160 [ 3.741681] do_sys_openat2+0xcc/0x108 [ 3.741716] __arm64_sys_openat+0x6c/0xc0 [ 3.741748] invoke_syscall.constprop.0+0x58/0xf8 [ 3.741811] do_el0_svc+0x48/0xd8 [ 3.741852] el0_svc+0x44/0x130 [ 3.741898] el0t_64_sync_handler+0x120/0x130 [ 3.741940] el0t_64_sync+0x190/0x198 [ 3.741993] Code: a9400e82 a9000c22 f9400681 d2800002 (f9401821) [ 3.742072] ---[ end trace 0000000000000000 ]---
Having the same issue on my MacBook Pro 13" 2019 (Intel Core i5) with MacOS Sonoma 14.2.1, here is my report:
Fatal error reported: Linux kernel v6.5.11 crash on virtualization.framework
[13371.149560] invalid opcode: 0(2002) - 2024-02-11T20:10:41.158474497Z: kernel BUG at fs/super.c:501!000 [#1] PREEMPT
SMP NOPTI
[13371.152602] C(2003) - 2024-02-11T20:10:41.163124497Z: invalid opcode: 0000 [#1] PREEMPT SMP NOPTIPU: 6 PID: 302 Comm: lifecycle-serve Tainted: G W O 6.5.11-linuxkit #1
[13371.152972] R(2004) - 2024-02-11T20:10:41.166166497Z: CPU: 6 PID: 302 Comm: lifecycle-serve Tainted: G W O 6.5.11-linuxkit #1IP: 0010:generic_shutdown_super+0x142/0x150
[13371.153092] C
ode: cc cc e8 d1 5a f7 ff 48 8b bb e0 00 00 00 eb db 48 8b 43 28(2005) - 2024-02-11T20:10:41.166536497Z: RIP: 0010:generic_shutdown_super+0x142/0x150 48 8d b3 98 03
00 00 48 c7 c7 38 37 39 ab 48 8b 10 e8 8e 18 de ff <0f> 0b 66 66 2e 0f 1f 84 00 00 00 00 00 90 90 90 90 90 90 90 90 90
[13371.153437] RSP: 0018:ffffaba480737c00 EFLAGS: 00010246
[13371.153550] RAX: 0000000000000037 RBX: ffff903d574e9800 RCX: 0000000000000000
[13371.153629] RDX: 0000000000000000 RSI: ffff903e7fd9c400 RDI: ffff903e7fd9c400
[13371.153781] RBP: ffffffffc03bb6e0 R08: 0000000000000000 R09: ffffaba480737ab8
[13371.154295] R10: 0000000000000003 R11: ffffffffabac9c68 R12: ffff903d4d87526c
[13371.159833] R13: ffff903d40d22101 R14: ffff903d4006d5b8 R15: ffff903d4d875268
[13371.160253] FS: 0000000000000000(0000) GS:ffff903e7fd80000(0000) knlGS:0000000000000000
[13371.160642] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[13371.160840] CR2: 00007f83c8b27b30 CR3: 0000000100c5a006 CR4: 00000000003706a0
[13371.160982] Call Trace:
[13371.161174] <TASK>
[13371.161256] ? die+0x36/0x90
[13371.161403] ? do_trap+0xda/0x100
[13371.161580] ? generic_shutdown_super+0x142/0x150
[13371.161708] ? do_error_trap+0x65/0x80
[13371.161751] ? generic_shutdown_super+0x142/0x150
[13371.161803] ? exc_invalid_op+0x50/0x70
[13371.161847] ? generic_shutdown_super+0x142/0x150
[13371.161905] ? asm_exc_invalid_op+0x1a/0x20
[13371.162110] ? generic_shutdown_super+0x142/0x150
[13371.162404] kill_anon_super+0x12/0x30
[13371.162462] deactivate_locked_super+0x30/0xa0
[13371.162516] cleanup_mnt+0xbd/0x150
[13371.162559] task_work_run+0x5a/0x90
[13371.162597] do_exit+0x35a/0xad0
[13371.162632] ? futex_unqueue+0x3c/0x60
[13371.162729] do_group_exit+0x31/0x80
[13371.162860] get_signal+0x98c/0x9c0
[13371.162913] arch_do_signal_or_restart+0x3e/0x270
[13371.162993] exit_to_user_mode_prepare+0xb1/0x110
[13371.163032] syscall_exit_to_user_mode+0x21/0x50
[13371.163082] do_syscall_64+0x6b/0x90
[13371.163158] entry_SYSCALL_64_after_hwframe+0x6e/0xd8
[13371.163206] RIP: 0033:0x472c83
[13371.163246] Code: Unable to access opcode bytes at 0x472c59.
[13371.163312] RSP: 002b:00007fd63ff05948 EFLAGS: 00000286 ORIG_RAX: 00000000000000ca
[13371.163368] RAX: fffffffffffffe00 RBX: 0000000000000000 RCX: 0000000000472c83
[13371.163416] RDX: 0000000000000000 RSI: 0000000000000080 RDI: 000000c000073948
[13371.163485] RBP: 00007fd63ff05990 R08: 0000000000000000 R09: 0000000000000000
[13371.163930] R10: 0000000000000000 R11: 0000000000000286 R12: 00000000004477c0
[13371.163951] R13: 0000000000000010 R14: 000000c000007d40 R15: 000000c000f44580
[13371.164081] </TASK>
[13371.164142] Modules linked in: xfrm_user xfrm_algo nfsd auth_rpcgss nfs lockd grace sunrpc fakeowner(O) shiftfs(O) grpcfuse(O) vmw_vsock_virtio_transport vmw_vsock_virtio_transport_common vsock [last unloaded: selfowner(O)]
[13371.165500] ---[ end trace 0000000000000000 ]---
This appears to be fixed in v4.30.0 (although looking at the changelogs, it seems it was fixed in 4.28?)
Description
Seeing kernel crash errors on an M1 Mac with one of our images. Was working prevously, but not sure is the OS or docker has been upgraded since.
Reproduce
$ mkdir test && touch test/go.py # cap SYS_ADMIN is required as the container creates read-write overlay mounts at runtime $ docker run --rm -ti --cap-add=SYS_ADMIN -v "$(PWD)/test":/repo:ro public.ecr.aws/bbc-ats/bbc-ats-lint Running linters ERRO[0000] error waiting for container:
Expected behavior
Container should run OK, not crash the Linux VM
docker version
docker info
Diagnostics ID
F123AF07-8555-41C8-8B6A-13C59C0AA3FB/20231009144422
Additional Info
Full error is:
Fatal error reported: Linux kernel v6.4.16 crash on virtualization.framework
[ 214.238908] Unable to handle kernel NULL pointer dereference at virtual address 000000000000046c [ 214.239039] Mem abort info: [ 214.239058] ESR = 0x0000000096000006 [ 214.239097] EC = 0x25: DABT (current EL), IL = 32 bits [ 214.239154] SET = 0, FnV = 0 [ 214.239209] EA = 0, S1PTW = 0 [ 214.239262] FSC = 0x06: level 2 translation fault [ 214.239305] Data abort info: [ 214.239345] ISV = 0, ISS = 0x00000006 [ 214.239389] CM = 0, WnR = 0 [ 214.239425] user pgtable: 4k pages, 48-bit VAs, pgdp=000000010a6ec000 [ 214.239551] [000000000000046c] pgd=0800000140ff4003, p4d=0800000140ff4003, pud=0800000154c05003, pmd=0000000000000000 [ 214.239638] Internal error: Oops: 0000000096000006 [#1] PREEMPT SMP [ 214.239738] Modules linked in: xfrm_user xfrm_algo nfsd auth_rpcgss nfs lockd grace sunrpc fakeowner(O) shiftfs(O) grpcfuse(O) vmw_vsock_virtio_transport vmw_vsock_virtio_transport_common vsock [ 214.239949] CPU: 6 PID: 2272 Comm: black Tainted: G O 6.4.16-linuxkit #1 [ 214.240036] pstate: 41401005 (nZcv daif +PAN -UAO -TCO +DIT +SSBS BTYPE=--) [ 214.240154] pc : errseq_sample+0x14/0x30 [ 214.240215] lr : do_dentry_open+0x60/0x38c [ 214.240247] sp : ffff80000be5b8a0 [ 214.240309] x29: ffff80000be5b8a0 x28: 0000000000000000 x27: 0000000000000000 [ 214.240399] x26: 0000000000000004 x25: ffffbc1e1a36f3f8 x24: 0000000000000004 [ 214.240476] x23: 0000000000000000 x22: ffff31421968e510 x21: ffff314221803b00 [ 214.240561] x20: ffff314221803b00 x19: 000000000000046c x18: 0000000000000000 [ 214.240634] x17: 0000000000000000 x16: ffffbc1e1908d658 x15: 0000000000000000 [ 214.240698] x14: 0000000000000000 x13: 0000000000000000 x12: 0000000000000000 [ 214.240829] x11: 0000000000000000 x10: 0000000000000000 x9 : ffffbc1e1909cf08 [ 214.240939] x8 : ffff31421968e600 x7 : 0000000000000000 x6 : 0000000000000064 [ 214.241035] x5 : 0000000000000000 x4 : 00000000ffffffff x3 : ffff314219648b00 [ 214.241126] x2 : 0000000100000000 x1 : 0000000000000000 x0 : 000000000000046c [ 214.241188] Call trace: [ 214.241222] errseq_sample+0x14/0x30 [ 214.241261] do_dentry_open+0x60/0x38c [ 214.241317] open_with_fake_path+0x58/0x84 [ 214.241363] fakeowner_open_realfile+0x60/0x8c [fakeowner] [ 214.241421] fakeowner_open+0x38/0x74 [fakeowner] [ 214.241482] do_dentry_open+0x2f8/0x38c [ 214.241507] open_with_fake_path+0x58/0x84 [ 214.241535] ovl_open_realfile+0xbc/0xe8 [ 214.241567] ovl_open+0x74/0xbc [ 214.241613] do_dentry_open+0x2f8/0x38c [ 214.241685] vfs_open+0x3c/0x4c [ 214.241717] path_openat+0x9d8/0xb30 [ 214.241758] do_filp_open+0x74/0xe4 [ 214.241806] do_sys_openat2+0xa8/0x10c [ 214.241875] do_sys_open+0x5c/0x8c [ 214.241911] __arm64_sys_openat+0x30/0x40 [ 214.241956] invoke_syscall.constprop.0+0x88/0xd8 [ 214.242002] do_el0_svc+0xc4/0x128 [ 214.242037] el0_svc+0x9c/0xcc [ 214.242084] el0t_64_sync_handler+0xac/0x13c [ 214.242150] el0t_64_sync+0x190/0x194 [ 214.242191] Code: a9be7bfd 910003fd f9000bf3 aa0003f3 (b9400260) [ 214.242249] ---[ end trace 0000000000000000 ]---