Stable version for ARM processors

mhomidi commented 3 weeks ago

Hi everyone,

I have started my work on Quark project recently and I am trying to run this project on an ARM processor. I tried to go through the instruction in the but during the make command I got some error related to the qvisdor compilation (It is on armdev_stale branch):


error: assigning to `&T` is undefined behavior, consider using an `UnsafeCell`
    --> src/qlib/kernel/socket/hostinet/uring_socket.rs:1143:29
     |
1143 | ...                   *(&opt[0] as *const _ as u64 as *mut i32) = -errno;
     |                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
     |
     = note: for more information, visit <https://doc.rust-lang.org/book/ch15-05-interior-mutability.html>
     = note: `#[deny(invalid_reference_casting)]` on by default

error: casting `&T` to `&mut T` is undefined behavior, even if the reference is unused, consider instead using an `UnsafeCell`
  --> src/qlib/vcpu_mgr.rs:72:25
   |
72 |         return unsafe { &mut *(&self.allocator as *const _ as u64 as *mut VcpuAllocator) };
   |                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
   |
   = note: for more information, visit <https://doc.rust-lang.org/book/ch15-05-interior-mutability.html>
error: assigning to `&T` is undefined behavior, consider using an `UnsafeCell`
  --> src/vmspace/kernel_io_thread.rs:42:13
   |
42 |             *(&self.eventfd as *const _ as u64 as *mut i32) = eventfd;
   |             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
   |
   = note: for more information, visit <https://doc.rust-lang.org/book/ch15-05-interior-mutability.html>

I also get a lot of warning that I removed them from the message.

When I checked these issues with main branch, I saw that they are changed a lot. I wonder if there is a stable version of QuarkContainer for ARM processor? (Even if it does not support everything that you have on x86 version)

My rust version is here:

> rustup --version
rustup 1.27.1 (54dd3d00f 2024-04-24)
info: This is the version for the rustup toolchain manager, not the rustc compiler.
info: The currently active `rustc` version is `rustc 1.76.0-nightly (d86d65bbc 2023-12-10)`

shrik3 commented 3 weeks ago

Hi, the preliminary arm support has been merged into main, and the armdev_stale branch is, as the name suggests, deprecated. Please simply use the main branch. Let me know if you hit any problem

shrik3 commented 3 weeks ago

heads up, one known issue is the vdso. this patch makes things work but is not optimal. Proper vdso parser is WIP

Also if you use armv8.3+ there might be issues with the PAN (privileged access never) feature. I have some local changes to work around it, feel free to ping if that's your case.

mhomidi commented 3 weeks ago

When I add the patch to the main branch after running docker run --rm --runtime=quark hello-world I get this problem:

 > docker run --rm --runtime=quark hello-world
OOM!!! cpu [1], size is 400000, alignment is 400000
docker: Error response from daemon: OCI runtime start failed: unable to retrieve OCI runtime error (open /run/containerd/io.containerd.runtime.v1.linux/moby/c08892fa2b0934d483eada3e16284305c80e3762e64f5133fd038266896b9086/log.json: no such file or directory): /usr/local/bin/quark did not terminate successfully: exit status 255: : unknown.

When I do not add patch I get this one:

Application error: get panic ...
docker: Error response from daemon: OCI runtime start failed: unable to retrieve OCI runtime error (open /run/containerd/io.containerd.runtime.v1.linux/moby/495af967123899950202f6c7675706618018569b284a6f929fe90b86dd71ad54/log.json: no such file or directory): /usr/local/bin/quark did not terminate successfully: exit status 255: : unknown.

For the panic I checked the quark.log and the panic error is about this:

[INFO] [0/28307824153745] SandboxConnect connect address is ^@qvisor-sandbox.dad571f2f2362f3fbd9e419d626670c9ce0a6bac8992538b7ac130120b13e05f
[Print] [1/4340480000|0] get panic : Some(AccountTaskEnter: Task[4340480000] switching from state Blocked (expected RunningSys) to Blocked)
[Print] [1/4340480000|0] panic occurred in file 'src/qlib/kernel/task.rs' at line 348

I also try to check the PAN but I got the same issue. (I added this patch for PAN) The error is as same as before:

OOM!!! cpu [2], size is 400000, alignment is 400000
docker: Error response from daemon: OCI runtime start failed: unable to retrieve OCI runtime error (open /run/containerd/io.containerd.runtime.v1.linux/moby/fbb61cdae8825f024cf410b0f0b8edd8c420414766fc97671ff5ccd90a030ffe/log.json: no such file or directory): /usr/local/bin/quark did not terminate successfully: exit status 255: : unknown.

UPDATE: Actually I could not attached the log file. I just tried to copy end of it:

[DEBUG] [0/28320703024610] Creating new sandbox for container fbb61cdae8825f024cf410b0f0b8edd8c420414766fc97671ff5ccd90a030ffe
[INFO] [0/28320703066181] commandline args is /usr/local/bin/quark boot --pipefd 8 
[DEBUG] [0/28320703099669] Creating the sandboxRootDir at /var/lib/quark/fbb61cdae8825f024cf410b0f0b8edd8c420414766fc97671ff5ccd90a030ffe
[INFO] [0/28320703103013] EnableNamespace ToEnterNS is []
[INFO] [0/28320703103396] Save container fbb61cdae8825f024cf410b0f0b8edd8c420414766fc97671ff5ccd90a030ffe
[DEBUG] [0/28320703119742] mounting 
 proc to 
 /var/lib/docker/overlay2/cb918ca3fca4d0659e47a250ca8c2dd7bab7db6358aa7a4b90d7456c51fa5269/merged/proc as proc with data ''
[DEBUG] [0/28320703120249] mounting 
 tmpfs to 
 /var/lib/docker/overlay2/cb918ca3fca4d0659e47a250ca8c2dd7bab7db6358aa7a4b90d7456c51fa5269/merged/dev as tmpfs with data 'mode=755,size=65536k'
[DEBUG] [0/28320703123371] mounting 
 devpts to 
 /var/lib/docker/overlay2/cb918ca3fca4d0659e47a250ca8c2dd7bab7db6358aa7a4b90d7456c51fa5269/merged/dev/pts as devpts with data 'newinstance,ptmxmode=0666,mode=0620,gid=5'
[DEBUG] [0/28320703126299] mounting 
 sysfs to 
 /var/lib/docker/overlay2/cb918ca3fca4d0659e47a250ca8c2dd7bab7db6358aa7a4b90d7456c51fa5269/merged/sys as sysfs with data ''
[DEBUG] [0/28320703129239] mounting 
 mqueue to 
 /var/lib/docker/overlay2/cb918ca3fca4d0659e47a250ca8c2dd7bab7db6358aa7a4b90d7456c51fa5269/merged/dev/mqueue as mqueue with data ''
[DEBUG] [0/28320703129444] mounting 
 shm to 
 /var/lib/docker/overlay2/cb918ca3fca4d0659e47a250ca8c2dd7bab7db6358aa7a4b90d7456c51fa5269/merged/dev/shm as tmpfs with data 'mode=1777,size=67108864'
[DEBUG] [0/28320703132495] mounting 
 /var/lib/docker/containers/fbb61cdae8825f024cf410b0f0b8edd8c420414766fc97671ff5ccd90a030ffe/resolv.conf to 
 /var/lib/docker/overlay2/cb918ca3fca4d0659e47a250ca8c2dd7bab7db6358aa7a4b90d7456c51fa5269/merged/etc/resolv.conf as bind with data ''
[DEBUG] [0/28320703133658] mounting 
 /var/lib/docker/containers/fbb61cdae8825f024cf410b0f0b8edd8c420414766fc97671ff5ccd90a030ffe/hostname to 
 /var/lib/docker/overlay2/cb918ca3fca4d0659e47a250ca8c2dd7bab7db6358aa7a4b90d7456c51fa5269/merged/etc/hostname as bind with data ''
[DEBUG] [0/28320703134019] mounting 
 /var/lib/docker/containers/fbb61cdae8825f024cf410b0f0b8edd8c420414766fc97671ff5ccd90a030ffe/hosts to 
 /var/lib/docker/overlay2/cb918ca3fca4d0659e47a250ca8c2dd7bab7db6358aa7a4b90d7456c51fa5269/merged/etc/hosts as bind with data ''
[DEBUG] [0/28320703134358] mknoding /dev/null
[DEBUG] [0/28320703134430] mknoding /dev/zero
[DEBUG] [0/28320703134463] mknoding /dev/full
[DEBUG] [0/28320703134496] mknoding /dev/tty
[DEBUG] [0/28320703134526] mknoding /dev/urandom
[DEBUG] [0/28320703134554] mknoding /dev/random
[ERROR] [0/28320703134605] mount to /var/lib/docker/overlay2/cb918ca3fca4d0659e47a250ca8c2dd7bab7db6358aa7a4b90d7456c51fa5269/merged /var/lib/quark/fbb61cdae8825f024cf410b0f0b8edd8c420414766fc97671ff5ccd90a030ffe/fbb61cdae8825f024cf410b0f0b8edd8c420414766fc97671ff5ccd90a030ffe
[INFO] [0/28320703140711] reset umask from 22 to 0, kernelMemRegionSize is 18
[INFO] [0/28320703142035] SetMemRegion phyAddr = 4000000000, hostAddr=4000000000; pageMmapsize = 6000 MB
[INFO] [0/28320703142265] set map region start=4000000000, end=4600000000
[INFO] [0/28320703143422] KernelMap1G start is 4000000000, end is 4600000000
[INFO] [0/28320703143444] KernelMap start is 3ffffff000, end is 4000000000
[INFO] [0/28320703147859] commandline args is /usr/local/bin/quark --root /var/run/docker/runtime-quark/moby --log /run/containerd/io.containerd.runtime.v1.linux/moby/fbb61cdae8825f024cf410b0f0b8edd8c420414766fc97671ff5ccd90a030ffe/log.json --log-format json start fbb61cdae8825f024cf410b0f0b8edd8c420414766fc97671ff5ccd90a030ffe 
[INFO] [0/28320703149253] Container:: start ....
[INFO] [0/28320703149314] Load metadata for container /var/run/docker/runtime-quark/moby fbb61cdae8825f024cf410b0f0b8edd8c420414766fc97671ff5ccd90a030ffe
[INFO] [0/28320703154317] metadatafile is /var/run/docker/runtime-quark/moby/fbb61cdae8825f024cf410b0f0b8edd8c420414766fc97671ff5ccd90a030ffe/meta.json
[INFO] [0/28320703163182] elfFile header Header { pt1: HeaderPt1 { magic: [127, 69, 76, 70], class: SixtyFour, data: LittleEndian, version: Current, os_abi: SystemV, abi_version: 0, padding: [0, 0, 0, 0, 0, 0, 0] }, pt2: Header64(HeaderPt2_ { type_: Executable, machine: AArch64, version: 1, entry_point: 274877911192, ph_offset: 64, sh_offset: 3639840, flags: 0, header_size: 64, ph_entry_size: 56, ph_count: 2, sh_entry_size: 64, sh_count: 6626, sh_str_index: 6625 }) }
[INFO] [0/28320703163282] program header: ProgramHeader64 { type_: Ok(Load), flags: Flags(7), offset: 65536, virtual_addr: 274877906944, physical_addr: 274877906944, file_size: 2414720, mem_size: 2420320, align: 65536 }
[INFO] [0/28320703163298] start mem 4000000000, len 24e000, offset 10000
[INFO] [0/28320703163370] program header: ProgramHeader64 { type_: Ok(OsSpecific(1685382481)), flags: Flags(6), offset: 0, virtual_addr: 0, physical_addr: 0, file_size: 0, mem_size: 0, align: 16 }
[INFO] [0/28320703171167] start enter guest[0]: entry is 4000001098, stack is 43400a0000
[INFO] [0/1273] shareSpace ready...
[INFO] [1/1406] cpu#1 start
[INFO] [1/1438] start enter guest[1]: entry is 4000001098, stack is 43400c0000
[INFO] [2/1449] cpu#2 start
[INFO] [2/1472] start enter guest[2]: entry is 4000001098, stack is 43400e0000
[INFO] [3/1489] cpu#3 start
[INFO] [3/1512] start enter guest[3]: entry is 4000001098, stack is 4340100000
[INFO] [4/1544] cpu#4 start
[INFO] [4/1578] start enter guest[4]: entry is 4000001098, stack is 4540120000
[INFO] [5/1611] cpu#5 start
[INFO] [6/1633] cpu#6 start
[INFO] [5/1641] start enter guest[5]: entry is 4000001098, stack is 4540140000
[INFO] [6/1663] start enter guest[6]: entry is 4000001098, stack is 4540160000
[INFO] [7/1686] cpu#7 start
[INFO] [8/1808] cpu#8 start
[INFO] [7/1815] start enter guest[7]: entry is 4000001098, stack is 4540180000
[INFO] [8/1839] start enter guest[8]: entry is 4000001098, stack is 45401c0000
[INFO] [9/1877] cpu#9 start
[INFO] [9/1929] start enter guest[9]: entry is 4000001098, stack is 45401e0000
[INFO] [10/1943] cpu#10 start
[INFO] [10/1976] start enter guest[10]: entry is 4000001098, stack is 4540200000
[INFO] [11/1989] cpu#11 start
[INFO] [11/2034] start enter guest[11]: entry is 4000001098, stack is 4340120000
[INFO] [12/2060] cpu#12 start
[INFO] [13/2124] cpu#13 start
[INFO] [12/2122] start enter guest[12]: entry is 4000001098, stack is 4340140000
[INFO] [14/2176] cpu#14 start
[INFO] [13/2170] start enter guest[13]: entry is 4000001098, stack is 4340160000
[INFO] [14/2208] start enter guest[14]: entry is 4000001098, stack is 4340180000
[INFO] [15/2229] cpu#15 start
[INFO] [16/2285] cpu#16 start
[INFO] [15/2265] start enter guest[15]: entry is 4000001098, stack is 43401a0000
[INFO] [16/2317] start enter guest[16]: entry is 4000001098, stack is 43401e0000
[INFO] [17/2345] cpu#17 start
[INFO] [18/2373] cpu#18 start
[INFO] [17/2396] start enter guest[17]: entry is 4000001098, stack is 4340200000
[INFO] [18/2404] start enter guest[18]: entry is 4000001098, stack is 4340220000
[INFO] [19/2444] cpu#19 start
[INFO] [19/2476] start enter guest[19]: entry is 4000001098, stack is 4340240000
[INFO] [21/2527] cpu#21 start
[INFO] [20/2690] cpu#20 start
[INFO] [21/2683] start enter guest[21]: entry is 4000001098, stack is 4340280000
[INFO] [20/2740] start enter guest[20]: entry is 4000001098, stack is 4340260000
[INFO] [24/2821] cpu#24 start
[INFO] [25/2889] cpu#25 start
[INFO] [24/2848] start enter guest[24]: entry is 4000001098, stack is 4340300000
[INFO] [25/2930] start enter guest[25]: entry is 4000001098, stack is 4340320000
[INFO] [29/3082] cpu#29 start
[INFO] [29/3119] start enter guest[29]: entry is 4000001098, stack is 43403a0000
[INFO] [31/3170] cpu#31 start
[INFO] [31/3214] start enter guest[31]: entry is 4000001098, stack is 43403e0000
[INFO] [38/3528] cpu#38 start
[INFO] [38/3556] start enter guest[38]: entry is 4000001098, stack is 45404e0000
[INFO] [35/3591] cpu#35 start
[INFO] [35/3622] start enter guest[35]: entry is 4000001098, stack is 4540480000
[INFO] [43/3738] cpu#43 start
[INFO] [28/3798] cpu#28 start
[INFO] [28/3819] start enter guest[28]: entry is 4000001098, stack is 4340380000
[INFO] [30/3844] cpu#30 start
[INFO] [33/3853] cpu#33 start
[INFO] [43/3822] start enter guest[43]: entry is 4000001098, stack is 45405a0000
[INFO] [30/3862] start enter guest[30]: entry is 4000001098, stack is 43403c0000
[INFO] [33/3879] start enter guest[33]: entry is 4000001098, stack is 4540440000
[INFO] [0/28320703191138] Start container fbb61cdae8825f024cf410b0f0b8edd8c420414766fc97671ff5ccd90a030ffe
[DEBUG] [0/28320703191403] executing hook ddd Hook { path: "/proc/1969107/exe", args: ["libnetwork-setkey", "-exec-root=/var/run/docker", "fbb61cdae8825f024cf410b0f0b8edd8c420414766fc97671ff5ccd90a030ffe", "5ec58dc378e4"], env: [], timeout: None }
[INFO] [44/5493] cpu#44 start
[INFO] [32/5491] cpu#32 start
[INFO] [26/5488] cpu#26 start
[INFO] [23/5522] cpu#23 start
[INFO] [39/5517] cpu#39 start
[INFO] [46/5489] cpu#46 start
[INFO] [34/5488] cpu#34 start
[INFO] [40/5534] cpu#40 start
[INFO] [22/5508] cpu#22 start
[INFO] [41/5546] cpu#41 start
[INFO] [44/5605] start enter guest[44]: entry is 4000001098, stack is 45405c0000
[INFO] [48/5688] cpu#48 start
[INFO] [26/5607] start enter guest[26]: entry is 4000001098, stack is 4340340000
[INFO] [32/5606] start enter guest[32]: entry is 4000001098, stack is 4540420000
[INFO] [34/5778] start enter guest[34]: entry is 4000001098, stack is 4540460000
[INFO] [39/5627] start enter guest[39]: entry is 4000001098, stack is 4540520000
[INFO] [22/5775] start enter guest[22]: entry is 4000001098, stack is 43402a0000
[INFO] [23/5614] start enter guest[23]: entry is 4000001098, stack is 43402c0000
[INFO] [46/5812] start enter guest[46]: entry is 4000001098, stack is 4540600000
[INFO] [48/5779] start enter guest[48]: entry is 4000001098, stack is 4540660000
[INFO] [40/5811] start enter guest[40]: entry is 4000001098, stack is 4540540000
[INFO] [36/5914] cpu#36 start
[INFO] [36/5937] start enter guest[36]: entry is 4000001098, stack is 45404a0000
[INFO] [41/5751] start enter guest[41]: entry is 4000001098, stack is 4540560000
[INFO] [47/6109] cpu#47 start
[INFO] [45/6158] cpu#45 start
[INFO] [37/6438] cpu#37 start
[INFO] [27/6442] cpu#27 start
[INFO] [50/6445] cpu#50 start
[INFO] [42/6506] cpu#42 start
[INFO] [45/6179] start enter guest[45]: entry is 4000001098, stack is 45405e0000
[INFO] [27/6488] start enter guest[27]: entry is 4000001098, stack is 4340360000
[INFO] [37/6477] start enter guest[37]: entry is 4000001098, stack is 45404c0000
[INFO] [49/6549] cpu#49 start
[INFO] [47/6133] start enter guest[47]: entry is 4000001098, stack is 4540640000
[INFO] [42/6585] start enter guest[42]: entry is 4000001098, stack is 4540580000
[INFO] [50/6568] start enter guest[50]: entry is 4000001098, stack is 45406a0000
[INFO] [49/6570] start enter guest[49]: entry is 4000001098, stack is 4540680000
[INFO] [53/7953] cpu#53 start
[INFO] [53/8004] start enter guest[53]: entry is 4000001098, stack is 4540700000
[INFO] [56/8351] cpu#56 start
[INFO] [55/8359] cpu#55 start
[INFO] [57/8358] cpu#57 start
[INFO] [55/8402] start enter guest[55]: entry is 4000001098, stack is 4540760000
[INFO] [58/8410] cpu#58 start
[INFO] [56/8400] start enter guest[56]: entry is 4000001098, stack is 4540780000
[INFO] [57/8405] start enter guest[57]: entry is 4000001098, stack is 45407a0000
[INFO] [58/8435] start enter guest[58]: entry is 4000001098, stack is 45407c0000
[INFO] [62/9195] cpu#62 start
[INFO] [62/9299] start enter guest[62]: entry is 4000001098, stack is 4340440000
[INFO] [54/9683] cpu#54 start
[INFO] [59/9687] cpu#59 start
[INFO] [54/9710] start enter guest[54]: entry is 4000001098, stack is 4540720000
[INFO] [59/9728] start enter guest[59]: entry is 4000001098, stack is 45407e0000
[INFO] [52/9806] cpu#52 start
[INFO] [60/9818] cpu#60 start
[INFO] [52/9852] start enter guest[52]: entry is 4000001098, stack is 45406e0000
[INFO] [60/9851] start enter guest[60]: entry is 4000001098, stack is 4540800000
[INFO] [61/10238] cpu#61 start
[INFO] [51/10269] cpu#51 start
[INFO] [61/10323] start enter guest[61]: entry is 4000001098, stack is 4340420000
[INFO] [51/10357] start enter guest[51]: entry is 4000001098, stack is 45406c0000
[INFO] [0/28320703928928] SandboxConnect connect address is  qvisor-sandbox.fbb61cdae8825f024cf410b0f0b8edd8c420414766fc97671ff5ccd90a030ffe
[ERROR] [2/30653513] OOM!!! cpu [2], size is 400000, alignment is 400000
[ERROR] [0/28320866235977] the error is IOError("ReadWithFds io::error is ECONNRESET")

shrik3 commented 3 weeks ago

Thanks for the report. could you increase the debugging level and post the log again? in config.json, set "DebugLevel" to "Debug".

[edit] oh i see you already have debug logs. This is weird, what CPU model are you using? Could you also try limiting the number of vcpus by passing e.g. -cpus=3 to the docker commandline?

mhomidi commented 3 weeks ago

lscpu of my system:

lscpu
Architecture:                    aarch64
CPU op-mode(s):                  64-bit
Byte Order:                      Little Endian
CPU(s):                          192
On-line CPU(s) list:             0-191
Thread(s) per core:              1
Core(s) per socket:              48
Socket(s):                       4
NUMA node(s):                    8
Vendor ID:                       0x48
Model:                           0
Stepping:                        0x1
CPU max MHz:                     2600.0000
CPU min MHz:                     200.0000
BogoMIPS:                        200.00
L1d cache:                       12 MiB
L1i cache:                       12 MiB
L2 cache:                        96 MiB
L3 cache:                        384 MiB
NUMA node0 CPU(s):               0-23
NUMA node1 CPU(s):               24-47
NUMA node2 CPU(s):               48-71
NUMA node3 CPU(s):               72-95
NUMA node4 CPU(s):               96-119
NUMA node5 CPU(s):               120-143
NUMA node6 CPU(s):               144-167
NUMA node7 CPU(s):               168-191
Vulnerability Itlb multihit:     Not affected
Vulnerability L1tf:              Not affected
Vulnerability Mds:               Not affected
Vulnerability Meltdown:          Not affected
Vulnerability Mmio stale data:   Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:        Mitigation; __user pointer sanitization
Vulnerability Spectre v2:        Not affected
Vulnerability Srbds:             Not affected
Vulnerability Tsx async abort:   Not affected
Flags:                           fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma dcpop asimddp asimdfhm ssbs

I also put the --cpus=3 to my docker and the result is something like this:


[DEBUG] [0/28587370136190] Creating new sandbox for container 58892083db0ffddca9a7e6358070e7605b9a7a2ce5c38c622c313eceb76b6cac
[INFO] [0/28587370177662] commandline args is /usr/local/bin/quark boot --pipefd 8 
[DEBUG] [0/28587370213392] Creating the sandboxRootDir at /var/lib/quark/58892083db0ffddca9a7e6358070e7605b9a7a2ce5c38c622c313eceb76b6cac
[INFO] [0/28587370216794] EnableNamespace ToEnterNS is []
[INFO] [0/28587370217028] Save container 58892083db0ffddca9a7e6358070e7605b9a7a2ce5c38c622c313eceb76b6cac
[DEBUG] [0/28587370233971] mounting 
 proc to 
 /var/lib/docker/overlay2/231ad2d4be4b03524a5289ae41e113679b657f918469ec4b93fe49e74475e78d/merged/proc as proc with data ''
[DEBUG] [0/28587370234472] mounting 
 tmpfs to 
 /var/lib/docker/overlay2/231ad2d4be4b03524a5289ae41e113679b657f918469ec4b93fe49e74475e78d/merged/dev as tmpfs with data 'mode=755,size=65536k'
[DEBUG] [0/28587370237391] mounting 
 devpts to 
 /var/lib/docker/overlay2/231ad2d4be4b03524a5289ae41e113679b657f918469ec4b93fe49e74475e78d/merged/dev/pts as devpts with data 'newinstance,ptmxmode=0666,mode=0620,gid=5'
[DEBUG] [0/28587370240184] mounting 
 sysfs to 
 /var/lib/docker/overlay2/231ad2d4be4b03524a5289ae41e113679b657f918469ec4b93fe49e74475e78d/merged/sys as sysfs with data ''
[DEBUG] [0/28587370243165] mounting 
 mqueue to 
 /var/lib/docker/overlay2/231ad2d4be4b03524a5289ae41e113679b657f918469ec4b93fe49e74475e78d/merged/dev/mqueue as mqueue with data ''
[DEBUG] [0/28587370243346] mounting 
 shm to 
 /var/lib/docker/overlay2/231ad2d4be4b03524a5289ae41e113679b657f918469ec4b93fe49e74475e78d/merged/dev/shm as tmpfs with data 'mode=1777,size=67108864'
[DEBUG] [0/28587370246136] mounting 
 /var/lib/docker/containers/58892083db0ffddca9a7e6358070e7605b9a7a2ce5c38c622c313eceb76b6cac/resolv.conf to 
 /var/lib/docker/overlay2/231ad2d4be4b03524a5289ae41e113679b657f918469ec4b93fe49e74475e78d/merged/etc/resolv.conf as bind with data ''
[DEBUG] [0/28587370247538] mounting 
 /var/lib/docker/containers/58892083db0ffddca9a7e6358070e7605b9a7a2ce5c38c622c313eceb76b6cac/hostname to 
 /var/lib/docker/overlay2/231ad2d4be4b03524a5289ae41e113679b657f918469ec4b93fe49e74475e78d/merged/etc/hostname as bind with data ''
[DEBUG] [0/28587370248161] mounting 
 /var/lib/docker/containers/58892083db0ffddca9a7e6358070e7605b9a7a2ce5c38c622c313eceb76b6cac/hosts to 
 /var/lib/docker/overlay2/231ad2d4be4b03524a5289ae41e113679b657f918469ec4b93fe49e74475e78d/merged/etc/hosts as bind with data ''
[DEBUG] [0/28587370248550] mknoding /dev/null
[DEBUG] [0/28587370248621] mknoding /dev/zero
[DEBUG] [0/28587370248655] mknoding /dev/full
[DEBUG] [0/28587370248680] mknoding /dev/tty
[DEBUG] [0/28587370248712] mknoding /dev/urandom
[DEBUG] [0/28587370248738] mknoding /dev/random
[ERROR] [0/28587370248787] mount to /var/lib/docker/overlay2/231ad2d4be4b03524a5289ae41e113679b657f918469ec4b93fe49e74475e78d/merged /var/lib/quark/58892083db0ffddca9a7e6358070e7605b9a7a2ce5c38c622c313eceb76b6cac/58892083db0ffddca9a7e6358070e7605b9a7a2ce5c38c622c313eceb76b6cac
[DEBUG] [0/28587370258857] mountinfo from sandbox process: [MountInfo { mnt_id: 2636, pid: 2635, majmin: "253:1", root: "/", mount_point: "/", mount_options: {"relatime": None, "rw": None}, opt_fields: [Master(1)], fs_type: "ext4", mount_source: Some("/dev/mapper/Atlas--179--vg-root"), super_options: {"errors": Some("remount-ro"), "stripe": Some("64"), "rw": None} }, MountInfo { mnt_id: 2637, pid: 2636, majmin: "0:6", root: "/", mount_point: "/dev", mount_options: {"noexec": None, "nosuid": None, "relatime": None, "rw": None}, opt_fields: [Master(2)], fs_type: "devtmpfs", mount_source: Some("udev"), super_options: {"nr_inodes": Some("131792870"), "rw": None, "mode": Some("755"), "size": Some("527171480k")} }, MountInfo { mnt_id: 2638, pid: 2637, majmin: "0:26", root: "/", mount_point: "/dev/pts", mount_options: {"noexec": None, "relatime": None, "nosuid": None, "rw": None}, opt_fields: [Master(3)], fs_type: "devpts", mount_source: Some("devpts"), super_options: {"rw": None, "mode": Some("620"), "ptmxmode": Some("000"), "gid": Some("5")} }, MountInfo { mnt_id: 2639, pid: 2637, majmin: "0:29", root: "/", mount_point: "/dev/shm", mount_options: {"nosuid": None, "nodev": None, "rw": None}, opt_fields: [Master(4)], fs_type: "tmpfs", mount_source: Some("tmpfs"), super_options: {"huge": Some("advise"), "rw": None} }, MountInfo { mnt_id: 2640, pid: 2637, majmin: "0:49", root: "/", mount_point: "/dev/hugepages", mount_options: {"rw": None, "relatime": None}, opt_fields: [Master(28)], fs_type: "hugetlbfs", mount_source: Some("hugetlbfs"), super_options: {"rw": None, "pagesize": Some("2M")} }, MountInfo { mnt_id: 2641, pid: 2637, majmin: "0:23", root: "/", mount_point: "/dev/mqueue", mount_options: {"relatime": None, "nosuid": None, "noexec": None, "rw": None, "nodev": None}, opt_fields: [Master(29)], fs_type: "mqueue", mount_source: Some("mqueue"), super_options: {"rw": None} }, MountInfo { mnt_id: 2642, pid: 2636, majmin: "0:27", root: "/", mount_point: "/run", mount_options: {"noexec": None, "rw": None, "nodev": None, "nosuid": None, "relatime": None}, opt_fields: [Master(5)], fs_type: "tmpfs", mount_source: Some("tmpfs"), super_options: {"mode": Some("755"), "size": Some("105446624k"), "rw": None} }, MountInfo { mnt_id: 2643, pid: 2642, majmin: "0:30", root: "/", mount_point: "/run/lock", mount_options: {"nodev": None, "nosuid": None, "relatime": None, "rw": None, "noexec": None}, opt_fields: [Master(6)], fs_type: "tmpfs", mount_source: Some("tmpfs"), super_options: {"rw": None, "size": Some("5120k")} }, MountInfo { mnt_id: 2644, pid: 2642, majmin: "0:50", root: "/", mount_point: "/run/rpc_pipefs", mount_options: {"relatime": None, "rw": None}, opt_fields: [Master(32)], fs_type: "rpc_pipefs", mount_source: Some("sunrpc"), super_options: {"rw": None} }, MountInfo { mnt_id: 2645, pid: 2642, majmin: "0:60", root: "/", mount_point: "/run/user/1680", mount_options: {"nosuid": None, "nodev": None, "rw": None, "relatime": None}, opt_fields: [Master(453)], fs_type: "tmpfs", mount_source: Some("tmpfs"), super_options: {"uid": Some("1680"), "size": Some("105446620k"), "gid": Some("1680"), "mode": Some("700"), "rw": None} }, MountInfo { mnt_id: 2646, pid: 2642, majmin: "0:4", root: "net:[4026531896]", mount_point: "/run/docker/netns/default", mount_options: {"rw": None}, opt_fields: [Master(538)], fs_type: "nsfs", mount_source: Some("nsfs"), super_options: {"rw": None} }, MountInfo { mnt_id: 2647, pid: 2642, majmin: "0:4", root: "net:[4026556749]", mount_point: "/run/docker/netns/3d5c9ebb9818", mount_options: {"rw": None}, opt_fields: [Master(572)], fs_type: "nsfs", mount_source: Some("nsfs"), super_options: {"rw": None} }, MountInfo { mnt_id: 2648, pid: 2642, majmin: "0:4", root: "net:[4026556549]", mount_point: "/run/docker/netns/15152e8b7fff", mount_options: {"rw": None}, opt_fields: [Master(776)], fs_type: "nsfs", mount_source: Some("nsfs"), super_options: {"rw": None} }, MountInfo { mnt_id: 2649, pid: 2636, majmin: "0:25", root: "/", mount_point: "/sys", mount_options: {"rw": None, "nodev": None, "relatime": None, "noexec": None, "nosuid": None}, opt_fields: [Master(7)], fs_type: "sysfs", mount_source: Some("sysfs"), super_options: {"rw": None} }, MountInfo { mnt_id: 2650, pid: 2649, majmin: "0:7", root: "/", mount_point: "/sys/kernel/security", mount_options: {"rw": None, "relatime": None, "nosuid": None, "noexec": None, "nodev": None}, opt_fields: [Master(8)], fs_type: "securityfs", mount_source: Some("securityfs"), super_options: {"rw": None} }, MountInfo { mnt_id: 2651, pid: 2649, majmin: "0:31", root: "/", mount_point: "/sys/fs/cgroup", mount_options: {"nosuid": None, "nodev": None, "ro": None, "noexec": None}, opt_fields: [Master(9)], fs_type: "tmpfs", mount_source: Some("tmpfs"), super_options: {"mode": Some("755"), "ro": None} }, MountInfo { mnt_id: 2652, pid: 2651, majmin: "0:32", root: "/", mount_point: "/sys/fs/cgroup/unified", mount_options: {"nodev": None, "rw": None, "noexec": None, "nosuid": None, "relatime": None}, opt_fields: [Master(10)], fs_type: "cgroup2", mount_source: Some("cgroup2"), super_options: {"rw": None} }, MountInfo { mnt_id: 2653, pid: 2651, majmin: "0:33", root: "/", mount_point: "/sys/fs/cgroup/systemd", mount_options: {"noexec": None, "nosuid": None, "rw": None, "relatime": None, "nodev": None}, opt_fields: [Master(11)], fs_type: "cgroup", mount_source: Some("cgroup"), super_options: {"xattr": None, "rw": None, "name": Some("systemd")} }, MountInfo { mnt_id: 2654, pid: 2651, majmin: "0:37", root: "/", mount_point: "/sys/fs/cgroup/net_cls,net_prio", mount_options: {"nosuid": None, "noexec": None, "rw": None, "nodev": None, "relatime": None}, opt_fields: [Master(16)], fs_type: "cgroup", mount_source: Some("cgroup"), super_options: {"net_cls": None, "net_prio": None, "rw": None} }, MountInfo { mnt_id: 2655, pid: 2651, majmin: "0:38", root: "/", mount_point: "/sys/fs/cgroup/cpu,cpuacct", mount_options: {"nosuid": None, "relatime": None, "rw": None, "nodev": None, "noexec": None}, opt_fields: [Master(17)], fs_type: "cgroup", mount_source: Some("cgroup"), super_options: {"cpu": None, "cpuacct": None, "rw": None} }, MountInfo { mnt_id: 2656, pid: 2651, majmin: "0:39", root: "/", mount_point: "/sys/fs/cgroup/freezer", mount_options: {"rw": None, "nodev": None, "nosuid": None, "noexec": None, "relatime": None}, opt_fields: [Master(18)], fs_type: "cgroup", mount_source: Some("cgroup"), super_options: {"rw": None, "freezer": None} }, MountInfo { mnt_id: 2657, pid: 2651, majmin: "0:40", root: "/", mount_point: "/sys/fs/cgroup/devices", mount_options: {"noexec": None, "nosuid": None, "rw": None, "relatime": None, "nodev": None}, opt_fields: [Master(19)], fs_type: "cgroup", mount_source: Some("cgroup"), super_options: {"rw": None, "devices": None} }, MountInfo { mnt_id: 2658, pid: 2651, majmin: "0:41", root: "/", mount_point: "/sys/fs/cgroup/perf_event", mount_options: {"relatime": None, "nosuid": None, "nodev": None, "rw": None, "noexec": None}, opt_fields: [Master(20)], fs_type: "cgroup", mount_source: Some("cgroup"), super_options: {"rw": None, "perf_event": None} }, MountInfo { mnt_id: 2659, pid: 2651, majmin: "0:42", root: "/", mount_point: "/sys/fs/cgroup/cpuset", mount_options: {"relatime": None, "noexec": None, "rw": None, "nosuid": None, "nodev": None}, opt_fields: [Master(21)], fs_type: "cgroup", mount_source: Some("cgroup"), super_options: {"cpuset": None, "rw": None} }, MountInfo { mnt_id: 2660, pid: 2651, majmin: "0:43", root: "/", mount_point: "/sys/fs/cgroup/blkio", mount_options: {"nosuid": None, "noexec": None, "relatime": None, "nodev": None, "rw": None}, opt_fields: [Master(22)], fs_type: "cgroup", mount_source: Some("cgroup"), super_options: {"rw": None, "blkio": None} }, MountInfo { mnt_id: 2661, pid: 2651, majmin: "0:44", root: "/", mount_point: "/sys/fs/cgroup/rdma", mount_options: {"nodev": None, "nosuid": None, "relatime": None, "rw": None, "noexec": None}, opt_fields: [Master(23)], fs_type: "cgroup", mount_source: Some("cgroup"), super_options: {"rdma": None, "rw": None} }, MountInfo { mnt_id: 2662, pid: 2651, majmin: "0:45", root: "/", mount_point: "/sys/fs/cgroup/memory", mount_options: {"relatime": None, "rw": None, "nosuid": None, "nodev": None, "noexec": None}, opt_fields: [Master(24)], fs_type: "cgroup", mount_source: Some("cgroup"), super_options: {"memory": None, "rw": None} }, MountInfo { mnt_id: 2663, pid: 2651, majmin: "0:46", root: "/", mount_point: "/sys/fs/cgroup/hugetlb", mount_options: {"rw": None, "nodev": None, "relatime": None, "noexec": None, "nosuid": None}, opt_fields: [Master(25)], fs_type: "cgroup", mount_source: Some("cgroup"), super_options: {"rw": None, "hugetlb": None} }, MountInfo { mnt_id: 2664, pid: 2651, majmin: "0:47", root: "/", mount_point: "/sys/fs/cgroup/pids", mount_options: {"nosuid": None, "nodev": None, "rw": None, "relatime": None, "noexec": None}, opt_fields: [Master(26)], fs_type: "cgroup", mount_source: Some("cgroup"), super_options: {"rw": None, "pids": None} }, MountInfo { mnt_id: 2665, pid: 2649, majmin: "0:34", root: "/", mount_point: "/sys/fs/pstore", mount_options: {"nodev": None, "nosuid": None, "rw": None, "relatime": None, "noexec": None}, opt_fields: [Master(12)], fs_type: "pstore", mount_source: Some("pstore"), super_options: {"rw": None} }, MountInfo { mnt_id: 2666, pid: 2649, majmin: "0:35", root: "/", mount_point: "/sys/firmware/efi/efivars", mount_options: {"noexec": None, "nosuid": None, "nodev": None, "rw": None, "relatime": None}, opt_fields: [Master(13)], fs_type: "efivarfs", mount_source: Some("efivarfs"), super_options: {"rw": None} }, MountInfo { mnt_id: 2667, pid: 2649, majmin: "0:36", root: "/", mount_point: "/sys/fs/bpf", mount_options: {"rw": None, "noexec": None, "relatime": None, "nosuid": None, "nodev": None}, opt_fields: [Master(14)], fs_type: "bpf", mount_source: None, super_options: {"mode": Some("700"), "rw": None} }, MountInfo { mnt_id: 2668, pid: 2649, majmin: "0:8", root: "/", mount_point: "/sys/kernel/debug", mount_options: {"relatime": None, "noexec": None, "nosuid": None, "rw": None, "nodev": None}, opt_fields: [Master(30)], fs_type: "debugfs", mount_source: Some("debugfs"), super_options: {"rw": None} }, MountInfo { mnt_id: 2669, pid: 2668, majmin: "0:12", root: "/", mount_point: "/sys/kernel/debug/tracing", mount_options: {"nodev": None, "relatime": None, "rw": None, "nosuid": None, "noexec": None}, opt_fields: [Master(661)], fs_type: "tracefs", mount_source: Some("tracefs"), super_options: {"rw": None} }, MountInfo { mnt_id: 2670, pid: 2649, majmin: "0:12", root: "/", mount_point: "/sys/kernel/tracing", mount_options: {"rw": None, "relatime": None, "nosuid": None, "noexec": None, "nodev": None}, opt_fields: [Master(31)], fs_type: "tracefs", mount_source: Some("tracefs"), super_options: {"rw": None} }, MountInfo { mnt_id: 2671, pid: 2649, majmin: "0:52", root: "/", mount_point: "/sys/fs/fuse/connections", mount_options: {"relatime": None, "rw": None, "nodev": None, "nosuid": None, "noexec": None}, opt_fields: [Master(34)], fs_type: "fusectl", mount_source: Some("fusectl"), super_options: {"rw": None} }, MountInfo { mnt_id: 2672, pid: 2649, majmin: "0:24", root: "/", mount_point: "/sys/kernel/config", mount_options: {"noexec": None, "rw": None, "nodev": None, "nosuid": None, "relatime": None}, opt_fields: [Master(35)], fs_type: "configfs", mount_source: Some("configfs"), super_options: {"rw": None} }, MountInfo { mnt_id: 2673, pid: 2636, majmin: "0:5", root: "/", mount_point: "/proc", mount_options: {"rw": None, "nodev": None, "noexec": None, "relatime": None, "nosuid": None}, opt_fields: [Master(15)], fs_type: "proc", mount_source: Some("proc"), super_options: {"rw": None} }, MountInfo { mnt_id: 2674, pid: 2673, majmin: "0:48", root: "/", mount_point: "/proc/sys/fs/binfmt_misc", mount_options: {"relatime": None, "rw": None}, opt_fields: [Master(27)], fs_type: "autofs", mount_source: Some("systemd-1"), super_options: {"pipe_ino": Some("873"), "fd": Some("28"), "minproto": Some("5"), "direct": None, "rw": None, "pgrp": Some("1"), "timeout": Some("0"), "maxproto": Some("5")} }, MountInfo { mnt_id: 2675, pid: 2674, majmin: "0:53", root: "/", mount_point: "/proc/sys/fs/binfmt_misc", mount_options: {"relatime": None, "noexec": None, "rw": None, "nosuid": None, "nodev": None}, opt_fields: [Master(114)], fs_type: "binfmt_misc", mount_source: Some("binfmt_misc"), super_options: {"rw": None} }, MountInfo { mnt_id: 2676, pid: 2673, majmin: "0:51", root: "/", mount_point: "/proc/fs/nfsd", mount_options: {"relatime": None, "rw": None}, opt_fields: [Master(33)], fs_type: "nfsd", mount_source: Some("nfsd"), super_options: {"rw": None} }, MountInfo { mnt_id: 2677, pid: 2636, majmin: "8:1", root: "/", mount_point: "/boot/efi", mount_options: {"relatime": None, "rw": None}, opt_fields: [Master(117)], fs_type: "vfat", mount_source: Some("/dev/sda1"), super_options: {"iocharset": Some("iso8859-1"), "rw": None, "shortname": Some("mixed"), "dmask": Some("0077"), "errors": Some("remount-ro"), "codepage": Some("437"), "fmask": Some("0077")} }, MountInfo { mnt_id: 2678, pid: 2636, majmin: "253:0", root: "/", mount_point: "/data01", mount_options: {"rw": None, "relatime": None}, opt_fields: [Master(120)], fs_type: "xfs", mount_source: Some("/dev/mapper/vg00-lv00"), super_options: {"rw": None, "logbsize": Some("32k"), "noquota": None, "inode64": None, "attr2": None, "logbufs": Some("8")} }, MountInfo { mnt_id: 2679, pid: 2636, majmin: "0:58", root: "/", mount_point: "/var/lib/lxcfs", mount_options: {"nodev": None, "rw": None, "relatime": None, "nosuid": None}, opt_fields: [Master(240)], fs_type: "fuse.lxcfs", mount_source: Some("lxcfs"), super_options: {"rw": None, "user_id": Some("0"), "group_id": Some("0"), "allow_other": None} }, MountInfo { mnt_id: 2680, pid: 2636, majmin: "0:61", root: "/", mount_point: "/mnt/efs_guiyang", mount_options: {"rw": None, "relatime": None}, opt_fields: [Master(462)], fs_type: "nfs", mount_source: Some("10.170.23.190:/"), super_options: {"wsize": Some("1048576"), "mountvers": Some("3"), "nolock": None, "mountport": Some("20048"), "proto": Some("tcp"), "addr": Some("10.170.23.190"), "retrans": Some("2"), "hard": None, "namlen": Some("255"), "local_lock": Some("all"), "vers": Some("3"), "rw": None, "rsize": Some("1048576"), "mountproto": Some("udp"), "timeo": Some("600"), "mountaddr": Some("10.170.23.190"), "sec": Some("sys")} }, MountInfo { mnt_id: 2681, pid: 2636, majmin: "0:62", root: "/", mount_point: "/mnt/pangu-ulan", mount_options: {"relatime": None, "rw": None}, opt_fields: [Master(471)], fs_type: "nfs", mount_source: Some("7.216.53.145:/"), super_options: {"namlen": Some("255"), "rw": None, "timeo": Some("600"), "local_lock": Some("all"), "vers": Some("3"), "mountport": Some("20048"), "rsize": Some("1048576"), "mountvers": Some("3"), "mountaddr": Some("7.216.53.145"), "addr": Some("7.216.53.145"), "wsize": Some("1048576"), "nolock": None, "proto": Some("tcp"), "mountproto": Some("udp"), "hard": None, "sec": Some("sys"), "retrans": Some("2")} }, MountInfo { mnt_id: 2682, pid: 2636, majmin: "0:63", root: "/", mount_point: "/mnt/data_gamma", mount_options: {"relatime": None, "rw": None}, opt_fields: [Master(480)], fs_type: "nfs", mount_source: Some("7.216.52.19:/"), super_options: {"mountvers": Some("3"), "mountport": Some("20048"), "retrans": Some("2"), "sec": Some("sys"), "proto": Some("tcp"), "addr": Some("7.216.52.19"), "hard": None, "timeo": Some("600"), "vers": Some("3"), "mountproto": Some("udp"), "mountaddr": Some("7.216.52.19"), "rsize": Some("1048576"), "wsize": Some("1048576"), "nolock": None, "namlen": Some("255"), "local_lock": Some("all"), "rw": None} }, MountInfo { mnt_id: 2683, pid: 2636, majmin: "0:64", root: "/", mount_point: "/var/lib/kubelet/pods/4e948f7d-c02a-46ff-8f13-7a2f9b1bbf1d/volumes/kubernetes.io~secret/etcd-certs", mount_options: {"relatime": None, "rw": None}, opt_fields: [Master(489)], fs_type: "tmpfs", mount_source: Some("tmpfs"), super_options: {"rw": None} }, MountInfo { mnt_id: 2684, pid: 2636, majmin: "0:65", root: "/", mount_point: "/var/lib/kubelet/pods/4e948f7d-c02a-46ff-8f13-7a2f9b1bbf1d/volumes/kubernetes.io~secret/calico-node-token-kjsvv", mount_options: {"relatime": None, "rw": None}, opt_fields: [Master(498)], fs_type: "tmpfs", mount_source: Some("tmpfs"), super_options: {"rw": None} }, MountInfo { mnt_id: 2685, pid: 2636, majmin: "0:66", root: "/", mount_point: "/var/lib/kubelet/pods/f727f1a4-ce0d-439e-be5b-c2ea6f4ae8ac/volumes/kubernetes.io~secret/coredns-token-xmm82", mount_options: {"rw": None, "relatime": None}, opt_fields: [Master(507)], fs_type: "tmpfs", mount_source: Some("tmpfs"), super_options: {"rw": None} }, MountInfo { mnt_id: 2686, pid: 2636, majmin: "0:67", root: "/", mount_point: "/var/lib/kubelet/pods/14136b14-da74-44aa-aec4-9d8674927a1b/volumes/kubernetes.io~secret/default-token-q7c4r", mount_options: {"relatime": None, "rw": None}, opt_fields: [Master(516)], fs_type: "tmpfs", mount_source: Some("tmpfs"), super_options: {"rw": None} }, MountInfo { mnt_id: 2687, pid: 2636, majmin: "0:113", root: "/", mount_point: "/mnt/10.155.193.134", mount_options: {"rw": None, "relatime": None}, opt_fields: [Master(624)], fs_type: "nfs", mount_source: Some("10.155.193.134:/"), super_options: {"proto": Some("tcp"), "vers": Some("3"), "addr": Some("10.155.193.134"), "nolock": None, "wsize": Some("1048576"), "timeo": Some("600"), "sec": Some("sys"), "mountvers": Some("3"), "hard": None, "mountport": Some("20048"), "rsize": Some("1048576"), "retrans": Some("2"), "rw": None, "mountproto": Some("udp"), "local_lock": Some("all"), "mountaddr": Some("10.155.193.134"), "namlen": Some("255")} }, MountInfo { mnt_id: 2688, pid: 2636, majmin: "0:114", root: "/", mount_point: "/mnt/10.170.23.20", mount_options: {"rw": None, "relatime": None}, opt_fields: [Master(633)], fs_type: "nfs", mount_source: Some("10.170.23.20:/"), super_options: {"namlen": Some("255"), "nolock": None, "mountaddr": Some("10.170.23.20"), "vers": Some("3"), "rw": None, "wsize": Some("1048576"), "local_lock": Some("all"), "mountvers": Some("3"), "retrans": Some("2"), "addr": Some("10.170.23.20"), "proto": Some("tcp"), "hard": None, "rsize": Some("1048576"), "sec": Some("sys"), "timeo": Some("600"), "mountproto": Some("udp"), "mountport": Some("20048")} }, MountInfo { mnt_id: 2689, pid: 2636, majmin: "0:121", root: "/", mount_point: "/mnt/7.216.107.52", mount_options: {"rw": None, "relatime": None}, opt_fields: [Master(526)], fs_type: "nfs", mount_source: Some("7.216.107.52:/"), super_options: {"mountvers": Some("3"), "sec": Some("sys"), "rsize": Some("1048576"), "local_lock": Some("all"), "vers": Some("3"), "hard": None, "mountproto": Some("udp"), "mountaddr": Some("7.216.107.52"), "namlen": Some("255"), "retrans": Some("2"), "mountport": Some("2049"), "timeo": Some("600"), "addr": Some("7.216.107.52"), "rw": None, "wsize": Some("1048576"), "proto": Some("tcp"), "nolock": None} }, MountInfo { mnt_id: 2690, pid: 2636, majmin: "0:76", root: "/", mount_point: "/var/lib/docker/overlay2/5cf367e7cff554824a4bceefa439292f19be608be65aea90d79840705acb5b68/merged", mount_options: {"rw": None, "relatime": None}, opt_fields: [Master(46)], fs_type: "overlay", mount_source: Some("overlay"), super_options: {"lowerdir": Some("/var/lib/docker/overlay2/l/NWOGTYWBJEDCG3NRY3ZGIMWOCI:/var/lib/docker/overlay2/l/6CTZAQ7TQLSLIXUGJJ2NT7MTFP"), "xino": Some("off"), "rw": None, "workdir": Some("/var/lib/docker/overlay2/5cf367e7cff554824a4bceefa439292f19be608be65aea90d79840705acb5b68/work"), "upperdir": Some("/var/lib/docker/overlay2/5cf367e7cff554824a4bceefa439292f19be608be65aea90d79840705acb5b68/diff")} }, MountInfo { mnt_id: 2691, pid: 2636, majmin: "0:88", root: "/", mount_point: "/var/lib/docker/containers/ebca0fc9a5671fa17e340d1a5fad1575c382a126872f6526cac39f3cc38eaa5f/mounts/shm", mount_options: {"rw": None, "relatime": None, "noexec": None, "nosuid": None, "nodev": None}, opt_fields: [Master(187)], fs_type: "tmpfs", mount_source: Some("shm"), super_options: {"rw": None, "size": Some("65536k")} }, MountInfo { mnt_id: 2692, pid: 2636, majmin: "0:100", root: "/", mount_point: "/var/lib/docker/overlay2/c00f81ff3cd065743e78ec85de4b1cde4a82260703cd2e485926b1095d4651a9/merged", mount_options: {"relatime": None, "rw": None}, opt_fields: [Master(527)], fs_type: "overlay", mount_source: Some("overlay"), super_options: {"workdir": Some("/var/lib/docker/overlay2/c00f81ff3cd065743e78ec85de4b1cde4a82260703cd2e485926b1095d4651a9/work"), "rw": None, "xino": Some("off"), "lowerdir": Some("/var/lib/docker/overlay2/l/A7XCDCQDYXOJUYF222JXPSUO3B:/var/lib/docker/overlay2/l/6CTZAQ7TQLSLIXUGJJ2NT7MTFP"), "upperdir": Some("/var/lib/docker/overlay2/c00f81ff3cd065743e78ec85de4b1cde4a82260703cd2e485926b1095d4651a9/diff")} }, MountInfo { mnt_id: 2693, pid: 2636, majmin: "0:101", root: "/", mount_point: "/var/lib/docker/containers/7965328ebad52b1eb67b25c6866632ec4ff46360c0d61a5bbd3b56d630ab78ae/mounts/shm", mount_options: {"relatime": None, "nodev": None, "nosuid": None, "rw": None, "noexec": None}, opt_fields: [Master(557)], fs_type: "tmpfs", mount_source: Some("shm"), super_options: {"rw": None, "size": Some("65536k")} }, MountInfo { mnt_id: 2694, pid: 2636, majmin: "0:132", root: "/", mount_point: "/var/lib/docker/overlay2/70d2ddc4a30739521fc32ebbc8e95982b9442ebabd809bdebf56601d31f889b6/merged", mount_options: {"relatime": None, "rw": None}, opt_fields: [Master(589)], fs_type: "overlay", mount_source: Some("overlay"), super_options: {"rw": None, "lowerdir": Some("/var/lib/docker/overlay2/l/34MMWLEZGA4357CLTQOFWWV4KZ:/var/lib/docker/overlay2/l/UDHNJX4ZB5ZVQDW73VGOI6TVFX:/var/lib/docker/overlay2/l/CEGUEZLKS6EF4RPHVXVZH4RK4W:/var/lib/docker/overlay2/l/MZFS7ERNFWIWQLWRV4JXT4CJFL:/var/lib/docker/overlay2/l/GK5W6EE7MW4NXHWIMSWHR75ZDO:/var/lib/docker/overlay2/l/HKISJNNSJTCARSKDWBR2ERGDHU"), "workdir": Some("/var/lib/docker/overlay2/70d2ddc4a30739521fc32ebbc8e95982b9442ebabd809bdebf56601d31f889b6/work"), "upperdir": Some("/var/lib/docker/overlay2/70d2ddc4a30739521fc32ebbc8e95982b9442ebabd809bdebf56601d31f889b6/diff"), "xino": Some("off")} }, MountInfo { mnt_id: 2695, pid: 2636, majmin: "253:1", root: "/var/lib/quark/f6a0559dee44b42218c077ac96eff8fa4916e7a2ccf9a41ab3b8dc91f7d83d16", mount_point: "/var/lib/quark/f6a0559dee44b42218c077ac96eff8fa4916e7a2ccf9a41ab3b8dc91f7d83d16", mount_options: {"rw": None, "relatime": None}, opt_fields: [Master(1)], fs_type: "ext4", mount_source: Some("/dev/mapper/Atlas--179--vg-root"), super_options: {"rw": None, "stripe": Some("64"), "errors": Some("remount-ro")} }, MountInfo { mnt_id: 2696, pid: 2695, majmin: "0:137", root: "/", mount_point: "/var/lib/quark/f6a0559dee44b42218c077ac96eff8fa4916e7a2ccf9a41ab3b8dc91f7d83d16/f6a0559dee44b42218c077ac96eff8fa4916e7a2ccf9a41ab3b8dc91f7d83d16", mount_options: {"rw": None, "relatime": None}, opt_fields: [Master(606)], fs_type: "overlay", mount_source: Some("overlay"), super_options: {"lowerdir": Some("/var/lib/docker/overlay2/l/35OPCWOFHTEZOECIDWNUC4XLA3:/var/lib/docker/overlay2/l/M6UYENVGI6HTDHORILUP6A5PYW"), "upperdir": Some("/var/lib/docker/overlay2/6d99e3934569d0770181a4a2e9f44832e72518599e361071036db3bdbeabb768/diff"), "rw": None, "xino": Some("off"), "workdir": Some("/var/lib/docker/overlay2/6d99e3934569d0770181a4a2e9f44832e72518599e361071036db3bdbeabb768/work")} }, MountInfo { mnt_id: 2697, pid: 2636, majmin: "0:137", root: "/", mount_point: "/var/lib/quark/f6a0559dee44b42218c077ac96eff8fa4916e7a2ccf9a41ab3b8dc91f7d83d16/f6a0559dee44b42218c077ac96eff8fa4916e7a2ccf9a41ab3b8dc91f7d83d16", mount_options: {"relatime": None, "rw": None}, opt_fields: [Master(606)], fs_type: "overlay", mount_source: Some("overlay"), super_options: {"workdir": Some("/var/lib/docker/overlay2/6d99e3934569d0770181a4a2e9f44832e72518599e361071036db3bdbeabb768/work"), "rw": None, "upperdir": Some("/var/lib/docker/overlay2/6d99e3934569d0770181a4a2e9f44832e72518599e361071036db3bdbeabb768/diff"), "xino": Some("off"), "lowerdir": Some("/var/lib/docker/overlay2/l/35OPCWOFHTEZOECIDWNUC4XLA3:/var/lib/docker/overlay2/l/M6UYENVGI6HTDHORILUP6A5PYW")} }, MountInfo { mnt_id: 2698, pid: 2636, majmin: "0:99", root: "/", mount_point: "/var/lib/docker/overlay2/5b279854b284d574c379b2e38099af0092f13a0ee0d6aaf9529bf05e2ec161d1/merged", mount_options: {"relatime": None, "rw": None}, opt_fields: [Master(754)], fs_type: "overlay", mount_source: Some("overlay"), super_options: {"workdir": Some("/var/lib/docker/overlay2/5b279854b284d574c379b2e38099af0092f13a0ee0d6aaf9529bf05e2ec161d1/work"), "xino": Some("off"), "lowerdir": Some("/var/lib/docker/overlay2/l/YXDEDKXB6Z54MNYCU72AIKTJIL:/var/lib/docker/overlay2/l/6CTZAQ7TQLSLIXUGJJ2NT7MTFP"), "rw": None, "upperdir": Some("/var/lib/docker/overlay2/5b279854b284d574c379b2e38099af0092f13a0ee0d6aaf9529bf05e2ec161d1/diff")} }, MountInfo { mnt_id: 2699, pid: 2636, majmin: "0:105", root: "/", mount_point: "/var/lib/docker/containers/360f0df0037fa208d9bb6a184001e12de198c29261f2f0a0b15dad041c864e5d/mounts/shm", mount_options: {"rw": None, "nodev": None, "relatime": None, "nosuid": None, "noexec": None}, opt_fields: [Master(765)], fs_type: "tmpfs", mount_source: Some("shm"), super_options: {"size": Some("65536k"), "rw": None} }, MountInfo { mnt_id: 2700, pid: 2636, majmin: "0:138", root: "/", mount_point: "/var/lib/docker/overlay2/baa7f8f2494e08741869e257884e863a419cf396414e3c6f261c4ae70ce8ab40/merged", mount_options: {"rw": None, "relatime": None}, opt_fields: [Master(607)], fs_type: "overlay", mount_source: Some("overlay"), super_options: {"rw": None, "upperdir": Some("/var/lib/docker/overlay2/baa7f8f2494e08741869e257884e863a419cf396414e3c6f261c4ae70ce8ab40/diff"), "lowerdir": Some("/var/lib/docker/overlay2/l/VWT7H7HRL66U7ZG6BXMCF22WDZ:/var/lib/docker/overlay2/l/UR2UMPASMWBIBJ256NAVZGJQLU:/var/lib/docker/overlay2/l/2DB2HVRZFJB72PYBNBPVZCRTIZ"), "workdir": Some("/var/lib/docker/overlay2/baa7f8f2494e08741869e257884e863a419cf396414e3c6f261c4ae70ce8ab40/work"), "xino": Some("off")} }, MountInfo { mnt_id: 2701, pid: 2636, majmin: "253:1", root: "/var/lib/quark/4f0bd31f3a4e6c02cd70e6042a8ba78b0096fe8d44d2330ff5183a14114634e4", mount_point: "/var/lib/quark/4f0bd31f3a4e6c02cd70e6042a8ba78b0096fe8d44d2330ff5183a14114634e4", mount_options: {"relatime": None, "rw": None}, opt_fields: [Master(1)], fs_type: "ext4", mount_source: Some("/dev/mapper/Atlas--179--vg-root"), super_options: {"rw": None, "errors": Some("remount-ro"), "stripe": Some("64")} }, MountInfo { mnt_id: 2702, pid: 2701, majmin: "0:147", root: "/", mount_point: "/var/lib/quark/4f0bd31f3a4e6c02cd70e6042a8ba78b0096fe8d44d2330ff5183a14114634e4/4f0bd31f3a4e6c02cd70e6042a8ba78b0096fe8d44d2330ff5183a14114634e4", mount_options: {"relatime": None, "rw": None}, opt_fields: [Master(744)], fs_type: "overlay", mount_source: Some("overlay"), super_options: {"xino": Some("off"), "workdir": Some("/var/lib/docker/overlay2/81a684608f5d112fc6476354f404a1b209fda4b065b3054ad5e9c1f2de403acf/work"), "upperdir": Some("/var/lib/docker/overlay2/81a684608f5d112fc6476354f404a1b209fda4b065b3054ad5e9c1f2de403acf/diff"), "rw": None, "lowerdir": Some("/var/lib/docker/overlay2/l/N32QMUBS7ACS3NQDTLYJIE6E7Z:/var/lib/docker/overlay2/l/M6UYENVGI6HTDHORILUP6A5PYW")} }, MountInfo { mnt_id: 2703, pid: 2636, majmin: "0:147", root: "/", mount_point: "/var/lib/quark/4f0bd31f3a4e6c02cd70e6042a8ba78b0096fe8d44d2330ff5183a14114634e4/4f0bd31f3a4e6c02cd70e6042a8ba78b0096fe8d44d2330ff5183a14114634e4", mount_options: {"relatime": None, "rw": None}, opt_fields: [Master(744)], fs_type: "overlay", mount_source: Some("overlay"), super_options: {"xino": Some("off"), "lowerdir": Some("/var/lib/docker/overlay2/l/N32QMUBS7ACS3NQDTLYJIE6E7Z:/var/lib/docker/overlay2/l/M6UYENVGI6HTDHORILUP6A5PYW"), "workdir": Some("/var/lib/docker/overlay2/81a684608f5d112fc6476354f404a1b209fda4b065b3054ad5e9c1f2de403acf/work"), "upperdir": Some("/var/lib/docker/overlay2/81a684608f5d112fc6476354f404a1b209fda4b065b3054ad5e9c1f2de403acf/diff"), "rw": None} }, MountInfo { mnt_id: 2704, pid: 2636, majmin: "253:1", root: "/var/lib/quark/79b37fb17dd0a1d23dcb96f37ad5edf4cf1936fd60f483a82207796dfef24efb", mount_point: "/var/lib/quark/79b37fb17dd0a1d23dcb96f37ad5edf4cf1936fd60f483a82207796dfef24efb", mount_options: {"relatime": None, "rw": None}, opt_fields: [Master(1)], fs_type: "ext4", mount_source: Some("/dev/mapper/Atlas--179--vg-root"), super_options: {"errors": Some("remount-ro"), "rw": None, "stripe": Some("64")} }, MountInfo { mnt_id: 2705, pid: 2704, majmin: "0:148", root: "/", mount_point: "/var/lib/quark/79b37fb17dd0a1d23dcb96f37ad5edf4cf1936fd60f483a82207796dfef24efb/79b37fb17dd0a1d23dcb96f37ad5edf4cf1936fd60f483a82207796dfef24efb", mount_options: {"relatime": None, "rw": None}, opt_fields: [Master(745)], fs_type: "overlay", mount_source: Some("overlay"), super_options: {"xino": Some("off"), "rw": None, "upperdir": Some("/var/lib/docker/overlay2/ec118f3baa9683bae9070965e50700c0470cb58257fc028dc12ac731632a7c10/diff"), "lowerdir": Some("/var/lib/docker/overlay2/l/ESIA3WXCJEAZHWPBFT7LNH2TRZ:/var/lib/docker/overlay2/l/M6UYENVGI6HTDHORILUP6A5PYW"), "workdir": Some("/var/lib/docker/overlay2/ec118f3baa9683bae9070965e50700c0470cb58257fc028dc12ac731632a7c10/work")} }, MountInfo { mnt_id: 2706, pid: 2636, majmin: "0:148", root: "/", mount_point: "/var/lib/quark/79b37fb17dd0a1d23dcb96f37ad5edf4cf1936fd60f483a82207796dfef24efb/79b37fb17dd0a1d23dcb96f37ad5edf4cf1936fd60f483a82207796dfef24efb", mount_options: {"rw": None, "relatime": None}, opt_fields: [Master(745)], fs_type: "overlay", mount_source: Some("overlay"), super_options: {"xino": Some("off"), "workdir": Some("/var/lib/docker/overlay2/ec118f3baa9683bae9070965e50700c0470cb58257fc028dc12ac731632a7c10/work"), "lowerdir": Some("/var/lib/docker/overlay2/l/ESIA3WXCJEAZHWPBFT7LNH2TRZ:/var/lib/docker/overlay2/l/M6UYENVGI6HTDHORILUP6A5PYW"), "rw": None, "upperdir": Some("/var/lib/docker/overlay2/ec118f3baa9683bae9070965e50700c0470cb58257fc028dc12ac731632a7c10/diff")} }, MountInfo { mnt_id: 2707, pid: 2636, majmin: "253:1", root: "/var/lib/quark/dad571f2f2362f3fbd9e419d626670c9ce0a6bac8992538b7ac130120b13e05f", mount_point: "/var/lib/quark/dad571f2f2362f3fbd9e419d626670c9ce0a6bac8992538b7ac130120b13e05f", mount_options: {"relatime": None, "rw": None}, opt_fields: [Master(1)], fs_type: "ext4", mount_source: Some("/dev/mapper/Atlas--179--vg-root"), super_options: {"stripe": Some("64"), "rw": None, "errors": Some("remount-ro")} }, MountInfo { mnt_id: 2708, pid: 2707, majmin: "0:149", root: "/", mount_point: "/var/lib/quark/dad571f2f2362f3fbd9e419d626670c9ce0a6bac8992538b7ac130120b13e05f/dad571f2f2362f3fbd9e419d626670c9ce0a6bac8992538b7ac130120b13e05f", mount_options: {"rw": None, "relatime": None}, opt_fields: [Master(746)], fs_type: "overlay", mount_source: Some("overlay"), super_options: {"upperdir": Some("/var/lib/docker/overlay2/83abe0de4dbf19ab9493fe0276299da9f690f3ddf79f5c1ab1de70c70182bc6f/diff"), "rw": None, "lowerdir": Some("/var/lib/docker/overlay2/l/FK473J5LB2JC76NQBNLAB3UNXJ:/var/lib/docker/overlay2/l/M6UYENVGI6HTDHORILUP6A5PYW"), "workdir": Some("/var/lib/docker/overlay2/83abe0de4dbf19ab9493fe0276299da9f690f3ddf79f5c1ab1de70c70182bc6f/work"), "xino": Some("off")} }, MountInfo { mnt_id: 2709, pid: 2636, majmin: "0:149", root: "/", mount_point: "/var/lib/quark/dad571f2f2362f3fbd9e419d626670c9ce0a6bac8992538b7ac130120b13e05f/dad571f2f2362f3fbd9e419d626670c9ce0a6bac8992538b7ac130120b13e05f", mount_options: {"rw": None, "relatime": None}, opt_fields: [Master(746)], fs_type: "overlay", mount_source: Some("overlay"), super_options: {"lowerdir": Some("/var/lib/docker/overlay2/l/FK473J5LB2JC76NQBNLAB3UNXJ:/var/lib/docker/overlay2/l/M6UYENVGI6HTDHORILUP6A5PYW"), "workdir": Some("/var/lib/docker/overlay2/83abe0de4dbf19ab9493fe0276299da9f690f3ddf79f5c1ab1de70c70182bc6f/work"), "upperdir": Some("/var/lib/docker/overlay2/83abe0de4dbf19ab9493fe0276299da9f690f3ddf79f5c1ab1de70c70182bc6f/diff"), "xino": Some("off"), "rw": None} }, MountInfo { mnt_id: 2710, pid: 2636, majmin: "253:1", root: "/var/lib/quark/7640b5c2192f4204a718c05e77323359184738705b98fd8cbd20b769d4cd1c6d", mount_point: "/var/lib/quark/7640b5c2192f4204a718c05e77323359184738705b98fd8cbd20b769d4cd1c6d", mount_options: {"rw": None, "relatime": None}, opt_fields: [Master(1)], fs_type: "ext4", mount_source: Some("/dev/mapper/Atlas--179--vg-root"), super_options: {"stripe": Some("64"), "rw": None, "errors": Some("remount-ro")} }, MountInfo { mnt_id: 2711, pid: 2710, majmin: "0:150", root: "/", mount_point: "/var/lib/quark/7640b5c2192f4204a718c05e77323359184738705b98fd8cbd20b769d4cd1c6d/7640b5c2192f4204a718c05e77323359184738705b98fd8cbd20b769d4cd1c6d", mount_options: {"relatime": None, "rw": None}, opt_fields: [Master(747)], fs_type: "overlay", mount_source: Some("overlay"), super_options: {"xino": Some("off"), "rw": None, "workdir": Some("/var/lib/docker/overlay2/994ab8eb1be63f2c46313c8c66ec7ee93d0a1eb619121714765f9e4a7da26819/work"), "lowerdir": Some("/var/lib/docker/overlay2/l/GTRFVQ647CBJIJWSIZEWHEFNCO:/var/lib/docker/overlay2/l/M6UYENVGI6HTDHORILUP6A5PYW"), "upperdir": Some("/var/lib/docker/overlay2/994ab8eb1be63f2c46313c8c66ec7ee93d0a1eb619121714765f9e4a7da26819/diff")} }, MountInfo { mnt_id: 2712, pid: 2636, majmin: "0:150", root: "/", mount_point: "/var/lib/quark/7640b5c2192f4204a718c05e77323359184738705b98fd8cbd20b769d4cd1c6d/7640b5c2192f4204a718c05e77323359184738705b98fd8cbd20b769d4cd1c6d", mount_options: {"relatime": None, "rw": None}, opt_fields: [Master(747)], fs_type: "overlay", mount_source: Some("overlay"), super_options: {"upperdir": Some("/var/lib/docker/overlay2/994ab8eb1be63f2c46313c8c66ec7ee93d0a1eb619121714765f9e4a7da26819/diff"), "xino": Some("off"), "workdir": Some("/var/lib/docker/overlay2/994ab8eb1be63f2c46313c8c66ec7ee93d0a1eb619121714765f9e4a7da26819/work"), "lowerdir": Some("/var/lib/docker/overlay2/l/GTRFVQ647CBJIJWSIZEWHEFNCO:/var/lib/docker/overlay2/l/M6UYENVGI6HTDHORILUP6A5PYW"), "rw": None} }, MountInfo { mnt_id: 2713, pid: 2636, majmin: "253:1", root: "/var/lib/quark/495af967123899950202f6c7675706618018569b284a6f929fe90b86dd71ad54", mount_point: "/var/lib/quark/495af967123899950202f6c7675706618018569b284a6f929fe90b86dd71ad54", mount_options: {"relatime": None, "rw": None}, opt_fields: [Master(1)], fs_type: "ext4", mount_source: Some("/dev/mapper/Atlas--179--vg-root"), super_options: {"errors": Some("remount-ro"), "stripe": Some("64"), "rw": None} }, MountInfo { mnt_id: 2714, pid: 2713, majmin: "0:151", root: "/", mount_point: "/var/lib/quark/495af967123899950202f6c7675706618018569b284a6f929fe90b86dd71ad54/495af967123899950202f6c7675706618018569b284a6f929fe90b86dd71ad54", mount_options: {"relatime": None, "rw": None}, opt_fields: [Master(47)], fs_type: "overlay", mount_source: Some("overlay"), super_options: {"rw": None, "workdir": Some("/var/lib/docker/overlay2/5974f4d9d6ba020a0be6ddd51c446fe2878239f0fe9e063d4b0041650aa8daf7/work"), "lowerdir": Some("/var/lib/docker/overlay2/l/2G56ZNAP6JATVEJJPIICWGGLIY:/var/lib/docker/overlay2/l/M6UYENVGI6HTDHORILUP6A5PYW"), "xino": Some("off"), "upperdir": Some("/var/lib/docker/overlay2/5974f4d9d6ba020a0be6ddd51c446fe2878239f0fe9e063d4b0041650aa8daf7/diff")} }, MountInfo { mnt_id: 2715, pid: 2636, majmin: "0:151", root: "/", mount_point: "/var/lib/quark/495af967123899950202f6c7675706618018569b284a6f929fe90b86dd71ad54/495af967123899950202f6c7675706618018569b284a6f929fe90b86dd71ad54", mount_options: {"relatime": None, "rw": None}, opt_fields: [Master(47)], fs_type: "overlay", mount_source: Some("overlay"), super_options: {"xino": Some("off"), "workdir": Some("/var/lib/docker/overlay2/5974f4d9d6ba020a0be6ddd51c446fe2878239f0fe9e063d4b0041650aa8daf7/work"), "upperdir": Some("/var/lib/docker/overlay2/5974f4d9d6ba020a0be6ddd51c446fe2878239f0fe9e063d4b0041650aa8daf7/diff"), "lowerdir": Some("/var/lib/docker/overlay2/l/2G56ZNAP6JATVEJJPIICWGGLIY:/var/lib/docker/overlay2/l/M6UYENVGI6HTDHORILUP6A5PYW"), "rw": None} }, MountInfo { mnt_id: 2716, pid: 2636, majmin: "253:1", root: "/var/lib/quark/c08892fa2b0934d483eada3e16284305c80e3762e64f5133fd038266896b9086", mount_point: "/var/lib/quark/c08892fa2b0934d483eada3e16284305c80e3762e64f5133fd038266896b9086", mount_options: {"rw": None, "relatime": None}, opt_fields: [Master(1)], fs_type: "ext4", mount_source: Some("/dev/mapper/Atlas--179--vg-root"), super_options: {"errors": Some("remount-ro"), "stripe": Some("64"), "rw": None} }, MountInfo { mnt_id: 2717, pid: 2716, majmin: "0:152", root: "/", mount_point: "/var/lib/quark/c08892fa2b0934d483eada3e16284305c80e3762e64f5133fd038266896b9086/c08892fa2b0934d483eada3e16284305c80e3762e64f5133fd038266896b9086", mount_options: {"relatime": None, "rw": None}, opt_fields: [Master(485)], fs_type: "overlay", mount_source: Some("overlay"), super_options: {"upperdir": Some("/var/lib/docker/overlay2/92a4e71b524875626edde68c19a6d2aae31f594831e6eede2c92f39262eeaf9c/diff"), "xino": Some("off"), "lowerdir": Some("/var/lib/docker/overlay2/l/GJLKRRVG2OWJHZDB25L2JPZDQQ:/var/lib/docker/overlay2/l/M6UYENVGI6HTDHORILUP6A5PYW"), "rw": None, "workdir": Some("/var/lib/docker/overlay2/92a4e71b524875626edde68c19a6d2aae31f594831e6eede2c92f39262eeaf9c/work")} }, MountInfo { mnt_id: 2718, pid: 2636, majmin: "0:152", root: "/", mount_point: "/var/lib/quark/c08892fa2b0934d483eada3e16284305c80e3762e64f5133fd038266896b9086/c08892fa2b0934d483eada3e16284305c80e3762e64f5133fd038266896b9086", mount_options: {"relatime": None, "rw": None}, opt_fields: [Master(485)], fs_type: "overlay", mount_source: Some("overlay"), super_options: {"rw": None, "lowerdir": Some("/var/lib/docker/overlay2/l/GJLKRRVG2OWJHZDB25L2JPZDQQ:/var/lib/docker/overlay2/l/M6UYENVGI6HTDHORILUP6A5PYW"), "workdir": Some("/var/lib/docker/overlay2/92a4e71b524875626edde68c19a6d2aae31f594831e6eede2c92f39262eeaf9c/work"), "xino": Some("off"), "upperdir": Some("/var/lib/docker/overlay2/92a4e71b524875626edde68c19a6d2aae31f594831e6eede2c92f39262eeaf9c/diff")} }, MountInfo { mnt_id: 2719, pid: 2636, majmin: "253:1", root: "/var/lib/quark/fbb61cdae8825f024cf410b0f0b8edd8c420414766fc97671ff5ccd90a030ffe", mount_point: "/var/lib/quark/fbb61cdae8825f024cf410b0f0b8edd8c420414766fc97671ff5ccd90a030ffe", mount_options: {"relatime": None, "rw": None}, opt_fields: [Master(1)], fs_type: "ext4", mount_source: Some("/dev/mapper/Atlas--179--vg-root"), super_options: {"errors": Some("remount-ro"), "rw": None, "stripe": Some("64")} }, MountInfo { mnt_id: 2720, pid: 2719, majmin: "0:153", root: "/", mount_point: "/var/lib/quark/fbb61cdae8825f024cf410b0f0b8edd8c420414766fc97671ff5ccd90a030ffe/fbb61cdae8825f024cf410b0f0b8edd8c420414766fc97671ff5ccd90a030ffe", mount_options: {"rw": None, "relatime": None}, opt_fields: [Master(528)], fs_type: "overlay", mount_source: Some("overlay"), super_options: {"lowerdir": Some("/var/lib/docker/overlay2/l/YO33ECSZZNUTWAYKNOBJVPSL5N:/var/lib/docker/overlay2/l/M6UYENVGI6HTDHORILUP6A5PYW"), "upperdir": Some("/var/lib/docker/overlay2/cb918ca3fca4d0659e47a250ca8c2dd7bab7db6358aa7a4b90d7456c51fa5269/diff"), "rw": None, "workdir": Some("/var/lib/docker/overlay2/cb918ca3fca4d0659e47a250ca8c2dd7bab7db6358aa7a4b90d7456c51fa5269/work"), "xino": Some("off")} }, MountInfo { mnt_id: 2721, pid: 2636, majmin: "0:153", root: "/", mount_point: "/var/lib/quark/fbb61cdae8825f024cf410b0f0b8edd8c420414766fc97671ff5ccd90a030ffe/fbb61cdae8825f024cf410b0f0b8edd8c420414766fc97671ff5ccd90a030ffe", mount_options: {"relatime": None, "rw": None}, opt_fields: [Master(528)], fs_type: "overlay", mount_source: Some("overlay"), super_options: {"workdir": Some("/var/lib/docker/overlay2/cb918ca3fca4d0659e47a250ca8c2dd7bab7db6358aa7a4b90d7456c51fa5269/work"), "lowerdir": Some("/var/lib/docker/overlay2/l/YO33ECSZZNUTWAYKNOBJVPSL5N:/var/lib/docker/overlay2/l/M6UYENVGI6HTDHORILUP6A5PYW"), "xino": Some("off"), "upperdir": Some("/var/lib/docker/overlay2/cb918ca3fca4d0659e47a250ca8c2dd7bab7db6358aa7a4b90d7456c51fa5269/diff"), "rw": None} }, MountInfo { mnt_id: 2722, pid: 2636, majmin: "253:1", root: "/var/lib/quark/d3a624bd476665dbfbd609d73fc9a05294e1ecd4a5a9599e9c5386d96e478acf", mount_point: "/var/lib/quark/d3a624bd476665dbfbd609d73fc9a05294e1ecd4a5a9599e9c5386d96e478acf", mount_options: {"rw": None, "relatime": None}, opt_fields: [Master(1)], fs_type: "ext4", mount_source: Some("/dev/mapper/Atlas--179--vg-root"), super_options: {"stripe": Some("64"), "rw": None, "errors": Some("remount-ro")} }, MountInfo { mnt_id: 2723, pid: 2722, majmin: "0:154", root: "/", mount_point: "/var/lib/quark/d3a624bd476665dbfbd609d73fc9a05294e1ecd4a5a9599e9c5386d96e478acf/d3a624bd476665dbfbd609d73fc9a05294e1ecd4a5a9599e9c5386d96e478acf", mount_options: {"rw": None, "relatime": None}, opt_fields: [Master(539)], fs_type: "overlay", mount_source: Some("overlay"), super_options: {"lowerdir": Some("/var/lib/docker/overlay2/l/J7FTHPPSFCYN4LAAMUOH3NYPXK:/var/lib/docker/overlay2/l/M6UYENVGI6HTDHORILUP6A5PYW"), "rw": None, "upperdir": Some("/var/lib/docker/overlay2/9ac6c91a2422aa2e6008150a3acb22a77d90c363022dbac58606ca05e6e55f62/diff"), "xino": Some("off"), "workdir": Some("/var/lib/docker/overlay2/9ac6c91a2422aa2e6008150a3acb22a77d90c363022dbac58606ca05e6e55f62/work")} }, MountInfo { mnt_id: 2724, pid: 2636, majmin: "0:154", root: "/", mount_point: "/var/lib/quark/d3a624bd476665dbfbd609d73fc9a05294e1ecd4a5a9599e9c5386d96e478acf/d3a624bd476665dbfbd609d73fc9a05294e1ecd4a5a9599e9c5386d96e478acf", mount_options: {"rw": None, "relatime": None}, opt_fields: [Master(539)], fs_type: "overlay", mount_source: Some("overlay"), super_options: {"xino": Some("off"), "workdir": Some("/var/lib/docker/overlay2/9ac6c91a2422aa2e6008150a3acb22a77d90c363022dbac58606ca05e6e55f62/work"), "upperdir": Some("/var/lib/docker/overlay2/9ac6c91a2422aa2e6008150a3acb22a77d90c363022dbac58606ca05e6e55f62/diff"), "rw": None, "lowerdir": Some("/var/lib/docker/overlay2/l/J7FTHPPSFCYN4LAAMUOH3NYPXK:/var/lib/docker/overlay2/l/M6UYENVGI6HTDHORILUP6A5PYW")} }, MountInfo { mnt_id: 2725, pid: 2636, majmin: "253:1", root: "/var/lib/quark/7647628ea8cf0a0a4939cc78682f8756103af1dc7aad358c61fc03675a4054ff", mount_point: "/var/lib/quark/7647628ea8cf0a0a4939cc78682f8756103af1dc7aad358c61fc03675a4054ff", mount_options: {"rw": None, "relatime": None}, opt_fields: [Master(1)], fs_type: "ext4", mount_source: Some("/dev/mapper/Atlas--179--vg-root"), super_options: {"rw": None, "errors": Some("remount-ro"), "stripe": Some("64")} }, MountInfo { mnt_id: 2726, pid: 2725, majmin: "0:155", root: "/", mount_point: "/var/lib/quark/7647628ea8cf0a0a4939cc78682f8756103af1dc7aad358c61fc03675a4054ff/7647628ea8cf0a0a4939cc78682f8756103af1dc7aad358c61fc03675a4054ff", mount_options: {"rw": None, "relatime": None}, opt_fields: [Master(558)], fs_type: "overlay", mount_source: Some("overlay"), super_options: {"rw": None, "lowerdir": Some("/var/lib/docker/overlay2/l/ZT5MTRJHUFPJUNZEZELWP5GQJA:/var/lib/docker/overlay2/l/M6UYENVGI6HTDHORILUP6A5PYW"), "xino": Some("off"), "upperdir": Some("/var/lib/docker/overlay2/c46519a94685b488a010bea16a8f27feb74e6b76d87859641712dbafab89bf14/diff"), "workdir": Some("/var/lib/docker/overlay2/c46519a94685b488a010bea16a8f27feb74e6b76d87859641712dbafab89bf14/work")} }, MountInfo { mnt_id: 2727, pid: 2636, majmin: "0:155", root: "/", mount_point: "/var/lib/quark/7647628ea8cf0a0a4939cc78682f8756103af1dc7aad358c61fc03675a4054ff/7647628ea8cf0a0a4939cc78682f8756103af1dc7aad358c61fc03675a4054ff", mount_options: {"relatime": None, "rw": None}, opt_fields: [Master(558)], fs_type: "overlay", mount_source: Some("overlay"), super_options: {"rw": None, "lowerdir": Some("/var/lib/docker/overlay2/l/ZT5MTRJHUFPJUNZEZELWP5GQJA:/var/lib/docker/overlay2/l/M6UYENVGI6HTDHORILUP6A5PYW"), "upperdir": Some("/var/lib/docker/overlay2/c46519a94685b488a010bea16a8f27feb74e6b76d87859641712dbafab89bf14/diff"), "workdir": Some("/var/lib/docker/overlay2/c46519a94685b488a010bea16a8f27feb74e6b76d87859641712dbafab89bf14/work"), "xino": Some("off")} }, MountInfo { mnt_id: 2728, pid: 2636, majmin: "253:1", root: "/var/lib/quark/c7c6b39e598a1cf221a5e1abf213be5fb45b3d33fb110eaeb125d9f5e07e46dd", mount_point: "/var/lib/quark/c7c6b39e598a1cf221a5e1abf213be5fb45b3d33fb110eaeb125d9f5e07e46dd", mount_options: {"relatime": None, "rw": None}, opt_fields: [Master(1)], fs_type: "ext4", mount_source: Some("/dev/mapper/Atlas--179--vg-root"), super_options: {"rw": None, "errors": Some("remount-ro"), "stripe": Some("64")} }, MountInfo { mnt_id: 2729, pid: 2728, majmin: "0:156", root: "/", mount_point: "/var/lib/quark/c7c6b39e598a1cf221a5e1abf213be5fb45b3d33fb110eaeb125d9f5e07e46dd/c7c6b39e598a1cf221a5e1abf213be5fb45b3d33fb110eaeb125d9f5e07e46dd", mount_options: {"rw": None, "relatime": None}, opt_fields: [Master(573)], fs_type: "overlay", mount_source: Some("overlay"), super_options: {"rw": None, "xino": Some("off"), "upperdir": Some("/var/lib/docker/overlay2/f555e3d1d9512b75ca982cbae18e62622e37aab86bee11a58f07a77ebbedc93a/diff"), "lowerdir": Some("/var/lib/docker/overlay2/l/X4DIKCBWC5XZLHBLEP64HOWN4Q:/var/lib/docker/overlay2/l/M6UYENVGI6HTDHORILUP6A5PYW"), "workdir": Some("/var/lib/docker/overlay2/f555e3d1d9512b75ca982cbae18e62622e37aab86bee11a58f07a77ebbedc93a/work")} }, MountInfo { mnt_id: 2730, pid: 2636, majmin: "0:156", root: "/", mount_point: "/var/lib/quark/c7c6b39e598a1cf221a5e1abf213be5fb45b3d33fb110eaeb125d9f5e07e46dd/c7c6b39e598a1cf221a5e1abf213be5fb45b3d33fb110eaeb125d9f5e07e46dd", mount_options: {"relatime": None, "rw": None}, opt_fields: [Master(573)], fs_type: "overlay", mount_source: Some("overlay"), super_options: {"upperdir": Some("/var/lib/docker/overlay2/f555e3d1d9512b75ca982cbae18e62622e37aab86bee11a58f07a77ebbedc93a/diff"), "rw": None, "lowerdir": Some("/var/lib/docker/overlay2/l/X4DIKCBWC5XZLHBLEP64HOWN4Q:/var/lib/docker/overlay2/l/M6UYENVGI6HTDHORILUP6A5PYW"), "xino": Some("off"), "workdir": Some("/var/lib/docker/overlay2/f555e3d1d9512b75ca982cbae18e62622e37aab86bee11a58f07a77ebbedc93a/work")} }, MountInfo { mnt_id: 2731, pid: 2636, majmin: "0:157", root: "/", mount_point: "/var/lib/docker/overlay2/231ad2d4be4b03524a5289ae41e113679b657f918469ec4b93fe49e74475e78d/merged", mount_options: {"relatime": None, "rw": None}, opt_fields: [Master(592)], fs_type: "overlay", mount_source: Some("overlay"), super_options: {"upperdir": Some("/var/lib/docker/overlay2/231ad2d4be4b03524a5289ae41e113679b657f918469ec4b93fe49e74475e78d/diff"), "rw": None, "lowerdir": Some("/var/lib/docker/overlay2/l/4EAU3T2Y66PGRN3LPFUMXPP7F3:/var/lib/docker/overlay2/l/M6UYENVGI6HTDHORILUP6A5PYW"), "workdir": Some("/var/lib/docker/overlay2/231ad2d4be4b03524a5289ae41e113679b657f918469ec4b93fe49e74475e78d/work"), "xino": Some("off")} }, MountInfo { mnt_id: 2732, pid: 2636, majmin: "253:1", root: "/var/lib/quark/58892083db0ffddca9a7e6358070e7605b9a7a2ce5c38c622c313eceb76b6cac", mount_point: "/var/lib/quark/58892083db0ffddca9a7e6358070e7605b9a7a2ce5c38c622c313eceb76b6cac", mount_options: {"rw": None, "relatime": None}, opt_fields: [Master(1)], fs_type: "ext4", mount_source: Some("/dev/mapper/Atlas--179--vg-root"), super_options: {"stripe": Some("64"), "rw": None, "errors": Some("remount-ro")} }, MountInfo { mnt_id: 2733, pid: 2732, majmin: "0:157", root: "/", mount_point: "/var/lib/quark/58892083db0ffddca9a7e6358070e7605b9a7a2ce5c38c622c313eceb76b6cac/58892083db0ffddca9a7e6358070e7605b9a7a2ce5c38c622c313eceb76b6cac", mount_options: {"rw": None, "relatime": None}, opt_fields: [Master(592)], fs_type: "overlay", mount_source: Some("overlay"), super_options: {"rw": None, "workdir": Some("/var/lib/docker/overlay2/231ad2d4be4b03524a5289ae41e113679b657f918469ec4b93fe49e74475e78d/work"), "lowerdir": Some("/var/lib/docker/overlay2/l/4EAU3T2Y66PGRN3LPFUMXPP7F3:/var/lib/docker/overlay2/l/M6UYENVGI6HTDHORILUP6A5PYW"), "xino": Some("off"), "upperdir": Some("/var/lib/docker/overlay2/231ad2d4be4b03524a5289ae41e113679b657f918469ec4b93fe49e74475e78d/diff")} }, MountInfo { mnt_id: 2734, pid: 2636, majmin: "0:157", root: "/", mount_point: "/var/lib/quark/58892083db0ffddca9a7e6358070e7605b9a7a2ce5c38c622c313eceb76b6cac/58892083db0ffddca9a7e6358070e7605b9a7a2ce5c38c622c313eceb76b6cac", mount_options: {"rw": None, "relatime": None}, opt_fields: [Master(592)], fs_type: "overlay", mount_source: Some("overlay"), super_options: {"lowerdir": Some("/var/lib/docker/overlay2/l/4EAU3T2Y66PGRN3LPFUMXPP7F3:/var/lib/docker/overlay2/l/M6UYENVGI6HTDHORILUP6A5PYW"), "workdir": Some("/var/lib/docker/overlay2/231ad2d4be4b03524a5289ae41e113679b657f918469ec4b93fe49e74475e78d/work"), "xino": Some("off"), "rw": None, "upperdir": Some("/var/lib/docker/overlay2/231ad2d4be4b03524a5289ae41e113679b657f918469ec4b93fe49e74475e78d/diff")} }, MountInfo { mnt_id: 2736, pid: 2732, majmin: "253:1", root: "/var/lib/quark/58892083db0ffddca9a7e6358070e7605b9a7a2ce5c38c622c313eceb76b6cac", mount_point: "/var/lib/quark/58892083db0ffddca9a7e6358070e7605b9a7a2ce5c38c622c313eceb76b6cac", mount_options: {"relatime": None, "rw": None}, opt_fields: [Master(1)], fs_type: "ext4", mount_source: Some("/dev/mapper/Atlas--179--vg-root"), super_options: {"rw": None, "stripe": Some("64"), "errors": Some("remount-ro")} }, MountInfo { mnt_id: 2737, pid: 2736, majmin: "0:157", root: "/", mount_point: "/var/lib/quark/58892083db0ffddca9a7e6358070e7605b9a7a2ce5c38c622c313eceb76b6cac/58892083db0ffddca9a7e6358070e7605b9a7a2ce5c38c622c313eceb76b6cac", mount_options: {"relatime": None, "rw": None}, opt_fields: [Master(592)], fs_type: "overlay", mount_source: Some("overlay"), super_options: {"upperdir": Some("/var/lib/docker/overlay2/231ad2d4be4b03524a5289ae41e113679b657f918469ec4b93fe49e74475e78d/diff"), "rw": None, "xino": Some("off"), "workdir": Some("/var/lib/docker/overlay2/231ad2d4be4b03524a5289ae41e113679b657f918469ec4b93fe49e74475e78d/work"), "lowerdir": Some("/var/lib/docker/overlay2/l/4EAU3T2Y66PGRN3LPFUMXPP7F3:/var/lib/docker/overlay2/l/M6UYENVGI6HTDHORILUP6A5PYW")} }, MountInfo { mnt_id: 2738, pid: 2731, majmin: "0:5", root: "/", mount_point: "/var/lib/docker/overlay2/231ad2d4be4b03524a5289ae41e113679b657f918469ec4b93fe49e74475e78d/merged/proc", mount_options: {"relatime": None, "rw": None, "nodev": None, "noexec": None, "nosuid": None}, opt_fields: [], fs_type: "proc", mount_source: Some("proc"), super_options: {"rw": None} }, MountInfo { mnt_id: 2739, pid: 2731, majmin: "0:159", root: "/", mount_point: "/var/lib/docker/overlay2/231ad2d4be4b03524a5289ae41e113679b657f918469ec4b93fe49e74475e78d/merged/dev", mount_options: {"rw": None, "nosuid": None}, opt_fields: [], fs_type: "tmpfs", mount_source: Some("tmpfs"), super_options: {"size": Some("65536k"), "mode": Some("755"), "rw": None} }, MountInfo { mnt_id: 2740, pid: 2739, majmin: "0:160", root: "/", mount_point: "/var/lib/docker/overlay2/231ad2d4be4b03524a5289ae41e113679b657f918469ec4b93fe49e74475e78d/merged/dev/pts", mount_options: {"nosuid": None, "rw": None, "relatime": None, "noexec": None}, opt_fields: [], fs_type: "devpts", mount_source: Some("devpts"), super_options: {"ptmxmode": Some("666"), "rw": None, "mode": Some("620"), "gid": Some("5")} }, MountInfo { mnt_id: 2741, pid: 2731, majmin: "0:161", root: "/", mount_point: "/var/lib/docker/overlay2/231ad2d4be4b03524a5289ae41e113679b657f918469ec4b93fe49e74475e78d/merged/sys", mount_options: {"ro": None, "nosuid": None, "noexec": None, "nodev": None, "relatime": None}, opt_fields: [], fs_type: "sysfs", mount_source: Some("sysfs"), super_options: {"ro": None} }, MountInfo { mnt_id: 2742, pid: 2739, majmin: "0:158", root: "/", mount_point: "/var/lib/docker/overlay2/231ad2d4be4b03524a5289ae41e113679b657f918469ec4b93fe49e74475e78d/merged/dev/mqueue", mount_options: {"noexec": None, "relatime": None, "rw": None, "nodev": None, "nosuid": None}, opt_fields: [], fs_type: "mqueue", mount_source: Some("mqueue"), super_options: {"rw": None} }, MountInfo { mnt_id: 2743, pid: 2739, majmin: "0:162", root: "/", mount_point: "/var/lib/docker/overlay2/231ad2d4be4b03524a5289ae41e113679b657f918469ec4b93fe49e74475e78d/merged/dev/shm", mount_options: {"rw": None, "relatime": None, "nodev": None, "noexec": None, "nosuid": None}, opt_fields: [], fs_type: "tmpfs", mount_source: Some("shm"), super_options: {"size": Some("65536k"), "rw": None} }, MountInfo { mnt_id: 2744, pid: 2731, majmin: "253:1", root: "/var/lib/docker/containers/58892083db0ffddca9a7e6358070e7605b9a7a2ce5c38c622c313eceb76b6cac/resolv.conf", mount_point: "/var/lib/docker/overlay2/231ad2d4be4b03524a5289ae41e113679b657f918469ec4b93fe49e74475e78d/merged/etc/resolv.conf", mount_options: {"relatime": None, "rw": None}, opt_fields: [Master(1)], fs_type: "ext4", mount_source: Some("/dev/mapper/Atlas--179--vg-root"), super_options: {"rw": None, "errors": Some("remount-ro"), "stripe": Some("64")} }, MountInfo { mnt_id: 2745, pid: 2731, majmin: "253:1", root: "/var/lib/docker/containers/58892083db0ffddca9a7e6358070e7605b9a7a2ce5c38c622c313eceb76b6cac/hostname", mount_point: "/var/lib/docker/overlay2/231ad2d4be4b03524a5289ae41e113679b657f918469ec4b93fe49e74475e78d/merged/etc/hostname", mount_options: {"rw": None, "relatime": None}, opt_fields: [Master(1)], fs_type: "ext4", mount_source: Some("/dev/mapper/Atlas--179--vg-root"), super_options: {"errors": Some("remount-ro"), "stripe": Some("64"), "rw": None} }, MountInfo { mnt_id: 2746, pid: 2731, majmin: "253:1", root: "/var/lib/docker/containers/58892083db0ffddca9a7e6358070e7605b9a7a2ce5c38c622c313eceb76b6cac/hosts", mount_point: "/var/lib/docker/overlay2/231ad2d4be4b03524a5289ae41e113679b657f918469ec4b93fe49e74475e78d/merged/etc/hosts", mount_options: {"rw": None, "relatime": None}, opt_fields: [Master(1)], fs_type: "ext4", mount_source: Some("/dev/mapper/Atlas--179--vg-root"), super_options: {"rw": None, "errors": Some("remount-ro"), "stripe": Some("64")} }, MountInfo { mnt_id: 2747, pid: 2737, majmin: "0:157", root: "/", mount_point: "/var/lib/quark/58892083db0ffddca9a7e6358070e7605b9a7a2ce5c38c622c313eceb76b6cac/58892083db0ffddca9a7e6358070e7605b9a7a2ce5c38c622c313eceb76b6cac", mount_options: {"relatime": None, "rw": None}, opt_fields: [Master(592)], fs_type: "overlay", mount_source: Some("overlay"), super_options: {"lowerdir": Some("/var/lib/docker/overlay2/l/4EAU3T2Y66PGRN3LPFUMXPP7F3:/var/lib/docker/overlay2/l/M6UYENVGI6HTDHORILUP6A5PYW"), "upperdir": Some("/var/lib/docker/overlay2/231ad2d4be4b03524a5289ae41e113679b657f918469ec4b93fe49e74475e78d/diff"), "xino": Some("off"), "rw": None, "workdir": Some("/var/lib/docker/overlay2/231ad2d4be4b03524a5289ae41e113679b657f918469ec4b93fe49e74475e78d/work")} }, MountInfo { mnt_id: 2748, pid: 2747, majmin: "0:5", root: "/", mount_point: "/var/lib/quark/58892083db0ffddca9a7e6358070e7605b9a7a2ce5c38c622c313eceb76b6cac/58892083db0ffddca9a7e6358070e7605b9a7a2ce5c38c622c313eceb76b6cac/proc", mount_options: {"noexec": None, "rw": None, "nodev": None, "relatime": None, "nosuid": None}, opt_fields: [], fs_type: "proc", mount_source: Some("proc"), super_options: {"rw": None} }, MountInfo { mnt_id: 2749, pid: 2747, majmin: "0:159", root: "/", mount_point: "/var/lib/quark/58892083db0ffddca9a7e6358070e7605b9a7a2ce5c38c622c313eceb76b6cac/58892083db0ffddca9a7e6358070e7605b9a7a2ce5c38c622c313eceb76b6cac/dev", mount_options: {"rw": None, "nosuid": None}, opt_fields: [], fs_type: "tmpfs", mount_source: Some("tmpfs"), super_options: {"size": Some("65536k"), "mode": Some("755"), "rw": None} }, MountInfo { mnt_id: 2750, pid: 2749, majmin: "0:160", root: "/", mount_point: "/var/lib/quark/58892083db0ffddca9a7e6358070e7605b9a7a2ce5c38c622c313eceb76b6cac/58892083db0ffddca9a7e6358070e7605b9a7a2ce5c38c622c313eceb76b6cac/dev/pts", mount_options: {"rw": None, "nosuid": None, "noexec": None, "relatime": None}, opt_fields: [], fs_type: "devpts", mount_source: Some("devpts"), super_options: {"rw": None, "gid": Some("5"), "mode": Some("620"), "ptmxmode": Some("666")} }, MountInfo { mnt_id: 2751, pid: 2749, majmin: "0:158", root: "/", mount_point: "/var/lib/quark/58892083db0ffddca9a7e6358070e7605b9a7a2ce5c38c622c313eceb76b6cac/58892083db0ffddca9a7e6358070e7605b9a7a2ce5c38c622c313eceb76b6cac/dev/mqueue", mount_options: {"nosuid": None, "rw": None, "noexec": None, "nodev": None, "relatime": None}, opt_fields: [], fs_type: "mqueue", mount_source: Some("mqueue"), super_options: {"rw": None} }, MountInfo { mnt_id: 2752, pid: 2749, majmin: "0:162", root: "/", mount_point: "/var/lib/quark/58892083db0ffddca9a7e6358070e7605b9a7a2ce5c38c622c313eceb76b6cac/58892083db0ffddca9a7e6358070e7605b9a7a2ce5c38c622c313eceb76b6cac/dev/shm", mount_options: {"rw": None, "noexec": None, "relatime": None, "nosuid": None, "nodev": None}, opt_fields: [], fs_type: "tmpfs", mount_source: Some("shm"), super_options: {"size": Some("65536k"), "rw": None} }, MountInfo { mnt_id: 2753, pid: 2747, majmin: "0:161", root: "/", mount_point: "/var/lib/quark/58892083db0ffddca9a7e6358070e7605b9a7a2ce5c38c622c313eceb76b6cac/58892083db0ffddca9a7e6358070e7605b9a7a2ce5c38c622c313eceb76b6cac/sys", mount_options: {"ro": None, "nosuid": None, "nodev": None, "relatime": None, "noexec": None}, opt_fields: [], fs_type: "sysfs", mount_source: Some("sysfs"), super_options: {"ro": None} }, MountInfo { mnt_id: 2754, pid: 2747, majmin: "253:1", root: "/var/lib/docker/containers/58892083db0ffddca9a7e6358070e7605b9a7a2ce5c38c622c313eceb76b6cac/resolv.conf", mount_point: "/var/lib/quark/58892083db0ffddca9a7e6358070e7605b9a7a2ce5c38c622c313eceb76b6cac/58892083db0ffddca9a7e6358070e7605b9a7a2ce5c38c622c313eceb76b6cac/etc/resolv.conf", mount_options: {"relatime": None, "rw": None}, opt_fields: [Master(1)], fs_type: "ext4", mount_source: Some("/dev/mapper/Atlas--179--vg-root"), super_options: {"rw": None, "stripe": Some("64"), "errors": Some("remount-ro")} }, MountInfo { mnt_id: 2755, pid: 2747, majmin: "253:1", root: "/var/lib/docker/containers/58892083db0ffddca9a7e6358070e7605b9a7a2ce5c38c622c313eceb76b6cac/hostname", mount_point: "/var/lib/quark/58892083db0ffddca9a7e6358070e7605b9a7a2ce5c38c622c313eceb76b6cac/58892083db0ffddca9a7e6358070e7605b9a7a2ce5c38c622c313eceb76b6cac/etc/hostname", mount_options: {"rw": None, "relatime": None}, opt_fields: [Master(1)], fs_type: "ext4", mount_source: Some("/dev/mapper/Atlas--179--vg-root"), super_options: {"rw": None, "errors": Some("remount-ro"), "stripe": Some("64")} }, MountInfo { mnt_id: 2756, pid: 2747, majmin: "253:1", root: "/var/lib/docker/containers/58892083db0ffddca9a7e6358070e7605b9a7a2ce5c38c622c313eceb76b6cac/hosts", mount_point: "/var/lib/quark/58892083db0ffddca9a7e6358070e7605b9a7a2ce5c38c622c313eceb76b6cac/58892083db0ffddca9a7e6358070e7605b9a7a2ce5c38c622c313eceb76b6cac/etc/hosts", mount_options: {"rw": None, "relatime": None}, opt_fields: [Master(1)], fs_type: "ext4", mount_source: Some("/dev/mapper/Atlas--179--vg-root"), super_options: {"errors": Some("remount-ro"), "rw": None, "stripe": Some("64")} }, MountInfo { mnt_id: 2757, pid: 2736, majmin: "0:157", root: "/", mount_point: "/var/lib/quark/58892083db0ffddca9a7e6358070e7605b9a7a2ce5c38c622c313eceb76b6cac/tmp", mount_options: {"relatime": None, "rw": None}, opt_fields: [Master(592)], fs_type: "overlay", mount_source: Some("overlay"), super_options: {"rw": None, "workdir": Some("/var/lib/docker/overlay2/231ad2d4be4b03524a5289ae41e113679b657f918469ec4b93fe49e74475e78d/work"), "xino": Some("off"), "lowerdir": Some("/var/lib/docker/overlay2/l/4EAU3T2Y66PGRN3LPFUMXPP7F3:/var/lib/docker/overlay2/l/M6UYENVGI6HTDHORILUP6A5PYW"), "upperdir": Some("/var/lib/docker/overlay2/231ad2d4be4b03524a5289ae41e113679b657f918469ec4b93fe49e74475e78d/diff")} }, MountInfo { mnt_id: 2758, pid: 2757, majmin: "0:5", root: "/", mount_point: "/var/lib/quark/58892083db0ffddca9a7e6358070e7605b9a7a2ce5c38c622c313eceb76b6cac/tmp/proc", mount_options: {"rw": None, "noexec": None, "nodev": None, "nosuid": None, "relatime": None}, opt_fields: [], fs_type: "proc", mount_source: Some("proc"), super_options: {"rw": None} }, MountInfo { mnt_id: 2759, pid: 2757, majmin: "0:159", root: "/", mount_point: "/var/lib/quark/58892083db0ffddca9a7e6358070e7605b9a7a2ce5c38c622c313eceb76b6cac/tmp/dev", mount_options: {"nosuid": None, "rw": None}, opt_fields: [], fs_type: "tmpfs", mount_source: Some("tmpfs"), super_options: {"size": Some("65536k"), "rw": None, "mode": Some("755")} }, MountInfo { mnt_id: 2760, pid: 2759, majmin: "0:160", root: "/", mount_point: "/var/lib/quark/58892083db0ffddca9a7e6358070e7605b9a7a2ce5c38c622c313eceb76b6cac/tmp/dev/pts", mount_options: {"relatime": None, "nosuid": None, "rw": None, "noexec": None}, opt_fields: [], fs_type: "devpts", mount_source: Some("devpts"), super_options: {"gid": Some("5"), "ptmxmode": Some("666"), "rw": None, "mode": Some("620")} }, MountInfo { mnt_id: 2761, pid: 2759, majmin: "0:158", root: "/", mount_point: "/var/lib/quark/58892083db0ffddca9a7e6358070e7605b9a7a2ce5c38c622c313eceb76b6cac/tmp/dev/mqueue", mount_options: {"nosuid": None, "relatime": None, "rw": None, "noexec": None, "nodev": None}, opt_fields: [], fs_type: "mqueue", mount_source: Some("mqueue"), super_options: {"rw": None} }, MountInfo { mnt_id: 2762, pid: 2759, majmin: "0:162", root: "/", mount_point: "/var/lib/quark/58892083db0ffddca9a7e6358070e7605b9a7a2ce5c38c622c313eceb76b6cac/tmp/dev/shm", mount_options: {"nodev": None, "noexec": None, "nosuid": None, "relatime": None, "rw": None}, opt_fields: [], fs_type: "tmpfs", mount_source: Some("shm"), super_options: {"rw": None, "size": Some("65536k")} }, MountInfo { mnt_id: 2763, pid: 2757, majmin: "0:161", root: "/", mount_point: "/var/lib/quark/58892083db0ffddca9a7e6358070e7605b9a7a2ce5c38c622c313eceb76b6cac/tmp/sys", mount_options: {"ro": None, "nosuid": None, "relatime": None, "noexec": None, "nodev": None}, opt_fields: [], fs_type: "sysfs", mount_source: Some("sysfs"), super_options: {"ro": None} }, MountInfo { mnt_id: 2764, pid: 2757, majmin: "253:1", root: "/var/lib/docker/containers/58892083db0ffddca9a7e6358070e7605b9a7a2ce5c38c622c313eceb76b6cac/resolv.conf", mount_point: "/var/lib/quark/58892083db0ffddca9a7e6358070e7605b9a7a2ce5c38c622c313eceb76b6cac/tmp/etc/resolv.conf", mount_options: {"rw": None, "relatime": None}, opt_fields: [Master(1)], fs_type: "ext4", mount_source: Some("/dev/mapper/Atlas--179--vg-root"), super_options: {"rw": None, "errors": Some("remount-ro"), "stripe": Some("64")} }, MountInfo { mnt_id: 2765, pid: 2757, majmin: "253:1", root: "/var/lib/docker/containers/58892083db0ffddca9a7e6358070e7605b9a7a2ce5c38c622c313eceb76b6cac/hostname", mount_point: "/var/lib/quark/58892083db0ffddca9a7e6358070e7605b9a7a2ce5c38c622c313eceb76b6cac/tmp/etc/hostname", mount_options: {"relatime": None, "rw": None}, opt_fields: [Master(1)], fs_type: "ext4", mount_source: Some("/dev/mapper/Atlas--179--vg-root"), super_options: {"errors": Some("remount-ro"), "stripe": Some("64"), "rw": None} }, MountInfo { mnt_id: 2766, pid: 2757, majmin: "253:1", root: "/var/lib/docker/containers/58892083db0ffddca9a7e6358070e7605b9a7a2ce5c38c622c313eceb76b6cac/hosts", mount_point: "/var/lib/quark/58892083db0ffddca9a7e6358070e7605b9a7a2ce5c38c622c313eceb76b6cac/tmp/etc/hosts", mount_options: {"relatime": None, "rw": None}, opt_fields: [Master(1)], fs_type: "ext4", mount_source: Some("/dev/mapper/Atlas--179--vg-root"), super_options: {"stripe": Some("64"), "rw": None, "errors": Some("remount-ro")} }]
[INFO] [0/28587370261113] commandline args is /usr/local/bin/quark --root /var/run/docker/runtime-quark/moby --log /run/containerd/io.containerd.runtime.v1.linux/moby/58892083db0ffddca9a7e6358070e7605b9a7a2ce5c38c622c313eceb76b6cac/log.json --log-format json start 58892083db0ffddca9a7e6358070e7605b9a7a2ce5c38c622c313eceb76b6cac 
[INFO] [0/28587370262382] Container:: start ....
[INFO] [0/28587370262438] Load metadata for container /var/run/docker/runtime-quark/moby 58892083db0ffddca9a7e6358070e7605b9a7a2ce5c38c622c313eceb76b6cac
[INFO] [0/28587370264544] reset umask from 22 to 0, kernelMemRegionSize is 18
[INFO] [0/28587370265970] SetMemRegion phyAddr = 4000000000, hostAddr=4000000000; pageMmapsize = 6000 MB
[INFO] [0/28587370266198] set map region start=4000000000, end=4600000000
[INFO] [0/28587370267121] KernelMap1G start is 4000000000, end is 4600000000
[INFO] [0/28587370267176] KernelMap start is 3ffffff000, end is 4000000000
[INFO] [0/28587370267173] metadatafile is /var/run/docker/runtime-quark/moby/58892083db0ffddca9a7e6358070e7605b9a7a2ce5c38c622c313eceb76b6cac/meta.json
[INFO] [0/28587370285806] elfFile header Header { pt1: HeaderPt1 { magic: [127, 69, 76, 70], class: SixtyFour, data: LittleEndian, version: Current, os_abi: SystemV, abi_version: 0, padding: [0, 0, 0, 0, 0, 0, 0] }, pt2: Header64(HeaderPt2_ { type_: Executable, machine: AArch64, version: 1, entry_point: 274877911192, ph_offset: 64, sh_offset: 3639840, flags: 0, header_size: 64, ph_entry_size: 56, ph_count: 2, sh_entry_size: 64, sh_count: 6626, sh_str_index: 6625 }) }
[INFO] [0/28587370285907] program header: ProgramHeader64 { type_: Ok(Load), flags: Flags(7), offset: 65536, virtual_addr: 274877906944, physical_addr: 274877906944, file_size: 2414720, mem_size: 2420320, align: 65536 }
[INFO] [0/28587370285921] start mem 4000000000, len 24e000, offset 10000
[INFO] [0/28587370286010] program header: ProgramHeader64 { type_: Ok(OsSpecific(1685382481)), flags: Flags(6), offset: 0, virtual_addr: 0, physical_addr: 0, file_size: 0, mem_size: 0, align: 16 }
[INFO] [0/28587370287442] start enter guest[0]: entry is 4000001098, stack is 45400a0000
[DEBUG] [0/4540080000|0] init singleton finished
[DEBUG] [0/4540080000|0] init time keeper finished
[DEBUG] [0/4540080000|0] init vsyscall finished
[DEBUG] [0/4540080000|0] init host epoll fd finished
[DEBUG] [0/4540080000|0] init vdso finished
[INFO] [0/1270] shareSpace ready...
[INFO] [1/1381] cpu#1 start
[INFO] [1/1409] start enter guest[1]: entry is 4000001098, stack is 45400c0000
[INFO] [2/1413] cpu#2 start
[INFO] [2/1441] start enter guest[2]: entry is 4000001098, stack is 45400e0000
[DEBUG] [1/45400a0000|0] heap start is 4340000000
[INFO] [0/28587370304074] Start container 58892083db0ffddca9a7e6358070e7605b9a7a2ce5c38c622c313eceb76b6cac
[DEBUG] [0/28587370304305] executing hook ddd Hook { path: "/proc/1969107/exe", args: ["libnetwork-setkey", "-exec-root=/var/run/docker", "58892083db0ffddca9a7e6358070e7605b9a7a2ce5c38c622c313eceb76b6cac", "5ec58dc378e4"], env: [], timeout: None }
[INFO] [0/28587371041470] SandboxConnect connect address is  qvisor-sandbox.58892083db0ffddca9a7e6358070e7605b9a7a2ce5c38c622c313eceb76b6cac
[ERROR] [2/28780854] OOM!!! cpu [2], size is 100, alignment is 8
[ERROR] [0/28587526309438] the error is IOError("ReadWithFds io::error is ECONNRESET")

QuarkContainer commented 3 weeks ago

@mhomidi , would you please change the ReserveCpuCount in config.json to 180 to have a try? So far Quark doesn't support 64+ cpu cores.

[edit] Sorry. I see your vcpu count has decreased by -cpus=3 . The instruction should be no use.

shrik3 commented 3 weeks ago

Vendor ID: 0x48

is that a Kunpeng cpu from HiSilicon? The code works fine on our Ampere Altra server (Neoverse-N1 or Arm8.2A), it also works on a raspberrypi XD.

Interesting that the code works differently on the same ISA... Also what's your host kernel (uname -a)?

chl337 commented 3 weeks ago

@mhomidi for completes could you also post on which kernel version are you running?

mhomidi commented 3 weeks ago

I am not sure about the vendor.

> uname -a
Linux Atlas-179 5.4.0-125-generic #141-Ubuntu SMP Wed Aug 10 14:03:41 UTC 2022 aarch64 aarch64 aarch64 GNU/Linux

shrik3 commented 3 weeks ago

feel free to take my local branch here: https://github.com/shrik3/Quark/tree/arm_dev_base

(you may need to change the makefile to fit your specific cpu model):

RFLAGS   = -C target-cpu=neoverse-n1

chl337 commented 3 weeks ago

@mhomidi IIRC the minimal number of vcpus needed is 2. Can you please try it?

mhomidi commented 3 weeks ago

Sorry for the late. I had an issue with the docker.

This is the log when --cpus=2:

[DEBUG] [0/28663389322953] Creating new sandbox for container 7b9ab0269bbd7f361bf5ffed59f0a7ad60980146bf8856f2a18d9dac9f080365
[INFO] [0/28663389364911] commandline args is /usr/local/bin/quark boot --pipefd 8 
[DEBUG] [0/28663389400250] Creating the sandboxRootDir at /var/lib/quark/7b9ab0269bbd7f361bf5ffed59f0a7ad60980146bf8856f2a18d9dac9f080365
[INFO] [0/28663389403989] EnableNamespace ToEnterNS is []
[INFO] [0/28663389404392] Save container 7b9ab0269bbd7f361bf5ffed59f0a7ad60980146bf8856f2a18d9dac9f080365
[DEBUG] [0/28663389421609] mounting 
 proc to 
 /var/lib/docker/overlay2/5f329cb63c8a0204d4b7b5607768616b21ba2fa6b45c9d9a6a1e4612c17aaa68/merged/proc as proc with data ''
[DEBUG] [0/28663389422137] mounting 
 tmpfs to 
 /var/lib/docker/overlay2/5f329cb63c8a0204d4b7b5607768616b21ba2fa6b45c9d9a6a1e4612c17aaa68/merged/dev as tmpfs with data 'mode=755,size=65536k'
[DEBUG] [0/28663389424540] mounting 
 devpts to 
 /var/lib/docker/overlay2/5f329cb63c8a0204d4b7b5607768616b21ba2fa6b45c9d9a6a1e4612c17aaa68/merged/dev/pts as devpts with data 'newinstance,ptmxmode=0666,mode=0620,gid=5'
[DEBUG] [0/28663389427197] mounting 
 sysfs to 
 /var/lib/docker/overlay2/5f329cb63c8a0204d4b7b5607768616b21ba2fa6b45c9d9a6a1e4612c17aaa68/merged/sys as sysfs with data ''
[DEBUG] [0/28663389429878] mounting 
 mqueue to 
 /var/lib/docker/overlay2/5f329cb63c8a0204d4b7b5607768616b21ba2fa6b45c9d9a6a1e4612c17aaa68/merged/dev/mqueue as mqueue with data ''
[DEBUG] [0/28663389430246] mounting 
 shm to 
 /var/lib/docker/overlay2/5f329cb63c8a0204d4b7b5607768616b21ba2fa6b45c9d9a6a1e4612c17aaa68/merged/dev/shm as tmpfs with data 'mode=1777,size=67108864'
[DEBUG] [0/28663389433308] mounting 
 /var/lib/docker/containers/7b9ab0269bbd7f361bf5ffed59f0a7ad60980146bf8856f2a18d9dac9f080365/resolv.conf to 
 /var/lib/docker/overlay2/5f329cb63c8a0204d4b7b5607768616b21ba2fa6b45c9d9a6a1e4612c17aaa68/merged/etc/resolv.conf as bind with data ''
[DEBUG] [0/28663389434824] mounting 
 /var/lib/docker/containers/7b9ab0269bbd7f361bf5ffed59f0a7ad60980146bf8856f2a18d9dac9f080365/hostname to 
 /var/lib/docker/overlay2/5f329cb63c8a0204d4b7b5607768616b21ba2fa6b45c9d9a6a1e4612c17aaa68/merged/etc/hostname as bind with data ''
[DEBUG] [0/28663389435517] mounting 
 /var/lib/docker/containers/7b9ab0269bbd7f361bf5ffed59f0a7ad60980146bf8856f2a18d9dac9f080365/hosts to 
 /var/lib/docker/overlay2/5f329cb63c8a0204d4b7b5607768616b21ba2fa6b45c9d9a6a1e4612c17aaa68/merged/etc/hosts as bind with data ''
[DEBUG] [0/28663389435960] mknoding /dev/null
[DEBUG] [0/28663389436035] mknoding /dev/zero
[DEBUG] [0/28663389436068] mknoding /dev/full
[DEBUG] [0/28663389436098] mknoding /dev/tty
[DEBUG] [0/28663389436134] mknoding /dev/urandom
[DEBUG] [0/28663389436162] mknoding /dev/random
[ERROR] [0/28663389436214] mount to /var/lib/docker/overlay2/5f329cb63c8a0204d4b7b5607768616b21ba2fa6b45c9d9a6a1e4612c17aaa68/merged /var/lib/quark/7b9ab0269bbd7f361bf5ffed59f0a7ad60980146bf8856f2a18d9dac9f080365/7b9ab0269bbd7f361bf5ffed59f0a7ad60980146bf8856f2a18d9dac9f080365
[INFO] [0/28663389443333] reset umask from 22 to 0, kernelMemRegionSize is 18
[INFO] [0/28663389444271] SetMemRegion phyAddr = 4000000000, hostAddr=4000000000; pageMmapsize = 6000 MB
[INFO] [0/28663389444497] set map region start=4000000000, end=4600000000
[INFO] [0/28663389445686] KernelMap1G start is 4000000000, end is 4600000000
[INFO] [0/28663389445708] KernelMap start is 3ffffff000, end is 4000000000
[INFO] [0/28663389451894] commandline args is /usr/local/bin/quark --root /var/run/docker/runtime-quark/moby --log /run/containerd/io.containerd.runtime.v1.linux/moby/7b9ab0269bbd7f361bf5ffed59f0a7ad60980146bf8856f2a18d9dac9f080365/log.json --log-format json start 7b9ab0269bbd7f361bf5ffed59f0a7ad60980146bf8856f2a18d9dac9f080365 
[INFO] [0/28663389453283] Container:: start ....
[INFO] [0/28663389453337] Load metadata for container /var/run/docker/runtime-quark/moby 7b9ab0269bbd7f361bf5ffed59f0a7ad60980146bf8856f2a18d9dac9f080365
[INFO] [0/28663389458444] metadatafile is /var/run/docker/runtime-quark/moby/7b9ab0269bbd7f361bf5ffed59f0a7ad60980146bf8856f2a18d9dac9f080365/meta.json
[INFO] [0/28663389463344] elfFile header Header { pt1: HeaderPt1 { magic: [127, 69, 76, 70], class: SixtyFour, data: LittleEndian, version: Current, os_abi: SystemV, abi_version: 0, padding: [0, 0, 0, 0, 0, 0, 0] }, pt2: Header64(HeaderPt2_ { type_: Executable, machine: AArch64, version: 1, entry_point: 274877911192, ph_offset: 64, sh_offset: 3639840, flags: 0, header_size: 64, ph_entry_size: 56, ph_count: 2, sh_entry_size: 64, sh_count: 6626, sh_str_index: 6625 }) }
[INFO] [0/28663389463401] program header: ProgramHeader64 { type_: Ok(Load), flags: Flags(7), offset: 65536, virtual_addr: 274877906944, physical_addr: 274877906944, file_size: 2414720, mem_size: 2420320, align: 65536 }
[INFO] [0/28663389463414] start mem 4000000000, len 24e000, offset 10000
[INFO] [0/28663389463489] program header: ProgramHeader64 { type_: Ok(OsSpecific(1685382481)), flags: Flags(6), offset: 0, virtual_addr: 0, physical_addr: 0, file_size: 0, mem_size: 0, align: 16 }
[INFO] [0/28663389464777] start enter guest[0]: entry is 4000001098, stack is 4340080000
[INFO] [0/1051] shareSpace ready...
[INFO] [1/1158] cpu#1 start
[INFO] [1/1183] start enter guest[1]: entry is 4000001098, stack is 45400c0000
[INFO] [0/28663389495626] Start container 7b9ab0269bbd7f361bf5ffed59f0a7ad60980146bf8856f2a18d9dac9f080365
[DEBUG] [0/28663389495912] executing hook ddd Hook { path: "/proc/3553117/exe", args: ["libnetwork-setkey", "-exec-root=/var/run/docker", "7b9ab0269bbd7f361bf5ffed59f0a7ad60980146bf8856f2a18d9dac9f080365", "6a063076712d"], env: [], timeout: None }
[INFO] [0/28663390489567] SandboxConnect connect address is  qvisor-sandbox.7b9ab0269bbd7f361bf5ffed59f0a7ad60980146bf8856f2a18d9dac9f080365
[ERROR] [1/9295787] OOM!!! cpu [1], size is 20000, alignment is 20000
[ERROR] [0/28663443396393] the error is IOError("ReadWithFds io::error is ECONNRESET")

chl337 commented 3 weeks ago

@mhomidi it seems as it fails in https://github.com/QuarkContainer/Quark/blob/2f6247a96acb44852b23da29228eef2902afa106/qlib/kernel/memmgr/buf_allocator.rs#L380 but I am not sure why...

P.S. Trivial, but does the host machine have enough resources? + I suppose, you haven't modified the Quark's config?

shrik3 commented 3 weeks ago

@mhomidi it seems as it fails in

https://github.com/QuarkContainer/Quark/blob/2f6247a96acb44852b23da29228eef2902afa106/qlib/kernel/memmgr/buf_allocator.rs#L380 but I am not sure why...

this does ring a bell. Basically the alloc is given a layout that is too much to handle. That's either a bad configuration from somewhere, or the passed layout is somehow corrupted. .. (but why am I now seeing a panic in the above log?)

mhomidi commented 3 weeks ago

@mhomidi it seems as it fails in

https://github.com/QuarkContainer/Quark/blob/2f6247a96acb44852b23da29228eef2902afa106/qlib/kernel/memmgr/buf_allocator.rs#L380

but I am not sure why... P.S. Trivial, but does the host machine have enough resources? + I suppose, you haven't modified the Quark's config?

I did not change anything in the configuration. I tried the kernel size in config but it does not change anything.

@shrik3

feel free to take my local branch here: https://github.com/shrik3/Quark/tree/arm_dev_base

(you may need to change the makefile to fit your specific cpu model):
RFLAGS   = -C target-cpu=neoverse-n1 
After using your code I figure out that the kernel always use hypercall 23 which is wait vcpu and no one comes and wake it up. Do you have any idea how this may happen?

[INFO] [0/29008753117166] reset umask from 22 to 0, kernelMemRegionSize is 18
[INFO] [0/29008753118680] SetMemRegion phyAddr = 4000000000, hostAddr=4000000000; pageMmapsize = 6000 MB
[INFO] [0/29008753119010] set map region start=4000000000, end=4600000000
[INFO] [0/29008753123253] KernelMap1G start is 4000000000, end is 4600000000
[INFO] [0/29008753123347] KernelMap start is 3ffffff000, end is 4000000000
[INFO] [0/29008753148853] elfFile header Header { pt1: HeaderPt1 { magic: [127, 69, 76, 70], class: SixtyFour, data: LittleEndian, version: Current, os_abi: SystemV, abi_version: 0, padding: [0, 0, 0, 0, 0, 0, 0] }, pt2: Header64(HeaderPt2_ { type_: Executable, machine: AArch64, version: 1, entry_point: 274877911192, ph_offset: 64, sh_offset: 27450920, flags: 0, header_size: 64, ph_entry_size: 56, ph_count: 2, sh_entry_size: 64, sh_count: 9628, sh_str_index: 9627 }) }
[INFO] [0/29008753148975] program header: ProgramHeader64 { type_: Ok(Load), flags: Flags(7), offset: 65536, virtual_addr: 274877906944, physical_addr: 274877906944, file_size: 2844048, mem_size: 2849665, align: 65536 }
[INFO] [0/29008753149013] start mem 4000000000, len 2b7000, offset 10000
[INFO] [0/29008753149290] program header: ProgramHeader64 { type_: Ok(OsSpecific(1685382481)), flags: Flags(6), offset: 0, virtual_addr: 0, physical_addr: 0, file_size: 0, mem_size: 0, align: 16 }
[INFO] [0/29008753151122] start enter guest[0]: entry is 4000001098, stack is 45400e0000
[ERROR] [0/29008753154652] hypercall id is 8
[DEBUG] [0/45400c0000|0] init singleton finished
[ERROR] [0/29008753154875] hypercall id is 15
[ERROR] [0/29008753154914] hypercall is Rdtsc(Rdtsc)
[ERROR] [0/29008753154971] hypercall id is 15
[ERROR] [0/29008753154995] hypercall is Rdtsc(Rdtsc)
[ERROR] [0/29008753155041] hypercall id is 15
[ERROR] [0/29008753155064] hypercall is Rdtsc(Rdtsc)
[ERROR] [0/29008753155106] hypercall id is 15
[ERROR] [0/29008753155138] hypercall is SetTscOffset(SetTscOffset { offset: 293952aac16c2 })
[ERROR] [0/1275] hypercall id is 19
[ERROR] [0/1309] hypercall id is 11
[ERROR] [0/1324] hypercall id is 11
[ERROR] [0/1342] hypercall id is 19
[ERROR] [0/1353] hypercall id is 11
[ERROR] [0/1363] hypercall id is 11
[ERROR] [0/1449] hypercall id is 8
[DEBUG] [0/45400c0000|0] init time keeper finished
[ERROR] [0/1480] hypercall id is 8
[DEBUG] [0/45400c0000|0] init vsyscall finished
[ERROR] [0/1552] hypercall id is 15
[ERROR] [0/1560] hypercall is EventfdWrite(EventfdWrite { fd: b })
[ERROR] [0/1577] hypercall id is 8
[DEBUG] [0/45400c0000|0] init host epoll fd finished
[ERROR] [0/1605] hypercall id is 8
[DEBUG] [0/45400c0000|0] init vdso finished
[ERROR] [0/1621] hypercall id is 24
[INFO] [0/1642] shareSpace ready...
[INFO] [1/1762] cpu#1 start
[INFO] [1/1808] start enter guest[1]: entry is 4000001098, stack is 4540100000
[ERROR] [1/3115] hypercall id is 8
[DEBUG] [1/45400e0000|0] heap start is 4340000000
[ERROR] [1/6232] hypercall id is 15
[ERROR] [1/6247] hypercall is EventfdWrite(EventfdWrite { fd: b })
[ERROR] [1/16301] hypercall id is 23
[INFO] [0/29008753826566] SandboxConnect connect address is  qvisor-sandbox.bffff8f5c4ff4e6554115cc9b9029fd1723a560fdd9d0b95867b8a48598eee3b
[ERROR] [0/201728] hypercall id is 16
[ERROR] [0/201772] iothread fd is 11
[ERROR] [1/28519055] hypercall id is 23
[ERROR] [1/28560345] hypercall id is 23
.
.
.
[ERROR] [1/28168275] hypercall id is 23
[ERROR] [1/28519055] hypercall id is 23
[ERROR] [1/28560345] hypercall id is 23
[ERROR] [1/28601277] hypercall id is 23
[ERROR] [1/28656796] hypercall id is 23
[ERROR] [0/29009044038117] the error is IOError("ReadWithFds io::error is ECONNRESET")

shrik3 commented 3 weeks ago

On Wed Jun 12, 2024 at 11:57 PM CEST, Hadi Omidi wrote:

After using your code I figure out that the kernel always use hypercall 23 which is wait vcpu and no one comes and wake it up. Do you have any idea how this may happen?

hypercall 23 is VCPU_WAIT, and this is normal.

[ERROR] [0/29009044038117] the error is IOError("ReadWithFds io::error is ECONNRESET")

so a host call is failing on the qvisor side with "connection reset by pair..."

this doesn't make sense for a hello-world ...

QuarkContainer / Quark

Stable version for ARM processors #1313