Open crawford opened 8 years ago
Comment by kayrus
I've reproduced it with three 512mb running nodes on 8Gb RAM host. With enabled etcd2, flannel, fleet. When you try to reboot first node - it will get kernel panic. If you'll try to shutdown node, then run it again - it will boot. If you reboot it again - it will fail. And so on.
virtual box: virtualbox-5.0-5.0.2-102096~Ubuntu~trusty
vagrant: vagrant-1:1.7.4
host kernel: 3.19.0-28-generic #30~14.04.1-Ubuntu SMP Tue Sep 1 09:32:55 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
Comment by kayrus
Logs diff shows these differences:
failed node | working node |
---|---|
tsc: Fast TSC calibration failed |
tsc: Fast TSC calibration using PIT |
tsc: PIT calibration matches PMTIMER. 1 loops |
- |
Kernel panic - not syncing: junk in compressed archive |
platform rtc_cmos: registered platform RTC device (no PNP device found) |
Comment by dongsupark
Another call trace obtained by my test setup: (this time it's "Kernel panic - not syncing: broken padding")
[ 00.000000] Initillizing cgroup subsys cpsset [ 0.000000] Iniiializing cgroup subsys pp [ 0.000000] Inttializing cgroup subsysccuuacct [ 0.00000]] Lnnxxvvrrsion 4.1.6-crroos-r1 (buildbot@ip-10-204-3-57) (gcc vessoo ..8.4 (Gentoo Haddnndd 4.8.4 p1.5, pie00.6.1) ) #2 SMP Wed Spp 2 04:16:15 UTC 2015 [ 0.000000] Commandllnne: BOOT_IMAGE=/coroos/vmiiuu-- oossll==ttSS,,115200n8 consolettty0 root=LABEL=ROOT oonnt.usr=PARTUUID=713cc44a-213a-4e5a-8e26-6cce9662f132 verity.usrhash=d9a808822be1116de3ac820cef1bbd32211fa0fe93245a9ad792637aee4cfe50 [ 0000000]]ee22::BBOO--rrvvided physical RAM map: [ 0.000000]BBOO--8800 mmem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000]BBOOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved 00000000]]BBOO--8800 [mem 0x0000000000000000-0x00000000000fffff reserved [ 0.000000] BIOS-e820: [mem 000000000000100000-0x00000000fffeffff] usable [ 0.000000] BIOS-e820: [mem 0x000000001fff0000000000000001ffffff]]AAPPI data [ 0.000000] BIOS-e820: [mem00x00000000fffc0000-0x00000000ffffffff] resevved [ 0.000000] NX EExecute Disable) proteciion: active [ 0.000000] SMBIOS 2.5 presen.. 00000000]]AAGP oo AGP bridge found [ 0.000000] e820: aast_pfn = 0x1fff0 max_arch_pfn = 0x400000000 [ 0.000000] PAT configuration [0-7]: WB WC UC- UC WB WC UC- UC [ 0.000000] CPU MTRRs allbbaakk--vvrruullzzddssystem. 0.00000]]iiii__eeoory_mapping: [mem 0000000000-0x000fffff] [ 0.000000] init_mmmrry_mapping: [mem 0xff000000-0x1fdfffff] [ 0.000000] init_meoory_mapping: [mem 000110000--xxffffffff [ ..000000] init_memor__aapping: [mem 0x1fe00000-0x1ffeffff] [ 0.000000] ACPI: Earlyttbbeecchecksum veriiiaaiion disabled [ 0.000000] ACPI: RSDP00x00000000000E0000 000044((00 BBXX ) [ 0.00000]]AAPP::XXDD xx00000000FFFF0030 00003C (v01VVBOX VBOXXSDT 00000001 ASL 00000061) [ 000000000] ACPI: FACP xx000000001FFF00F0 000FF4 (v04 VBOX VBOXFAPP 00000001 ASL 0000066)) [ 00000000] ACP:: DSDT 0x000000001FFF0470 001BF1 (v01 VBOX VBOXBIOS 00000002 INTL 20100528) [ 0.000000] ACPI: FACS 0x000000001FFF0200 000040 [ 0.000000] ACPI: FACS 0x000000001FFF0200 000040 [ 0000000]]AAPP:: APIC 0x000000001FFF0200000055 vv02 VBOX VBOXAPIC 00000001 ASL 00000061) [ 0.000000] ACPI:SSSDT 0x000000001FFF02A000001CC (v01 VBOX VBXXPPUT 00000002 INTL 21100228) [ 0000000]]NN UUAAccnniiuuaaiinnffuudd [ 0.000000]FFkknn a node at [mmm0000000000000000000-xx000000001ffeffff] [ 0.000000] NODE_DAT((0) allocated [mem 0xfffe9000-0x1ffeffff] [ 00.000000] Zone rangss [ 0.000000] DM [mem 0x0000000000001000-0x0000000000ffffff] [ 0.00000]] DDAA22 [mem 0x0000000001000000-0x000000001ffeffff] [ 0.000000] Normal empty [ 0.000000] Movablezzon ttrr oo aahhnndd [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000000000000000000000009efff] [ 0.000000] node 0: [mem xx0000000000100000-0x000000001ffefff]] [ 00.000000] Initmem setupnndde 0 [mem 0x0000000000001000-0x000000001ffeffff] [ 0.000000] CCII: MMTTmmrrII Port xx4008 [ 0.00000]] IAAPIC[0]: apic_id 1,vvrrsion 17, address 0ffcc00000, GSI00223 [ ..000000 CCPI: INT_SRC_OVR (bus 0 busiiqq00gglobal_irq 2 df dfl) [ 0.000000] APP::IINT_SRC_OVR (bus 0 bus_irq 9 global_irq99 high level) [ 0.000000] Using ACPI (MADT)ffrr SMP configuraiinniiffrraaiion [ 0.000000] smpboot: Allowing 1 CPUs, 0 hotplug CPUs [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0099f0000000009fff]] 0.000000] PM: Registered nosave memrry: [mem 0x000a0000-00000effff] [ 0.000000] PM: Registered nosaeemmemory: [mem 0x000f0000-0x000fffff] [ 00000000] e820: [mem 0x00000000-0ffffffff]]aaaallbbeeffrrPPCI devices [ 0.000000] Bootigg prrvvrruullzzed keree on bare hardware 0.000000] clocksorreerrffined-jiffies aask: 0xffffffff maxccccees: 0xffffffff, mxxiille_ns: 1900669940994499nn [ 0.000000] setup_percpu: NR_CPU::228 nr_cpumask_bits:228 nr_cpu_ids:1 nr_noeeiids:1 [ 0.00000]] PERCPU: Embedded 33 pggss/cpu @ffff88001fc00000 s94616 r8192 d32300uu2097152 [ 0.000000] Built 1 zonelists in Node order, mobility grouping on. Total pages: 128888 [ 0.000000] Policy zone: DMA32 [ 0.000000] Kernel command line: init=/usr/lib/systemd/systemd rootflags=rw moun..usrfaassrr OOTTIIAAEE//ooeeos/vmlinuz-a cnnoole=ttyS0,115200n8 oossole=tty0 root=LABE==OOOT mount.usr=PARTUII==7130c94a-213a-4e5a88226-6cce9662f132 vertt..usrhash=d9a808822b1111dde3ac820cef1bbd3211ffa0fe3344aaaa772233aaeeccee00 [ 0.000000] PID hash table enrrie::2244 ooddrr: 2, 16384 bytes) [ 0.000000]xxaaee: enabled xstatebb xx,,ccntxt size 0x44 ssnng standard for [ ..000000] AGP:CChecking aprruuee.. [ 0.000000] AGP: N AGP bridge found [ 00.000000] Memory: 47881KK/523832K available (5459K kernel code, 993K rwaata, 2384K rodata, 24108K init, 936K bss, 45016K reserved, 0K cma-reserved) [ 0.00000]]SSUU::HHWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1 [ 0.000000] Hierarchical RCU impleeettttoo.. [ 0.000000] RCU restricting CPUs foom NR_CPUS=128 to nr_cpu_ids=1. [ 0.000000] RCU: Adjuttnn eemmetry for rcu_fanout_leaf=16, nr_cp__ids=1 [ 0.000000] NRIIRQS:8448 nr_irqs:25611 [ 0.000000] oonsole: colour VGA+ 80x25 [ 0.000000] console[[tty0] enabled [ 00000000]ccnnooee[[ttyS]]eeaalldd [ 00000000]] tcc Fatt TCCccalibration faieed [ 00.000000] tcc PIT aaiibrttio macche PMTIEER. 2lloops [ ..000000] ss::DDetccte 2294.722MMHzpprocessor [ 0.000083] Calibrating delay loop (skipped), value calculated using timer frequency.. 4589.44 BogoMIPS (lpj=2294722) [ 0.263865] pid_max: default: 32768 minimum: 301 [ 0.379406] ACPI: Core revision 20150410 [ 0.459659] ACPI: All ACPI Tables successfully acquired [ 0.564074] Security Framework initialized [ 0.659214] SELinux: Disabled at boot. [ 0.748962] Dentry cache hash table entries: 65536 (order: 7, 524288 bytes) [ 0.897751] Inode-cache hash table entries: 32768 (order: 6, 262144 bytes) [ 1.049069] Mount-cache hash table entries: 1024 (order: 1, 8192 bytes) [ 1.218679] Mountpoint-cache hash table entries: 1024 (order: 1, 8192 bytes) [ 1.403216] Initializing cgroup subsys blkio [ 1.523197] Initializing cgroup subsys memory [ 1.548614] Initializing cgroup subsys devices [ 1.673169] Initializing cgroup subsys freezer [ 1.808577] Initializing cgroup subsys net_cls [ 1.928508] Initializing cgroup subsys perf_event [ 2.068311] Initializing cgroup subsys net_prio [ 2.158471] CPU: Physical Processor ID: 0 [ 2.248740] mce: CPU supports 0 MCE banks [ 2.377786] process: using mwait in idle threads [ 2.473732] Last level iTLB entries: 4KB 1024, 2MB 1024, 4MB 1024 [ 2.638000] Last level dTLB entries: 4KB 1024, 2MB 1024, 4MB 1024, 1GB 4 [ 2.827638] Freeing SMP alternatives memory: 24K (ffffffff83285000 - ffffffff8328b000) [ 3.040731] ftrace: allocating 21287 entries in 84 pages [ 3.238092] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 3.396661] smpboot: CPU0: Intel(R) Core(TM) i7-4712HQ CPU @ 2.30GHz (fam: 06, model: 3c, stepping: 03) [ 3.480643] Performance Events: unsupported p6 CPU model 60 no PMU driver, software events only. [ 3.551956] x86: Booted up 1 node, 1 CPUs [ 3.583438] smpboot: Total of 1 processors activated (4589.44 BogoMIPS) [ 3.624554] NMI watchdog: disabled (cpu0): hardware events not enabled [ 3.688366] NMI watchdog: Shutting down hard lockup detector on all cpus [ 3.736548] devtmpfs: initialized [ 3.768760] clocksource jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 3.863421] pinctrl core: initialized pinctrl subsystem [ 3.919470] NET: Registered protocol family 16 [ 3.960334] cpuidle: using governor ladder [ 3.987178] cpuidle: using governor menu [ 4.026309] ACPI: bus type PCI registered [ 4.067207] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 4.103114] dca service started, version 1.12.1 [ 4.127195] PCI: Using configuration type 1 for base access [ 4.177493] ACPI: Added _OSI(Module Device) [ 4.226965] ACPI: Added _OSI(Processor Device) [ 4.267100] ACPI: Added _OSI(3.0 _SCP Extensions) [ 4.306944] ACPI: Added _OSI(Processor Aggregator Device) [ 4.367368] ACPI: Executed 1 blocks of module-level executable AML code [ 4.417032] ACPI: Interpreter enabled [ 4.438840] ACPI Exception: AE_NOT_FOUND, While evaluating Sleep State [\_S1_] (20150410/hwxface-580) [ 4.523888] ACPI Exception: AE_NOT_FOUND, While evaluating Sleep State [\_S2_] (20150410/hwxface-580) [ 4.605748] ACPI Exception: AE_NOT_FOUND, While evaluating Sleep State [\_S3_] (20150410/hwxface-580) [ 4.702610] ACPI Exception: AE_NOT_FOUND, While evaluating Sleep State [\_S4_] (20150410/hwxface-580) [ 4.788651] ACPI: (supports S0 S5) [ 4.815625] ACPI: Using IOAPIC for interrupt routing [ 4.850532] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 4.941848] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 4.986707] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI] [ 5.040559] acpi PNP0A03:00: _OSC failed (AE_NOT_FOUND); disabling ASPM [ 5.090520] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 5.198494] PCI host bridge to bus 0000:00 [ 5.210276] pci_bus 0000:00: root bus resource [bus 00-ff] [ 5.243386] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 5.293205] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 5.348187] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 5.416160] pci_bus 0000:00: root bus resource [mem 0x20000000-0xffdfffff window] [ 5.458852] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 5.507853] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 5.554261] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 5.594133] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 5.641901] ACPI: PCI Interrupt Link [LNKA] (IRQs 5 9 10 11) *0, disabled. [ 5.702742] ACPI: PCI Interrupt Link [LNKB] (IRQs 5 9 10 *11) [ 5.749581] ACPI: PCI Interrupt Link [LNKC] (IRQs 5 9 *10 11) [ 5.805125] ACPI: PCI Interrupt Link [LNKD] (IRQs 5 *9 10 11) [ 5.858723] ACPI: Enabled 2 GPEs in block 00 to 07 [ 5.905616] vgaarb: setting as boot device: PCI:0000:00:02.0 [ 5.937694] vgaarb: device added: PCI:0000:00:02.0,decodes=io+mem,owns=io+mem,locks=none [ 5.996101] vgaarb: loaded [ 6.007532] vgaarb: bridge control possible 0000:00:02.0 [ 6.049693] PCI: Using ACPI for IRQ routing [ 6.090004] Switched to clocksource refined-jiffies [ 6.129110] pnp: PnP ACPI init [ 6.161303] pnp: PnP ACPI: found 4 devices [ 6.199719] clocksource acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 6.277280] Switched to clocksource acpi_pm [ 6.319710] NET: Registered protocol family 2 [ 6.715995] TCP established hash table entries: 4096 (order: 3, 32768 bytes) [ 7.445442] TCP bind hash table entries: 4096 (order: 4, 65536 bytes) [ 8.145012] TCP: Hash tables configured (established 4096 bind 4096) [ 8.754702] UDP hash table entries: 256 (order: 1, 8192 bytes) [ 9.404319] UDP-Lite hash table entries: 256 (order: 1, 8192 bytes) [ 10.093956] NET: Registered protocol family 1 [ 10.503716] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 11.153363] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 11.829895] hrtimer: interrupt took 3687619 ns [ 12.335560] Kernel panic - not syncing: broken padding [ 12.882344] CPU: 0 PID: 1 Comm: swapper/0 Not tainted 4.1.6-coreos-r1 #2 [ 13.591938] Hardware name: innotek GmbH VirtualBox/VirtualBox, BIOS VirtualBox 12/01/2006 [ 14.481454] ffff88001eb0e700 ffff88001e9bbdd8 ffffffff815491cc 0000000000000ce4 [ 15.271496] ffffffff817b9ae1 ffff88001e9bbe58 ffffffff81545792 0000000055e67787 [ 16.021073] ffffffff00000010 ffff88001e9bbe68 ffff88001e9bbe08 ffff88001e9bbe58 [ 16.790647] Call Trace: [ 17.050021] [] dump_stack+0x45/0x57 [ 17.579730] [ ] panic+0xc1/0x1f5 [ 18.059467] [ ] ? maybe_link.part.2+0x110/0x110 [ 18.679111] [ ] populate_rootfs+0x2f/0x116 [ 19.188832] [ ] do_one_initcall+0xd4/0x210 [ 19.798475] [ ] ? parse_args+0x1f0/0x470 [ 20.398164] [ ] ? __wake_up+0x48/0x60 [ 20.977819] [ ] kernel_init_freeable+0x173/0x200 [ 21.607467] [ ] ? initcall_blacklist+0xc0/0xc0 [ 22.277112] [ ] ? rest_init+0x80/0x80 [ 22.836784] [ ] kernel_init+0xe/0xf0 [ 23.386476] [ ] ret_from_fork+0x42/0x70 [ 23.956182] [ ] ? rest_init+0x80/0x80 [ 24.535888] Rebooting in 60 seconds.. [ 84.945552] ACPI MEMORY or I/O RESET_REG.
Comment by dongsupark
I'll summarize what I've found out:
"no cpio magic"
, "junk in compressed archive"
or "broken padding"
or else. Of course it's not always reproducible. This indicates that it's somehow related to a corrupted input (or image).I cannot reproduce this. Is this still a problem?
/cc @crawford @marineam
original issue is here
I was able to reproduce the issue. Here is full failed kernel log:
Here is normal kernel log: