search

coreos / bugs

Issue tracker for CoreOS Container Linux
https://coreos.com/os/eol/
146 stars 30 forks source link

CoreOS EC2 breaks after reboot or stop/start. #178

Closed chrisjenx closed 9 years ago

chrisjenx commented 9 years ago

First boot of an instance is fine, can ssh into it using our private key, deploy using fleetctl and so forth.

But if I reboot or stop/start I am never able to connect to the box again, nor do any of the containers seem to start again, I get invalid public key on trying to SSH to the box. It seem's like the EC2 cloud config is not applied correctly, but that is just a guess. (the logs say they apply it).

System log and cloud config is provided below.

System log.

Xen Minimal OS!
  start_info: 0xcf6000(VA)
    nr_pages: 0x6cc00
  shared_inf: 0x7dd99000(MA)
     pt_base: 0xcf9000(VA)
nr_pt_frames: 0xb
    mfn_list: 0x990000(VA)
   mod_start: 0x0(VA)
     mod_len: 0
       flags: 0x0
    cmd_line: root=/dev/sda ro 4
  stack:      0x94f860-0x96f860
MM: Init
      _text: 0x0(VA)
     _etext: 0x5ffbd(VA)
   _erodata: 0x78000(VA)
     _edata: 0x80ae0(VA)
stack start: 0x94f860(VA)
       _end: 0x98fe68(VA)
  start_pfn: d07
    max_pfn: 6cc00
Mapping memory range 0x1000000 - 0x6cc00000
setting 0x0-0x78000 readonly
skipped 0x1000
MM: Initialise page allocator for 1066000(1066000)-6cc00000(6cc00000)
MM: done
Demand map pfns at 6cc01000-206cc01000.
Heap resides at 206cc02000-406cc02000.
Initialising timer interface
Initialising console ... done.
gnttab_table mapped at 0x6cc01000.
Initialising scheduler
Thread "Idle": pointer: 0x206cc02010, stack: 0x13f0000
Initialising xenbus
Thread "xenstore": pointer: 0x206cc027c0, stack: 0x6c800000
Dummy main: start_info=0x96f960
Thread "main": pointer: 0x206cc02f70, stack: 0x6c810000
"main" "root=/dev/sda" "ro" "4" 
vbd 2048 is hd0
******************* BLKFRONT for device/vbd/2048 **********

backend at /local/domain/0/backend/vbd/1225/2048
41943040 sectors of 512 bytes
**************************
vbd 2051 is hd1
******************* BLKFRONT for device/vbd/2051 **********

backend at /local/domain/0/backend/vbd/1225/2051
1835008 sectors of 512 bytes
**************************
vbd 2064 is hd2
******************* BLKFRONT for device/vbd/2064 **********

backend at /local/domain/0/backend/vbd/1225/2064
312705024 sectors of 512 bytes
**************************


    GNU GRUB  version 0.97  (1781760K lower / 0K upper memory)

+-------------------------------------------------------------------------+||||||||||||||||||||||||+-------------------------------------------------------------------------+

    Use the ^ and v keys to select which entry is highlighted.

    Press enter to boot the selected OS, 'e' to edit the

    commands before booting, or 'c' for a command-line.  CoreOS A Root                                                            CoreOS B Root                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        Booting 'CoreOS B Root'

root            (hd0,0)

 Filesystem type is fat, partition type 0xc

kernel          /syslinux/vmlinuz.B console=tty0 ro noswap cros_legacy  console

=hvc0 root=LABEL=ROOT rootflags=subvol=root usr=PARTUUID=e03dd35c-7c2d-4a47-b3f

e-27f15780a57c

close blk: backend at /local/domain/0/backend/vbd/1225/2048
close blk: backend at /local/domain/0/backend/vbd/1225/2051
close blk: backend at /local/domain/0/backend/vbd/1225/2064
[    0.000000] Initializing cgroup subsys cpuset
[    0.000000] Initializing cgroup subsys cpu
[    0.000000] Initializing cgroup subsys cpuacct
[    0.000000] Linux version 3.16.2+ (buildbot@ip-10-204-3-57) (gcc version 4.7.3 (Gentoo Hardened 4.7.3-r1 p1.4, pie-0.5.5) ) #2 SMP Thu Oct 16 01:11:04 UTC 2014
[    0.000000] Command line: console=tty0 ro noswap cros_legacy  console=hvc0 root=LABEL=ROOT rootflags=subvol=root usr=PARTUUID=e03dd35c-7c2d-4a47-b3fe-27f15780a57c
[    0.000000] ACPI in unprivileged domain disabled
[    0.000000] e820: BIOS-provided physical RAM map:
[    0.000000] Xen: [mem 0x0000000000000000-0x000000000009ffff] usable
[    0.000000] Xen: [mem 0x00000000000a0000-0x00000000000fffff] reserved
[    0.000000] Xen: [mem 0x0000000000100000-0x000000006d3fffff] usable
[    0.000000] NX (Execute Disable) protection: active
[    0.000000] MPS support code is not built-in.
[    0.000000] Using acpi=off or acpi=noirq or pci=noacpi may have problem
[    0.000000] DMI not present or invalid.
[    0.000000] AGP: No AGP bridge found
[    0.000000] e820: last_pfn = 0x6d400 max_arch_pfn = 0x400000000
[    0.000000] init_memory_mapping: [mem 0x00000000-0x000fffff]
[    0.000000] init_memory_mapping: [mem 0x6ca00000-0x6cbfffff]
[    0.000000] init_memory_mapping: [mem 0x6c000000-0x6c9fffff]
[    0.000000] init_memory_mapping: [mem 0x00100000-0x6bffffff]
[    0.000000] init_memory_mapping: [mem 0x6cc00000-0x6d3fffff]
[    0.000000] NUMA turned off
[    0.000000] Faking a node at [mem 0x0000000000000000-0x000000006d3fffff]
[    0.000000] Initmem setup node 0 [mem 0x00000000-0x6d3fffff]
[    0.000000]   NODE_DATA [mem 0x6c8b0000-0x6c8b6fff]
[    0.000000] Zone ranges:
[    0.000000]   DMA      [mem 0x00001000-0x00ffffff]
[    0.000000]   DMA32    [mem 0x01000000-0xffffffff]
[    0.000000]   Normal   empty
[    0.000000] Movable zone start for each node
[    0.000000] Early memory node ranges
[    0.000000]   node   0: [mem 0x00001000-0x0009ffff]
[    0.000000]   node   0: [mem 0x00100000-0x6d3fffff]
[    0.000000] smpboot: Allowing 1 CPUs, 0 hotplug CPUs
[    0.000000] No local APIC present
[    0.000000] APIC: disable apic facility
[    0.000000] APIC: switched to apic NOOP
[    0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000fffff]
[    0.000000] e820: [mem 0x6d400000-0xffffffff] available for PCI devices
[    0.000000] Booting paravirtualized kernel on Xen
[    0.000000] Xen version: 4.2.amazon (preserve-AD)
[    0.000000] setup_percpu: NR_CPUS:128 nr_cpumask_bits:128 nr_cpu_ids:1 nr_node_ids:1
[    0.000000] PERCPU: Embedded 28 pages/cpu @ffff88006c200000 s84672 r8192 d21824 u2097152
[    0.000000] Built 1 zonelists in Node order, mobility grouping on.  Total pages: 440378
[    0.000000] Policy zone: DMA32
[    0.000000] Kernel command line: init=/usr/lib/systemd/systemd console=tty0 ro noswap cros_legacy  console=hvc0 root=LABEL=ROOT rootflags=subvol=root usr=PARTUUID=e03dd35c-7c2d-4a47-b3fe-27f15780a57c
[    0.000000] PID hash table entries: 4096 (order: 3, 32768 bytes)
[    0.000000] AGP: Checking aperture...
[    0.000000] AGP: No AGP bridge found
[    0.000000] Memory: 1712904K/1789564K available (4989K kernel code, 890K rwdata, 2220K rodata, 20116K init, 920K bss, 76660K reserved)
[    0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1
[    0.000000] Hierarchical RCU implementation.
[    0.000000]  RCU restricting CPUs from NR_CPUS=128 to nr_cpu_ids=1.
[    0.000000] RCU: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=1
[    0.000000] NR_IRQS:8448 nr_irqs:256 16
[    0.000000] xen:events: Using 2-level ABI
[    0.000000] Console: colour dummy device 80x25
[    0.000000] console [tty0] enabled
[    0.000000] console [hvc0] enabled
[    0.000000] allocated 7340032 bytes of page_cgroup
[    0.000000] please try 'cgroup_disable=memory' option if you don't want memory cgroups
[    0.000000] installing Xen timer for CPU 0
[    0.000000] tsc: Detected 1800.072 MHz processor
[    0.001000] Calibrating delay loop (skipped), value calculated using timer frequency.. 3600.14 BogoMIPS (lpj=1800072)
[    0.001000] pid_max: default: 32768 minimum: 301
[    0.001000] Dentry cache hash table entries: 262144 (order: 9, 2097152 bytes)
[    0.001321] Inode-cache hash table entries: 131072 (order: 8, 1048576 bytes)
[    0.001649] Mount-cache hash table entries: 4096 (order: 3, 32768 bytes)
[    0.001667] Mountpoint-cache hash table entries: 4096 (order: 3, 32768 bytes)
[    0.002055] Initializing cgroup subsys memory
[    0.002133] Initializing cgroup subsys devices
[    0.002153] Initializing cgroup subsys freezer
[    0.002163] Initializing cgroup subsys net_cls
[    0.002175] Initializing cgroup subsys blkio
[    0.002186] Initializing cgroup subsys perf_event
[    0.002193] Initializing cgroup subsys net_prio
[    0.002291] ENERGY_PERF_BIAS: Set to 'normal', was 'performance'
[    0.002291] ENERGY_PERF_BIAS: View and update with x86_energy_perf_policy(8)
[    0.002305] CPU: Physical Processor ID: 0
[    0.002309] CPU: Processor Core ID: 0
[    0.003383] Last level iTLB entries: 4KB 512, 2MB 8, 4MB 8
[    0.003383] Last level dTLB entries: 4KB 512, 2MB 0, 4MB 0, 1GB 4
[    0.003383] tlb_flushall_shift: 6
[    0.039000] Freeing SMP alternatives memory: 20K (ffffffff82e85000 - ffffffff82e8a000)
[    0.042106] ftrace: allocating 19287 entries in 76 pages
[    0.048164] cpu 0 spinlock event irq 17
[    0.060772] Performance Events: unsupported p6 CPU model 62 no PMU driver, software events only.
[    0.062584] x86: Booted up 1 node, 1 CPUs
[    0.062658] NMI watchdog: disabled (cpu0): hardware events not enabled
[    0.062775] devtmpfs: initialized
[    0.066675] NET: Registered protocol family 16
[    0.066702] xen:grant_table: Grant tables using version 1 layout
[    0.066724] Grant table initialized
[    0.067020] dca service started, version 1.12.1
[    0.067678] PCI: setting up Xen PCI frontend stub
[    0.068833] ACPI: Interpreter disabled.
[    0.068856] xen:balloon: Initialising balloon driver
[    0.070031] xen_balloon: Initialising balloon driver
[    0.074115] vgaarb: loaded
[    0.074196] PCI: System does not support PCI
[    0.074200] PCI: System does not support PCI
[    0.074415] Switched to clocksource xen
[    0.079800] pnp: PnP ACPI: disabled
[    0.081509] NET: Registered protocol family 2
[    0.081764] TCP established hash table entries: 16384 (order: 5, 131072 bytes)
[    0.081852] TCP bind hash table entries: 16384 (order: 6, 262144 bytes)
[    0.081910] TCP: Hash tables configured (established 16384 bind 16384)
[    0.081970] TCP: reno registered
[    0.081987] UDP hash table entries: 1024 (order: 3, 32768 bytes)
[    0.082005] UDP-Lite hash table entries: 1024 (order: 3, 32768 bytes)
[    0.082084] NET: Registered protocol family 1
[    0.972080] platform rtc_cmos: registered platform RTC device (no PNP device found)
[    0.972178] RAPL PMU detected, hw unit 2^-16 Joules, API unit is 2^-32 Joules, 3 fixed counters 163840 ms ovfl timer
[    0.972514] futex hash table entries: 256 (order: 2, 16384 bytes)
[    0.972535] Initialise system trusted keyring
[    0.972824] HugeTLB registered 2 MB page size, pre-allocated 0 pages
[    0.974679] VFS: Disk quotas dquot_6.5.2
[    0.974729] Dquot-cache hash table entries: 512 (order 0, 4096 bytes)
[    0.974831] msgmni has been set to 3345
[    0.975100] alg: No test for stdrng (krng)
[    0.975172] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 252)
[    0.975231] io scheduler noop registered
[    0.975239] io scheduler deadline registered
[    0.975305] io scheduler cfq registered (default)
[    0.975384] pci_hotplug: PCI Hot Plug PCI Core version: 0.5
[    0.975408] pciehp: PCI Express Hot Plug Controller Driver version: 0.4
[    0.975545] ioatdma: Intel(R) QuickData Technology Driver 4.00
[    0.975900] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled
[    0.976335] libphy: Fixed MDIO Bus: probed
[    0.976384] i8042: PNP: No PS/2 controller found. Probing ports directly.
[    3.032526] i8042: No controller found
[    3.136988] rtc_cmos rtc_cmos: rtc core: registered rtc_cmos as rtc0
[    3.137090] rtc_cmos: probe of rtc_cmos failed with error -38
[    3.137236] TCP: cubic registered
[    3.137468] NET: Registered protocol family 10
[    3.137787] NET: Registered protocol family 17
[    3.137808] mce: Unable to init device /dev/mcelog (rc: -5)
[    3.150201] Loading compiled-in X.509 certificates
[    3.150229] registered taskstats version 1
[    3.150310] xenbus_probe_frontend: Device with no driver: device/vbd/2048
[    3.150316] xenbus_probe_frontend: Device with no driver: device/vbd/2064
[    3.150321] xenbus_probe_frontend: Device with no driver: device/vbd/2051
[    3.150326] xenbus_probe_frontend: Device with no driver: device/vif/0
[    3.150407] /build/amd64-usr/var/tmp/portage/sys-kernel/coreos-kernel-3.16.2-r3/work/coreos-kernel-3.16.2/drivers/rtc/hctosys.c: unable to open rtc device (rtc0)
[    3.173409] Freeing unused kernel memory: 20116K (ffffffff81ae0000 - ffffffff82e85000)
[    3.173431] Write protecting the kernel read-only data: 10240k
[    3.187476] Freeing unused kernel memory: 1144K (ffff8800014e2000 - ffff880001600000)
[    3.188965] Freeing unused kernel memory: 1876K (ffff88000182b000 - ffff880001a00000)
[    3.200869] random: systemd urandom read with 14 bits of entropy available
[    3.201974] systemd[1]: systemd 215 running in system mode. (-PAM -AUDIT -SELINUX +IMA -SYSVINIT +LIBCRYPTSETUP -GCRYPT -ACL -XZ +SECCOMP -APPARMOR)
[    3.202077] systemd[1]: Detected virtualization 'xen'.
[    3.202108] systemd[1]: Detected architecture 'x86-64'.
[    3.202119] systemd[1]: Running in initial RAM disk.

Welcome to dracut-037-r3 (Initramfs)!

[    3.203329] systemd[1]: Inserted module 'autofs4'
[    3.203573] systemd[1]: No hostname configured.
[    3.203588] systemd[1]: Set hostname to <localhost>.
[    3.203639] systemd[1]: Initializing machine ID from random generator.
[    3.424955] systemd[1]: Cannot add dependency job for unit systemd-journald-dev-log.socket, ignoring: Unit systemd-journald-dev-log.socket failed to load: No such file or directory.
[    3.425349] systemd[1]: Expecting device dev-disk-by\x2dpartuuid-e03dd35c\x2d7c2d\x2d4a47\x2db3fe\x2d27f15780a57c.device...
         Expecting device dev-disk-by\x2dpartuuid-e03dd35c\x2...0a57c.device...
[    3.425507] systemd[1]: Expecting device dev-disk-by\x2dlabel-ROOT.device...
         Expecting device dev-disk-by\x2dlabel-ROOT.device...
[    3.425587] systemd[1]: Starting Timers.
[  OK  ] Reached target Timers.
[    3.425662] systemd[1]: Reached target Timers.
[    3.425694] systemd[1]: Starting Dispatch Password Requests to Console Directory Watch.
[    3.425819] systemd[1]: Started Dispatch Password Requests to Console Directory Watch.
[    3.425843] systemd[1]: Starting Paths.
[  OK  ] Reached target Paths.
[    3.425925] systemd[1]: Reached target Paths.
[    3.425947] systemd[1]: Starting Encrypted Volumes.
[  OK  ] Reached target Encrypted Volumes.
[    3.426022] systemd[1]: Reached target Encrypted Volumes.
[    3.426043] systemd[1]: Starting -.slice.
[  OK  ] Created slice -.slice.
[    3.442133] systemd[1]: Created slice -.slice.
[    3.442171] systemd[1]: Starting udev Kernel Socket.
[  OK  ] Listening on udev Kernel Socket.
[    3.442298] systemd[1]: Listening on udev Kernel Socket.
[    3.442324] systemd[1]: Starting Journal Socket.
[  OK  ] Listening on Journal Socket.
[    3.442485] systemd[1]: Listening on Journal Socket.
[    3.442514] systemd[1]: Starting System Slice.
[  OK  ] Created slice System Slice.
[    3.442897] systemd[1]: Created slice System Slice.
[    3.443139] systemd[1]: Starting dracut cmdline hook...
         Starting dracut cmdline hook...
[    3.444470] systemd[1]: Starting system-systemd\x2dfsck.slice.
[  OK  ] Created slice system-systemd\x2dfsck.slice.
[    3.444943] systemd[1]: Created slice system-systemd\x2dfsck.slice.
[    3.445111] systemd[1]: Starting Create list of required static device nodes for the current kernel...
         Starting Create list of required static device nodes...rrent kernel...
[    3.454709] systemd[1]: Started Load Kernel Modules.
[    3.454756] systemd[1]: Starting Apply Kernel Variables...
         Starting Apply Kernel Variables...
[    3.456079] systemd[1]: Starting Journal Service...
         Starting Journal Service...
[  OK  ] Started Journal Service.
[    3.457534] systemd[1]: Started Journal Service.
[  OK  ] Reached target Slices.
[  OK  ] Listening on udev Control Socket.
[  OK  ] Reached target Sockets.
[  OK  ] Reached target Swap.
[  OK  ] Reached target Local File Systems.
[    3.481747] systemd[1]: Starting Slices.
[    3.482116] systemd[1]: Reached target Slices.
[    3.482427] systemd[1]: Starting udev Control Socket.
[    3.482723] systemd[1]: Listening on udev Control Socket.
[    3.483009] systemd[1]: Starting Sockets.
[    3.483317] systemd[1]: Reached target Sockets.
[    3.483614] systemd[1]: Starting Swap.
[    3.483918] systemd[1]: Reached target Swap.
[    3.484259] systemd[1]: Starting Local File Systems.
[    3.484546] systemd[1]: Reached target Local File Systems.
[  OK  ] Started Apply Kernel Variables.
[  OK  ] Started Create list of required static device nodes ...current kernel.
         Starting Create Static Device Nodes in /dev...
[  OK  ] Started Create Static Device Nodes in /dev.
[    3.538017] systemd[1]: Started Apply Kernel Variables.
[    3.538431] systemd[1]: Started Create list of required static device nodes for the current kernel.
[    3.538741] systemd[1]: Starting Create Static Device Nodes in /dev...
[    3.539202] systemd[1]: Started Create Static Device Nodes in /dev.
[    3.586637] dracut-cmdline[78]: dracut-dracut-037-r3
[  OK  ] Started dracut cmdline hook.
         Starting dracut pre-udev hook...
[    3.867430] systemd[1]: Started dracut cmdline hook.
[    3.867838] systemd[1]: Starting dracut pre-udev hook...
[  OK  ] Started dracut pre-udev hook.
         Starting udev Kernel Device Manager...
[    3.995416] systemd-udevd[170]: starting version 215
[    3.999445] systemd[1]: Started dracut pre-udev hook.
[    3.999829] systemd[1]: Starting udev Kernel Device Manager...
[    4.000168] systemd-udevd[170]: specified group 'tty' unknown
[    4.000502] systemd-udevd[170]: specified group 'uucp' unknown
[    4.000811] systemd-udevd[170]: specified group 'kmem' unknown
[    4.001126] systemd-udevd[170]: specified group 'input' unknown
[    4.001422] systemd-udevd[170]: specified group 'video' unknown
[    4.009864] systemd-udevd[170]: specified group 'audio' unknown
[    4.010178] systemd-udevd[170]: specified group 'lp' unknown
[    4.010474] systemd-udevd[170]: specified group 'disk' unknown
[    4.010771] systemd-udevd[170]: IMPORT{builtin}: 'uaccess' unknown /usr/lib64/udev/rules.d/73-seat-late.rules:15
[  OK  ] Started udev Kernel Device Manager.
         Starting dracut pre-trigger hook...
[    4.046198] systemd[1]: Started udev Kernel Device Manager.
[    4.046626] systemd[1]: Starting dracut pre-trigger hook...
[    4.096822] dracut-pre-trigger[171]: rd.md=0: removing MD RAID activation
[  OK  ] Started dracut pre-trigger hook.
         Starting udev Coldplug all Devices...
[  OK  ] Started udev Coldplug all Devices.
         Starting dracut initqueue hook...
[  OK  ] Reached target System Initialization.
[  OK  ] Reached target Basic System.
[    4.299706] systemd[1]: Started dracut pre-trigger hook.
[    4.300162] systemd[1]: Starting udev Coldplug all Devices...
[    4.300473] systemd[1]: Started udev Coldplug all Devices.
[    4.300777] systemd[1]: Starting dracut initqueue hook...
[    4.301087] systemd[1]: Starting System Initialization.
[    4.301394] systemd[1]: Reached target System Initialization.
[    4.301694] systemd[1]: Starting Basic System.
[    4.301992] systemd[1]: Reached target Basic System.
[    4.511888] microcode: CPU0 sig=0x306e4, pf=0x1, revision=0x415
[    4.540038] xen_netfront: Initialising Xen virtual ethernet driver
[    4.637233] systemd-udevd[170]: specified group 'tty' unknown
[    4.637580] systemd-udevd[170]: specified group 'uucp' unknown
[    4.637959] systemd-udevd[170]: specified group 'kmem' unknown
[    4.638312] systemd-udevd[170]: specified group 'input' unknown
[    4.638607] systemd-udevd[170]: specified group 'video' unknown
[    4.638974] systemd-udevd[170]: specified group 'audio' unknown
[    4.639322] systemd-udevd[170]: specified group 'lp' unknown
[    4.639620] systemd-udevd[170]: specified group 'disk' unknown
[    4.639913] systemd-udevd[170]: IMPORT{builtin}: 'uaccess' unknown /usr/lib64/udev/rules.d/73-seat-late.rules:15
[    4.657559] microcode: Microcode Update Driver: v2.00 <tigran@aivazian.fsnet.co.uk>, Peter Oruba
[    4.696123] SSE version of gcm_enc/dec engaged.
[    4.709252] alg: No test for __gcm-aes-aesni (__driver-gcm-aes-aesni)
[    4.755024] blkfront: xvda: barrier or flush: disabled; persistent grants: disabled; indirect descriptors: enabled;
[    4.763953]  xvda: xvda1 xvda2 xvda3 xvda4 xvda6 xvda7 xvda9
[    4.764921] Setting capacity to 41943040
[    4.764934] xvda: detected capacity change from 0 to 21474836480
[    4.765743] blkfront: xvdb: flush diskcache: enabled; persistent grants: disabled; indirect descriptors: enabled;
[    4.776781]  xvdb: unknown partition table
[    4.777836] vbd vbd-2051: 16 xlvbd_add at /local/domain/0/backend/vbd/1225/2051
[    4.832240] Setting capacity to 312705024
[    4.832253] xvdb: detected capacity change from 0 to 160104972288
[  OK  ] Found device /dev/disk/by-partuuid/e03dd35c-7c2d-4a4...e-27f15780a57c.
[    4.953847] systemd[1]: Found device /dev/disk/by-partuuid/e03dd35c-7c2d-4a47-b3fe-27f15780a57c.
[    4.982073] raid6: sse2x1    2144 MB/s
[    5.006789] raid6: sse2x2    2757 MB/s
[    5.023147] raid6: sse2x4    3300 MB/s
[    5.023156] raid6: using algorithm sse2x4 (3300 MB/s)
[    5.023161] raid6: using ssse3x2 recovery algorithm
[    5.024269] xor: measuring software checksum speed
[    5.041060]    prefetch64-sse:  5240.000 MB/sec
[    5.051064]    generic_sse:  5044.000 MB/sec
[    5.051073] xor: using function: prefetch64-sse (5240.000 MB/sec)
[    5.085979] Btrfs loaded
[    5.087689] BTRFS: device label ROOT devid 1 transid 37950 /dev/xvda9
[  OK  ] Found device /dev/disk/by-label/ROOT.
         Starting File System Check on /dev/disk/by-label/ROOT...
[    5.112285] systemd[1]: Found device /dev/disk/by-label/ROOT.
[    5.112678] systemd[1]: Starting File System Check on /dev/disk/by-label/ROOT...
[  OK  ] Started dracut initqueue hook.
         Starting dracut pre-mount hook...
[    5.122161] systemd[1]: Started dracut initqueue hook.
[    5.122503] systemd[1]: Starting dracut pre-mount hook...
[    5.199477] systemd-fsck[253]: /sbin/fsck.btrfs: BTRFS file system.
[    5.199558] systemd-fsck[253]: /sbin/fsck.btrfs: BTRFS file system.
[  OK  ] Started File System Check on /dev/disk/by-label/ROOT.
[    5.200781] systemd[1]: Started File System Check on /dev/disk/by-label/ROOT.
[  OK  ] Started dracut pre-mount hook.
[    5.227662] systemd[1]: Started dracut pre-mount hook.
[    5.230262] systemd[1]: Mounting /sysroot...
         Mounting /sysroot...
[    5.271042] BTRFS info (device xvda9): disk space caching is enabled
[    5.288069] BTRFS: detected SSD devices, enabling SSD mode
[    5.288895] random: nonblocking pool is initialized
[  OK  ] Mounted /sysroot.
         Starting Remount Root File System...
[    5.298152] systemd[1]: Mounted /sysroot.
[    5.298516] systemd[1]: Starting Remount Root File System...
[    5.304726] BTRFS info (device xvda9): disk space caching is enabled
[  OK  ] Started Remount Root File System.
         Mounting /sysroot/usr...
[    5.321017] systemd[1]: Started Remount Root File System.
[    5.321391] systemd[1]: Mounting /sysroot/usr...
[    5.397117] EXT4-fs (xvda4): mounted filesystem without journal. Opts: (null)
[  OK  ] Mounted /sysroot/usr.
[    5.398517] systemd[1]: Mounted /sysroot/usr.
[    5.399289] systemd[1]: Starting Initrd Root File System.
[  OK  ] Reached target Initrd Root File System.
[    5.399796] systemd[1]: Reached target Initrd Root File System.
         Starting Reload Configuration from the Real Root...
[    5.400220] systemd[1]: Starting Reload Configuration from the Real Root...
[    5.433693] systemd[1]: Reloading.
[  OK  ] Started Reload Configuration from the Real Root.
[    5.998763] systemd[1]: Started Reload Configuration from the Real Root.
[    6.015025] systemd[1]: Starting Initrd File Systems.
[  OK  ] Reached target Initrd File Systems.
[    6.015569] systemd[1]: Reached target Initrd File Systems.
[    6.024602] systemd[1]: Started dracut mount hook.
[    6.024996] systemd[1]: Starting Initrd Default Target.
[  OK  ] Reached target Initrd Default Target.
[    6.025511] systemd[1]: Reached target Initrd Default Target.
[    6.026113] systemd[1]: Starting dracut pre-pivot and cleanup hook...
         Starting dracut pre-pivot and cleanup hook...
[    6.308526] BTRFS info (device xvda9): disk space caching is enabled
[    6.363522] dracut-pre-pivot[338]: bootengine: machine-id is valid
[  OK  ] Started dracut pre-pivot and cleanup hook.
[    6.403543] systemd[1]: Started dracut pre-pivot and cleanup hook.
[    6.406420] systemd[1]: Starting Cleaning Up and Shutting Down Daemons...
         Starting Cleaning Up and Shutting Down Daemons...
[    6.426538] systemd[1]: Cannot add dependency job for unit systemd-journald-dev-log.socket, ignoring: Unit systemd-journald-dev-log.socket failed to load: No such file or directory.
[  OK  ] Stopped Cleaning Up and Shutting Down Daemons.
[    6.429906] systemd[1]: Stopped Cleaning Up and Shutting Down Daemons.
[    6.430944] systemd[1]: Stopping dracut pre-pivot and cleanup hook...
         Stopping dracut pre-pivot and cleanup hook...
[  OK  ] Stopped dracut pre-pivot and cleanup hook.
[    6.431916] systemd[1]: Stopped dracut pre-pivot and cleanup hook.
[    6.432359] systemd[1]: Stopping dracut pre-mount hook...
         Stopping dracut pre-mount hook...
[  OK  ] Stopped dracut pre-mount hook.
[    6.433321] systemd[1]: Stopped dracut pre-mount hook.
[    6.433694] systemd[1]: Stopping dracut initqueue hook...
         Stopping dracut initqueue hook...
[  OK  ] Stopped dracut initqueue hook.
[  OK  ] Stopped target Initrd Default Target.
[  OK  ] Stopped target Basic System.
[  OK  ] Stopped target Slices.
[  OK  ] Stopped target Paths.
[  OK  ] Stopped target Timers.
[    6.443560] systemd[1]: Stopped dracut initqueue hook.
[    6.443923] systemd[1]: Stopping Initrd Default Target.
[  OK  ] Stopped target Sockets.
[    6.444392] systemd[1]: Stopped target Initrd Default Target.
[    6.444748] systemd[1]: Stopping Basic System.
[  OK  ] Stopped target System Initialization.
[    6.445217] systemd[1]: Stopped target Basic System.
[    6.445571] systemd[1]: Stopping Slices.
         Stopping Apply Kernel Variables...
[  OK  ] Stopped Apply Kernel Variables.
[    6.446603] systemd[1]: Stopped target Slices.
[    6.446960] systemd[1]: Stopping Paths.
[  OK  ] Stopped target Encrypted Volumes.
[    6.447446] systemd[1]: Stopped target Paths.
[    6.447801] systemd[1]: Stopping Dispatch Password Requests to Console Directory Watch.
         Stopping udev Coldplug all Devices...
[  OK  ] Stopped udev Coldplug all Devices.
[    6.448806] systemd[1]: Stopped Dispatch Password Requests to Console Directory Watch.
[    6.449195] systemd[1]: Stopping Timers.
         Stopping dracut pre-trigger hook...
[  OK  ] Stopped dracut pre-trigger hook.
[    6.450025] systemd[1]: Stopped target Timers.
[  OK  ] Stopped target Swap.
[    6.460367] systemd[1]: Stopping Sockets.
[    6.460721] systemd[1]: Stopped target Sockets.
[  OK  ] Stopped target Local File Systems.
[    6.461183] systemd[1]: Stopping System Initialization.
[    6.461534] systemd[1]: Stopped target System Initialization.
         Stopping udev Kernel Device Manager...
[  OK  ] Stopped udev Kernel Device Manager.
[    6.482896] systemd[1]: Stopping Apply Kernel Variables...
[    6.484468] systemd[1]: Stopped Apply Kernel Variables.
         Stopping dracut pre-udev hook...
[  OK  ] Stopped dracut pre-udev hook.
[    6.485423] systemd[1]: Stopping Encrypted Volumes.
[    6.485815] systemd[1]: Stopped target Encrypted Volumes.
         Stopping dracut cmdline hook...
[  OK  ] Stopped dracut cmdline hook.
[    6.511892] systemd[1]: Stopping udev Coldplug all Devices...
         Stopping Create Static Device Nodes in /dev...
[  OK  ] Stopped Create Static Device Nodes in /dev.
[    6.512934] systemd[1]: Stopped udev Coldplug all Devices.
[    6.513378] systemd[1]: Stopping dracut pre-trigger hook...
         Stopping Create list of required static device nodes...rrent kernel...
[  OK  ] Stopped Create list of required static device nodes ...current kernel.
[    6.514406] systemd[1]: Stopping Swap.
[    6.514795] systemd[1]: Stopped target Swap.
[  OK  ] Closed udev Kernel Socket.
[    6.515291] systemd[1]: Stopping Local File Systems.
[    6.515667] systemd[1]: Stopped target Local File Systems.
[  OK  ] Closed udev Control Socket.
[    6.516146] systemd[1]: Stopping udev Kernel Device Manager...
[    6.516550] systemd[1]: Stopped udev Kernel Device Manager.
         Starting Cleanup udevd DB...
[  OK  ] Started Cleanup udevd DB.
[    6.542698] systemd[1]: Stopping dracut pre-udev hook...
[    6.543165] systemd[1]: Stopped dracut pre-udev hook.
[  OK  ] Reached target Switch Root.
[    6.543623] systemd[1]: Stopped dracut cmdline hook.
[    6.543997] systemd[1]: Stopping Create Static Device Nodes in /dev...
         Starting Switch Root...
[    6.555524] systemd[1]: Stopped Create Static Device Nodes in /dev.
[    6.555860] systemd[1]: Stopping Create list of required static device nodes for the current kernel...
[    6.556215] systemd[1]: Stopped Create list of required static device nodes for the current kernel.
[    6.556555] systemd[1]: Stopping udev Kernel Socket.
[    6.556919] systemd[1]: Closed udev Kernel Socket.
[    6.557295] systemd[1]: Stopping udev Control Socket.
[    6.557592] systemd[1]: Closed udev Control Socket.
[    6.557889] systemd[1]: Starting Cleanup udevd DB...
[    6.558228] systemd[1]: Started Cleanup udevd DB.
[    6.558531] systemd[1]: Starting Switch Root.
[    6.558825] systemd[1]: Reached target Switch Root.
[    6.559156] systemd[1]: Starting Switch Root...
[    6.570161] systemd[1]: Switching root.
[    6.602581] systemd-journald[81]: Received SIGTERM from PID 1 (systemd).

Welcome to CoreOS 444.5.0!

[  OK  ] Stopped Switch Root.
[  OK  ] Stopped target Switch Root.
[  OK  ] Stopped target Initrd File Systems.
         Stopping File System Check on /dev/disk/by-label/ROOT...
[  OK  ] Stopped File System Check on /dev/disk/by-label/ROOT.
[  OK  ] Removed slice system-systemd\x2dfsck.slice.
         Stopping remount-sysroot.service...
[  OK  ] Stopped remount-sysroot.service.
[  OK  ] Stopped target Initrd Root File System.
[  OK  ] Created slice User and Session Slice.
[  OK  ] Created slice system-getty.slice.
[  OK  ] Created slice system-serial\x2dgetty.slice.
         Expecting device dev-hvc0.device...
[  OK  ] Created slice system-system\x2dcloudinit.slice.
[  OK  ] Reached target Slices.
[  OK  ] Listening on /dev/initctl Compatibility Named Pipe.
[  OK  ] Listening on Delayed Shutdown Socket.
         Starting Setup Virtual Console...
         Starting Create list of required static device nodes...rrent kernel...
         Starting Apply Kernel Variables...
         Mounting POSIX Message Queue File System...
[  OK  ] Stopped Trigger Flushing of Journal to Persistent Storage.
         Stopping Journal Service...
[  OK  ] Stopped Journal Service.
[  OK  ] Listening on Journal Socket (/dev/log).
         Starting Journal Service...
[  OK  ] Started Journal Service.
         Mounting Debug File System...
         Mounting Huge Pages File System...
[  OK  ] Reached target Swap.
         Starting Load/Save Random Seed...
         Mounting External Media Directory...
         Mounting /proc/xen...
         Mounting Temporary Directory...
[  OK  ] Created slice system-addon\x2dconfig.slice.
[  OK  ] Created slice system-addon\x2drun.slice.
         Expecting device dev-disk-by\x2dlabel-OEM.device...
[  OK  ] Listening on udev Kernel Socket.
[  OK  ] Listening on udev Control Socket.
         Starting udev Coldplug all Devices...
[  OK  ] Mounted POSIX Message Queue File System.
[  OK  ] Mounted Debug File System.
[  OK  ] Mounted Huge Pages File System.
[  OK  ] Mounted External Media Directory.
[  OK  ] Mounted Temporary Directory.
[  OK  ] Started Setup Virtual Console.
[  OK  ] Started Create list of required static device nodes ...current kernel.
[  OK  ] Started Apply Kernel Variables.
[  OK  ] Started Load/Save Random Seed.
[  OK  ] Mounted /proc/xen.
[  OK  ] Started udev Coldplug all Devices.
         Starting udev Wait for Complete Device Initialization...
         Starting Set fake PV driver version for XenServer...
         Starting Create Static Device Nodes in /dev...
[  OK  ] Started Set fake PV driver version for XenServer.
[  OK  ] Started Create Static Device Nodes in /dev.
         Starting udev Kernel Device Manager...
[  OK  ] Reached target Local File Systems (Pre).
[    7.506028] systemd-udevd[404]: starting version 215
[  OK  ] Started udev Kernel Device Manager.
[  OK  ] Started udev Wait for Complete Device Initialization.
         Starting Activation of LVM2 logical volumes...
[  OK  ] Started Activation of LVM2 logical volumes.
[  OK  ] Reached target Encrypted Volumes.
         Starting Activation of LVM2 logical volumes...
[  OK  ] Found device /dev/hvc0.
[  OK  ] Started Activation of LVM2 logical volumes.
         Starting Activation of LVM2 logical volumes...
[  OK  ] Found device /dev/disk/by-label/OEM.
         Mounting /usr/share/oem...
[   10.917291] EXT4-fs (xvda6): mounted filesystem with ordered data mode. Opts: commit=600
[  OK  ] Mounted /usr/share/oem.
[  OK  ] Reached target Local File Systems.
         Starting Create Volatile Files and Directories...
         Starting Create missing system files...
[  OK  ] Started Create missing system files.
[  OK  ] Started Create Volatile Files and Directories.
         Starting Clean up broken links in /etc/ssl/certs...
         Starting Rebuild Dynamic Linker Cache...
         Starting Update UTMP about System Boot/Shutdown...
[  OK  ] Started Activation of LVM2 logical volumes.
[  OK  ] Reached target Remote File Systems.
         Starting Trigger Flushing of Journal to Persistent Storage...
[  OK  ] Started Update UTMP about System Boot/Shutdown.
[   11.174573] systemd-journald[377]: Received request to flush runtime journal from PID 1
[  OK  ] Started Trigger Flushing of Journal to Persistent Storage.
[  OK  ] Started Clean up broken links in /etc/ssl/certs.
[  OK  ] Started Rebuild Dynamic Linker Cache.
[  OK  ] Reached target System Initialization.
[DEPEND] Dependency failed for Load user-provided cloud configs.
[  OK  ] Reached target Paths.
[  OK  ] Listening on D-Bus System Message Bus Socket.
[  OK  ] Listening on OpenSSH Server Socket.
[  OK  ] Reached target Timers.
[  OK  ] Listening on Docker Socket for the API.
[  OK  ] Reached target Sockets.
[  OK  ] Reached target Basic System.
         Starting Generate sshd host keys...
         Starting Install an ssh key from /proc/cmdline...
         Starting Generate /run/issue...
         Starting Network Time Service...
[  OK  ] Started Network Time Service.
         Starting Update Engine...
         Starting Resize Btrfs Filesystems...
         Starting Login Service...
         Starting D-Bus System Message Bus...
[  OK  ] Started D-Bus System Message Bus.
         Starting Load cloud-config from /usr/share/oem/cloud-config.yml...
         Starting Network Service...
[  OK  ] Started Install an ssh key from /proc/cmdline.
[  OK  ] Started Generate /run/issue.
[  OK  ] Started Resize Btrfs Filesystems.
[  OK  ] Started Login Service.
[  OK  ] Started Generate sshd host keys.
[  OK  ] Started Network Service.
[  OK  ] Reached target Network.
         Starting Network Name Resolution...
         Starting Hostname Service...
[  OK  ] Started Network Name Resolution.
[  OK  ] Started Hostname Service.
         Starting Authorization Manager...
[  OK  ] Started Update Engine.
         Starting Cluster reboot manager...
[  OK  ] Started Cluster reboot manager.
         Starting Generate /run/coreos/motd...
[  OK  ] Started Authorization Manager.
[  OK  ] Stopped Cloudinit from EC2-style metadata.
         Starting Cloudinit from EC2-style metadata...
[  OK  ] Started Generate /run/coreos/motd.
         Starting Permit User Sessions...
[  OK  ] Started Permit User Sessions.
         Starting Serial Getty on hvc0...
[  OK  ] Started Serial Getty on hvc0.
         Starting Getty on tty1...
[  OK  ] Started Getty on tty1.
[  OK  ] Reached target Login Prompts.
         Starting etcd...
[  OK  ] Started etcd.
[  OK  ] Listening on fleet.socket.
         Starting fleet daemon...
[  OK  ] Started fleet daemon.
         Stopping Cluster reboot manager...
[  OK  ] Stopped Cluster reboot manager.
         Starting Cluster reboot manager...
[  OK  ] Started Cluster reboot manager.
[  OK  ] Started Cloudinit from EC2-style metadata.
[  OK  ] Started Load cloud-config from /usr/share/oem/cloud-config.yml.
[  OK  ] Reached target Load system-provided cloud configs.
[  OK  ] Reached target Multi-User System.


This is ip-10-3-0-130.eu-west-1.compute.internal.eu-west-1.compute.internal (Linux x86_64 3.16.2+) 14:09:41
SSH host key: <omitted> (DSA)
SSH host key: <omitted> (ED25519)
SSH host key: <omitted> (RSA)
eth0: 10.3.0.130 fe80::60:7eff:fe8b:ede1

ip-10-3-0-130 login: [   19.342091] ip_tables: (C) 2000-2006 Netfilter Core Team
[   20.534119] Bridge firewalling registered
[   20.540954] IPv6: ADDRCONF(NETDEV_UP): docker0: link is not ready
[   20.597359] nf_conntrack version 0.5.0 (16384 buckets, 65536 max)
[   23.991103] device veth8da0 entered promiscuous mode
[   23.991756] IPv6: ADDRCONF(NETDEV_UP): veth8da0: link is not ready
[   24.024198] IPv6: ADDRCONF(NETDEV_CHANGE): veth8da0: link becomes ready
[   24.024249] docker0: port 1(veth8da0) entered forwarding state
[   24.024261] docker0: port 1(veth8da0) entered forwarding state
[   24.024287] IPv6: ADDRCONF(NETDEV_CHANGE): docker0: link becomes ready
[   24.668670] IPv6: ADDRCONF(NETDEV_UP): veth2f37: link is not ready
[   24.675247] device veth647e entered promiscuous mode
[   24.688096] docker0: port 2(veth647e) entered forwarding state
[   24.688120] docker0: port 2(veth647e) entered forwarding state
[   24.689513] IPv6: ADDRCONF(NETDEV_CHANGE): veth2f37: link becomes ready
[   25.862354] IPv6: ADDRCONF(NETDEV_UP): veth754d: link is not ready
[   25.889338] IPv6: ADDRCONF(NETDEV_CHANGE): veth754d: link becomes ready
[   25.921108] device veth7580 entered promiscuous mode
[   25.921148] docker0: port 3(veth7580) entered forwarding state
[   25.921159] docker0: port 3(veth7580) entered forwarding state
[   28.353910] device veth4f10 entered promiscuous mode
[   28.354577] IPv6: ADDRCONF(NETDEV_UP): veth4f10: link is not ready
[   28.358754] IPv6: ADDRCONF(NETDEV_CHANGE): veth4f10: link becomes ready
[   28.358801] docker0: port 4(veth4f10) entered forwarding state
[   28.358812] docker0: port 4(veth4f10) entered forwarding state
[   39.075579] docker0: port 1(veth8da0) entered forwarding state
[   39.716059] docker0: port 2(veth647e) entered forwarding state
[   40.928075] docker0: port 3(veth7580) entered forwarding state
[   43.360035] docker0: port 4(veth4f10) entered forwarding state

Cloud-Config

#cloud-config

coreos:
  update:
    reboot-strategy: best-effort
  etcd:
    addr: $private_ipv4:4001
    peer-addr: $private_ipv4:7001
    discovery: https://discovery.etcd.io/omitted
    #discovery: https://discovery.etcd.io/<token>
  fleet:
    public-ip: $public_ipv4
    metadata: enviroment=dev,provider=aws,platform=cloud,region=eu-west,public_ip=$public_ipv4
  units:
    - name: etcd.service
      command: start
    - name: fleet.service
      command: start

Any input as to what is happening would be great.

Thanks.

crawford commented 9 years ago

What version of CoreOS is this? Could you try setting a password, rebooting, and then logging in with that password? If you are able to get in, can you attach the output of journalctl -b _EXE=/usr/bin/coreos-cloudinit --no-pager?

chrisjenx commented 9 years ago

Never mind, its up. Organic interface error. Route 53 was being really special. Sorry guys!