fluent / fluent-bit

Fast and Lightweight Logs and Metrics processor for Linux, BSD, OSX and Windows
https://fluentbit.io
Apache License 2.0
5.82k stars 1.58k forks source link

[Alpine Linux] fluent-bit spamming "[msgpack2json] unknown msgpack type 1970433894" to log #377

Closed taoeffect closed 7 years ago

taoeffect commented 7 years ago

I have no idea what's going on.

In the span of a few seconds fluent-bit generated a 16M log file consisting of:

[2017/09/17 23:32:37] [ warn] [msgpack2json] unknown msgpack type 1970433894
[2017/09/17 23:32:37] [ warn] [msgpack2json] unknown msgpack type 1970433894
[2017/09/17 23:32:37] [ warn] [msgpack2json] unknown msgpack type 1970433894
[2017/09/17 23:32:37] [ warn] [msgpack2json] unknown msgpack type 1970433894
[2017/09/17 23:32:37] [ warn] [msgpack2json] unknown msgpack type 1970433894
[2017/09/17 23:32:37] [ warn] [msgpack2json] unknown msgpack type 1970433894
[2017/09/17 23:32:37] [ warn] [msgpack2json] unknown msgpack type 1970433894
[2017/09/17 23:32:37] [ warn] [msgpack2json] unknown msgpack type 1970433894
[2017/09/17 23:32:37] [ warn] [msgpack2json] unknown msgpack type 1970433894
[2017/09/17 23:32:37] [ warn] [msgpack2json] unknown msgpack type 1970433894
[2017/09/17 23:32:37] [ warn] [msgpack2json] unknown msgpack type 1970433894
[2017/09/17 23:32:37] [ warn] [msgpack2json] unknown msgpack type 1970433894
[2017/09/17 23:32:37] [ warn] [msgpack2json] unknown msgpack type 1970433894

Am using Docker version 17.06.2-ce, build cec0b72, with fluent-bit version v0.12.2.

My dedicated container for logs contains this:

CMD fluent-bit -c /home/unpriv/log_to_files.conf

The config file contains this:

[SERVICE]
    Log_File  /home/unpriv/logs/fluent-bit.log

[INPUT]
    Name    forward
    Listen  0.0.0.0
    Port    24224

[OUTPUT]
    Name    file
    Match   *

[OUTPUT]
    Name    stdout

The /home/unpriv/logs/fluent-bit.log is the file containing the 16M worth of spammed msgpack2json messages.

The docker-compose.yml contains this:

version: "3.3"
services:
  mariadb:
    build: mariadb
    logging:
      driver: fluentd
      options:
        tag: "{{.Name}}-{{.ImageID}}"
    ports:
      - "3306:3306"
    depends_on:
      - logs
    environment:
      - MYSQL_ROOT_PASSWORD=password
    volumes:
      - ./vol/mariadb:/var/lib/mysql:delegated
  logs:
    build: logs
    restart: on-failure
    ports:
      - "24224:24224"
    volumes:
      - ./vol/logs:/home/unpriv/logs

It wasn't doing this before. No idea why it's happening now.

taoeffect commented 7 years ago

Now it's doing:

[2017/09/18 00:47:31] [ warn] [msgpack2json] unknown msgpack type 876228640
[2017/09/18 00:47:31] [ warn] [msgpack2json] unknown msgpack type 876228640
[2017/09/18 00:47:31] [ warn] [msgpack2json] unknown msgpack type 876228640
[2017/09/18 00:47:31] [ warn] [msgpack2json] unknown msgpack type 876228640
[2017/09/18 00:47:31] [ warn] [msgpack2json] unknown msgpack type 876228640

It seems to be caused by the output of the mariadb service, which has a Dockerfile of:

FROM localhost:5000/customalpine
RUN apk --update add mariadb mariadb-client && rm -f /var/cache/apk/*
VOLUME /var/lib/mysql
EXPOSE 3306
COPY configs/ /etc/mysql/
COPY scripts/docker-entrypoint.sh /usr/local/bin
ENTRYPOINT ["/usr/local/bin/docker-entrypoint.sh"]

Where docker-entrypoint.sh is:

#!/bin/sh

MYSQL_DATADIR=${MYSQL_DATADIR:-"/var/lib/mysql"}

if [ -d "/run/mysqld" ]; then
  echo "[i] MySQL directory already present, skipping creation"
else
  echo "[i] mysqld not found, creating...."
  mkdir -p /run/mysqld
  chown -R mysql:mysql /run/mysqld

  echo 'Initializing database'
  mkdir -p "$MYSQL_DATADIR/mysql"
  chown -R mysql:mysql $MYSQL_DATADIR
  mysql_install_db --user=mysql --datadir="$MYSQL_DATADIR" --rpm

  echo 'Database initialized'

  tfile=`mktemp`
  [ -f "$tfile" ] || echo "failed to mktemp!" && exit 1

  cat << EOF > $tfile
DELETE FROM mysql.user WHERE user NOT IN ('mysql.sys', 'mysqlxsys', 'root') OR host NOT IN ('localhost');
SET PASSWORD FOR 'root'@'localhost'=PASSWORD('${MYSQL_ROOT_PASSWORD}');
GRANT ALL ON *.* TO 'root'@'localhost' WITH GRANT OPTION;
DROP DATABASE IF EXISTS test;
FLUSH PRIVILEGES;
EOF

  /usr/bin/mysqld --user=mysql --bootstrap --verbose=1 --datadir="$MYSQL_DATADIR" < $tfile
  rm -f "$tfile"
fi

exec /usr/bin/mysqld --user=mysql --console --datadir="$MYSQL_DATADIR"

And configs contains configs/my.cnf:

[mysqld]
pid-file        = /var/run/mysqld/mysqld.pid
socket          = /var/run/mysqld/mysqld.sock
symbolic-links  = 0

# Custom config should go here
!includedir /etc/mysql/conf.d/

And configs/conf.d/docker.cnf:

[mysqld]
skip-host-cache
skip-name-resolve

I find it so strange that it wasn't doing this yesterday but now it's doing it every time and I can't get it to stop.

taoeffect commented 7 years ago

Should have mentioned, this is how the logs starts:

[2017/09/18 00:47:09] [ info] [engine] started
[2017/09/18 00:47:09] [ info] [in_fw] binding 0.0.0.0:24224
[2017/09/18 00:47:19] [ warn] [pack] invalid UTF-8 bytes, skipping
[2017/09/18 00:47:19] [ warn] unknown time format bf59ce92
[2017/09/18 00:47:19] [ warn] [msgpack2json] unknown msgpack type 876228640
[2017/09/18 00:47:19] [ warn] [msgpack2json] unknown msgpack type 876228640

Am using alpine linux, and all Docker containers should have these environment variables set:

ENV TERM=xterm LC_ALL=en_US.UTF-8 LANG=en_US.UTF-8
taoeffect commented 7 years ago

OK, I figured it out! 🎉 EDIT: Nope, didn't, see comment after next comment. 😢

It turns out that just having that ENV line in the parent customalpine Dockerfile was the problem!

Removing this from the custom alpine Dockerfile fixed the problem:

ENV TERM=xterm LC_ALL=en_US.UTF-8 LANG=en_US.UTF-8

Weirdly, having that in the Dockerfiles for the child log and mariadb services (or not having it!) didn't cause the problem to reappear.

So, I have no idea why this is. It's some very weirdly specific Docker+Alpine+Fluent-bit bug.

taoeffect commented 7 years ago

It's some very weirdly specific Docker+Alpine+Fluent-bit bug.

However, it still makes me nervous that this is at all possible. The problem could one day re-appear and result in a flooded log file. That problem shouldn't happen regardless.

The way syslog and others handle this is they output a single line to the file along the lines of:

... previous log message repeated 2000 times ...
taoeffect commented 7 years ago

Looks like I celebrated too soon.

Sometimes it works. Sometimes it doesn't. I've no idea what's going on or why.

[2017/09/18 05:08:16] [ info] [engine] started
[2017/09/18 05:08:16] [ info] [in_fw] binding 0.0.0.0:24224
[2017/09/18 05:08:31] [ warn] [pack] invalid UTF-8 bytes, skipping
[2017/09/18 05:08:31] [ warn] [pack] invalid UTF-8 bytes, skipping
[2017/09/18 05:08:31] [ warn] unknown time format bf59ce92
[2017/09/18 05:08:31] [ warn] [msgpack2json] unknown msgpack type 1714435937
[2017/09/18 05:08:31] [ warn] [msgpack2json] unknown msgpack type 1714435937

Here's the very end of the output from the mariadb log:

forums_mariadb_1-sha256:84e8a: [1505711309.000000, {"container_name":"/forums_mariadb_1", "source":"stderr", "log":"2017-09-18  5:08:29 139803331611464 [Note] InnoDB: innodb_empty_free_list_algorithm has been changed to legacy because of small buffer pool size. In order to use backoff, increase buffer pool at least up to 20MB.", "container_id":"caa70f6cbe6e7f5bf8bc48ac272ea371ffc037c3a6f6b02e5c73d0759d1a562a"}]
forums_mariadb_1-sha256:84e8a: [1505711309.000000, {"source":"stderr", "log":"", "container_id":"caa70f6cbe6e7f5bf8bc48ac272ea371ffc037c3a6f6b02e5c73d0759d1a562a", "container_name":"/forums_mariadb_1"}]
forums_mariadb_1-sha256:84e8a: [1505711309.000000, {"container_id":"caa70f6cbe6e7f5bf8bc48ac272ea371ffc037c3a6f6b02e5c73d0759d1a562a", "container_name":"/forums_mariadb_1", "source":"stderr", "log":"2017-09-18  5:08:29 139803331611464 [Note] InnoDB: Using mutexes to ref count buffer pool pages"}

The only anomaly that I see is that the second-to-last message contains an empty "log" value. Perhaps that's what's causing it..?

EDIT: I caught something else in another run:

forums_mariadb_1-sha256:b7f51: [1505712064.000000, {"container_id":"1c48013ede6b3941ad66f5e6b5871175905eda0a3cb33b4a95ffd926da5b642c", "container_name":"/forums_mariadb_1", "source":"stderr", "log":"2017-09-18  5:21:04 139901711743816 [Note] InnoDB: Completed initialization of buffer pool"}]
forums_mariadb_1-sha256:b7f51: [1505712064.000000, {"container_id":"1c48013ede6b3941ad66f5e6b5871175905eda0a3cb33b4a95ffd926da5b642c", "container_name":"/forums_mariadb_1", "source":"stderr", "log":"2017-09-18  5:21:04 1\u00a4\u0000s"}]
forums_mariadb_1-sha256:b7f51: [1505712064.000000, {"source":"stderr", "log":"2017-09-18  5:21:04 139901711743816 [Note] InnoDB: 128 rollback segment(s) are active.", "container_id":"1c48013ede6b3941ad66f5e6b5871175905eda0a3cb33b4", 6051919806976817155:0}

There are two anomalies in the logs:

edsiper commented 7 years ago

@taoeffect please provide the output of:

fluent-bit -c yourconf.conf --sosreport
marcbachmann commented 7 years ago

I'm having the same problem. I can reproduce it using that config:

[SERVICE]
    Flush        5
    Daemon       Off
    Log_Level    info
    Parsers_File /etc/fluent-bit/parsers.conf
    Log_File     /var/log/system-log-daemon

[INPUT]
    Name         kmsg
    Tag          kernel

[OUTPUT]
    Name         File
    Match        *
    Path         /var/log/messages

Somehow there's an issue in the file output. Everything works if I just use stdout. So if I replace the output with this one, it works:

[OUTPUT]
    Name         stdout
    Match        *

.... edit: weird, on the same system, same fluent-bit process, this one is working without any issues:

[SERVICE]
    Flush        1
    Daemon       Off
    Log_Level    info
    Parsers_File /etc/fluent-bit/parsers.conf
    Log_File     /var/log/docker-log-daemon

[INPUT]
    Name         forward
    Listen       127.0.0.1
    Port         24224
    Chunk_Size   32
    Buffer_Size  64
    Parser       docker

[FILTER]
    Name         kubernetes
    Match        *
    Merge_JSON_Log On
    Dummy_Meta   On

[FILTER]
    Name         record_modifier
    Match        *
    Remove_key   kubernetes

[OUTPUT]
    Name         File
    Match        *
    Path         /var/log/docker_messages
taoeffect commented 7 years ago

@taoeffect please provide the output of:

@edsiper Under what circumstances do I run that command? And do I run it while fluent-bit is already running? Remember, the error is intermittent and doesn't always happen.

marcbachmann commented 7 years ago

In my case the error occurs every time. Here's the whole output:

dmesg output before execution ``` [ 0.000000] Linux version 4.10.8-docker-1 (travis@scaleway-qa.pr-860) (gcc version 5.4.0 20160609 (Ubuntu 5.4.0-6ubuntu1~16.04.4) ) #1 SMP Tue Apr 4 16:18:07 UTC 2017 [ 0.000000] Command line: showopts console=ttyS1,9600n8 nousb vga=0 root=/dev/nbd0 scaleway boot=local [ 0.000000] x86/fpu: Legacy x87 FPU detected. [ 0.000000] e820: BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009dfff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009e000-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000e0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x000000007dafefff] usable [ 0.000000] BIOS-e820: [mem 0x000000007daff000-0x000000007f3befff] reserved [ 0.000000] BIOS-e820: [mem 0x000000007f3bf000-0x000000007f7befff] ACPI NVS [ 0.000000] BIOS-e820: [mem 0x000000007f7bf000-0x000000007f7fefff] ACPI data [ 0.000000] BIOS-e820: [mem 0x000000007f7ff000-0x000000007f7fffff] usable [ 0.000000] BIOS-e820: [mem 0x000000007f800000-0x000000007fffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000e0000000-0x00000000efffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fec80000-0x00000000fecfffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fed01000-0x00000000fed01fff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fed03000-0x00000000fed03fff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fed08000-0x00000000fed08fff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fed0c000-0x00000000fed0ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fed1c000-0x00000000fed1cfff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fef00000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000ffa00000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x000000027fffffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: Online Labs SR/SR, BIOS 00.00.00.0007 03/04/2016 [ 0.000000] e820: update [mem 0x00000000-0x00000fff] usable ==> reserved [ 0.000000] e820: remove [mem 0x000a0000-0x000fffff] usable [ 0.000000] e820: last_pfn = 0x280000 max_arch_pfn = 0x400000000 [ 0.000000] MTRR default type: uncachable [ 0.000000] MTRR fixed ranges enabled: [ 0.000000] 00000-9FFFF write-back [ 0.000000] A0000-FFFFF uncachable [ 0.000000] MTRR variable ranges enabled: [ 0.000000] 0 base 0FF000000 mask FFF000000 write-protect [ 0.000000] 1 base 000000000 mask F80000000 write-back [ 0.000000] 2 base 100000000 mask F00000000 write-back [ 0.000000] 3 base 200000000 mask F80000000 write-back [ 0.000000] 4 base 07F800000 mask FFF800000 uncachable [ 0.000000] 5 disabled [ 0.000000] 6 disabled [ 0.000000] 7 disabled [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WC UC- WT [ 0.000000] e820: last_pfn = 0x7f800 max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000fe1d0-0x000fe1df] mapped at [ffff8800000fe1d0] [ 0.000000] Base memory trampoline at [ffff880000088000] 88000 size 24576 [ 0.000000] BRK [0x01df9000, 0x01df9fff] PGTABLE [ 0.000000] BRK [0x01dfa000, 0x01dfafff] PGTABLE [ 0.000000] BRK [0x01dfb000, 0x01dfbfff] PGTABLE [ 0.000000] BRK [0x01dfc000, 0x01dfcfff] PGTABLE [ 0.000000] BRK [0x01dfd000, 0x01dfdfff] PGTABLE [ 0.000000] BRK [0x01dfe000, 0x01dfefff] PGTABLE [ 0.000000] RAMDISK: [mem 0x7d440000-0x7da61fff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000FE020 000024 (v02 INSYDE) [ 0.000000] ACPI: XSDT 0x000000007F7FE120 000094 (v01 INSYDE MOHOPEAK 00000001 01000013) [ 0.000000] ACPI: FACP 0x000000007F7F9000 00010C (v05 INSYDE MOHOPEAK 00000001 ACPI 00040000) [ 0.000000] ACPI: DSDT 0x000000007F7F2000 002D5E (v02 INSYDE MOHOPEAK 00000000 ACPI 00040000) [ 0.000000] ACPI: FACS 0x000000007F5AE000 000040 [ 0.000000] ACPI: UEFI 0x000000007F7FD000 000236 (v01 INSYDE MOHOPEAK 00000001 ACPI 00040000) [ 0.000000] ACPI: BDAT 0x000000007F7FB000 000030 (v01 INSYDE MOHOPEAK 00000000 ACPI 00040000) [ 0.000000] ACPI: BOOT 0x000000007F7FA000 000028 (v01 INSYDE MOHOPEAK 00000001 ACPI 00040000) [ 0.000000] ACPI: HPET 0x000000007F7F8000 000038 (v01 INSYDE MOHOPEAK 00000001 ACPI 00040000) [ 0.000000] ACPI: APIC 0x000000007F7F7000 000092 (v03 INSYDE MOHOPEAK 00000001 ACPI 00040000) [ 0.000000] ACPI: MCFG 0x000000007F7F6000 00003C (v01 INSYDE MOHOPEAK 00000001 ACPI 00040000) [ 0.000000] ACPI: WDAT 0x000000007F7F5000 0001C4 (v01 INSYDE MOHOPEAK 00000001 ACPI 00040000) [ 0.000000] ACPI: UEFI 0x000000007F7F1000 000042 (v01 INSYDE MOHOPEAK 00000000 ACPI 00040000) [ 0.000000] ACPI: SSDT 0x000000007F7F0000 0009F1 (v01 INSYDE MOHOPEAK 00003000 ACPI 00040000) [ 0.000000] ACPI: HEST 0x000000007F7EF000 0000A8 (v01 INSYDE MOHOPEAK 00000001 ACPI 00040000) [ 0.000000] ACPI: ERST 0x000000007F7EE000 000230 (v01 INSYDE MOHOPEAK 00000001 ACPI 00040000) [ 0.000000] ACPI: BERT 0x000000007F7ED000 000030 (v01 INSYDE MOHOPEAK 00000001 ACPI 00040000) [ 0.000000] ACPI: EINJ 0x000000007F7EC000 000150 (v01 INSYDE MOHOPEAK 00000001 ACPI 00040000) [ 0.000000] ACPI: Local APIC address 0xfee00000 [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x000000027fffffff] [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009dfff] [ 0.000000] node 0: [mem 0x0000000000100000-0x000000007dafefff] [ 0.000000] node 0: [mem 0x000000007f7ff000-0x000000007f7fffff] [ 0.000000] node 0: [mem 0x0000000100000000-0x000000027fffffff] [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x000000027fffffff] [ 0.000000] On node 0 totalpages: 2087581 [ 0.000000] DMA zone: 64 pages used for memmap [ 0.000000] DMA zone: 37 pages reserved [ 0.000000] DMA zone: 3997 pages, LIFO batch:0 [ 0.000000] DMA32 zone: 7980 pages used for memmap [ 0.000000] DMA32 zone: 510720 pages, LIFO batch:31 [ 0.000000] Normal zone: 24576 pages used for memmap [ 0.000000] Normal zone: 1572864 pages, LIFO batch:31 [ 0.000000] ACPI: PM-Timer IO Port: 0x408 [ 0.000000] ACPI: Local APIC address 0xfee00000 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] high edge lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 1, version 32, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: IRQ0 used by override. [ 0.000000] ACPI: IRQ9 used by override. [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] smpboot: Allowing 8 CPUs, 4 hotplug CPUs [ 0.000000] e820: [mem 0x80000000-0xdfffffff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on bare hardware [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604462750000 ns [ 0.000000] setup_percpu: NR_CPUS:128 nr_cpumask_bits:128 nr_cpu_ids:8 nr_node_ids:1 [ 0.000000] percpu: Embedded 32 pages/cpu @ffff88027fc00000 s91480 r8192 d31400 u262144 [ 0.000000] pcpu-alloc: s91480 r8192 d31400 u262144 alloc=1*2097152 [ 0.000000] pcpu-alloc: [0] 0 1 2 3 4 5 6 7 [ 0.000000] Built 1 zonelists in Zone order, mobility grouping on. Total pages: 2054924 [ 0.000000] Kernel command line: showopts console=ttyS1,9600n8 nousb vga=0 root=/dev/nbd0 scaleway boot=local [ 0.000000] log_buf_len individual max cpu contribution: 4096 bytes [ 0.000000] log_buf_len total cpu_extra contributions: 28672 bytes [ 0.000000] log_buf_len min size: 16384 bytes [ 0.000000] log_buf_len: 65536 bytes [ 0.000000] early log buf free: 8204(50%) [ 0.000000] PID hash table entries: 4096 (order: 3, 32768 bytes) [ 0.000000] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes) [ 0.000000] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes) [ 0.000000] Memory: 8119080K/8350324K available (7401K kernel code, 603K rwdata, 2140K rodata, 924K init, 452K bss, 231244K reserved, 0K cma-reserved) [ 0.000000] Hierarchical RCU implementation. [ 0.000000] Build-time adjustment of leaf fanout to 64. [ 0.000000] RCU restricting CPUs from NR_CPUS=128 to nr_cpu_ids=8. [ 0.000000] RCU: Adjusting geometry for rcu_fanout_leaf=64, nr_cpu_ids=8 [ 0.000000] NR_IRQS:8448 nr_irqs:488 16 [ 0.000000] Console: colour dummy device 80x25 [ 0.000000] console [ttyS1] enabled [ 0.000000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 133484882848 ns [ 0.000000] hpet clockevent registered [ 0.000000] tsc: Fast TSC calibration using PIT [ 0.000000] tsc: Detected 2393.905 MHz processor [ 0.000004] Calibrating delay loop (skipped), value calculated using timer frequency.. 4787.81 BogoMIPS (lpj=23939050) [ 0.128569] pid_max: default: 32768 minimum: 301 [ 0.184050] ACPI: Core revision 20160930 [ 0.234268] ACPI: 2 ACPI AML tables successfully acquired and loaded [ 0.310695] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes) [ 0.393313] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes) [ 0.481473] CPU: Physical Processor ID: 0 [ 0.529631] CPU: Processor Core ID: 0 [ 0.573612] mce: CPU supports 6 MCE banks [ 0.621768] process: using mwait in idle threads [ 0.677230] Last level iTLB entries: 4KB 48, 2MB 0, 4MB 0 [ 0.742086] Last level dTLB entries: 4KB 128, 2MB 16, 4MB 16, 1GB 0 [ 0.817596] Freeing SMP alternatives memory: 32K [ 0.874026] smpboot: Max logical packages: 2 [ 0.925754] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 1.097929] TSC deadline timer enabled [ 1.097933] smpboot: CPU0: Intel(R) Atom(TM) CPU C2550 @ 2.40GHz (family: 0x6, model: 0x4d, stepping: 0x8) [ 1.216145] Performance Events: PEBS fmt2+, 8-deep LBR, Silvermont events, 8-deep LBR, full-width counters, Intel PMU driver. [ 1.352025] ... version: 3 [ 1.400172] ... bit width: 40 [ 1.449359] ... generic registers: 2 [ 1.497501] ... value mask: 000000ffffffffff [ 1.561309] ... max period: 0000007fffffffff [ 1.625118] ... fixed-purpose events: 3 [ 1.673260] ... event mask: 0000000700000003 [ 1.737407] smp: Bringing up secondary CPUs ... [ 1.791971] x86: Booting SMP configuration: [ 1.842208] .... node #0, CPUs: #1 #2 #3 [ 2.292362] smp: Brought up 1 node, 4 CPUs [ 2.394920] smpboot: Total of 4 processors activated (19153.61 BogoMIPS) [ 2.476544] devtmpfs: initialized [ 2.516665] PM: Registering ACPI NVS region [mem 0x7f3bf000-0x7f7befff] (4194304 bytes) [ 2.613033] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604462750000 ns [ 2.731160] futex hash table entries: 2048 (order: 5, 131072 bytes) [ 2.806526] xor: measuring software checksum speed [ 2.964093] prefetch64-sse: 9251.600 MB/sec [ 3.118532] generic_sse: 8212.000 MB/sec [ 3.169811] xor: using function: prefetch64-sse (9251.600 MB/sec) [ 3.243241] NET: Registered protocol family 16 [ 3.336635] cpuidle: using governor ladder [ 3.385878] Simple Boot Flag at 0x44 set to 0x80 [ 3.441347] ACPI FADT declares the system doesn't support PCIe ASPM, so disable it [ 3.532313] ACPI: bus type PCI registered [ 3.580528] PCI: Using configuration type 1 for base access [ 3.697746] HugeTLB registered 2 MB page size, pre-allocated 0 pages [ 3.943198] raid6: sse2x1 gen() 676 MB/s [ 4.163195] raid6: sse2x1 xor() 2455 MB/s [ 4.383236] raid6: sse2x2 gen() 1076 MB/s [ 4.603277] raid6: sse2x2 xor() 2920 MB/s [ 4.823335] raid6: sse2x4 gen() 1809 MB/s [ 5.043358] raid6: sse2x4 xor() 2582 MB/s [ 5.094640] raid6: using algorithm sse2x4 gen() 1809 MB/s [ 5.159491] raid6: .... xor() 2582 MB/s, rmw enabled [ 5.219121] raid6: using ssse3x2 recovery algorithm [ 5.277746] ACPI: Added _OSI(Module Device) [ 5.327985] ACPI: Added _OSI(Processor Device) [ 5.381351] ACPI: Added _OSI(3.0 _SCP Extensions) [ 5.437855] ACPI: Added _OSI(Processor Aggregator Device) [ 5.502947] ACPI: Executed 1 blocks of module-level executable AML code [ 5.584852] ACPI: [Firmware Bug]: BIOS _OSI(Linux) query ignored [ 5.657678] ACPI: Dynamic OEM Table Load: [ 5.705834] ACPI: SSDT 0xFFFF880276BF6200 000107 (v01 PmRef Cpu0Ist 00003000 INTL 20130117) [ 5.808526] ACPI: Dynamic OEM Table Load: [ 5.856679] ACPI: SSDT 0xFFFF880276BF6000 0001FA (v01 PmRef Cpu0Cst 00003001 INTL 20130117) [ 5.959714] ACPI: Dynamic OEM Table Load: [ 6.007881] ACPI: SSDT 0xFFFF880276B04800 00047A (v01 PmRef ApIst 00003000 INTL 20130117) [ 6.110862] ACPI: Dynamic OEM Table Load: [ 6.159016] ACPI: SSDT 0xFFFF880276AF1E00 000119 (v01 PmRef ApCst 00003000 INTL 20130117) [ 6.262658] ACPI: Interpreter enabled [ 6.306651] ACPI: (supports S0 S5) [ 6.347491] ACPI: Using IOAPIC for interrupt routing [ 6.407169] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 6.521996] ACPI: Power Resource [FN00] (off) [ 6.574989] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 6.649252] acpi PNP0A08:00: _OSC: OS supports [Segments MSI] [ 6.718444] acpi PNP0A08:00: PCIe port services disabled; not requesting _OSC control [ 6.812754] PCI host bridge to bus 0000:00 [ 6.861950] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 6.943516] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 7.025083] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 7.115008] pci_bus 0000:00: root bus resource [mem 0x80000000-0xdfffffff window] [ 7.204927] pci_bus 0000:00: root bus resource [mem 0x280000000-0xfffffffff window] [ 7.296932] pci_bus 0000:00: root bus resource [bus 00-ff] [ 7.362845] pci 0000:00:00.0: [8086:1f02] type 00 class 0x060000 [ 7.362995] pci 0000:00:0e.0: [8086:1f14] type 00 class 0x060000 [ 7.363130] pci 0000:00:0f.0: [8086:1f16] type 00 class 0x080600 [ 7.363182] pci 0000:00:0f.0: PME# supported from D0 D3hot D3cold [ 7.363308] pci 0000:00:13.0: [8086:1f15] type 00 class 0x088000 [ 7.363326] pci 0000:00:13.0: reg 0x10: [mem 0x80025000-0x800253ff 64bit] [ 7.363518] pci 0000:00:14.0: [8086:1f45] type 00 class 0x020000 [ 7.363535] pci 0000:00:14.0: reg 0x10: [mem 0x80000000-0x8001ffff 64bit] [ 7.363544] pci 0000:00:14.0: reg 0x18: [io 0x1040-0x105f] [ 7.363563] pci 0000:00:14.0: reg 0x20: [mem 0x80020000-0x80023fff 64bit] [ 7.363613] pci 0000:00:14.0: PME# supported from D0 D3hot D3cold [ 7.363682] pci 0000:00:14.0: System wakeup disabled by ACPI [ 7.431738] pci 0000:00:16.0: [8086:1f2c] type 00 class 0x0c0320 [ 7.431760] pci 0000:00:16.0: reg 0x10: [mem 0x80026000-0x800263ff] [ 7.431860] pci 0000:00:16.0: PME# supported from D0 D3hot D3cold [ 7.431929] pci 0000:00:16.0: System wakeup disabled by ACPI [ 7.499980] pci 0000:00:18.0: [8086:1f32] type 00 class 0x010601 [ 7.499996] pci 0000:00:18.0: reg 0x10: [io 0x1068-0x106f] [ 7.500004] pci 0000:00:18.0: reg 0x14: [io 0x1074-0x1077] [ 7.500012] pci 0000:00:18.0: reg 0x18: [io 0x1060-0x1067] [ 7.500020] pci 0000:00:18.0: reg 0x1c: [io 0x1070-0x1073] [ 7.500028] pci 0000:00:18.0: reg 0x20: [io 0x1020-0x103f] [ 7.500036] pci 0000:00:18.0: reg 0x24: [mem 0x80024000-0x800247ff] [ 7.500079] pci 0000:00:18.0: PME# supported from D3hot [ 7.500196] pci 0000:00:1f.0: [8086:1f38] type 00 class 0x060100 [ 7.500377] pci 0000:00:1f.3: [8086:1f3c] type 00 class 0x0c0500 [ 7.500393] pci 0000:00:1f.3: reg 0x10: [mem 0x80027000-0x8002701f] [ 7.500424] pci 0000:00:1f.3: reg 0x20: [io 0x1000-0x101f] [ 7.500561] pci_bus 0000:00: on NUMA node 0 [ 7.500946] ACPI: PCI Interrupt Link [LNKA] (IRQs 3 4 5 6 10 *11 12 14 15) [ 7.583667] ACPI: PCI Interrupt Link [LNKB] (IRQs 3 4 5 6 *10 11 12 14 15) [ 7.666389] ACPI: PCI Interrupt Link [LNKC] (IRQs 3 4 5 6 *10 11 12 14 15) [ 7.749115] ACPI: PCI Interrupt Link [LNKD] (IRQs 3 4 5 6 10 11 12 14 15) *0, disabled. [ 7.845418] ACPI: PCI Interrupt Link [LNKE] (IRQs 3 4 5 6 10 11 12 14 15) *0, disabled. [ 7.941718] ACPI: PCI Interrupt Link [LNKF] (IRQs 3 4 5 6 10 *11 12 14 15) [ 8.024443] ACPI: PCI Interrupt Link [LNKG] (IRQs 3 4 5 6 10 11 12 14 15) *0, disabled. [ 8.120746] ACPI: PCI Interrupt Link [LNKH] (IRQs 3 4 5 6 10 11 12 14 15) *0 [ 8.205748] ACPI: Enabled 1 GPEs in block 00 to 3F [ 8.263637] vgaarb: loaded [ 8.296246] SCSI subsystem initialized [ 8.341322] libata version 3.00 loaded. [ 8.341373] ACPI: bus type USB registered [ 8.389562] usbcore: registered new interface driver usbfs [ 8.455481] usbcore: registered new interface driver hub [ 8.519322] usbcore: registered new device driver usb [ 8.580041] pps_core: LinuxPPS API ver. 1 registered [ 8.639679] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 8.749450] PTP clock support registered [ 8.796649] PCI: Using ACPI for IRQ routing [ 8.846894] PCI: pci_cache_line_size set to 64 bytes [ 8.846921] e820: reserve RAM buffer [mem 0x0009e000-0x0009ffff] [ 8.846924] e820: reserve RAM buffer [mem 0x7daff000-0x7fffffff] [ 8.846925] e820: reserve RAM buffer [mem 0x7f800000-0x7fffffff] [ 8.847283] clocksource: Switched to clocksource hpet [ 8.908050] VFS: Disk quotas dquot_6.6.0 [ 8.955185] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 9.037874] pnp: PnP ACPI init [ 9.074773] system 00:00: [mem 0xe0000000-0xefffffff] has been reserved [ 9.154253] system 00:00: [mem 0x7f800000-0x7fffffff] has been reserved [ 9.233734] system 00:00: Plug and Play ACPI device, IDs PNP0c01 (active) [ 9.233802] system 00:01: [io 0x0400-0x047f] has been reserved [ 9.304928] system 00:01: [io 0x0500-0x05fe] has been reserved [ 9.376051] system 00:01: [io 0x0680-0x069f] has been reserved [ 9.447176] system 00:01: [mem 0xfed00000-0xfedfffff] could not be reserved [ 9.530833] system 00:01: [mem 0x000c0000-0x000dffff] could not be reserved [ 9.614488] system 00:01: [mem 0x000e0000-0x000fffff] could not be reserved [ 9.698145] system 00:01: [mem 0xffa00000-0xffffffff] has been reserved [ 9.777626] system 00:01: Plug and Play ACPI device, IDs PNP0c02 (active) [ 9.777676] pnp 00:02: Plug and Play ACPI device, IDs PNP0b00 (active) [ 9.777760] pnp 00:03: Plug and Play ACPI device, IDs PNP0501 (active) [ 9.777835] pnp 00:04: Plug and Play ACPI device, IDs PNP0501 (active) [ 9.778098] pnp: PnP ACPI: found 5 devices [ 9.834973] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 9.941622] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 9.941625] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 9.941627] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 9.941629] pci_bus 0000:00: resource 7 [mem 0x80000000-0xdfffffff window] [ 9.941632] pci_bus 0000:00: resource 8 [mem 0x280000000-0xfffffffff window] [ 9.941728] NET: Registered protocol family 2 [ 9.994282] TCP established hash table entries: 65536 (order: 7, 524288 bytes) [ 10.081222] TCP bind hash table entries: 65536 (order: 8, 1048576 bytes) [ 10.162004] TCP: Hash tables configured (established 65536 bind 65536) [ 10.240505] UDP hash table entries: 4096 (order: 5, 131072 bytes) [ 10.313771] UDP-Lite hash table entries: 4096 (order: 5, 131072 bytes) [ 10.392332] NET: Registered protocol family 1 [ 10.444935] PCI: CLS 64 bytes, default 64 [ 10.445032] Unpacking initramfs... [ 10.614060] Freeing initrd memory: 6280K [ 10.661193] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 10.738591] software IO TLB [mem 0x79440000-0x7d440000] (64MB) mapped at [ffff880079440000-ffff88007d43ffff] [ 10.857876] AVX or AES-NI instructions are not detected. [ 10.921701] CPU feature 'AVX registers' is not supported. [ 10.986560] AVX2 or AES-NI instructions are not detected. [ 11.051802] audit: initializing netlink subsys (disabled) [ 11.116938] Initialise system trusted keyrings [ 11.170448] workingset: timestamp_bits=46 max_order=21 bucket_order=0 [ 11.249689] async_tx: api initialized (async) [ 11.302024] Key type asymmetric registered [ 11.351217] Asymmetric key parser 'x509' registered [ 11.409836] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 249) [ 11.498757] io scheduler noop registered [ 11.545870] io scheduler deadline registered (default) [ 11.607602] io scheduler cfq registered [ 11.653864] input: Sleep Button as /devices/LNXSYSTM:00/LNXSYBUS:00/PNP0C0E:00/input/input0 [ 11.754238] ACPI: Sleep Button [SLPB] [ 11.798276] input: Power Button as /devices/LNXSYSTM:00/LNXSYBUS:00/PNP0C0C:00/input/input1 [ 11.898646] ACPI: Power Button [PWRB] [ 11.942685] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input2 [ 12.031566] ACPI: Power Button [PWRF] [ 12.075993] tsc: Refined TSC clocksource calibration: 2393.903 MHz [ 12.076116] Monitor-Mwait will be used to enter C-1 state [ 12.076138] Monitor-Mwait will be used to enter C-2 state [ 12.150266] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x2281b7b41ef, max_idle_ns: 440795228866 ns [ 12.271070] (NULL device *): hwmon_device_register() is deprecated. Please convert the driver to use hwmon_device_register_with_info(). [ 12.417550] thermal LNXTHERM:00: registered as thermal_zone0 [ 12.485542] ACPI: Thermal Zone [TZ01] (21 C) [ 12.556281] Serial: 8250/16550 driver, 4 ports, IRQ sharing disabled [ 12.652932] 00:03: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 12.761111] 00:04: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 12.851756] nbd: registered device at major 43 [ 12.908456] zram: Added device: zram0 [ 12.952502] ahci 0000:00:18.0: version 3.0 [ 12.962838] ahci 0000:00:18.0: AHCI 0001.0300 32 slots 2 ports 3 Gbps 0x3 impl SATA mode [ 13.060080] ahci 0000:00:18.0: flags: 64bit ncq led clo pio deso sadm sds apst [ 13.148424] scsi host0: ahci [ 13.183168] scsi host1: ahci [ 13.217824] ata1: SATA max UDMA/133 abar m2048@0x80024000 port 0x80024100 irq 24 [ 13.306705] ata2: SATA max UDMA/133 abar m2048@0x80024000 port 0x80024180 irq 24 [ 13.395830] libphy: Fixed MDIO Bus: probed [ 13.445146] tun: Universal TUN/TAP device driver, 1.6 [ 13.505830] tun: (C) 1999-2004 Max Krasnyansky [ 13.580159] clocksource: Switched to clocksource tsc [ 13.580178] CAN device driver interface [ 13.580180] e1000: Intel(R) PRO/1000 Network Driver - version 7.3.21-k8-NAPI [ 13.580180] e1000: Copyright (c) 1999-2006 Intel Corporation. [ 13.580203] e1000e: Intel(R) PRO/1000 Network Driver - 3.2.6-k [ 13.580203] e1000e: Copyright(c) 1999 - 2015 Intel Corporation. [ 13.580226] igb: Intel(R) Gigabit Ethernet Network Driver - version 5.4.0-k [ 13.580227] igb: Copyright (c) 2007-2014 Intel Corporation. [ 13.619035] igb 0000:00:14.0: added PHC on eth0 [ 13.619037] igb 0000:00:14.0: Intel(R) Gigabit Ethernet Network Connection [ 13.619113] igb 0000:00:14.0: eth0: PBA No: 001900-000 [ 13.619115] igb 0000:00:14.0: Using MSI-X interrupts. 4 rx queue(s), 4 tx queue(s) [ 13.619144] igbvf: Intel(R) Gigabit Virtual Function Network Driver - version 2.4.0-k [ 13.619145] igbvf: Copyright (c) 2009 - 2012 Intel Corporation. [ 13.619167] ixgbe: Intel(R) 10 Gigabit PCI Express Network Driver - version 4.4.0-k [ 13.619167] ixgbe: Copyright (c) 1999-2016 Intel Corporation. [ 13.687404] ixgbevf: Intel(R) 10 Gigabit PCI Express Virtual Function Network Driver - version 3.2.2-k [ 13.687405] ixgbevf: Copyright (c) 2009 - 2015 Intel Corporation. [ 13.687497] i40e: Intel(R) Ethernet Connection XL710 Network Driver - version 1.6.25-k [ 13.687498] i40e: Copyright (c) 2013 - 2014 Intel Corporation. [ 13.687561] ixgb: Intel(R) PRO/10GbE Network Driver - version 1.0.135-k2-NAPI [ 13.687562] ixgb: Copyright (c) 1999-2008 Intel Corporation. [ 13.687583] i40evf: Intel(R) 40-10 Gigabit Virtual Function Network Driver - version 1.6.25-k [ 13.687583] Copyright (c) 2013 - 2015 Intel Corporation. [ 13.687639] Intel(R) Ethernet Switch Host Interface Driver - version 0.21.2-k [ 13.687640] Copyright (c) 2013 - 2016 Intel Corporation. [ 13.687729] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver [ 13.687730] ehci-pci: EHCI PCI platform driver [ 13.687895] ehci-pci 0000:00:16.0: EHCI Host Controller [ 13.687904] ehci-pci 0000:00:16.0: new USB bus registered, assigned bus number 1 [ 13.687922] ehci-pci 0000:00:16.0: debug port 2 [ 13.691829] ehci-pci 0000:00:16.0: cache line size of 64 is not supported [ 13.691846] ehci-pci 0000:00:16.0: irq 23, io mem 0x80026000 [ 13.717281] ehci-pci 0000:00:16.0: USB 2.0 started, EHCI 1.00 [ 13.717515] hub 1-0:1.0: USB hub found [ 13.717529] hub 1-0:1.0: 8 ports detected [ 13.717843] usbcore: registered new interface driver usb-storage [ 13.717890] i8042: PNP: No PS/2 controller found. [ 13.717993] mousedev: PS/2 mouse device common for all mice [ 13.718352] usbcore: registered new interface driver usbhid [ 13.718353] usbhid: USB HID core driver [ 13.718378] Netfilter messages via NETLINK v0.30. [ 13.718383] nfnl_acct: registering with nfnetlink. [ 13.770568] nf_conntrack version 0.5.0 (65536 buckets, 262144 max) [ 13.839639] ip_set: protocol 6 [ 13.839644] IPVS: Registered protocols (TCP) [ 13.909149] ata2: SATA link down (SStatus 0 SControl 300) [ 13.909686] IPVS: Connection hash table configured (size=4096, memory=64Kbytes) [ 13.909733] IPVS: Creating netns size=1656 id=0 [ 13.909752] IPVS: ipvs loaded. [ 13.909811] ip_tables: (C) 2000-2006 Netfilter Core Team [ 14.059163] ata1: SATA link down (SStatus 0 SControl 300) [ 14.087328] usb 1-1: new high-speed USB device number 2 using ehci-pci [ 14.185812] NET: Registered protocol family 10 [ 14.268424] Segment Routing with IPv6 [ 14.268449] mip6: Mobile IPv6 [ 14.268479] ip6_tables: (C) 2000-2006 Netfilter Core Team [ 14.268693] sit: IPv6, IPv4 and MPLS over IPv4 tunneling driver [ 14.269055] NET: Registered protocol family 17 [ 14.269062] NET: Registered protocol family 15 [ 14.269071] bridge: filtering via arp/ip/ip6tables is no longer available by default. Update your scripts to load br_netfilter if you need this. [ 14.269072] Ebtables v2.0 registered [ 14.269073] can: controller area network core (rev 20120528 abi 9) [ 14.269091] NET: Registered protocol family 29 [ 14.269092] can: raw protocol (rev 20120528) [ 14.269093] can: broadcast manager protocol (rev 20161123 t) [ 14.269095] can: netlink gateway (rev 20130117) max_hops=1 [ 14.297937] hub 1-1:1.0: USB hub found [ 14.298013] hub 1-1:1.0: 4 ports detected [ 14.330139] 8021q: 802.1Q VLAN Support v1.8 [ 18.201680] SSE version of gcm_enc/dec engaged. [ 18.257129] registered taskstats version 1 [ 18.306334] Loading compiled-in X.509 certificates [ 18.364457] Key type encrypted registered [ 18.412803] hctosys: unable to open rtc device (rtc0) [ 18.475219] Freeing unused kernel memory: 924K [ 18.528604] Write protecting the kernel read-only data: 12288k [ 18.599271] Freeing unused kernel memory: 776K [ 18.661071] Freeing unused kernel memory: 1956K [ 19.297830] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready [ 19.371227] 8021q: adding VLAN 0 to HW filter on device eth0 [ 19.455968] igb 0000:00:14.0 eth0: igb: eth0 NIC Link is Up 2500 Mbps Full Duplex, Flow Control: RX/TX [ 19.584620] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready [ 24.171110] EXT4-fs (nbd0): couldn't mount as ext3 due to feature incompatibilities [ 24.171236] EXT4-fs (nbd0): couldn't mount as ext2 due to feature incompatibilities [ 24.187743] random: fast init done [ 24.599570] EXT4-fs (nbd0): recovery complete [ 24.599989] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ 31.605546] random: crng init done [ 39.777894] EXT4-fs (nbd1): mounted filesystem with ordered data mode. Opts: (null) [ 41.351612] Bridge firewalling registered [ 41.433712] Initializing XFRM netlink socket [ 41.495216] IPv6: ADDRCONF(NETDEV_UP): docker0: link is not ready ```
--sosreport output before execution ``` Fluent-Bit v0.12.1 Copyright (C) Treasure Data Fluent Bit Enterprise - SOS Report ================================== The following report aims to be used by Fluent Bit and Fluentd Enterprise Customers of Treasure Data. For more details visit: https://fluentd.treasuredata.com [Fluent Bit] Edition Community Edition Version 0.12.1 Built Flags JSMN_PARENT_LINKS JSMN_STRICT FLB_HAVE_TLS FLB_HAVE_SQLDB FLB_HAVE_TRACE FLB_HAVE_FLUSH_LIBCO FLB_HAVE_FORK FLB_HAVE_PROXY_GO FLB_HAVE_REGEX FLB_HAVE_C_TLS FLB_HAVE_SETJMP FLB_HAVE_ACCEPT4 FLB_HAVE_INOTIFY [Operating System] Name Linux Release 4.10.8-docker-1 Version #1 SMP Tue Apr 4 16:18:07 UTC 2017 [Hardware] Architecture x86_64 Processors 4 [Built Plugins] Inputs cpu mem kmsg tail proc disk netif dummy head health serial stdin tcp mqtt lib forward random syslog Filters grep stdout kubernetes parser record_modifier Outputs counter es exit file forward http influxdb kafka-rest nats null plot stdout td lib flowcounter [SERVER] Runtime configuration Flush 5 Daemon Off Log_Level Info [INPUT] Instance Name kmsg.0 (kmsg, id=0) Flags Threaded No Tag kernel Routes file.0 [OUTPUT] Instance Name file.0 (file, mask_id=1) Match * TLS Active No Retry Limit 1 Path /var/log/messages-new ```
Execute command & cancel using ctrl+c ``` scw-9aa909:~# /bin/fluent-bit -c /etc/fluent-bit/system-logs.conf Fluent-Bit v0.12.1 Copyright (C) Treasure Data ^C[engine] caught signal Segmentation fault ```
dmesg output after cancelling ``` [ 24.171110] EXT4-fs (nbd0): couldn't mount as ext3 due to feature incompatibilities [ 24.171236] EXT4-fs (nbd0): couldn't mount as ext2 due to feature incompatibilities [ 24.187743] random: fast init done [ 24.599570] EXT4-fs (nbd0): recovery complete [ 24.599989] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ 31.605546] random: crng init done [ 39.777894] EXT4-fs (nbd1): mounted filesystem with ordered data mode. Opts: (null) [ 41.351612] Bridge firewalling registered [ 41.433712] Initializing XFRM netlink socket [ 41.495216] IPv6: ADDRCONF(NETDEV_UP): docker0: link is not ready [ 170.143472] traps: fluent-bit[3501] general protection ip:7fc6bdbe1b54 sp:55e179d11bb0 error:0 [ 170.143477] in ld-musl-x86_64.so.1[7fc6bdbbe000+89000] ```



If it helps, here's the whole server setup I'm using: https://github.com/marcbachmann/image-alpine

I could also give access to an instance.

edsiper commented 7 years ago

thanks for the information.

Just a disclaimer: Alpine Linux is not supported, use our official Debian images instead

taoeffect commented 7 years ago

Just a disclaimer: Alpine Linux is not supported, use our official Debian images instead

Is that a "WONTFIX"?

Would be a shame, since it excludes a large portion of potential users. Docker makes little sense with Debian, and a lot of sense with Alpine. :-\

marcbachmann commented 7 years ago

Is that a "WONTFIX"?

Regarding the issue, I don't think it's alpine specific. With the csv file format output I sometimes get a segfault right away but not that huge amount of errors like with the default formatter. ltsv is a bit more stable, there are just less errors.

edsiper commented 7 years ago

About Alpine + Fluent Bit: we had many issues, specifically with:

We invested a lot of time to deal with Alpine, we will reconsider to support it once issues mentioned above becomes fixed.

about the main issue in question, I will troubleshoot.

edsiper commented 7 years ago

@taoeffect @marcbachmann

I am trying to reproduce the problem.. would you please provide specific Dockerfiles or full compose file to test ?

marcbachmann commented 7 years ago

Dockerfile

FROM alpine:3.6
ENV FLB_MAJOR 0
ENV FLB_MINOR 12
ENV FLB_PATCH 2
ENV FLB_VERSION 0.12.2

RUN apk --no-cache add file build-base ca-certificates cmake && \
    wget -O "/tmp/fluent-bit-$FLB_VERSION.tar.gz" "http://fluentbit.io/releases/$FLB_MAJOR.$FLB_MINOR/fluent-bit-$FLB_VERSION.tar.gz" && \
    cd /tmp && \
    tar zxfv "fluent-bit-$FLB_VERSION.tar.gz" && \
    cd "fluent-bit-$FLB_VERSION/build/" && \
    cmake -DFLB_DEBUG=On -DFLB_TRACE=On -DFLB_JEMALLOC=On -DFLB_BUFFERING=On -DCMAKE_INSTALL_PREFIX=/ ../ && \
    make && make install

log.conf

[SERVICE]
    Flush        5
    Daemon       Off
    Log_Level    info
    Parsers_File /etc/fluent-bit/parsers.conf

[INPUT]
    Name   kmsg
    Tag    kernel

[OUTPUT]
    Name         File
    Match        *
    Path         /var/log/messages
docker build -t fluent-bit .
docker run -it --rm --privileged -v $PWD/log.conf:/log.conf fluent-bit fluent-bit -c /log.conf

--privileged in the run command is used to get access to /proc/kmsg

edsiper commented 7 years ago

thanks for the test case.

After digging on it for an hour I could not find the root of the cause, it's not just the message on the screen, there some weird behaviors of the default memory allocator (or maybe valgrind on Alpine?):

==14== Thread 1:
==14== Syscall param epoll_pwait(sigmask) points to unaddressable byte(s)
==14==    at 0x4020D39: epoll_pwait (in /lib/ld-musl-x86_64.so.1)
==14==    by 0x2EBB52: mk_event_wait (mk_event.c:154)
==14==    by 0x13D13C: flb_engine_start (flb_engine.c:466)
==14==    by 0x131283: main (fluent-bit.c:729)
==14==  Address 0x0 is not stack'd, malloc'd or (recently) free'd
==14== 
==14== Invalid read of size 1
==14==    at 0x40484A0: __fwritex (in /lib/ld-musl-x86_64.so.1)
==14==    by 0x3: ???
==14==  Address 0x4ea8b01 is 33 bytes inside a block of size 317 alloc'd
==14==    at 0x4C929C5: malloc (vg_replace_malloc.c:299)
==14==    by 0x133824: flb_malloc (flb_mem.h:57)
==14==    by 0x135BEA: flb_input_flush (flb_input.c:890)
==14==    by 0x13E66F: flb_engine_dispatch (flb_engine_dispatch.c:168)
==14==    by 0x13BD4F: flb_engine_flush (flb_engine.c:83)
==14==    by 0x13D1D0: flb_engine_handle_event (flb_engine.c:257)
==14==    by 0x13D1D0: flb_engine_start (flb_engine.c:469)
==14==    by 0x131283: main (fluent-bit.c:729)
==14== 
==14== Invalid read of size 1
==14==    at 0x4050213: ??? (in /lib/ld-musl-x86_64.so.1)
==14==    by 0x40484C1: __fwritex (in /lib/ld-musl-x86_64.so.1)
==14==    by 0x3: ???
==14==  Address 0x4ea8afc is 28 bytes inside a block of size 317 alloc'd
==14==    at 0x4C929C5: malloc (vg_replace_malloc.c:299)
==14==    by 0x133824: flb_malloc (flb_mem.h:57)
==14==    by 0x135BEA: flb_input_flush (flb_input.c:890)
==14==    by 0x13E66F: flb_engine_dispatch (flb_engine_dispatch.c:168)
==14==    by 0x13BD4F: flb_engine_flush (flb_engine.c:83)
==14==    by 0x13D1D0: flb_engine_handle_event (flb_engine.c:257)
==14==    by 0x13D1D0: flb_engine_start (flb_engine.c:469)
==14==    by 0x131283: main (fluent-bit.c:729)

since Alpine is not supported I suggest you use our Fluent Bit image (debian based). I understand the current image side is big (140MB) but next version will be around 80MB.

edsiper commented 7 years ago

Fluent Bit 0.12.3 has been released:

The new Docker image is about 79MB (used to be > 100MB):

Since Alpine is not supported I am closing this ticket.

taoeffect commented 7 years ago

Thanks @edsiper! Am looking forward to trying this out, and will let you know if I run into the same problem again. Appreciate the small image size.