search

influxdata / telegraf

Agent for collecting, processing, aggregating, and writing metrics, logs, and other arbitrary data.
https://influxdata.com/telegraf
MIT License
14.9k stars 5.6k forks source link

telegraf crashes with fatal error: heapBitsBulkBarrier: unaligned arguments #2724

Closed dirkaholic closed 7 years ago

dirkaholic commented 7 years ago

Bug report

We have telegraf running on all servers in our infrastructure. As they are provisioned via puppet they should have a similar configuration. We have one group of servers that act as appplication servers running nginx with a php application. On these hosts we have the logparser plugin configured in order to analyze the nginx logs. It works perfectly on 3 app servers but on the fourth telegraf is constantly crashing with "fatal error: heapBitsBulkBarrier: unaligned arguments" in the logs.

Relevant telegraf.conf:

###############################################################################
#                            INPUT PLUGINS                                    #
###############################################################################

# Read metrics about cpu usage
[[inputs.cpu]]
  ## Whether to report per-cpu stats or not
  percpu = true
  ## Whether to report total system cpu stats or not
  totalcpu = true
  ## If true, collect raw CPU time metrics.
  collect_cpu_time = false

# Read metrics about disk usage by mount point
[[inputs.disk]]
  ## By default, telegraf gather stats for all mountpoints.
  ## Setting mountpoints will restrict the stats to the specified mountpoints.
  # mount_points = ["/"]

  ## Ignore some mountpoints by filesystem type. For example (dev)tmpfs (usually
  ## present on /run, /var/run, /dev/shm or /dev).
  ignore_fs = ["tmpfs", "devtmpfs", "devfs"]

# Read metrics about disk IO by device
[[inputs.diskio]]
  ## By default, telegraf will gather stats for all devices including
  ## disk partitions.
  ## Setting devices will restrict the stats to the specified devices.
  # devices = ["sda", "sdb"]
  ## Uncomment the following line if you need disk serial numbers.
  # skip_serial_number = false

# Get kernel statistics from /proc/stat
[[inputs.kernel]]
  # no configuration

# Read metrics about memory usage
[[inputs.mem]]
  # no configuration

# Get the number of processes and group them by status
[[inputs.processes]]
  # no configuration

# Read metrics about swap memory usage
[[inputs.swap]]
  # no configuration

# Read metrics about system load & uptime
[[inputs.system]]
  # no configuration

# Read metrics about network interface usage
[[inputs.net]]
  ## By default, telegraf gathers stats from any up interface (excluding loopback)
  ## Setting interfaces will tell it to gather these explicit interfaces,
  ## regardless of status.
  ##
  # interfaces = ["eth0"]

# Monitor sensors, requires lm-sensors package
[[inputs.sensors]]
  ## Remove numbers from field names.
  ## If true, a field name like 'temp1_input' will be changed to 'temp_input'.
  # remove_numbers = true
  [inputs.sensors.tags]
    crit_alert_temp = "85"

# Read Nginx's basic status information (ngx_http_stub_status_module)
[[inputs.nginx]]
  ## An array of Nginx stub_status URI to gather stats.
  urls = ["http://localhost/nginx_status"]

# Stream and parse nginx log file(s).
[[inputs.logparser]]
  ## Log files to parse.
  ## These accept standard unix glob matching rules, but with the addition of
  ## ** as a "super asterisk". ie:
  ##   /var/log/**.log     -> recursively find all .log files in /var/log
  ##   /var/log/*/*.log    -> find all .log files with a parent dir in /var/log
  ##   /var/log/apache.log -> only tail the apache log file
  files = ["/var/log/nginx/access.log", "/var/log/nginx/access.log.1"]
  ## Read file from beginning.
  from_beginning = false

  ## Parse logstash-style "grok" patterns:
  ##   Telegraf built-in parsing patterns: https://goo.gl/dkay10
  [inputs.logparser.grok]
    ## This is a list of patterns to check the given log file(s) for.
    ## Note that adding patterns here increases processing time. The most
    ## efficient configuration is to have one pattern per logparser.
    ## Other common built-in patterns are:
    ##   %{COMMON_LOG_FORMAT}   (plain apache & nginx access logs)
    ##   %{COMBINED_LOG_FORMAT} (access logs + referrer & agent)
    patterns = ["%{COMMON_LOG_FORMAT}"]
    ## Name of the outputted measurement name.
    measurement = "nginx_access_log"

System info:

Telegraf version: Telegraf v1.2.1 (git: release-1.2 3b6ffb344e5c03c1595d862282a6823ecb438cff) operating system name: Ubuntu 14.04.5 LTS

Steps to reproduce:

Not really clear as it happens on only one server although we have four of them running with the same config.

Expected behavior:

Actual behavior:

Additional info:

Full log output:

2017-04-26T08:55:18Z I! Starting Telegraf (version 1.2.1)
2017-04-26T08:55:18Z I! Loaded outputs: influxdb
2017-04-26T08:55:18Z I! Loaded inputs: inputs.mem inputs.processes inputs.system inputs.sensors inputs.nginx inputs.logparser inputs.cpu inputs.disk inputs.diskio inputs.kernel inputs.swap inputs.net
2017-04-26T08:55:18Z I! Tags enabled: host=app-prod-ex41ssd-hetzner17-49 role=app
2017-04-26T08:55:18Z I! Agent Config: Interval:10s, Quiet:false, Hostname:"app-prod-ex41ssd-hetzner17-49", Flush Interval:10s 
2017/04/26 10:55:18 Seeked /var/log/nginx/access.log - &{Offset:0 Whence:2}
2017/04/26 10:55:18 Seeked /var/log/nginx/access.log.1 - &{Offset:0 Whence:2}
fatal error: heapBitsBulkBarrier: unaligned arguments

goroutine 95094 [running]:
runtime.throw(0x10aa248, 0x28)
    /usr/local/go/src/runtime/panic.go:566 +0x95 fp=0xc420adc928 sp=0xc420adc908
runtime.heapBitsBulkBarrier(0xc42064c281, 0x18)
    /usr/local/go/src/runtime/mbitmap.go:565 +0x2c9 fp=0xc420adc988 sp=0xc420adc928
runtime.typedmemmove(0xfd8340, 0xc42064c281, 0xc420adca78)
    /usr/local/go/src/runtime/mbarrier.go:183 +0x65 fp=0xc420adc9c0 sp=0xc420adc988
runtime.convT2I(0x18f5700, 0xc420adca78, 0xc42064c281, 0xc42064c261, 0x1d)
    /usr/local/go/src/runtime/iface.go:190 +0x5f fp=0xc420adc9f0 sp=0xc420adc9c0
github.com/influxdata/influxdb/models.NewTags(0xc42067fda0, 0xc420adcba8, 0xc42064c181, 0x1d)
    /home/ubuntu/telegraf-build/src/github.com/influxdata/influxdb/models/points.go:1557 +0x342 fp=0xc420adcb30 sp=0xc420adc9f0
github.com/influxdata/influxdb/client/v2.NewPoint(0xc420299090, 0x10, 0xc42067fda0, 0xc42067fe00, 0xc420adcc48, 0x1, 0x1, 0xc420299010, 0x0, 0x0)
    /home/ubuntu/telegraf-build/src/github.com/influxdata/influxdb/client/v2/client.go:301 +0x74 fp=0xc420adcbd8 sp=0xc420adcb30
github.com/influxdata/telegraf/metric.(*metric).Point(0xc4208d7800, 0xc420299010)
    /home/ubuntu/telegraf-build/src/github.com/influxdata/telegraf/metric/metric.go:141 +0x120 fp=0xc420adcc70 sp=0xc420adcbd8
github.com/influxdata/telegraf/plugins/outputs/influxdb.(*InfluxDB).Write(0xc42000a900, 0xc420ac2000, 0x12f, 0x12f, 0xc420a68e58, 0x6b3f41)
    /home/ubuntu/telegraf-build/src/github.com/influxdata/telegraf/plugins/outputs/influxdb/influxdb.go:194 +0x154 fp=0xc420adcda8 sp=0xc420adcc70
github.com/influxdata/telegraf/internal/models.(*RunningOutput).write(0xc420184000, 0xc420ac2000, 0x12f, 0x12f, 0x12f, 0x0)
    /home/ubuntu/telegraf-build/src/github.com/influxdata/telegraf/internal/models/running_output.go:173 +0xa1 fp=0xc420adce58 sp=0xc420adcda8
github.com/influxdata/telegraf/internal/models.(*RunningOutput).Write(0xc420184000, 0x1157720, 0xc4207a79b0)
    /home/ubuntu/telegraf-build/src/github.com/influxdata/telegraf/internal/models/running_output.go:157 +0x49c fp=0xc420adcf38 sp=0xc420adce58
github.com/influxdata/telegraf/agent.(*Agent).flush.func1(0xc4207a79b0, 0xc420184000)
    /home/ubuntu/telegraf-build/src/github.com/influxdata/telegraf/agent/agent.go:232 +0x68 fp=0xc420adcfa0 sp=0xc420adcf38
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:2086 +0x1 fp=0xc420adcfa8 sp=0xc420adcfa0
created by github.com/influxdata/telegraf/agent.(*Agent).flush
    /home/ubuntu/telegraf-build/src/github.com/influxdata/telegraf/agent/agent.go:237 +0xb4

goroutine 1 [semacquire, 376 minutes]:
sync.runtime_Semacquire(0xc420701f5c)
    /usr/local/go/src/runtime/sema.go:47 +0x30
sync.(*WaitGroup).Wait(0xc420701f50)
    /usr/local/go/src/sync/waitgroup.go:131 +0x97
github.com/influxdata/telegraf/agent.(*Agent).Run(0xc4204a8080, 0xc420630960, 0x0, 0x0)
    /home/ubuntu/telegraf-build/src/github.com/influxdata/telegraf/agent/agent.go:383 +0x4de
main.reloadLoop(0xc42010a060, 0x0, 0x0)
    /home/ubuntu/telegraf-build/src/github.com/influxdata/telegraf/cmd/telegraf/telegraf.go:284 +0xcec
main.main()
    /home/ubuntu/telegraf-build/src/github.com/influxdata/telegraf/cmd/telegraf/telegraf.go:342 +0x85

goroutine 17 [syscall, 376 minutes, locked to thread]:
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:2086 +0x1

goroutine 5 [syscall, 376 minutes]:
os/signal.signal_recv(0x0)
    /usr/local/go/src/runtime/sigqueue.go:116 +0x157
os/signal.loop()
    /usr/local/go/src/os/signal/signal_unix.go:22 +0x22
created by os/signal.init.1
    /usr/local/go/src/os/signal/signal_unix.go:28 +0x41

goroutine 178 [select]:
github.com/influxdata/telegraf/agent.(*Agent).flusher.func1(0xc420351d00, 0xc420630960, 0xc420631e60, 0xc4204a8080)
    /home/ubuntu/telegraf-build/src/github.com/influxdata/telegraf/agent/agent.go:257 +0x2fa
created by github.com/influxdata/telegraf/agent.(*Agent).flusher
    /home/ubuntu/telegraf-build/src/github.com/influxdata/telegraf/agent/agent.go:286 +0xee

goroutine 50 [semacquire]:
sync.runtime_Semacquire(0xc4207a79bc)
    /usr/local/go/src/runtime/sema.go:47 +0x30
sync.(*WaitGroup).Wait(0xc4207a79b0)
    /usr/local/go/src/sync/waitgroup.go:131 +0x97
github.com/influxdata/telegraf/agent.(*Agent).flush(0xc4204a8080)
    /home/ubuntu/telegraf-build/src/github.com/influxdata/telegraf/agent/agent.go:240 +0xed
github.com/influxdata/telegraf/agent.(*Agent).flusher(0xc4204a8080, 0xc420630960, 0xc420630ba0, 0xc4201f6700, 0x1156268)
    /home/ubuntu/telegraf-build/src/github.com/influxdata/telegraf/agent/agent.go:299 +0x21f
github.com/influxdata/telegraf/agent.(*Agent).Run.func1(0xc420701f50, 0xc4204a8080, 0xc420630960, 0xc420630ba0)
    /home/ubuntu/telegraf-build/src/github.com/influxdata/telegraf/agent/agent.go:353 +0x77
created by github.com/influxdata/telegraf/agent.(*Agent).Run
    /home/ubuntu/telegraf-build/src/github.com/influxdata/telegraf/agent/agent.go:357 +0x320

goroutine 36 [select, 376 minutes, locked to thread]:
runtime.gopark(0x11575f0, 0x0, 0x107ae04, 0x6, 0x18, 0x2)
    /usr/local/go/src/runtime/proc.go:259 +0x13a
runtime.selectgoImpl(0xc42046cf30, 0x0, 0x18)
    /usr/local/go/src/runtime/select.go:423 +0x1235
runtime.selectgo(0xc42046cf30)
    /usr/local/go/src/runtime/select.go:238 +0x1c
runtime.ensureSigM.func1()
    /usr/local/go/src/runtime/signal1_unix.go:304 +0x2f3
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:2086 +0x1

goroutine 34 [IO wait]:
net.runtime_pollWait(0x7fa4bc8fc3b8, 0x72, 0x3)
    /usr/local/go/src/runtime/netpoll.go:160 +0x59
net.(*pollDesc).wait(0xc420406840, 0x72, 0xc42003a6f0, 0xc420012160)
    /usr/local/go/src/net/fd_poll_runtime.go:73 +0x38
net.(*pollDesc).waitRead(0xc420406840, 0x18ee740, 0xc420012160)
    /usr/local/go/src/net/fd_poll_runtime.go:78 +0x34
net.(*netFD).Read(0xc4204067e0, 0xc420497000, 0x400, 0x400, 0x0, 0x18ee740, 0xc420012160)
    /usr/local/go/src/net/fd_unix.go:243 +0x1a1
net.(*conn).Read(0xc4204a80e0, 0xc420497000, 0x400, 0x400, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/net.go:173 +0x70
crypto/tls.(*block).readFromUntil(0xc42025fc20, 0x18eb6c0, 0xc4204a80e0, 0x5, 0xc4204a80e0, 0x28)
    /usr/local/go/src/crypto/tls/conn.go:476 +0x91
crypto/tls.(*Conn).readRecord(0xc420072380, 0x1157717, 0xc420072488, 0xc42003adc8)
    /usr/local/go/src/crypto/tls/conn.go:578 +0xc4
crypto/tls.(*Conn).Read(0xc420072380, 0xc4203e7000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
    /usr/local/go/src/crypto/tls/conn.go:1113 +0x116
net/http.(*persistConn).Read(0xc42000aa00, 0xc4203e7000, 0x1000, 0x1000, 0x30, 0xc42003ab58, 0x43fcfc)
    /usr/local/go/src/net/http/transport.go:1261 +0x154
bufio.(*Reader).fill(0xc420630420)
    /usr/local/go/src/bufio/bufio.go:97 +0x10c
bufio.(*Reader).Peek(0xc420630420, 0x1, 0x0, 0x1, 0x0, 0xc420859b60, 0x0)
    /usr/local/go/src/bufio/bufio.go:129 +0x62
net/http.(*persistConn).readLoop(0xc42000aa00)
    /usr/local/go/src/net/http/transport.go:1418 +0x1a1
created by net/http.(*Transport).dialConn
    /usr/local/go/src/net/http/transport.go:1062 +0x4e9

goroutine 35 [select]:
net/http.(*persistConn).writeLoop(0xc42000aa00)
    /usr/local/go/src/net/http/transport.go:1646 +0x3bd
created by net/http.(*Transport).dialConn
    /usr/local/go/src/net/http/transport.go:1063 +0x50e

goroutine 37 [select, 376 minutes]:
main.reloadLoop.func2(0xc4206309c0, 0xc420630960, 0xc42010c070, 0xc42010a060)
    /home/ubuntu/telegraf-build/src/github.com/influxdata/telegraf/cmd/telegraf/telegraf.go:252 +0x264
created by main.reloadLoop
    /home/ubuntu/telegraf-build/src/github.com/influxdata/telegraf/cmd/telegraf/telegraf.go:266 +0x98d

goroutine 38 [select]:
github.com/influxdata/telegraf/plugins/inputs/logparser.(*LogParserPlugin).parser(0xc42008e120)
    /home/ubuntu/telegraf-build/src/github.com/influxdata/telegraf/plugins/inputs/logparser/logparser.go:192 +0x3da
created by github.com/influxdata/telegraf/plugins/inputs/logparser.(*LogParserPlugin).Start
    /home/ubuntu/telegraf-build/src/github.com/influxdata/telegraf/plugins/inputs/logparser/logparser.go:131 +0x62d

goroutine 39 [select]:
github.com/hpcloud/tail.(*Tail).waitForChanges(0xc4201b9810, 0xc420012100, 0x18e8c00)
    /home/ubuntu/telegraf-build/src/github.com/hpcloud/tail/tail.go:349 +0xe5a
github.com/hpcloud/tail.(*Tail).tailFileSync(0xc4201b9810)
    /home/ubuntu/telegraf-build/src/github.com/hpcloud/tail/tail.go:310 +0x313
created by github.com/hpcloud/tail.TailFile
    /home/ubuntu/telegraf-build/src/github.com/hpcloud/tail/tail.go:132 +0x1f8

goroutine 40 [chan receive]:
github.com/influxdata/telegraf/plugins/inputs/logparser.(*LogParserPlugin).receiver(0xc42008e120, 0xc4201b9810)
    /home/ubuntu/telegraf-build/src/github.com/influxdata/telegraf/plugins/inputs/logparser/logparser.go:168 +0xab
created by github.com/influxdata/telegraf/plugins/inputs/logparser.(*LogParserPlugin).Start
    /home/ubuntu/telegraf-build/src/github.com/influxdata/telegraf/plugins/inputs/logparser/logparser.go:154 +0x9ff

goroutine 41 [select, 240 minutes]:
github.com/hpcloud/tail.(*Tail).waitForChanges(0xc4201b98c0, 0xc420012100, 0x18e8c00)
    /home/ubuntu/telegraf-build/src/github.com/hpcloud/tail/tail.go:349 +0xe5a
github.com/hpcloud/tail.(*Tail).tailFileSync(0xc4201b98c0)
    /home/ubuntu/telegraf-build/src/github.com/hpcloud/tail/tail.go:310 +0x313
created by github.com/hpcloud/tail.TailFile
    /home/ubuntu/telegraf-build/src/github.com/hpcloud/tail/tail.go:132 +0x1f8

goroutine 42 [chan receive, 240 minutes]:
github.com/influxdata/telegraf/plugins/inputs/logparser.(*LogParserPlugin).receiver(0xc42008e120, 0xc4201b98c0)
    /home/ubuntu/telegraf-build/src/github.com/influxdata/telegraf/plugins/inputs/logparser/logparser.go:168 +0xab
created by github.com/influxdata/telegraf/plugins/inputs/logparser.(*LogParserPlugin).Start
    /home/ubuntu/telegraf-build/src/github.com/influxdata/telegraf/plugins/inputs/logparser/logparser.go:154 +0x9ff

goroutine 43 [select]:
github.com/hpcloud/tail/watch.(*InotifyTracker).run(0xc420826400)
    /home/ubuntu/telegraf-build/src/github.com/hpcloud/tail/watch/inotify_tracker.go:235 +0x45b
created by github.com/hpcloud/tail/watch.glob..func1
    /home/ubuntu/telegraf-build/src/github.com/hpcloud/tail/watch/inotify_tracker.go:53 +0x233

goroutine 44 [syscall]:
syscall.Syscall6(0xe8, 0xa, 0xc42086bc14, 0x7, 0xffffffffffffffff, 0x0, 0x0, 0xc42087bde0, 0x2, 0x2)
    /usr/local/go/src/syscall/asm_linux_amd64.s:44 +0x5
syscall.EpollWait(0xa, 0xc42086bc14, 0x7, 0x7, 0xffffffffffffffff, 0x2, 0xc42087be40, 0x2)
    /usr/local/go/src/syscall/zsyscall_linux_amd64.go:365 +0x77
github.com/hpcloud/tail/vendor/gopkg.in/fsnotify%2ev1.(*fdPoller).wait(0xc4206b1520, 0xc420777600, 0x0, 0xc4200f9b01)
    /home/ubuntu/telegraf-build/src/github.com/hpcloud/tail/vendor/gopkg.in/fsnotify.v1/inotify_poller.go:85 +0x91
github.com/hpcloud/tail/vendor/gopkg.in/fsnotify%2ev1.(*Watcher).readEvents(0xc4206ca5f0)
    /home/ubuntu/telegraf-build/src/github.com/hpcloud/tail/vendor/gopkg.in/fsnotify.v1/inotify.go:179 +0x196
created by github.com/hpcloud/tail/vendor/gopkg.in/fsnotify%2ev1.NewWatcher
    /home/ubuntu/telegraf-build/src/github.com/hpcloud/tail/vendor/gopkg.in/fsnotify.v1/inotify.go:58 +0x2cc

goroutine 45 [select]:
github.com/hpcloud/tail/watch.(*InotifyFileWatcher).ChangeEvents.func1(0xc4206b1380, 0xc4201b9888, 0xc4206b1560)
    /home/ubuntu/telegraf-build/src/github.com/hpcloud/tail/watch/inotify.go:87 +0x408
created by github.com/hpcloud/tail/watch.(*InotifyFileWatcher).ChangeEvents
    /home/ubuntu/telegraf-build/src/github.com/hpcloud/tail/watch/inotify.go:124 +0x155

goroutine 46 [select, 240 minutes]:
github.com/hpcloud/tail/watch.(*InotifyFileWatcher).ChangeEvents.func1(0xc4206b1420, 0xc4201b9938, 0xc4206b15a0)
    /home/ubuntu/telegraf-build/src/github.com/hpcloud/tail/watch/inotify.go:87 +0x408
created by github.com/hpcloud/tail/watch.(*InotifyFileWatcher).ChangeEvents
    /home/ubuntu/telegraf-build/src/github.com/hpcloud/tail/watch/inotify.go:124 +0x155

goroutine 51 [select]:
github.com/influxdata/telegraf/agent.(*Agent).gatherer(0xc4204a8080, 0xc420630960, 0xc4201d7300, 0x2540be400, 0xc420630ba0)
    /home/ubuntu/telegraf-build/src/github.com/influxdata/telegraf/agent/agent.go:129 +0x37e
github.com/influxdata/telegraf/agent.(*Agent).Run.func3(0xc420701f50, 0xc4204a8080, 0xc420630960, 0xc420630ba0, 0xc4201d7300, 0x2540be400)
    /home/ubuntu/telegraf-build/src/github.com/influxdata/telegraf/agent/agent.go:379 +0x7f
created by github.com/influxdata/telegraf/agent.(*Agent).Run
    /home/ubuntu/telegraf-build/src/github.com/influxdata/telegraf/agent/agent.go:380 +0x49b

goroutine 52 [select]:
github.com/influxdata/telegraf/agent.(*Agent).gatherer(0xc4204a8080, 0xc420630960, 0xc4201d7380, 0x2540be400, 0xc420630ba0)
    /home/ubuntu/telegraf-build/src/github.com/influxdata/telegraf/agent/agent.go:129 +0x37e
github.com/influxdata/telegraf/agent.(*Agent).Run.func3(0xc420701f50, 0xc4204a8080, 0xc420630960, 0xc420630ba0, 0xc4201d7380, 0x2540be400)
    /home/ubuntu/telegraf-build/src/github.com/influxdata/telegraf/agent/agent.go:379 +0x7f
created by github.com/influxdata/telegraf/agent.(*Agent).Run
    /home/ubuntu/telegraf-build/src/github.com/influxdata/telegraf/agent/agent.go:380 +0x49b

goroutine 53 [select]:
github.com/influxdata/telegraf/agent.(*Agent).gatherer(0xc4204a8080, 0xc420630960, 0xc4201d7400, 0x2540be400, 0xc420630ba0)
    /home/ubuntu/telegraf-build/src/github.com/influxdata/telegraf/agent/agent.go:129 +0x37e
github.com/influxdata/telegraf/agent.(*Agent).Run.func3(0xc420701f50, 0xc4204a8080, 0xc420630960, 0xc420630ba0, 0xc4201d7400, 0x2540be400)
    /home/ubuntu/telegraf-build/src/github.com/influxdata/telegraf/agent/agent.go:379 +0x7f
created by github.com/influxdata/telegraf/agent.(*Agent).Run
    /home/ubuntu/telegraf-build/src/github.com/influxdata/telegraf/agent/agent.go:380 +0x49b

goroutine 54 [select]:
github.com/influxdata/telegraf/agent.(*Agent).gatherer(0xc4204a8080, 0xc420630960, 0xc4201d7500, 0x2540be400, 0xc420630ba0)
    /home/ubuntu/telegraf-build/src/github.com/influxdata/telegraf/agent/agent.go:129 +0x37e
github.com/influxdata/telegraf/agent.(*Agent).Run.func3(0xc420701f50, 0xc4204a8080, 0xc420630960, 0xc420630ba0, 0xc4201d7500, 0x2540be400)
    /home/ubuntu/telegraf-build/src/github.com/influxdata/telegraf/agent/agent.go:379 +0x7f
created by github.com/influxdata/telegraf/agent.(*Agent).Run
    /home/ubuntu/telegraf-build/src/github.com/influxdata/telegraf/agent/agent.go:380 +0x49b

goroutine 55 [select]:
github.com/influxdata/telegraf/agent.(*Agent).gatherer(0xc4204a8080, 0xc420630960, 0xc4201d7580, 0x2540be400, 0xc420630ba0)
    /home/ubuntu/telegraf-build/src/github.com/influxdata/telegraf/agent/agent.go:129 +0x37e
github.com/influxdata/telegraf/agent.(*Agent).Run.func3(0xc420701f50, 0xc4204a8080, 0xc420630960, 0xc420630ba0, 0xc4201d7580, 0x2540be400)
    /home/ubuntu/telegraf-build/src/github.com/influxdata/telegraf/agent/agent.go:379 +0x7f
created by github.com/influxdata/telegraf/agent.(*Agent).Run
    /home/ubuntu/telegraf-build/src/github.com/influxdata/telegraf/agent/agent.go:380 +0x49b

goroutine 56 [select]:
github.com/influxdata/telegraf/agent.(*Agent).gatherer(0xc4204a8080, 0xc420630960, 0xc4201d76c0, 0x2540be400, 0xc420630ba0)
    /home/ubuntu/telegraf-build/src/github.com/influxdata/telegraf/agent/agent.go:129 +0x37e
github.com/influxdata/telegraf/agent.(*Agent).Run.func3(0xc420701f50, 0xc4204a8080, 0xc420630960, 0xc420630ba0, 0xc4201d76c0, 0x2540be400)
    /home/ubuntu/telegraf-build/src/github.com/influxdata/telegraf/agent/agent.go:379 +0x7f
created by github.com/influxdata/telegraf/agent.(*Agent).Run
    /home/ubuntu/telegraf-build/src/github.com/influxdata/telegraf/agent/agent.go:380 +0x49b

goroutine 57 [select]:
github.com/influxdata/telegraf/agent.(*Agent).gatherer(0xc4204a8080, 0xc420630960, 0xc4201d7780, 0x2540be400, 0xc420630ba0)
    /home/ubuntu/telegraf-build/src/github.com/influxdata/telegraf/agent/agent.go:129 +0x37e
github.com/influxdata/telegraf/agent.(*Agent).Run.func3(0xc420701f50, 0xc4204a8080, 0xc420630960, 0xc420630ba0, 0xc4201d7780, 0x2540be400)
    /home/ubuntu/telegraf-build/src/github.com/influxdata/telegraf/agent/agent.go:379 +0x7f
created by github.com/influxdata/telegraf/agent.(*Agent).Run
    /home/ubuntu/telegraf-build/src/github.com/influxdata/telegraf/agent/agent.go:380 +0x49b

goroutine 58 [select]:
github.com/influxdata/telegraf/agent.(*Agent).gatherer(0xc4204a8080, 0xc420630960, 0xc4201d7800, 0x2540be400, 0xc420630ba0)
    /home/ubuntu/telegraf-build/src/github.com/influxdata/telegraf/agent/agent.go:129 +0x37e
github.com/influxdata/telegraf/agent.(*Agent).Run.func3(0xc420701f50, 0xc4204a8080, 0xc420630960, 0xc420630ba0, 0xc4201d7800, 0x2540be400)
    /home/ubuntu/telegraf-build/src/github.com/influxdata/telegraf/agent/agent.go:379 +0x7f
created by github.com/influxdata/telegraf/agent.(*Agent).Run
    /home/ubuntu/telegraf-build/src/github.com/influxdata/telegraf/agent/agent.go:380 +0x49b

goroutine 59 [select]:
github.com/influxdata/telegraf/agent.(*Agent).gatherer(0xc4204a8080, 0xc420630960, 0xc4201d7880, 0x2540be400, 0xc420630ba0)
    /home/ubuntu/telegraf-build/src/github.com/influxdata/telegraf/agent/agent.go:129 +0x37e
github.com/influxdata/telegraf/agent.(*Agent).Run.func3(0xc420701f50, 0xc4204a8080, 0xc420630960, 0xc420630ba0, 0xc4201d7880, 0x2540be400)
    /home/ubuntu/telegraf-build/src/github.com/influxdata/telegraf/agent/agent.go:379 +0x7f
created by github.com/influxdata/telegraf/agent.(*Agent).Run
    /home/ubuntu/telegraf-build/src/github.com/influxdata/telegraf/agent/agent.go:380 +0x49b

goroutine 60 [select]:
github.com/influxdata/telegraf/agent.(*Agent).gatherer(0xc4204a8080, 0xc420630960, 0xc4201d7900, 0x2540be400, 0xc420630ba0)
    /home/ubuntu/telegraf-build/src/github.com/influxdata/telegraf/agent/agent.go:129 +0x37e
github.com/influxdata/telegraf/agent.(*Agent).Run.func3(0xc420701f50, 0xc4204a8080, 0xc420630960, 0xc420630ba0, 0xc4201d7900, 0x2540be400)
    /home/ubuntu/telegraf-build/src/github.com/influxdata/telegraf/agent/agent.go:379 +0x7f
created by github.com/influxdata/telegraf/agent.(*Agent).Run
    /home/ubuntu/telegraf-build/src/github.com/influxdata/telegraf/agent/agent.go:380 +0x49b

goroutine 61 [select]:
github.com/influxdata/telegraf/agent.(*Agent).gatherer(0xc4204a8080, 0xc420630960, 0xc4201d7980, 0x2540be400, 0xc420630ba0)
    /home/ubuntu/telegraf-build/src/github.com/influxdata/telegraf/agent/agent.go:129 +0x37e
github.com/influxdata/telegraf/agent.(*Agent).Run.func3(0xc420701f50, 0xc4204a8080, 0xc420630960, 0xc420630ba0, 0xc4201d7980, 0x2540be400)
    /home/ubuntu/telegraf-build/src/github.com/influxdata/telegraf/agent/agent.go:379 +0x7f
created by github.com/influxdata/telegraf/agent.(*Agent).Run
    /home/ubuntu/telegraf-build/src/github.com/influxdata/telegraf/agent/agent.go:380 +0x49b

goroutine 62 [select]:
github.com/influxdata/telegraf/agent.(*Agent).gatherer(0xc4204a8080, 0xc420630960, 0xc4201d7a40, 0x2540be400, 0xc420630ba0)
    /home/ubuntu/telegraf-build/src/github.com/influxdata/telegraf/agent/agent.go:129 +0x37e
github.com/influxdata/telegraf/agent.(*Agent).Run.func3(0xc420701f50, 0xc4204a8080, 0xc420630960, 0xc420630ba0, 0xc4201d7a40, 0x2540be400)
    /home/ubuntu/telegraf-build/src/github.com/influxdata/telegraf/agent/agent.go:379 +0x7f
created by github.com/influxdata/telegraf/agent.(*Agent).Run
    /home/ubuntu/telegraf-build/src/github.com/influxdata/telegraf/agent/agent.go:380 +0x49b

goroutine 93263 [select]:
net/http.(*persistConn).writeLoop(0xc4204a3000)
    /usr/local/go/src/net/http/transport.go:1646 +0x3bd
created by net/http.(*Transport).dialConn
    /usr/local/go/src/net/http/transport.go:1063 +0x50e

goroutine 93262 [IO wait]:
net.runtime_pollWait(0x7fa4bc8fc2f8, 0x72, 0xe)
    /usr/local/go/src/runtime/netpoll.go:160 +0x59
net.(*pollDesc).wait(0xc4201bc140, 0x72, 0xc4208a89d0, 0xc420012160)
    /usr/local/go/src/net/fd_poll_runtime.go:73 +0x38
net.(*pollDesc).waitRead(0xc4201bc140, 0x18ee740, 0xc420012160)
    /usr/local/go/src/net/fd_poll_runtime.go:78 +0x34
net.(*netFD).Read(0xc4201bc0e0, 0xc420cf9000, 0x1000, 0x1000, 0x0, 0x18ee740, 0xc420012160)
    /usr/local/go/src/net/fd_unix.go:243 +0x1a1
net.(*conn).Read(0xc420cf4008, 0xc420cf9000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/net.go:173 +0x70
net/http.(*persistConn).Read(0xc4204a3000, 0xc420cf9000, 0x1000, 0x1000, 0x7ecd00, 0xc4208a8b58, 0x4083fd)
    /usr/local/go/src/net/http/transport.go:1261 +0x154
bufio.(*Reader).fill(0xc4207760c0)
    /usr/local/go/src/bufio/bufio.go:97 +0x10c
bufio.(*Reader).Peek(0xc4207760c0, 0x1, 0xc4208a8bbd, 0x1, 0x0, 0xc420c20240, 0x0)
    /usr/local/go/src/bufio/bufio.go:129 +0x62
net/http.(*persistConn).readLoop(0xc4204a3000)
    /usr/local/go/src/net/http/transport.go:1418 +0x1a1
created by net/http.(*Transport).dialConn
    /usr/local/go/src/net/http/transport.go:1062 +0x4e9
danielnelson commented 7 years ago

I think this may be fixed in master with the new influxdb client. Could you test with a nightly builds?

dirkaholic commented 7 years ago

I've installed the nighlty build on this single server. It never survived 24 hours lately so I should know more tomorrow.

dirkaholic commented 7 years ago

When using the latest nightly build it crashes as well but with another error message:

panic: runtime error: index out of range
fatal error: panic on system stack

runtime stack:
runtime.throw(0x138d2ba, 0x15)
    /usr/local/go/src/runtime/panic.go:596 +0x95
panic(0x11c9a60, 0x1bab1f0)
    /usr/local/go/src/runtime/panic.go:420 +0x7d6
runtime.panicindex()
    /usr/local/go/src/runtime/panic.go:28 +0x5e
runtime.sellock(0xc42003cf48, 0x2, 0x2, 0xc42003cfa9, 0x2, 0x2)
    /usr/local/go/src/runtime/select.go:167 +0xa7
runtime.selectgoImpl.func1()
    /usr/local/go/src/runtime/select.go:478 +0x57
runtime.systemstack(0xc420508520)
    /usr/local/go/src/runtime/asm_amd64.s:327 +0x79
runtime.mstart()
    /usr/local/go/src/runtime/proc.go:1132

goroutine 101 [running]:
runtime.systemstack_switch()
    /usr/local/go/src/runtime/asm_amd64.s:281 fp=0xc42003cc10 sp=0xc42003cc08
runtime.selectgoImpl(0xc42003cf30, 0x0, 0x18)
    /usr/local/go/src/runtime/select.go:479 +0x140c fp=0xc42003ce88 sp=0xc42003cc10
runtime.selectgo(0xc42003cf30)
    /usr/local/go/src/runtime/select.go:238 +0x1c fp=0xc42003ceb0 sp=0xc42003ce88
github.com/influxdata/telegraf/agent.(*Agent).flusher.func1(0xc420152ca0, 0xc420074a80, 0xc420a301e0, 0xc42000e048)
    /root/go/src/github.com/influxdata/telegraf/agent/agent.go:263 +0x2f3 fp=0xc42003cfc0 sp=0xc42003ceb0
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:2197 +0x1 fp=0xc42003cfc8 sp=0xc42003cfc0
created by github.com/influxdata/telegraf/agent.(*Agent).flusher
    /root/go/src/github.com/influxdata/telegraf/agent/agent.go:292 +0xe5

goroutine 1 [semacquire, 71 minutes]:
sync.runtime_Semacquire(0xc42001737c)
    /usr/local/go/src/runtime/sema.go:47 +0x34
sync.(*WaitGroup).Wait(0xc420017370)
    /usr/local/go/src/sync/waitgroup.go:131 +0x7a
github.com/influxdata/telegraf/agent.(*Agent).Run(0xc42000e048, 0xc420074a80, 0x0, 0x0)
    /root/go/src/github.com/influxdata/telegraf/agent/agent.go:400 +0x4b3
main.reloadLoop(0xc420074900, 0x1bff5e0, 0x0, 0x0, 0x1bff5e0, 0x0, 0x0, 0x1bff5e0, 0x0, 0x0, ...)
    /root/go/src/github.com/influxdata/telegraf/cmd/telegraf/telegraf.go:222 +0xa06
main.main()
    /root/go/src/github.com/influxdata/telegraf/cmd/telegraf/telegraf.go:392 +0x65b

goroutine 17 [syscall, 71 minutes, locked to thread]:
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:2197 +0x1

goroutine 6 [syscall, 71 minutes]:
os/signal.signal_recv(0x0)
    /usr/local/go/src/runtime/sigqueue.go:116 +0x104
os/signal.loop()
    /usr/local/go/src/os/signal/signal_unix.go:22 +0x22
created by os/signal.init.1
    /usr/local/go/src/os/signal/signal_unix.go:28 +0x41

goroutine 44 [select, 71 minutes]:
github.com/influxdata/tail/watch.(*InotifyFileWatcher).ChangeEvents.func1(0xc4202a3620, 0xc4203f23e8, 0xc4203a5de0)
    /root/go/src/github.com/influxdata/tail/watch/inotify.go:87 +0x475
created by github.com/influxdata/tail/watch.(*InotifyFileWatcher).ChangeEvents
    /root/go/src/github.com/influxdata/tail/watch/inotify.go:137 +0x155

goroutine 45 [select]:
github.com/influxdata/telegraf/agent.(*Agent).flusher(0xc42000e048, 0xc420074a80, 0xc420074cc0, 0x180001, 0x0)
    /root/go/src/github.com/influxdata/telegraf/agent/agent.go:297 +0x3d4
github.com/influxdata/telegraf/agent.(*Agent).Run.func1(0xc420017370, 0xc42000e048, 0xc420074a80, 0xc420074cc0)
    /root/go/src/github.com/influxdata/telegraf/agent/agent.go:370 +0x6d
created by github.com/influxdata/telegraf/agent.(*Agent).Run
    /root/go/src/github.com/influxdata/telegraf/agent/agent.go:374 +0x2e3

goroutine 42 [IO wait]:
net.runtime_pollWait(0x7f8015e32538, 0x72, 0x3)
    /usr/local/go/src/runtime/netpoll.go:164 +0x59
net.(*pollDesc).wait(0xc4201d25a8, 0x72, 0x1ae7240, 0x1ade880)
    /usr/local/go/src/net/fd_poll_runtime.go:75 +0x38
net.(*pollDesc).waitRead(0xc4201d25a8, 0xc420130800, 0x400)
    /usr/local/go/src/net/fd_poll_runtime.go:80 +0x34
net.(*netFD).Read(0xc4201d2540, 0xc420130800, 0x400, 0x400, 0x0, 0x1ae7240, 0x1ade880)
    /usr/local/go/src/net/fd_unix.go:250 +0x1b7
net.(*conn).Read(0xc42000e0d0, 0xc420130800, 0x400, 0x400, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/net.go:181 +0x70
crypto/tls.(*block).readFromUntil(0xc420505dd0, 0x1ae3c80, 0xc42000e0d0, 0x5, 0xc42000e0d0, 0x28)
    /usr/local/go/src/crypto/tls/conn.go:488 +0x98
crypto/tls.(*Conn).readRecord(0xc4201b4000, 0x13ed617, 0xc4201b4120, 0x0)
    /usr/local/go/src/crypto/tls/conn.go:590 +0xc4
crypto/tls.(*Conn).Read(0xc4201b4000, 0xc420261000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
    /usr/local/go/src/crypto/tls/conn.go:1134 +0x11d
net/http.(*persistConn).Read(0xc42025e360, 0xc420261000, 0x1000, 0x1000, 0x5, 0xc420268f20, 0x1e)
    /usr/local/go/src/net/http/transport.go:1316 +0x14b
bufio.(*Reader).fill(0xc420390a80)
    /usr/local/go/src/bufio/bufio.go:97 +0x117
bufio.(*Reader).Peek(0xc420390a80, 0x1, 0x0, 0x1, 0x0, 0xc420a30600, 0x0)
    /usr/local/go/src/bufio/bufio.go:129 +0x67
net/http.(*persistConn).readLoop(0xc42025e360)
    /usr/local/go/src/net/http/transport.go:1474 +0x196
created by net/http.(*Transport).dialConn
    /usr/local/go/src/net/http/transport.go:1117 +0xa35

goroutine 43 [select]:
net/http.(*persistConn).writeLoop(0xc42025e360)
    /usr/local/go/src/net/http/transport.go:1704 +0x43a
created by net/http.(*Transport).dialConn
    /usr/local/go/src/net/http/transport.go:1118 +0xa5a

goroutine 20 [select, 71 minutes, locked to thread]:
runtime.gopark(0x13ed5d8, 0x0, 0x1373266, 0x6, 0x18, 0x2)
    /usr/local/go/src/runtime/proc.go:271 +0x13a
runtime.selectgoImpl(0xc420270f50, 0x0, 0x18)
    /usr/local/go/src/runtime/select.go:423 +0x1364
runtime.selectgo(0xc420270f50)
    /usr/local/go/src/runtime/select.go:238 +0x1c
runtime.ensureSigM.func1()
    /usr/local/go/src/runtime/signal_unix.go:434 +0x2dd
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:2197 +0x1

goroutine 21 [select, 71 minutes]:
main.reloadLoop.func1(0xc420074ae0, 0xc420074a80, 0xc4201d6d20, 0xc420074900)
    /root/go/src/github.com/influxdata/telegraf/cmd/telegraf/telegraf.go:183 +0x24a
created by main.reloadLoop
    /root/go/src/github.com/influxdata/telegraf/cmd/telegraf/telegraf.go:197 +0x6cb

goroutine 22 [select]:
github.com/influxdata/telegraf/plugins/inputs/logparser.(*LogParserPlugin).parser(0xc42007a000)
    /root/go/src/github.com/influxdata/telegraf/plugins/inputs/logparser/logparser.go:211 +0x47a
created by github.com/influxdata/telegraf/plugins/inputs/logparser.(*LogParserPlugin).Start
    /root/go/src/github.com/influxdata/telegraf/plugins/inputs/logparser/logparser.go:132 +0x5d3

goroutine 23 [select]:
github.com/influxdata/tail.(*Tail).waitForChanges(0xc4203f22c0, 0xc4200160d0, 0x1ae0cc0)
    /root/go/src/github.com/influxdata/tail/tail.go:360 +0x6f1
github.com/influxdata/tail.(*Tail).tailFileSync(0xc4203f22c0)
    /root/go/src/github.com/influxdata/tail/tail.go:315 +0x705
created by github.com/influxdata/tail.TailFile
    /root/go/src/github.com/influxdata/tail/tail.go:133 +0x1d9

goroutine 24 [chan receive]:
github.com/influxdata/telegraf/plugins/inputs/logparser.(*LogParserPlugin).receiver(0xc42007a000, 0xc4203f22c0)
    /root/go/src/github.com/influxdata/telegraf/plugins/inputs/logparser/logparser.go:186 +0x9e
created by github.com/influxdata/telegraf/plugins/inputs/logparser.(*LogParserPlugin).tailNewfiles
    /root/go/src/github.com/influxdata/telegraf/plugins/inputs/logparser/logparser.go:172 +0x416

goroutine 25 [select, 71 minutes]:
github.com/influxdata/tail.(*Tail).waitForChanges(0xc4203f2370, 0xc4200160d0, 0x1ae0cc0)
    /root/go/src/github.com/influxdata/tail/tail.go:360 +0x6f1
github.com/influxdata/tail.(*Tail).tailFileSync(0xc4203f2370)
    /root/go/src/github.com/influxdata/tail/tail.go:315 +0x705
created by github.com/influxdata/tail.TailFile
    /root/go/src/github.com/influxdata/tail/tail.go:133 +0x1d9

goroutine 26 [chan receive, 71 minutes]:
github.com/influxdata/telegraf/plugins/inputs/logparser.(*LogParserPlugin).receiver(0xc42007a000, 0xc4203f2370)
    /root/go/src/github.com/influxdata/telegraf/plugins/inputs/logparser/logparser.go:186 +0x9e
created by github.com/influxdata/telegraf/plugins/inputs/logparser.(*LogParserPlugin).tailNewfiles
    /root/go/src/github.com/influxdata/telegraf/plugins/inputs/logparser/logparser.go:172 +0x416

goroutine 27 [select]:
github.com/influxdata/tail/watch.(*InotifyTracker).run(0xc4203faf40)
    /root/go/src/github.com/influxdata/tail/watch/inotify_tracker.go:224 +0x4aa
created by github.com/influxdata/tail/watch.glob..func1
    /root/go/src/github.com/influxdata/tail/watch/inotify_tracker.go:54 +0x218

goroutine 28 [syscall]:
syscall.Syscall6(0xe8, 0x8, 0xc42094dcb4, 0x7, 0xffffffffffffffff, 0x0, 0x0, 0x2, 0x2, 0xc42095dec0)
    /usr/local/go/src/syscall/asm_linux_amd64.s:44 +0x5
syscall.EpollWait(0x8, 0xc42094dcb4, 0x7, 0x7, 0xffffffffffffffff, 0x4975b9, 0x0, 0x0)
    /usr/local/go/src/syscall/zsyscall_linux_amd64.go:365 +0x77
github.com/influxdata/tail/vendor/gopkg.in/fsnotify%2ev1.(*fdPoller).wait(0xc4202a3740, 0xc42029fb00, 0x0, 0xc42088ae01)
    /root/go/src/github.com/influxdata/tail/vendor/gopkg.in/fsnotify.v1/inotify_poller.go:85 +0x91
github.com/influxdata/tail/vendor/gopkg.in/fsnotify%2ev1.(*Watcher).readEvents(0xc42037cd70)
    /root/go/src/github.com/influxdata/tail/vendor/gopkg.in/fsnotify.v1/inotify.go:179 +0x196
created by github.com/influxdata/tail/vendor/gopkg.in/fsnotify%2ev1.NewWatcher
    /root/go/src/github.com/influxdata/tail/vendor/gopkg.in/fsnotify.v1/inotify.go:58 +0x26d

goroutine 29 [select]:
github.com/influxdata/tail/watch.(*InotifyFileWatcher).ChangeEvents.func1(0xc4202a3580, 0xc4203f2338, 0xc4202a37a0)
    /root/go/src/github.com/influxdata/tail/watch/inotify.go:87 +0x475
created by github.com/influxdata/tail/watch.(*InotifyFileWatcher).ChangeEvents
    /root/go/src/github.com/influxdata/tail/watch/inotify.go:137 +0x155

goroutine 46 [select]:
github.com/influxdata/telegraf/agent.(*Agent).gatherer(0xc42000e048, 0xc420074a80, 0xc42005d7c0, 0x2540be400, 0xc420074cc0)
    /root/go/src/github.com/influxdata/telegraf/agent/agent.go:129 +0x3a8
github.com/influxdata/telegraf/agent.(*Agent).Run.func3(0xc420017370, 0xc42000e048, 0xc420074a80, 0xc420074cc0, 0xc42005d7c0, 0x2540be400)
    /root/go/src/github.com/influxdata/telegraf/agent/agent.go:396 +0x7d
created by github.com/influxdata/telegraf/agent.(*Agent).Run
    /root/go/src/github.com/influxdata/telegraf/agent/agent.go:397 +0x46a

goroutine 47 [select]:
github.com/influxdata/telegraf/agent.(*Agent).gatherer(0xc42000e048, 0xc420074a80, 0xc42005d840, 0x2540be400, 0xc420074cc0)
    /root/go/src/github.com/influxdata/telegraf/agent/agent.go:129 +0x3a8
github.com/influxdata/telegraf/agent.(*Agent).Run.func3(0xc420017370, 0xc42000e048, 0xc420074a80, 0xc420074cc0, 0xc42005d840, 0x2540be400)
    /root/go/src/github.com/influxdata/telegraf/agent/agent.go:396 +0x7d
created by github.com/influxdata/telegraf/agent.(*Agent).Run
    /root/go/src/github.com/influxdata/telegraf/agent/agent.go:397 +0x46a

goroutine 48 [select]:
github.com/influxdata/telegraf/agent.(*Agent).gatherer(0xc42000e048, 0xc420074a80, 0xc42005d900, 0x2540be400, 0xc420074cc0)
    /root/go/src/github.com/influxdata/telegraf/agent/agent.go:129 +0x3a8
github.com/influxdata/telegraf/agent.(*Agent).Run.func3(0xc420017370, 0xc42000e048, 0xc420074a80, 0xc420074cc0, 0xc42005d900, 0x2540be400)
    /root/go/src/github.com/influxdata/telegraf/agent/agent.go:396 +0x7d
created by github.com/influxdata/telegraf/agent.(*Agent).Run
    /root/go/src/github.com/influxdata/telegraf/agent/agent.go:397 +0x46a

goroutine 49 [select]:
github.com/influxdata/telegraf/agent.(*Agent).gatherer(0xc42000e048, 0xc420074a80, 0xc42005d980, 0x2540be400, 0xc420074cc0)
    /root/go/src/github.com/influxdata/telegraf/agent/agent.go:129 +0x3a8
github.com/influxdata/telegraf/agent.(*Agent).Run.func3(0xc420017370, 0xc42000e048, 0xc420074a80, 0xc420074cc0, 0xc42005d980, 0x2540be400)
    /root/go/src/github.com/influxdata/telegraf/agent/agent.go:396 +0x7d
created by github.com/influxdata/telegraf/agent.(*Agent).Run
    /root/go/src/github.com/influxdata/telegraf/agent/agent.go:397 +0x46a

goroutine 50 [select]:
github.com/influxdata/telegraf/agent.(*Agent).gatherer(0xc42000e048, 0xc420074a80, 0xc42005da00, 0x2540be400, 0xc420074cc0)
    /root/go/src/github.com/influxdata/telegraf/agent/agent.go:129 +0x3a8
github.com/influxdata/telegraf/agent.(*Agent).Run.func3(0xc420017370, 0xc42000e048, 0xc420074a80, 0xc420074cc0, 0xc42005da00, 0x2540be400)
    /root/go/src/github.com/influxdata/telegraf/agent/agent.go:396 +0x7d
created by github.com/influxdata/telegraf/agent.(*Agent).Run
    /root/go/src/github.com/influxdata/telegraf/agent/agent.go:397 +0x46a

goroutine 51 [select]:
github.com/influxdata/telegraf/agent.(*Agent).gatherer(0xc42000e048, 0xc420074a80, 0xc42005dac0, 0x2540be400, 0xc420074cc0)
    /root/go/src/github.com/influxdata/telegraf/agent/agent.go:129 +0x3a8
github.com/influxdata/telegraf/agent.(*Agent).Run.func3(0xc420017370, 0xc42000e048, 0xc420074a80, 0xc420074cc0, 0xc42005dac0, 0x2540be400)
    /root/go/src/github.com/influxdata/telegraf/agent/agent.go:396 +0x7d
created by github.com/influxdata/telegraf/agent.(*Agent).Run
    /root/go/src/github.com/influxdata/telegraf/agent/agent.go:397 +0x46a

goroutine 52 [select]:
github.com/influxdata/telegraf/agent.(*Agent).gatherer(0xc42000e048, 0xc420074a80, 0xc42005db40, 0x2540be400, 0xc420074cc0)
    /root/go/src/github.com/influxdata/telegraf/agent/agent.go:129 +0x3a8
github.com/influxdata/telegraf/agent.(*Agent).Run.func3(0xc420017370, 0xc42000e048, 0xc420074a80, 0xc420074cc0, 0xc42005db40, 0x2540be400)
    /root/go/src/github.com/influxdata/telegraf/agent/agent.go:396 +0x7d
created by github.com/influxdata/telegraf/agent.(*Agent).Run
    /root/go/src/github.com/influxdata/telegraf/agent/agent.go:397 +0x46a

goroutine 53 [select]:
github.com/influxdata/telegraf/agent.gatherWithTimeout(0xc420074a80, 0xc42005dbc0, 0xc4201e0680, 0x2540be400)
    /root/go/src/github.com/influxdata/telegraf/agent/agent.go:157 +0x28f
github.com/influxdata/telegraf/agent.(*Agent).gatherer(0xc42000e048, 0xc420074a80, 0xc42005dbc0, 0x2540be400, 0xc420074cc0)
    /root/go/src/github.com/influxdata/telegraf/agent/agent.go:124 +0x2a4
github.com/influxdata/telegraf/agent.(*Agent).Run.func3(0xc420017370, 0xc42000e048, 0xc420074a80, 0xc420074cc0, 0xc42005dbc0, 0x2540be400)
    /root/go/src/github.com/influxdata/telegraf/agent/agent.go:396 +0x7d
created by github.com/influxdata/telegraf/agent.(*Agent).Run
    /root/go/src/github.com/influxdata/telegraf/agent/agent.go:397 +0x46a

goroutine 54 [select]:
github.com/influxdata/telegraf/agent.(*Agent).gatherer(0xc42000e048, 0xc420074a80, 0xc42005dcc0, 0x2540be400, 0xc420074cc0)
    /root/go/src/github.com/influxdata/telegraf/agent/agent.go:129 +0x3a8
github.com/influxdata/telegraf/agent.(*Agent).Run.func3(0xc420017370, 0xc42000e048, 0xc420074a80, 0xc420074cc0, 0xc42005dcc0, 0x2540be400)
    /root/go/src/github.com/influxdata/telegraf/agent/agent.go:396 +0x7d
created by github.com/influxdata/telegraf/agent.(*Agent).Run
    /root/go/src/github.com/influxdata/telegraf/agent/agent.go:397 +0x46a

goroutine 55 [select]:
github.com/influxdata/telegraf/agent.gatherWithTimeout(0xc420074a80, 0xc42005dd40, 0xc420254260, 0x2540be400)
    /root/go/src/github.com/influxdata/telegraf/agent/agent.go:157 +0x28f
github.com/influxdata/telegraf/agent.(*Agent).gatherer(0xc42000e048, 0xc420074a80, 0xc42005dd40, 0x2540be400, 0xc420074cc0)
    /root/go/src/github.com/influxdata/telegraf/agent/agent.go:124 +0x2a4
github.com/influxdata/telegraf/agent.(*Agent).Run.func3(0xc420017370, 0xc42000e048, 0xc420074a80, 0xc420074cc0, 0xc42005dd40, 0x2540be400)
    /root/go/src/github.com/influxdata/telegraf/agent/agent.go:396 +0x7d
created by github.com/influxdata/telegraf/agent.(*Agent).Run
    /root/go/src/github.com/influxdata/telegraf/agent/agent.go:397 +0x46a

goroutine 56 [select]:
github.com/influxdata/telegraf/agent.(*Agent).gatherer(0xc42000e048, 0xc420074a80, 0xc42005ddc0, 0x2540be400, 0xc420074cc0)
    /root/go/src/github.com/influxdata/telegraf/agent/agent.go:129 +0x3a8
github.com/influxdata/telegraf/agent.(*Agent).Run.func3(0xc420017370, 0xc42000e048, 0xc420074a80, 0xc420074cc0, 0xc42005ddc0, 0x2540be400)
    /root/go/src/github.com/influxdata/telegraf/agent/agent.go:396 +0x7d
created by github.com/influxdata/telegraf/agent.(*Agent).Run
    /root/go/src/github.com/influxdata/telegraf/agent/agent.go:397 +0x46a

goroutine 57 [select]:
github.com/influxdata/telegraf/agent.gatherWithTimeout(0xc420074a80, 0xc42005de40, 0xc420297a40, 0x2540be400)
    /root/go/src/github.com/influxdata/telegraf/agent/agent.go:157 +0x28f
github.com/influxdata/telegraf/agent.(*Agent).gatherer(0xc42000e048, 0xc420074a80, 0xc42005de40, 0x2540be400, 0xc420074cc0)
    /root/go/src/github.com/influxdata/telegraf/agent/agent.go:124 +0x2a4
github.com/influxdata/telegraf/agent.(*Agent).Run.func3(0xc420017370, 0xc42000e048, 0xc420074a80, 0xc420074cc0, 0xc42005de40, 0x2540be400)
    /root/go/src/github.com/influxdata/telegraf/agent/agent.go:396 +0x7d
created by github.com/influxdata/telegraf/agent.(*Agent).Run
    /root/go/src/github.com/influxdata/telegraf/agent/agent.go:397 +0x46a

goroutine 18584 [runnable]:
syscall.Wait4(0x33c, 0xc42003ebfc, 0x0, 0xc420cb4000, 0xc42003ecb0, 0xc420809200, 0x0)
    /usr/local/go/src/syscall/syscall_linux.go:254 +0x91
os.(*Process).wait(0xc420809200, 0x411452, 0xc4207f6cb0, 0xc420809140)
    /usr/local/go/src/os/exec_unix.go:38 +0xbb
os.(*Process).Wait(0xc420809200, 0x0, 0x0, 0x13ed1c8)
    /usr/local/go/src/os/exec.go:115 +0x2b
os/exec.(*Cmd).Wait(0xc420889600, 0x0, 0x0)
    /usr/local/go/src/os/exec/exec.go:435 +0x62
os/exec.(*Cmd).Run(0xc420889600, 0xc420279a90, 0x2)
    /usr/local/go/src/os/exec/exec.go:280 +0x5c
os/exec.(*Cmd).Output(0xc420889600, 0xc, 0xc42003ede8, 0x2, 0x2, 0xc420889600)
    /usr/local/go/src/os/exec/exec.go:474 +0x11c
github.com/shirou/gopsutil/internal/common.DoSysctrl(0x138d005, 0x15, 0xc4207f6c20, 0x1, 0x1, 0x1ae4540, 0xc420809110)
    /root/go/src/github.com/shirou/gopsutil/internal/common/common_linux.go:20 +0x120
github.com/shirou/gopsutil/host.Info(0xc420803560, 0x0, 0x0)
    /root/go/src/github.com/shirou/gopsutil/host/host_linux.go:79 +0x1d4
github.com/influxdata/telegraf/plugins/inputs/system.(*SystemStats).Gather(0x1bff5e0, 0x1af34c0, 0xc4201e0680, 0x113f140, 0xc420a31f20)
    /root/go/src/github.com/influxdata/telegraf/plugins/inputs/system/system.go:30 +0x55
github.com/influxdata/telegraf/agent.gatherWithTimeout.func1(0xc420e6c3c0, 0xc42005dbc0, 0xc4201e0680)
    /root/go/src/github.com/influxdata/telegraf/agent/agent.go:153 +0x49
created by github.com/influxdata/telegraf/agent.gatherWithTimeout
    /root/go/src/github.com/influxdata/telegraf/agent/agent.go:154 +0xcc

goroutine 17300 [IO wait]:
net.runtime_pollWait(0x7f8015e325f8, 0x72, 0xc)
    /usr/local/go/src/runtime/netpoll.go:164 +0x59
net.(*pollDesc).wait(0xc420386068, 0x72, 0x1ae7240, 0x1ade880)
    /usr/local/go/src/net/fd_poll_runtime.go:75 +0x38
net.(*pollDesc).waitRead(0xc420386068, 0xc42093b000, 0x1000)
    /usr/local/go/src/net/fd_poll_runtime.go:80 +0x34
net.(*netFD).Read(0xc420386000, 0xc42093b000, 0x1000, 0x1000, 0x0, 0x1ae7240, 0x1ade880)
    /usr/local/go/src/net/fd_unix.go:250 +0x1b7
net.(*conn).Read(0xc420256000, 0xc42093b000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/net.go:181 +0x70
net/http.(*persistConn).Read(0xc420710000, 0xc42093b000, 0x1000, 0x1000, 0x4, 0x455bd0, 0xc420a30360)
    /usr/local/go/src/net/http/transport.go:1316 +0x14b
bufio.(*Reader).fill(0xc420a300c0)
    /usr/local/go/src/bufio/bufio.go:97 +0x117
bufio.(*Reader).Peek(0xc420a300c0, 0x1, 0xc4209f2be5, 0x1, 0x0, 0xc420c421e0, 0x0)
    /usr/local/go/src/bufio/bufio.go:129 +0x67
net/http.(*persistConn).readLoop(0xc420710000)
    /usr/local/go/src/net/http/transport.go:1474 +0x196
created by net/http.(*Transport).dialConn
    /usr/local/go/src/net/http/transport.go:1117 +0xa35

goroutine 17301 [select]:
net/http.(*persistConn).writeLoop(0xc420710000)
    /usr/local/go/src/net/http/transport.go:1704 +0x43a
created by net/http.(*Transport).dialConn
    /usr/local/go/src/net/http/transport.go:1118 +0xa5a

goroutine 18448 [runnable]:
runtime.Gosched()
    /usr/local/go/src/runtime/proc.go:240 +0x1e
syscall.ParseDirent(0xc420ca7048, 0x5d0, 0xfb8, 0xffffffffffffffff, 0xc420cb0000, 0x0, 0x64, 0xc420d27b58, 0x1c, 0x6, ...)
    /usr/local/go/src/syscall/dirent.go:99 +0x2b7
os.(*File).readdirnames(0xc420bfc200, 0xffffffffffffffff, 0x18, 0x54dad3, 0xc420d27bb8, 0xc420d27bb0, 0xc4207461e0)
    /usr/local/go/src/os/dir_unix.go:77 +0x14a
os.(*File).Readdirnames(0xc420bfc200, 0xffffffffffffffff, 0xc420bfc200, 0x0, 0x0, 0x0, 0x0)
    /usr/local/go/src/os/dir.go:45 +0x3e
path/filepath.glob(0xc420153270, 0x9, 0x1385434, 0x4, 0xc420c70c00, 0x3d, 0x40, 0xc420c70c00, 0x3d, 0x40, ...)
    /usr/local/go/src/path/filepath/match.go:324 +0x172
path/filepath.Glob(0x1385427, 0x11, 0xc4208e6648, 0xc4208e65c8, 0x411452, 0x7f8015e7f4b0, 0xc42079ece8)
    /usr/local/go/src/path/filepath/match.go:265 +0x181
github.com/influxdata/telegraf/plugins/inputs/system.(*Processes).gatherFromProc(0xc420255d00, 0xc4206c02d0, 0x0, 0xc42029f680)
    /root/go/src/github.com/influxdata/telegraf/plugins/inputs/system/processes.go:134 +0x4b
github.com/influxdata/telegraf/plugins/inputs/system.(*Processes).Gather(0xc420255d00, 0x1af34c0, 0xc420297a40, 0x434248, 0x13ed450)
    /root/go/src/github.com/influxdata/telegraf/plugins/inputs/system/processes.go:55 +0xf2
github.com/influxdata/telegraf/agent.gatherWithTimeout.func1(0xc420930300, 0xc42005de40, 0xc420297a40)
    /root/go/src/github.com/influxdata/telegraf/agent/agent.go:153 +0x49
created by github.com/influxdata/telegraf/agent.gatherWithTimeout
    /root/go/src/github.com/influxdata/telegraf/agent/agent.go:154 +0xcc

goroutine 18531 [runnable]:
runtime.SetFinalizer(0x12bb300, 0xc4207065d0, 0x1168940, 0x13ed178)
    /usr/local/go/src/runtime/mfinal.go:295 +0x808
os.newProcess(0x33e, 0x0, 0xc420820d40)
    /usr/local/go/src/os/exec.go:25 +0x75
os.startProcess(0x137e583, 0xd, 0xc420820d40, 0x4, 0x4, 0xc420e81738, 0xc4209cf800, 0x18, 0x18)
    /usr/local/go/src/os/exec_posix.go:49 +0x2f0
os.StartProcess(0x137e583, 0xd, 0xc420820d40, 0x4, 0x4, 0xc420e81738, 0x0, 0x0, 0xc420a2a160)
    /usr/local/go/src/os/exec.go:94 +0x64
os/exec.(*Cmd).Start(0xc420a2a160, 0xc420971570, 0xc420a2a160)
    /usr/local/go/src/os/exec/exec.go:359 +0x3d2
github.com/shirou/gopsutil/internal/common.CombinedOutputTimeout(0xc420a2a160, 0xb2d05e00, 0xc42028f920, 0x3, 0x3, 0xc420a2a160, 0xc42089c001)
    /root/go/src/github.com/shirou/gopsutil/internal/common/common.go:311 +0x84
github.com/shirou/gopsutil/internal/common.Invoke.Command(0x137e583, 0xd, 0xc42028f920, 0x3, 0x3, 0x411ca8, 0x30, 0x115bf20, 0xc4206cd201, 0xc42028f920)
    /root/go/src/github.com/shirou/gopsutil/internal/common/common.go:41 +0x71
github.com/shirou/gopsutil/internal/common.(*Invoke).Command(0x1bff5e0, 0x137e583, 0xd, 0xc42028f920, 0x3, 0x3, 0xb, 0x10000c42033c928, 0xd, 0xe, ...)
    <autogenerated>:8 +0x81
github.com/shirou/gopsutil/disk.GetDiskSerialNumber(0xc420a31ecd, 0x4, 0x0, 0x40)
    /root/go/src/github.com/shirou/gopsutil/disk/disk_linux.go:373 +0x1e5
github.com/shirou/gopsutil/disk.IOCounters(0x0, 0x0, 0x0, 0x47e925, 0x0, 0xb)
    /root/go/src/github.com/shirou/gopsutil/disk/disk_linux.go:358 +0x6b0
github.com/influxdata/telegraf/plugins/inputs/system.(*systemPS).DiskIO(0xc420016f60, 0x0, 0x0, 0x0, 0xb, 0xc420e396f7, 0x709)
    /root/go/src/github.com/influxdata/telegraf/plugins/inputs/system/ps.go:138 +0x43
github.com/influxdata/telegraf/plugins/inputs/system.(*DiskIOStats).Gather(0xc4201d2000, 0x1af34c0, 0xc420254260, 0x113f140, 0xc420e645a0)
    /root/go/src/github.com/influxdata/telegraf/plugins/inputs/system/disk.go:128 +0x66
github.com/influxdata/telegraf/agent.gatherWithTimeout.func1(0xc420a31d40, 0xc42005dd40, 0xc420254260)
    /root/go/src/github.com/influxdata/telegraf/agent/agent.go:153 +0x49
created by github.com/influxdata/telegraf/agent.gatherWithTimeout
    /root/go/src/github.com/influxdata/telegraf/agent/agent.go:154 +0xcc
danielnelson commented 7 years ago

Anything interesting in the output of dmesg?

dirkaholic commented 7 years ago

Started it again in order to watch the dmesg output at the time of the crash.

telegraf showed yet another error message now which starts with:

unexpected fault address 0x0
fatal error: fault
[signal SIGSEGV: segmentation violation code=0x80 addr=0x0 pc=0x42e833]

As I see segfaults on other programs too when running dmesg -T I will initiate a hardware check on the server as I assume there might be a problem.

danielnelson commented 7 years ago

Yeah it seems this way, I'm going to close this issue for now and we can alway reopen if it seems like it is not a system problem.