search

justwatchcom / sql_exporter

Flexible SQL Exporter for Prometheus.
MIT License
402 stars 109 forks source link

fatal error: concurrent map iteration and map write #27

Closed willbuckner closed 2 years ago

willbuckner commented 5 years ago

Hello,

I am seeing many random crashes after long periods of running.

Version: 0.2.0 Environment: Linux 4.14/x86_64 go version: go1.8.3

The crash is:

fatal error: concurrent map iteration and map write

Anonymized logs of crash are here:


Jan 12 23:16:25 <HOST> prometheus-sql-exporter[<PID>]: {"caller":"level.go:84"... (a normal event being logged)}
Jan 12 23:16:26 <HOST> prometheus-sql-exporter[<PID>]: fatal error: concurrent map iteration and map write
Jan 12 23:16:26 <HOST> prometheus-sql-exporter[<PID>]: goroutine 20829277 [running]:
Jan 12 23:16:26 <HOST> prometheus-sql-exporter[<PID>]: runtime.throw(0x9198cb, 0x26)
Jan 12 23:16:26 <HOST> prometheus-sql-exporter[<PID>]:         /usr/local/go/src/runtime/panic.go:596 +0x95 fp=0xc421023db0 sp=0xc421023d90
Jan 12 23:16:26 <HOST> prometheus-sql-exporter[<PID>]: runtime.mapiternext(0xc421023f28)
Jan 12 23:16:26 <HOST> prometheus-sql-exporter[<PID>]:         /usr/local/go/src/runtime/hashmap.go:737 +0x7ee fp=0xc421023e60 sp=0xc421023db0
Jan 12 23:16:26 <HOST> prometheus-sql-exporter[<PID>]: runtime.mapiterinit(0x8866a0, 0xc420135ce0, 0xc421023f28)
Jan 12 23:16:26 <HOST> prometheus-sql-exporter[<PID>]:         /usr/local/go/src/runtime/hashmap.go:727 +0x2b3 fp=0xc421023eb8 sp=0xc421023e60
Jan 12 23:16:26 <HOST> prometheus-sql-exporter[<PID>]: main.(*Exporter).Collect(0xc4201350e0, 0xc4210b63c0)
Jan 12 23:16:26 <HOST> prometheus-sql-exporter[<PID>]:         /build/prometheus-sql-exporter/tmp/build/src/github.com/justwatchcom/sql_exporter/exporter.go:77 +0x100 fp=0xc421023f98 sp=0xc421023eb8
Jan 12 23:16:26 <HOST> prometheus-sql-exporter[<PID>]: github.com/justwatchcom/sql_exporter/vendor/github.com/prometheus/client_golang/prometheus.(*Registry).Gather.func2(0xc421249790, 0xc4210b63c0, 0xbcac60, 0xc4201350e0)
Jan 12 23:16:26 <HOST> prometheus-sql-exporter[<PID>]:         /build/prometheus-sql-exporter/tmp/build/src/github.com/justwatchcom/sql_exporter/vendor/github.com/prometheus/client_golang/prometheus/registry.go:382 +0x61 fp=0xc421023fc0 sp=0xc421023f98
Jan 12 23:16:26 <HOST> prometheus-sql-exporter[<PID>]: runtime.goexit()
Jan 12 23:16:26 <HOST> prometheus-sql-exporter[<PID>]:         /usr/local/go/src/runtime/asm_amd64.s:2197 +0x1 fp=0xc421023fc8 sp=0xc421023fc0
Jan 12 23:16:26 <HOST> prometheus-sql-exporter[<PID>]: created by github.com/justwatchcom/sql_exporter/vendor/github.com/prometheus/client_golang/prometheus.(*Registry).Gather
Jan 12 23:16:26 <HOST> prometheus-sql-exporter[<PID>]:         /build/prometheus-sql-exporter/tmp/build/src/github.com/justwatchcom/sql_exporter/vendor/github.com/prometheus/client_golang/prometheus/registry.go:383 +0x2ec
Jan 12 23:16:26 <HOST> prometheus-sql-exporter[<PID>]: goroutine 1 [IO wait, 1735 minutes]:
Jan 12 23:16:26 <HOST> prometheus-sql-exporter[<PID>]: net.runtime_pollWait(0x7ffb89c2a700, 0x72, 0x0)
Jan 12 23:16:26 <HOST> prometheus-sql-exporter[<PID>]:         /usr/local/go/src/runtime/netpoll.go:164 +0x59
Jan 12 23:16:26 <HOST> prometheus-sql-exporter[<PID>]: net.(*pollDesc).wait(0xc4202e7db8, 0x72, 0x0, 0xc421595d60)
Jan 12 23:16:26 <HOST> prometheus-sql-exporter[<PID>]:         /usr/local/go/src/net/fd_poll_runtime.go:75 +0x38
Jan 12 23:16:26 <HOST> prometheus-sql-exporter[<PID>]: net.(*pollDesc).waitRead(0xc4202e7db8, 0xffffffffffffffff, 0x0)
Jan 12 23:16:26 <HOST> prometheus-sql-exporter[<PID>]:         /usr/local/go/src/net/fd_poll_runtime.go:80 +0x34
Jan 12 23:16:26 <HOST> prometheus-sql-exporter[<PID>]: net.(*netFD).accept(0xc4202e7d50, 0x0, 0xbc84e0, 0xc421595d60)
Jan 12 23:16:26 <HOST> prometheus-sql-exporter[<PID>]:         /usr/local/go/src/net/fd_unix.go:430 +0x1e5
Jan 12 23:16:26 <HOST> prometheus-sql-exporter[<PID>]: net.(*TCPListener).accept(0xc42012c3b0, 0xc4200ad2a0, 0x87e080, 0xffffffffffffffff)
Jan 12 23:16:26 <HOST> prometheus-sql-exporter[<PID>]:         /usr/local/go/src/net/tcpsock_posix.go:136 +0x2e
Jan 12 23:16:26 <HOST> prometheus-sql-exporter[<PID>]: net.(*TCPListener).AcceptTCP(0xc42012c3b0, 0xc42006fbc8, 0xc42006fbd0, 0xc42006fbc0)
Jan 12 23:16:26 <HOST> prometheus-sql-exporter[<PID>]:         /usr/local/go/src/net/tcpsock.go:215 +0x49
Jan 12 23:16:26 <HOST> prometheus-sql-exporter[<PID>]: net/http.tcpKeepAliveListener.Accept(0xc42012c3b0, 0x924870, 0xc4200ad220, 0xbcde60, 0xc420345470)
Jan 12 23:16:26 <HOST> prometheus-sql-exporter[<PID>]:         /usr/local/go/src/net/http/server.go:3044 +0x2f
Jan 12 23:16:26 <HOST> prometheus-sql-exporter[<PID>]: net/http.(*Server).Serve(0xc4203182c0, 0xbcd820, 0xc42012c3b0, 0x0, 0x0)
Jan 12 23:16:26 <HOST> prometheus-sql-exporter[<PID>]:         /usr/local/go/src/net/http/server.go:2643 +0x228
Jan 12 23:16:26 <HOST> prometheus-sql-exporter[<PID>]: net/http.(*Server).ListenAndServe(0xc4203182c0, 0xc4203182c0, 0xc4201286d0)
Jan 12 23:16:26 <HOST> prometheus-sql-exporter[<PID>]:         /usr/local/go/src/net/http/server.go:2585 +0xb0
Jan 12 23:16:26 <HOST> prometheus-sql-exporter[<PID>]: net/http.ListenAndServe(0x7fffe54dff20, 0x16, 0x0, 0x0, 0x0, 0x0)
Jan 12 23:16:26 <HOST> prometheus-sql-exporter[<PID>]:         /usr/local/go/src/net/http/server.go:2787 +0x7f
Jan 12 23:16:26 <HOST> prometheus-sql-exporter[<PID>]: main.main()
Jan 12 23:16:26 <HOST> prometheus-sql-exporter[<PID>]:         /build/prometheus-sql-exporter/tmp/build/src/github.com/justwatchcom/sql_exporter/main.go:80 +0xa7d
Jan 12 23:16:26 <HOST> prometheus-sql-exporter[<PID>]: goroutine 17 [syscall, 88007 minutes, locked to thread]:
Jan 12 23:16:26 <HOST> prometheus-sql-exporter[<PID>]: runtime.goexit()
Jan 12 23:16:26 <HOST> prometheus-sql-exporter[<PID>]:         /usr/local/go/src/runtime/asm_amd64.s:2197 +0x1
Jan 12 23:16:26 <HOST> prometheus-sql-exporter[<PID>]: goroutine 19 [chan receive]:
Jan 12 23:16:26 <HOST> prometheus-sql-exporter[<PID>]: main.(*Job).runOnce(0xc42013f800, 0x312c8110, 0xc01540)
Jan 12 23:16:26 <HOST> prometheus-sql-exporter[<PID>]:         /build/prometheus-sql-exporter/tmp/build/src/github.com/justwatchcom/sql_exporter/job.go:167 +0x10a
Jan 12 23:16:26 <HOST> prometheus-sql-exporter[<PID>]: main.(*Job).(main.runOnce)-fm(0xc4210c6000, 0xc420061d48)
Jan 12 23:16:26 <HOST> prometheus-sql-exporter[<PID>]:         /build/prometheus-sql-exporter/tmp/build/src/github.com/justwatchcom/sql_exporter/job.go:116 +0x2a
Jan 12 23:16:26 <HOST> prometheus-sql-exporter[<PID>]: github.com/justwatchcom/sql_exporter/vendor/github.com/cenkalti/backoff.RetryNotify(0xc420061ea8, 0xbca8a0, 0xc4210c6000, 0x0, 0xc1da20, 0xc4210c6000)
Jan 12 23:16:26 <HOST> prometheus-sql-exporter[<PID>]:         /build/prometheus-sql-exporter/tmp/build/src/github.com/justwatchcom/sql_exporter/vendor/github.com/cenkalti/backoff/retry.go:32 +0x3f
Jan 12 23:16:26 <HOST> prometheus-sql-exporter[<PID>]: github.com/justwatchcom/sql_exporter/vendor/github.com/cenkalti/backoff.Retry(0xc420061ea8, 0xbca8a0, 0xc4210c6000, 0x4, 0x0)
Jan 12 23:16:26 <HOST> prometheus-sql-exporter[<PID>]:         /build/prometheus-sql-exporter/tmp/build/src/github.com/justwatchcom/sql_exporter/vendor/github.com/cenkalti/backoff/retry.go:22 +0x48
Jan 12 23:16:26 <HOST> prometheus-sql-exporter[<PID>]: main.(*Job).Run(0xc42013f800)
Jan 12 23:16:26 <HOST> prometheus-sql-exporter[<PID>]:         /build/prometheus-sql-exporter/tmp/build/src/github.com/justwatchcom/sql_exporter/job.go:116 +0x466
Jan 12 23:16:26 <HOST> prometheus-sql-exporter[<PID>]: created by main.NewExporter
Jan 12 23:16:26 <HOST> prometheus-sql-exporter[<PID>]:         /build/prometheus-sql-exporter/tmp/build/src/github.com/justwatchcom/sql_exporter/exporter.go:42 +0x20c
Jan 12 23:16:26 <HOST> prometheus-sql-exporter[<PID>]: goroutine 8 [chan receive, 88007 minutes]:
Jan 12 23:16:26 <HOST> prometheus-sql-exporter[<PID>]: database/sql.(*DB).connectionOpener(0xc4200acfa0)
Jan 12 23:16:26 <HOST> prometheus-sql-exporter[<PID>]:         /usr/local/go/src/database/sql/sql.go:837 +0x4a
Jan 12 23:16:26 <HOST> prometheus-sql-exporter[<PID>]: created by database/sql.Open
Jan 12 23:16:26 <HOST> prometheus-sql-exporter[<PID>]:         /usr/local/go/src/database/sql/sql.go:582 +0x212
Jan 12 23:16:26 <HOST> prometheus-sql-exporter[<PID>]: goroutine 19552027 [IO wait]:
Jan 12 23:16:26 <HOST> prometheus-sql-exporter[<PID>]: net.runtime_pollWait(0x7ffb89c2a280, 0x72, 0x8)
Jan 12 23:16:26 <HOST> prometheus-sql-exporter[<PID>]:         /usr/local/go/src/runtime/netpoll.go:164 +0x59
Jan 12 23:16:26 <HOST> prometheus-sql-exporter[<PID>]: net.(*pollDesc).wait(0xc4200ba148, 0x72, 0xbc9aa0, 0xbc5610)
Jan 12 23:16:26 <HOST> prometheus-sql-exporter[<PID>]:         /usr/local/go/src/net/fd_poll_runtime.go:75 +0x38
Jan 12 23:16:26 <HOST> prometheus-sql-exporter[<PID>]: net.(*pollDesc).waitRead(0xc4200ba148, 0xc42074a000, 0x1000)
Jan 12 23:16:26 <HOST> prometheus-sql-exporter[<PID>]:         /usr/local/go/src/net/fd_poll_runtime.go:80 +0x34
Jan 12 23:16:26 <HOST> prometheus-sql-exporter[<PID>]: net.(*netFD).Read(0xc4200ba0e0, 0xc42074a000, 0x1000, 0x1000, 0x0, 0xbc9aa0, 0xbc5610)
Jan 12 23:16:26 <HOST> prometheus-sql-exporter[<PID>]:         /usr/local/go/src/net/fd_unix.go:250 +0x1b7
Jan 12 23:16:26 <HOST> prometheus-sql-exporter[<PID>]: net.(*conn).Read(0xc42118a000, 0xc42074a000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
Jan 12 23:16:26 <HOST> prometheus-sql-exporter[<PID>]:         /usr/local/go/src/net/net.go:181 +0x70
Jan 12 23:16:26 <HOST> prometheus-sql-exporter[<PID>]: net/http.(*connReader).Read(0xc4205ba040, 0xc42074a000, 0x1000, 0x1000, 0x1b80, 0x8c81c0, 0xc420648c01)
Jan 12 23:16:26 <HOST> prometheus-sql-exporter[<PID>]:         /usr/local/go/src/net/http/server.go:754 +0x140
Jan 12 23:16:26 <HOST> prometheus-sql-exporter[<PID>]: bufio.(*Reader).fill(0xc42108a060)
Jan 12 23:16:26 <HOST> prometheus-sql-exporter[<PID>]:         /usr/local/go/src/bufio/bufio.go:97 +0x117
Jan 12 23:16:26 <HOST> prometheus-sql-exporter[<PID>]: bufio.(*Reader).ReadSlice(0xc42108a060, 0xa, 0xf, 0xe, 0xc4200719f8, 0x410096, 0x7ffb89cc6138)
Jan 12 23:16:26 <HOST> prometheus-sql-exporter[<PID>]:         /usr/local/go/src/bufio/bufio.go:338 +0xbb
Jan 12 23:16:26 <HOST> prometheus-sql-exporter[<PID>]: bufio.(*Reader).ReadLine(0xc42108a060, 0xc420090e00, 0x100, 0xf8, 0x8f9520, 0xc420648c01, 0x17ffb89cc1000)
Jan 12 23:16:26 <HOST> prometheus-sql-exporter[<PID>]:         /usr/local/go/src/bufio/bufio.go:367 +0x37
Jan 12 23:16:26 <HOST> prometheus-sql-exporter[<PID>]: net/textproto.(*Reader).readLineSlice(0xc420a94420, 0xc420071ac8, 0xc420071ac8, 0x410df8, 0x100, 0x8f9520)
Jan 12 23:16:26 <HOST> prometheus-sql-exporter[<PID>]:         /usr/local/go/src/net/textproto/reader.go:55 +0x5f
Jan 12 23:16:26 <HOST> prometheus-sql-exporter[<PID>]: net/textproto.(*Reader).ReadLine(0xc420a94420, 0xc420090e00, 0x72, 0x8000000000000000, 0x0)
Jan 12 23:16:26 <HOST> prometheus-sql-exporter[<PID>]:         /usr/local/go/src/net/textproto/reader.go:36 +0x2f
Jan 12 23:16:26 <HOST> prometheus-sql-exporter[<PID>]: net/http.readRequest(0xc42108a060, 0x0, 0xc420090e00, 0x0, 0x0)
Jan 12 23:16:26 <HOST> prometheus-sql-exporter[<PID>]:         /usr/local/go/src/net/http/request.go:918 +0xa5
Jan 12 23:16:26 <HOST> prometheus-sql-exporter[<PID>]: net/http.(*conn).readRequest(0xc4200ac3c0, 0xbcdda0, 0xc4205ba000, 0x0, 0x0, 0x0)
Jan 12 23:16:26 <HOST> prometheus-sql-exporter[<PID>]:         /usr/local/go/src/net/http/server.go:934 +0x213
Jan 12 23:16:26 <HOST> prometheus-sql-exporter[<PID>]: net/http.(*conn).serve(0xc4200ac3c0, 0xbcdda0, 0xc4205ba000)
Jan 12 23:16:26 <HOST> prometheus-sql-exporter[<PID>]:         /usr/local/go/src/net/http/server.go:1763 +0x49a
Jan 12 23:16:26 <HOST> prometheus-sql-exporter[<PID>]: created by net/http.(*Server).Serve
Jan 12 23:16:26 <HOST> prometheus-sql-exporter[<PID>]:         /usr/local/go/src/net/http/server.go:2668 +0x2ce
Jan 12 23:16:26 <HOST> systemd[1]: prometheus-sql-exporter.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
Jan 12 23:16:26 <HOST> prometheus-sql-exporter[<PID>]: goroutine 20829272 [IO wait]:
Jan 12 23:16:26 <HOST> prometheus-sql-exporter[<PID>]: net.runtime_pollWait(0x7ffb89c2a580, 0x72, 0x7)
Jan 12 23:16:26 <HOST> prometheus-sql-exporter[<PID>]:         /usr/local/go/src/runtime/netpoll.go:164 +0x59
Jan 12 23:16:26 <HOST> prometheus-sql-exporter[<PID>]: net.(*pollDesc).wait(0xc4200ba0d8, 0x72, 0xbc9aa0, 0xbc5610)
Jan 12 23:16:26 <HOST> prometheus-sql-exporter[<PID>]:         /usr/local/go/src/net/fd_poll_runtime.go:75 +0x38
Jan 12 23:16:26 <HOST> prometheus-sql-exporter[<PID>]: net.(*pollDesc).waitRead(0xc4200ba0d8, 0xc42084e611, 0x1)
Jan 12 23:16:26 <HOST> prometheus-sql-exporter[<PID>]:         /usr/local/go/src/net/fd_poll_runtime.go:80 +0x34
Jan 12 23:16:26 <HOST> prometheus-sql-exporter[<PID>]: net.(*netFD).Read(0xc4200ba070, 0xc42084e611, 0x1, 0x1, 0x0, 0xbc9aa0, 0xbc5610)
Jan 12 23:16:26 <HOST> prometheus-sql-exporter[<PID>]:         /usr/local/go/src/net/fd_unix.go:250 +0x1b7
Jan 12 23:16:26 <HOST> prometheus-sql-exporter[<PID>]: net.(*conn).Read(0xc42012c0b0, 0xc42084e611, 0x1, 0x1, 0x0, 0x0, 0x0)
Jan 12 23:16:26 <HOST> prometheus-sql-exporter[<PID>]:         /usr/local/go/src/net/net.go:181 +0x70
Jan 12 23:16:26 <HOST> prometheus-sql-exporter[<PID>]: net/http.(*connReader).backgroundRead(0xc42084e600)
Jan 12 23:16:26 <HOST> prometheus-sql-exporter[<PID>]:         /usr/local/go/src/net/http/server.go:656 +0x58
Jan 12 23:16:26 <HOST> prometheus-sql-exporter[<PID>]: created by net/http.(*connReader).startBackgroundRead
Jan 12 23:16:26 <HOST> prometheus-sql-exporter[<PID>]:         /usr/local/go/src/net/http/server.go:652 +0xdf
Jan 12 23:16:26 <HOST> prometheus-sql-exporter[<PID>]: goroutine 20298495 [runnable]:
Jan 12 23:16:26 <HOST> prometheus-sql-exporter[<PID>]: github.com/justwatchcom/sql_exporter/vendor/github.com/prometheus/client_golang/prometheus.checkMetricConsistency(0xc420d6ab40, 0xc4213dfa40, 0xc420a94540, 0xc420a94570, 0xc42010bd01, 0x1)
Jan 12 23:16:26 <HOST> prometheus-sql-exporter[<PID>]:         /build/prometheus-sql-exporter/tmp/build/src/github.com/justwatchcom/sql_exporter/vendor/github.com/prometheus/client_golang/prometheus/registry.go:696 +0x337
Jan 12 23:16:26 <HOST> prometheus-sql-exporter[<PID>]: github.com/justwatchcom/sql_exporter/vendor/github.com/prometheus/client_golang/prometheus.(*Registry).Gather(0xc420012940, 0x0, 0x0, 0x0, 0x0, 0x0)
Jan 12 23:16:26 <HOST> prometheus-sql-exporter[<PID>]:         /build/prometheus-sql-exporter/tmp/build/src/github.com/justwatchcom/sql_exporter/vendor/github.com/prometheus/client_golang/prometheus/registry.go:494 +0xa8a
Jan 12 23:16:26 <HOST> prometheus-sql-exporter[<PID>]: github.com/justwatchcom/sql_exporter/vendor/github.com/prometheus/client_golang/prometheus/promhttp.HandlerFor.func1(0xbcd4a0, 0xc4208280e0, 0xc42000b700)
Jan 12 23:16:26 <HOST> prometheus-sql-exporter[<PID>]:         /build/prometheus-sql-exporter/tmp/build/src/github.com/justwatchcom/sql_exporter/vendor/github.com/prometheus/client_golang/prometheus/promhttp/http.go:82 +0x4c
Jan 12 23:16:26 <HOST> prometheus-sql-exporter[<PID>]: net/http.HandlerFunc.ServeHTTP(0xc42013bd40, 0xbcd4a0, 0xc4208280e0, 0xc42000b700)
Jan 12 23:16:26 <HOST> prometheus-sql-exporter[<PID>]:         /usr/local/go/src/net/http/server.go:1942 +0x44
Jan 12 23:16:26 <HOST> prometheus-sql-exporter[<PID>]: net/http.(*ServeMux).ServeHTTP(0xc01280, 0xbcd4a0, 0xc4208280e0, 0xc42000b700)
Jan 12 23:16:26 <HOST> systemd[1]: prometheus-sql-exporter.service: Unit entered failed state.
Jan 12 23:16:26 <HOST> prometheus-sql-exporter[<PID>]:         /usr/local/go/src/net/http/server.go:2238 +0x130
Jan 12 23:16:26 <HOST> prometheus-sql-exporter[<PID>]: net/http.serverHandler.ServeHTTP(0xc4203182c0, 0xbcd4a0, 0xc4208280e0, 0xc42000b700)
Jan 12 23:16:26 <HOST> prometheus-sql-exporter[<PID>]:         /usr/local/go/src/net/http/server.go:2568 +0x92
Jan 12 23:16:26 <HOST> prometheus-sql-exporter[<PID>]: net/http.(*conn).serve(0xc4200ad220, 0xbcdda0, 0xc42084e5c0)
Jan 12 23:16:26 <HOST> prometheus-sql-exporter[<PID>]:         /usr/local/go/src/net/http/server.go:1825 +0x612
Jan 12 23:16:26 <HOST> prometheus-sql-exporter[<PID>]: created by net/http.(*Server).Serve
Jan 12 23:16:26 <HOST> prometheus-sql-exporter[<PID>]:         /usr/local/go/src/net/http/server.go:2668 +0x2ce
Jan 12 23:16:26 <HOST> prometheus-sql-exporter[<PID>]: goroutine 4546870 [select]:
Jan 12 23:16:26 <HOST> prometheus-sql-exporter[<PID>]: database/sql.(*DB).connectionCleaner(0xc4200acfa0, 0x1f3305bc00)
Jan 12 23:16:26 <HOST> prometheus-sql-exporter[<PID>]:         /usr/local/go/src/database/sql/sql.go:759 +0x59b
Jan 12 23:16:26 <HOST> prometheus-sql-exporter[<PID>]: created by database/sql.(*DB).startCleanerLocked
Jan 12 23:16:26 <HOST> prometheus-sql-exporter[<PID>]:         /usr/local/go/src/database/sql/sql.go:746 +0xb5
Jan 12 23:16:26 <HOST> prometheus-sql-exporter[<PID>]: goroutine 18957801 [IO wait]:
Jan 12 23:16:26 <HOST> prometheus-sql-exporter[<PID>]: net.runtime_pollWait(0x7ffb89c2a1c0, 0x72, 0x6)
Jan 12 23:16:26 <HOST> prometheus-sql-exporter[<PID>]:         /usr/local/go/src/runtime/netpoll.go:164 +0x59
Jan 12 23:16:26 <HOST> prometheus-sql-exporter[<PID>]: net.(*pollDesc).wait(0xc4201380d8, 0x72, 0xbc9aa0, 0xbc5610)
Jan 12 23:16:26 <HOST> prometheus-sql-exporter[<PID>]:         /usr/local/go/src/net/fd_poll_runtime.go:75 +0x38
Jan 12 23:16:26 <HOST> prometheus-sql-exporter[<PID>]: net.(*pollDesc).waitRead(0xc4201380d8, 0xc4211e5000, 0x1000)
Jan 12 23:16:26 <HOST> prometheus-sql-exporter[<PID>]:         /usr/local/go/src/net/fd_poll_runtime.go:80 +0x34
Jan 12 23:16:26 <HOST> prometheus-sql-exporter[<PID>]: net.(*netFD).Read(0xc420138070, 0xc4211e5000, 0x1000, 0x1000, 0x0, 0xbc9aa0, 0xbc5610)
Jan 12 23:16:26 <HOST> prometheus-sql-exporter[<PID>]:         /usr/local/go/src/net/fd_unix.go:250 +0x1b7
Jan 12 23:16:26 <HOST> prometheus-sql-exporter[<PID>]: net.(*conn).Read(0xc42012c000, 0xc4211e5000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
Jan 12 23:16:26 <HOST> prometheus-sql-exporter[<PID>]:         /usr/local/go/src/net/net.go:181 +0x70
Jan 12 23:16:26 <HOST> prometheus-sql-exporter[<PID>]: net/http.(*connReader).Read(0xc4214ec140, 0xc4211e5000, 0x1000, 0x1000, 0x1b80, 0x8c81c0, 0xc420084401)
Jan 12 23:16:26 <HOST> prometheus-sql-exporter[<PID>]:         /usr/local/go/src/net/http/server.go:754 +0x140
Jan 12 23:16:26 <HOST> prometheus-sql-exporter[<PID>]: bufio.(*Reader).fill(0xc420506000)
Jan 12 23:16:26 <HOST> prometheus-sql-exporter[<PID>]:         /usr/local/go/src/bufio/bufio.go:97 +0x117
Jan 12 23:16:26 <HOST> prometheus-sql-exporter[<PID>]: bufio.(*Reader).ReadSlice(0xc420506000, 0xc42095a00a, 0xc420cd11e0, 0x924e98, 0xc420960840, 0x20f459c0, 0xc420f45a28)
Jan 12 23:16:26 <HOST> prometheus-sql-exporter[<PID>]:         /usr/local/go/src/bufio/bufio.go:338 +0xbb
Jan 12 23:16:26 <HOST> prometheus-sql-exporter[<PID>]: bufio.(*Reader).ReadLine(0xc420506000, 0xc420091c00, 0x100, 0xf8, 0x8f9520, 0xc420084400, 0x7ffb89cc1000)
Jan 12 23:16:26 <HOST> prometheus-sql-exporter[<PID>]:         /usr/local/go/src/bufio/bufio.go:367 +0x37
Jan 12 23:16:26 <HOST> prometheus-sql-exporter[<PID>]: net/textproto.(*Reader).readLineSlice(0xc4205501b0, 0xc420f45ac8, 0xc420f45ac8, 0x410df8, 0x100, 0x8f9520)
Jan 12 23:16:26 <HOST> prometheus-sql-exporter[<PID>]:         /usr/local/go/src/net/textproto/reader.go:55 +0x5f
Jan 12 23:16:26 <HOST> prometheus-sql-exporter[<PID>]: net/textproto.(*Reader).ReadLine(0xc4205501b0, 0xc420091c00, 0x72, 0x8000000000000000, 0x0)
Jan 12 23:16:26 <HOST> prometheus-sql-exporter[<PID>]:         /usr/local/go/src/net/textproto/reader.go:36 +0x2f
Jan 12 23:16:26 <HOST> prometheus-sql-exporter[<PID>]: net/http.readRequest(0xc420506000, 0x0, 0xc420091c00, 0x0, 0x0)
Jan 12 23:16:26 <HOST> prometheus-sql-exporter[<PID>]:         /usr/local/go/src/net/http/request.go:918 +0xa5
Jan 12 23:16:26 <HOST> prometheus-sql-exporter[<PID>]: net/http.(*conn).readRequest(0xc4201200a0, 0xbcdda0, 0xc4214ec100, 0x0, 0x0, 0x0)
Jan 12 23:16:26 <HOST> prometheus-sql-exporter[<PID>]:         /usr/local/go/src/net/http/server.go:934 +0x213
Jan 12 23:16:26 <HOST> prometheus-sql-exporter[<PID>]: net/http.(*conn).serve(0xc4201200a0, 0xbcdda0, 0xc4214ec100)
Jan 12 23:16:26 <HOST> prometheus-sql-exporter[<PID>]:         /usr/local/go/src/net/http/server.go:1763 +0x49a
Jan 12 23:16:26 <HOST> prometheus-sql-exporter[<PID>]: created by net/http.(*Server).Serve
Jan 12 23:16:26 <HOST> prometheus-sql-exporter[<PID>]:         /usr/local/go/src/net/http/server.go:2668 +0x2ce
Jan 12 23:16:26 <HOST> prometheus-sql-exporter[<PID>]: goroutine 19516641 [IO wait]:
Jan 12 23:16:26 <HOST> prometheus-sql-exporter[<PID>]: net.runtime_pollWait(0x7ffb89c2a100, 0x72, 0x9)
Jan 12 23:16:26 <HOST> prometheus-sql-exporter[<PID>]:         /usr/local/go/src/runtime/netpoll.go:164 +0x59
Jan 12 23:16:26 <HOST> prometheus-sql-exporter[<PID>]: net.(*pollDesc).wait(0xc420138228, 0x72, 0xbc9aa0, 0xbc5610)
Jan 12 23:16:26 <HOST> prometheus-sql-exporter[<PID>]:         /usr/local/go/src/net/fd_poll_runtime.go:75 +0x38
Jan 12 23:16:26 <HOST> prometheus-sql-exporter[<PID>]: net.(*pollDesc).waitRead(0xc420138228, 0xc420a92000, 0x1000)
Jan 12 23:16:26 <HOST> prometheus-sql-exporter[<PID>]:         /usr/local/go/src/net/fd_poll_runtime.go:80 +0x34
Jan 12 23:16:26 <HOST> prometheus-sql-exporter[<PID>]: net.(*netFD).Read(0xc4201381c0, 0xc420a92000, 0x1000, 0x1000, 0x0, 0xbc9aa0, 0xbc5610)
Jan 12 23:16:26 <HOST> prometheus-sql-exporter[<PID>]:         /usr/local/go/src/net/fd_unix.go:250 +0x1b7
Jan 12 23:16:26 <HOST> prometheus-sql-exporter[<PID>]: net.(*conn).Read(0xc42118a208, 0xc420a92000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
Jan 12 23:16:26 <HOST> prometheus-sql-exporter[<PID>]:         /usr/local/go/src/net/net.go:181 +0x70
Jan 12 23:16:26 <HOST> prometheus-sql-exporter[<PID>]: net/http.(*connReader).Read(0xc420fc7440, 0xc420a92000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
Jan 12 23:16:26 <HOST> prometheus-sql-exporter[<PID>]:         /usr/local/go/src/net/http/server.go:754 +0x140
Jan 12 23:16:26 <HOST> prometheus-sql-exporter[<PID>]: bufio.(*Reader).fill(0xc420adc060)
Jan 12 23:16:26 <HOST> prometheus-sql-exporter[<PID>]:         /usr/local/go/src/bufio/bufio.go:97 +0x117
Jan 12 23:16:26 <HOST> prometheus-sql-exporter[<PID>]: bufio.(*Reader).ReadSlice(0xc420adc060, 0xa, 0x9, 0x8, 0xc42006d9f8, 0x410096, 0x7ffb89c67998)
Jan 12 23:16:26 <HOST> prometheus-sql-exporter[<PID>]:         /usr/local/go/src/bufio/bufio.go:338 +0xbb
Jan 12 23:16:26 <HOST> prometheus-sql-exporter[<PID>]: bufio.(*Reader).ReadLine(0xc420adc060, 0xc420492800, 0x100, 0xf8, 0x8f9520, 0xc420145401, 0x17ffb89cc1000)
Jan 12 23:16:26 <HOST> prometheus-sql-exporter[<PID>]:         /usr/local/go/src/bufio/bufio.go:367 +0x37
Jan 12 23:16:26 <HOST> prometheus-sql-exporter[<PID>]: net/textproto.(*Reader).readLineSlice(0xc420551920, 0xc42006dac8, 0xc42006dac8, 0x410df8, 0x100, 0x8f9520)
Jan 12 23:16:26 <HOST> prometheus-sql-exporter[<PID>]:         /usr/local/go/src/net/textproto/reader.go:55 +0x5f
Jan 12 23:16:26 <HOST> prometheus-sql-exporter[<PID>]: net/textproto.(*Reader).ReadLine(0xc420551920, 0xc420492800, 0x72, 0x8000000000000000, 0x0)
Jan 12 23:16:26 <HOST> prometheus-sql-exporter[<PID>]:         /usr/local/go/src/net/textproto/reader.go:36 +0x2f
Jan 12 23:16:26 <HOST> prometheus-sql-exporter[<PID>]: net/http.readRequest(0xc420adc060, 0x0, 0xc420492800, 0x0, 0x0)
Jan 12 23:16:26 <HOST> prometheus-sql-exporter[<PID>]:         /usr/local/go/src/net/http/request.go:918 +0xa5
Jan 12 23:16:26 <HOST> prometheus-sql-exporter[<PID>]: net/http.(*conn).readRequest(0xc420120320, 0xbcdda0, 0xc420fc7400, 0x0, 0x0, 0x0)
Jan 12 23:16:26 <HOST> prometheus-sql-exporter[<PID>]:         /usr/local/go/src/net/http/server.go:934 +0x213
Jan 12 23:16:26 <HOST> prometheus-sql-exporter[<PID>]: net/http.(*conn).serve(0xc420120320, 0xbcdda0, 0xc420fc7400)
Jan 12 23:16:26 <HOST> prometheus-sql-exporter[<PID>]:         /usr/local/go/src/net/http/server.go:1763 +0x49a
Jan 12 23:16:26 <HOST> prometheus-sql-exporter[<PID>]: created by net/http.(*Server).Serve
Jan 12 23:16:26 <HOST> prometheus-sql-exporter[<PID>]:         /usr/local/go/src/net/http/server.go:2668 +0x2ce
Jan 12 23:16:26 <HOST> prometheus-sql-exporter[<PID>]: goroutine 20828607 [runnable]:
Jan 12 23:16:26 <HOST> prometheus-sql-exporter[<PID>]: encoding/json.(*Encoder).Encode(0xc420f49c80, 0x8886e0, 0xc4212005a0, 0x864180, 0xc420b27140)
Jan 12 23:16:26 <HOST> prometheus-sql-exporter[<PID>]:         /usr/local/go/src/encoding/json/stream.go:188 +0x386
Jan 12 23:16:26 <HOST> prometheus-sql-exporter[<PID>]: github.com/justwatchcom/sql_exporter/vendor/github.com/go-kit/kit/log.(*jsonLogger).Log(0xc420128700, 0xc4201083c0, 0xc, 0xc, 0xc420108300, 0x8)
Jan 12 23:16:26 <HOST> prometheus-sql-exporter[<PID>]:         /build/prometheus-sql-exporter/tmp/build/src/github.com/justwatchcom/sql_exporter/vendor/github.com/go-kit/kit/log/json_logger.go:34 +0x180
Jan 12 23:16:26 <HOST> prometheus-sql-exporter[<PID>]: github.com/justwatchcom/sql_exporter/vendor/github.com/go-kit/kit/log.(*context).Log(0xc4201347e0, 0xc420108300, 0x8, 0xc, 0xc4206ee0c0, 0xc420648c00)
Jan 12 23:16:26 <HOST> prometheus-sql-exporter[<PID>]:         /build/prometheus-sql-exporter/tmp/build/src/github.com/justwatchcom/sql_exporter/vendor/github.com/go-kit/kit/log/log.go:124 +0x1ed
Jan 12 23:16:26 <HOST> prometheus-sql-exporter[<PID>]: github.com/justwatchcom/sql_exporter/vendor/github.com/go-kit/kit/log/level.(*logger).Log(0xc42013a480, 0xc420108300, 0x8, 0xc, 0xc42102a480, 0x2)
Jan 12 23:16:26 <HOST> prometheus-sql-exporter[<PID>]:         /build/prometheus-sql-exporter/tmp/build/src/github.com/justwatchcom/sql_exporter/vendor/github.com/go-kit/kit/log/level/level.go:84 +0xa5
Jan 12 23:16:26 <HOST> prometheus-sql-exporter[<PID>]: github.com/justwatchcom/sql_exporter/vendor/github.com/go-kit/kit/log.(*context).Log(0xc421200570, 0xc42102a480, 0x2, 0x2, 0x0, 0x0)
Jan 12 23:16:26 <HOST> prometheus-sql-exporter[<PID>]:         /build/prometheus-sql-exporter/tmp/build/src/github.com/justwatchcom/sql_exporter/vendor/github.com/go-kit/kit/log/log.go:124 +0x1ed
Jan 12 23:16:26 <HOST> prometheus-sql-exporter[<PID>]: main.(*Job).runOnceConnection(0xc42013f800, 0xc420016690, 0xc4200ba1c0)
Jan 12 23:16:26 <HOST> prometheus-sql-exporter[<PID>]:         /build/prometheus-sql-exporter/tmp/build/src/github.com/justwatchcom/sql_exporter/job.go:151 +0x652
Jan 12 23:16:26 <HOST> prometheus-sql-exporter[<PID>]: created by main.(*Job).runOnce
Jan 12 23:16:26 <HOST> prometheus-sql-exporter[<PID>]:         /build/prometheus-sql-exporter/tmp/build/src/github.com/justwatchcom/sql_exporter/job.go:161 +0x96
Jan 12 23:16:26 <HOST> prometheus-sql-exporter[<PID>]: goroutine 20829273 [semacquire]:
Jan 12 23:16:26 <HOST> prometheus-sql-exporter[<PID>]: sync.runtime_Semacquire(0xc42124979c)
Jan 12 23:16:26 <HOST> prometheus-sql-exporter[<PID>]:         /usr/local/go/src/runtime/sema.go:47 +0x34
Jan 12 23:16:26 <HOST> prometheus-sql-exporter[<PID>]: sync.(*WaitGroup).Wait(0xc421249790)
Jan 12 23:16:26 <HOST> prometheus-sql-exporter[<PID>]:         /usr/local/go/src/sync/waitgroup.go:131 +0x7a
Jan 12 23:16:26 <HOST> prometheus-sql-exporter[<PID>]: github.com/justwatchcom/sql_exporter/vendor/github.com/prometheus/client_golang/prometheus.(*Registry).Gather.func1(0xc421249790, 0xc4210b63c0)
Jan 12 23:16:26 <HOST> prometheus-sql-exporter[<PID>]:         /build/prometheus-sql-exporter/tmp/build/src/github.com/justwatchcom/sql_exporter/vendor/github.com/prometheus/client_golang/prometheus/registry.go:376 +0x2b
Jan 12 23:16:26 <HOST> prometheus-sql-exporter[<PID>]: created by github.com/justwatchcom/sql_exporter/vendor/github.com/prometheus/client_golang/prometheus.(*Registry).Gather
Jan 12 23:16:26 <HOST> prometheus-sql-exporter[<PID>]:         /build/prometheus-sql-exporter/tmp/build/src/github.com/justwatchcom/sql_exporter/vendor/github.com/prometheus/client_golang/prometheus/registry.go:378 +0x244

Thanks!

dewey commented 2 years ago

I'm not sure if this is still relevant in the current version. I'm going to close this, but if this is still happening please feel free to open a new issue. I haven't observed that in production yet.