search

letsencrypt / ct-woodpecker

A tool to monitor a certificate transparency log for operational problems
Mozilla Public License 2.0
182 stars 19 forks source link

Add support for gathering pprof data #81

Closed pgporada closed 5 years ago

pgporada commented 5 years ago

This would help troubleshoot either a potential memory leak or if the config we're using is the result of the out of memory issues. I'm thinking it's the config. We received this on a server earlier today. Relevant details about the server are

1 vCPU
1.75GB RAM
{
  "metricsAddr": ":1971",
  "dbURI": "<redacted>",
  "fetchConfig": {
    "interval": "120s",
    "timeout": "100s"
  },
  "submitConfig": {
    "interval": "600s",
    "timeout": "500s",
    "certIssuerKeyPath": "<redacted>",
    "certIssuerPath": "<redacted>"
  },
  "inclusionConfig": {
    "interval": "60s",
    "fetchBatchSize": 1000,
    "maxGetEntries": 10000
  },
"logs": [<redacted>]
}

top reported that ct-woodpecker was consuming 60% of available RAM.

Jan 31 19:11:03 [localhost] ct-woodpecker: fatal error: runtime: out of memory
Jan 31 19:11:03 [localhost] ct-woodpecker: runtime stack:
Jan 31 19:11:03 [localhost] ct-woodpecker: runtime.throw(0xb65037, 0x16)
Jan 31 19:11:03 [localhost] ct-woodpecker: /usr/local/go/src/runtime/panic.go:608 +0x72
Jan 31 19:11:03 [localhost] ct-woodpecker: runtime.sysMap(0xc044000000, 0x4000000, 0x1307ff8)
Jan 31 19:11:03 [localhost] ct-woodpecker: /usr/local/go/src/runtime/mem_linux.go:156 +0xc7
Jan 31 19:11:03 [localhost] ct-woodpecker: runtime.(*mheap).sysAlloc(0x12ee120, 0x4000000, 0x76220, 0x759bb1)
Jan 31 19:11:03 [localhost] ct-woodpecker: /usr/local/go/src/runtime/malloc.go:619 +0x1c7
Jan 31 19:11:03 [localhost] ct-woodpecker: runtime.(*mheap).grow(0x12ee120, 0x1, 0x0)
Jan 31 19:11:03 [localhost] ct-woodpecker: /usr/local/go/src/runtime/mheap.go:920 +0x42
Jan 31 19:11:03 [localhost] ct-woodpecker: runtime.(*mheap).allocSpanLocked(0x12ee120, 0x1, 0x1308008, 0xe0003f540f)
Jan 31 19:11:03 [localhost] ct-woodpecker: /usr/local/go/src/runtime/mheap.go:848 +0x337
Jan 31 19:11:03 [localhost] ct-woodpecker: runtime.(*mheap).alloc_m(0x12ee120, 0x1, 0x3c, 0xc000558600)
Jan 31 19:11:03 [localhost] ct-woodpecker: /usr/local/go/src/runtime/mheap.go:692 +0x119
Jan 31 19:11:03 [localhost] ct-woodpecker: runtime.(*mheap).alloc.func1()
Jan 31 19:11:03 [localhost] ct-woodpecker: /usr/local/go/src/runtime/mheap.go:759 +0x4c
Jan 31 19:11:03 [localhost] ct-woodpecker: runtime.(*mheap).alloc(0x12ee120, 0x1, 0x101003c, 0x4638c1)
Jan 31 19:11:03 [localhost] ct-woodpecker: /usr/local/go/src/runtime/mheap.go:758 +0x8a
Jan 31 19:11:03 [localhost] ct-woodpecker: runtime.(*mcentral).grow(0x12f0358, 0x0)
Jan 31 19:11:03 [localhost] ct-woodpecker: /usr/local/go/src/runtime/mcentral.go:232 +0x94
Jan 31 19:11:03 [localhost] ct-woodpecker: runtime.(*mcentral).cacheSpan(0x12f0358, 0x7f3ecb87c350)
Jan 31 19:11:03 [localhost] ct-woodpecker: /usr/local/go/src/runtime/mcentral.go:106 +0x2f8
Jan 31 19:11:03 [localhost] ct-woodpecker: runtime.(*mcache).refill(0x7f3ef748a000, 0x7f3ecabaa03c)
Jan 31 19:11:03 [localhost] ct-woodpecker: /usr/local/go/src/runtime/mcache.go:122 +0x95
Jan 31 19:11:03 [localhost] ct-woodpecker: runtime.(*mcache).nextFree.func1()
Jan 31 19:11:03 [localhost] ct-woodpecker: /usr/local/go/src/runtime/malloc.go:749 +0x32
Jan 31 19:11:03 [localhost] ct-woodpecker: runtime.systemstack(0x0)
Jan 31 19:11:03 [localhost] ct-woodpecker: /usr/local/go/src/runtime/asm_amd64.s:351 +0x66
Jan 31 19:11:03 [localhost] ct-woodpecker: runtime.mstart()
Jan 31 19:11:03 [localhost] ct-woodpecker: /usr/local/go/src/runtime/proc.go:1229
Jan 31 19:11:03 [localhost] ct-woodpecker: goroutine 44 [running]:
Jan 31 19:11:03 [localhost] ct-woodpecker: runtime.systemstack_switch()
Jan 31 19:11:03 [localhost] ct-woodpecker: /usr/local/go/src/runtime/asm_amd64.s:311 fp=0xc0092d53b8 sp=0xc0092d53b0 pc=0x4617e0
Jan 31 19:11:03 [localhost] ct-woodpecker: runtime.(*mcache).nextFree(0x7f3ef748a000, 0x3c, 0x0, 0x0, 0x197)
Jan 31 19:11:03 [localhost] ct-woodpecker: /usr/local/go/src/runtime/malloc.go:748 +0xb6 fp=0xc0092d5410 sp=0xc0092d53b8 pc=0x4166f6
Jan 31 19:11:03 [localhost] ct-woodpecker: runtime.mallocgc(0x380, 0xad54c0, 0xa2d301, 0xc043ffc820)
Jan 31 19:11:03 [localhost] ct-woodpecker: /usr/local/go/src/runtime/malloc.go:903 +0x793 fp=0xc0092d54b0 sp=0xc0092d5410 pc=0x417043
Jan 31 19:11:03 [localhost] ct-woodpecker: runtime.growslice(0xad54c0, 0xc043f83340, 0x8, 0x8, 0x9, 0x2, 0x0, 0x0)
Jan 31 19:11:03 [localhost] ct-woodpecker: /usr/local/go/src/runtime/slice.go:204 +0x145 fp=0xc0092d5518 sp=0xc0092d54b0 pc=0x44c8e5
Jan 31 19:11:03 [localhost] systemd: ct-woodpecker.service: main process exited, code=exited, status=2/INVALIDARGUMENT
Jan 31 19:11:03 [localhost] ct-woodpecker: github.com/letsencrypt/ct-woodpecker/vendor/github.com/google/certificate-transparency-go/x509.parseCertificate(0xc043e82900, 0x541, 0x541, 0xa32120)
Jan 31 19:11:03 [localhost] ct-woodpecker: /go/src/github.com/letsencrypt/ct-woodpecker/vendor/github.com/google/certificate-transparency-go/x509/x509.go:1723 +0x3005 fp=0xc0092d58e8 sp=0xc0092d5518 pc=0x751b45
Jan 31 19:11:03 [localhost] ct-woodpecker: github.com/letsencrypt/ct-woodpecker/vendor/github.com/google/certificate-transparency-go/x509.ParseCertificate(0xc043edcb00, 0x541, 0x541, 0x60, 0xb09280, 0x1)
Jan 31 19:11:03 [localhost] ct-woodpecker: /go/src/github.com/letsencrypt/ct-woodpecker/vendor/github.com/google/certificate-transparency-go/x509/x509.go:1955 +0x126 fp=0xc0092d5970 sp=0xc0092d58e8 pc=0x752156
Jan 31 19:11:03 [localhost] ct-woodpecker: github.com/letsencrypt/ct-woodpecker/vendor/github.com/google/certificate-transparency-go.(*MerkleTreeLeaf).X509Certificate(0xc043fdd188, 0xc043c3b4a0, 0xc0421bcf00, 0x4b5)
Jan 31 19:11:03 [localhost] ct-woodpecker: /go/src/github.com/letsencrypt/ct-woodpecker/vendor/github.com/google/certificate-transparency-go/types.go:380 +0xa4 fp=0xc0092d59b8 sp=0xc0092d5970 pc=0x75ba54
Jan 31 19:11:03 [localhost] ct-woodpecker: github.com/letsencrypt/ct-woodpecker/vendor/github.com/google/certificate-transparency-go.(*RawLogEntry).ToLogEntry(0xc043fdd180, 0xc0092d5b78, 0xc043fdd180, 0x0)
Jan 31 19:11:03 [localhost] ct-woodpecker: /go/src/github.com/letsencrypt/ct-woodpecker/vendor/github.com/google/certificate-transparency-go/serialization.go:302 +0x371 fp=0xc0092d5ab0 sp=0xc0092d59b8 pc=0x759bb1
Jan 31 19:11:03 [localhost] ct-woodpecker: github.com/letsencrypt/ct-woodpecker/vendor/github.com/google/certificate-transparency-go.LogEntryFromLeaf(0x2ddb5a6, 0xc0092d5b78, 0xc043c3b400, 0x0, 0x0)
Jan 31 19:11:03 [localhost] ct-woodpecker: /go/src/github.com/letsencrypt/ct-woodpecker/vendor/github.com/google/certificate-transparency-go/serialization.go:338 +0x73 fp=0xc0092d5ae8 sp=0xc0092d5ab0 pc=0x759d73
Jan 31 19:11:03 [localhost] ct-woodpecker: github.com/letsencrypt/ct-woodpecker/vendor/github.com/google/certificate-transparency-go/client.(*LogClient).GetEntries(0xc000031080, 0xc09060, 0xc000034028, 0x2ddb420, 0x2ddb808, 0x3e8, 0x3e8, 0x3e8, 0x0, 0x0)
Jan 31 19:11:03 [localhost] ct-woodpecker: /go/src/github.com/letsencrypt/ct-woodpecker/vendor/github.com/google/certificate-transparency-go/client/getentries.go:68 +0x19a fp=0xc0092d5be8 sp=0xc0092d5ae8 pc=0x7fed8a
Jan 31 19:11:03 [localhost] systemd: Unit ct-woodpecker.service entered failed state.
Jan 31 19:11:03 [localhost] ct-woodpecker: github.com/letsencrypt/ct-woodpecker/monitor.(*inclusionChecker).getEntries(0xc000149180, 0x2ddb420, 0x32c56cd, 0xc025104c80, 0x0, 0x0, 0xc099e0, 0x1305f20, 0xc00008f180)
Jan 31 19:11:03 [localhost] ct-woodpecker: /go/src/github.com/letsencrypt/ct-woodpecker/monitor/inclusion_checker.go:208 +0x24d fp=0xc0092d5cd0 sp=0xc0092d5be8 pc=0x9270dd
Jan 31 19:11:03 [localhost] ct-woodpecker: github.com/letsencrypt/ct-woodpecker/monitor.(*inclusionChecker).checkInclusion(0xc000149180, 0xc0092d5f58, 0x2)
Jan 31 19:11:03 [localhost] ct-woodpecker: /go/src/github.com/letsencrypt/ct-woodpecker/monitor/inclusion_checker.go:171 +0x61a fp=0xc0092d5ee0 sp=0xc0092d5cd0 pc=0x9263da
Jan 31 19:11:03 [localhost] ct-woodpecker: github.com/letsencrypt/ct-woodpecker/monitor.(*inclusionChecker).run.func1(0xc000149180)
Jan 31 19:11:03 [localhost] ct-woodpecker: /go/src/github.com/letsencrypt/ct-woodpecker/monitor/inclusion_checker.go:116 +0x109 fp=0xc0092d5fd8 sp=0xc0092d5ee0 pc=0x92af89
Jan 31 19:11:03 [localhost] ct-woodpecker: runtime.goexit()
Jan 31 19:11:03 [localhost] ct-woodpecker: /usr/local/go/src/runtime/asm_amd64.s:1333 +0x1 fp=0xc0092d5fe0 sp=0xc0092d5fd8 pc=0x4638c1
Jan 31 19:11:03 [localhost] ct-woodpecker: created by github.com/letsencrypt/ct-woodpecker/monitor.(*inclusionChecker).run
Jan 31 19:11:03 [localhost] ct-woodpecker: /go/src/github.com/letsencrypt/ct-woodpecker/monitor/inclusion_checker.go:109 +0x3f
Jan 31 19:11:03 [localhost] ct-woodpecker: goroutine 1 [chan receive, 9 minutes]:
Jan 31 19:11:03 [localhost] ct-woodpecker: github.com/letsencrypt/ct-woodpecker/cmd.WaitForSignal(0xc00008f180, 0xc000184f58)
Jan 31 19:11:03 [localhost] ct-woodpecker: /go/src/github.com/letsencrypt/ct-woodpecker/cmd/common.go:21 +0x15a
Jan 31 19:11:03 [localhost] ct-woodpecker: main.main()
Jan 31 19:11:03 [localhost] ct-woodpecker: /go/src/github.com/letsencrypt/ct-woodpecker/cmd/ct-woodpecker/main.go:51 +0x1ed
Jan 31 19:11:03 [localhost] ct-woodpecker: goroutine 5 [syscall, 9 minutes]:
Jan 31 19:11:03 [localhost] ct-woodpecker: os/signal.signal_recv(0x0)
Jan 31 19:11:03 [localhost] ct-woodpecker: /usr/local/go/src/runtime/sigqueue.go:139 +0x9c
Jan 31 19:11:03 [localhost] ct-woodpecker: os/signal.loop()
Jan 31 19:11:03 [localhost] ct-woodpecker: /usr/local/go/src/os/signal/signal_unix.go:23 +0x22
Jan 31 19:11:03 [localhost] ct-woodpecker: created by os/signal.init.0
Jan 31 19:11:03 [localhost] ct-woodpecker: /usr/local/go/src/os/signal/signal_unix.go:29 +0x41
Jan 31 19:11:03 [localhost] ct-woodpecker: goroutine 8 [chan receive]:
Jan 31 19:11:03 [localhost] ct-woodpecker: github.com/letsencrypt/ct-woodpecker/vendor/github.com/golang/glog.(*loggingT).flushDaemon(0x12e7fe0)
Jan 31 19:11:03 [localhost] ct-woodpecker: /go/src/github.com/letsencrypt/ct-woodpecker/vendor/github.com/golang/glog/glog.go:882 +0x8b
Jan 31 19:11:03 [localhost] ct-woodpecker: created by github.com/letsencrypt/ct-woodpecker/vendor/github.com/golang/glog.init.0
Jan 31 19:11:03 [localhost] ct-woodpecker: /go/src/github.com/letsencrypt/ct-woodpecker/vendor/github.com/golang/glog/glog.go:410 +0x203
Jan 31 19:11:03 [localhost] ct-woodpecker: goroutine 23 [select, 9 minutes]:
Jan 31 19:11:03 [localhost] systemd: ct-woodpecker.service failed.
Jan 31 19:11:03 [localhost] ct-woodpecker: database/sql.(*DB).connectionOpener(0xc00010c180, 0xc09020, 0xc0000310c0)
Jan 31 19:11:03 [localhost] ct-woodpecker: /usr/local/go/src/database/sql/sql.go:1001 +0xe8
Jan 31 19:11:03 [localhost] ct-woodpecker: created by database/sql.OpenDB
Jan 31 19:11:03 [localhost] ct-woodpecker: /usr/local/go/src/database/sql/sql.go:671 +0x15d
Jan 31 19:11:03 [localhost] ct-woodpecker: goroutine 24 [select, 9 minutes]:
Jan 31 19:11:03 [localhost] ct-woodpecker: database/sql.(*DB).connectionResetter(0xc00010c180, 0xc09020, 0xc0000310c0)
Jan 31 19:11:03 [localhost] ct-woodpecker: /usr/local/go/src/database/sql/sql.go:1014 +0xfb
Jan 31 19:11:03 [localhost] ct-woodpecker: created by database/sql.OpenDB
Jan 31 19:11:03 [localhost] ct-woodpecker: /usr/local/go/src/database/sql/sql.go:672 +0x193
Jan 31 19:11:03 [localhost] ct-woodpecker: goroutine 25 [select, 9 minutes]:
Jan 31 19:11:03 [localhost] ct-woodpecker: database/sql.(*DB).connectionOpener(0xc00010c300, 0xc09020, 0xc000031400)
Jan 31 19:11:03 [localhost] ct-woodpecker: /usr/local/go/src/database/sql/sql.go:1001 +0xe8
Jan 31 19:11:03 [localhost] ct-woodpecker: created by database/sql.OpenDB
Jan 31 19:11:03 [localhost] ct-woodpecker: /usr/local/go/src/database/sql/sql.go:671 +0x15d
Jan 31 19:11:03 [localhost] ct-woodpecker: goroutine 26 [select, 9 minutes]:
Jan 31 19:11:03 [localhost] ct-woodpecker: database/sql.(*DB).connectionResetter(0xc00010c300, 0xc09020, 0xc000031400)
Jan 31 19:11:03 [localhost] ct-woodpecker: /usr/local/go/src/database/sql/sql.go:1014 +0xfb
Jan 31 19:11:03 [localhost] ct-woodpecker: created by database/sql.OpenDB
Jan 31 19:11:03 [localhost] ct-woodpecker: /usr/local/go/src/database/sql/sql.go:672 +0x193
Jan 31 19:11:03 [localhost] ct-woodpecker: goroutine 27 [select, 9 minutes]:
Jan 31 19:11:03 [localhost] ct-woodpecker: database/sql.(*DB).connectionOpener(0xc00010c480, 0xc09020, 0xc000031700)
Jan 31 19:11:03 [localhost] ct-woodpecker: /usr/local/go/src/database/sql/sql.go:1001 +0xe8
Jan 31 19:11:03 [localhost] ct-woodpecker: created by database/sql.OpenDB
Jan 31 19:11:03 [localhost] ct-woodpecker: /usr/local/go/src/database/sql/sql.go:671 +0x15d
Jan 31 19:11:03 [localhost] ct-woodpecker: goroutine 28 [select, 9 minutes]:
Jan 31 19:11:03 [localhost] ct-woodpecker: database/sql.(*DB).connectionResetter(0xc00010c480, 0xc09020, 0xc000031700)
Jan 31 19:11:03 [localhost] ct-woodpecker: /usr/local/go/src/database/sql/sql.go:1014 +0xfb
Jan 31 19:11:03 [localhost] ct-woodpecker: created by database/sql.OpenDB
Jan 31 19:11:03 [localhost] ct-woodpecker: /usr/local/go/src/database/sql/sql.go:672 +0x193
Jan 31 19:11:03 [localhost] ct-woodpecker: goroutine 29 [select, 9 minutes]:
Jan 31 19:11:03 [localhost] ct-woodpecker: database/sql.(*DB).connectionOpener(0xc00010c600, 0xc09020, 0xc000031a40)
Jan 31 19:11:03 [localhost] ct-woodpecker: /usr/local/go/src/database/sql/sql.go:1001 +0xe8
Jan 31 19:11:03 [localhost] ct-woodpecker: created by database/sql.OpenDB
Jan 31 19:11:03 [localhost] ct-woodpecker: /usr/local/go/src/database/sql/sql.go:671 +0x15d
Jan 31 19:11:03 [localhost] ct-woodpecker: goroutine 30 [select, 9 minutes]:
Jan 31 19:11:03 [localhost] ct-woodpecker: database/sql.(*DB).connectionResetter(0xc00010c600, 0xc09020, 0xc000031a40)
Jan 31 19:11:03 [localhost] ct-woodpecker: /usr/local/go/src/database/sql/sql.go:1014 +0xfb
Jan 31 19:11:03 [localhost] ct-woodpecker: created by database/sql.OpenDB
Jan 31 19:11:03 [localhost] ct-woodpecker: /usr/local/go/src/database/sql/sql.go:672 +0x193
Jan 31 19:11:03 [localhost] ct-woodpecker: goroutine 31 [select, 9 minutes]:
Jan 31 19:11:03 [localhost] ct-woodpecker: database/sql.(*DB).connectionOpener(0xc00010c780, 0xc09020, 0xc000031d40)
Jan 31 19:11:03 [localhost] ct-woodpecker: /usr/local/go/src/database/sql/sql.go:1001 +0xe8
Jan 31 19:11:03 [localhost] ct-woodpecker: created by database/sql.OpenDB
Jan 31 19:11:03 [localhost] ct-woodpecker: /usr/local/go/src/database/sql/sql.go:671 +0x15d
Jan 31 19:11:03 [localhost] ct-woodpecker: goroutine 32 [select, 9 minutes]:
Jan 31 19:11:03 [localhost] ct-woodpecker: database/sql.(*DB).connectionResetter(0xc00010c780, 0xc09020, 0xc000031d40)
Jan 31 19:11:03 [localhost] ct-woodpecker: /usr/local/go/src/database/sql/sql.go:1014 +0xfb
Jan 31 19:11:03 [localhost] ct-woodpecker: created by database/sql.OpenDB
Jan 31 19:11:03 [localhost] ct-woodpecker: /usr/local/go/src/database/sql/sql.go:672 +0x193
Jan 31 19:11:03 [localhost] ct-woodpecker: goroutine 33 [select, 9 minutes]:
Jan 31 19:11:03 [localhost] ct-woodpecker: database/sql.(*DB).connectionOpener(0xc00010c900, 0xc09020, 0xc00029a080)
Jan 31 19:11:03 [localhost] ct-woodpecker: /usr/local/go/src/database/sql/sql.go:1001 +0xe8
Jan 31 19:11:03 [localhost] ct-woodpecker: created by database/sql.OpenDB
Jan 31 19:11:03 [localhost] ct-woodpecker: /usr/local/go/src/database/sql/sql.go:671 +0x15d
Jan 31 19:11:03 [localhost] ct-woodpecker: goroutine 34 [select, 9 minutes]:
Jan 31 19:11:03 [localhost] ct-woodpecker: database/sql.(*DB).connectionResetter(0xc00010c900, 0xc09020, 0xc00029a080)
Jan 31 19:11:03 [localhost] ct-woodpecker: /usr/local/go/src/database/sql/sql.go:1014 +0xfb
Jan 31 19:11:03 [localhost] ct-woodpecker: created by database/sql.OpenDB
Jan 31 19:11:03 [localhost] ct-woodpecker: /usr/local/go/src/database/sql/sql.go:672 +0x193
Jan 31 19:11:03 [localhost] ct-woodpecker: goroutine 35 [select, 9 minutes]:
Jan 31 19:11:03 [localhost] ct-woodpecker: database/sql.(*DB).connectionOpener(0xc00010ca80, 0xc09020, 0xc00029a380)
Jan 31 19:11:03 [localhost] ct-woodpecker: /usr/local/go/src/database/sql/sql.go:1001 +0xe8
Jan 31 19:11:03 [localhost] ct-woodpecker: created by database/sql.OpenDB
Jan 31 19:11:03 [localhost] ct-woodpecker: /usr/local/go/src/database/sql/sql.go:671 +0x15d
Jan 31 19:11:03 [localhost] ct-woodpecker: goroutine 36 [select, 9 minutes]:
Jan 31 19:11:03 [localhost] ct-woodpecker: database/sql.(*DB).connectionResetter(0xc00010ca80, 0xc09020, 0xc00029a380)
Jan 31 19:11:03 [localhost] ct-woodpecker: /usr/local/go/src/database/sql/sql.go:1014 +0xfb
Jan 31 19:11:03 [localhost] ct-woodpecker: created by database/sql.OpenDB
Jan 31 19:11:03 [localhost] ct-woodpecker: /usr/local/go/src/database/sql/sql.go:672 +0x193
Jan 31 19:11:03 [localhost] ct-woodpecker: goroutine 37 [select, 9 minutes]:
Jan 31 19:11:03 [localhost] ct-woodpecker: database/sql.(*DB).connectionOpener(0xc00010cc00, 0xc09020, 0xc00029a680)
Jan 31 19:11:03 [localhost] ct-woodpecker: /usr/local/go/src/database/sql/sql.go:1001 +0xe8
Jan 31 19:11:03 [localhost] ct-woodpecker: created by database/sql.OpenDB
Jan 31 19:11:03 [localhost] ct-woodpecker: /usr/local/go/src/database/sql/sql.go:671 +0x15d
Jan 31 19:11:03 [localhost] ct-woodpecker: goroutine 38 [select, 9 minutes]:
Jan 31 19:11:03 [localhost] ct-woodpecker: database/sql.(*DB).connectionResetter(0xc00010cc00, 0xc09020, 0xc00029a680)
Jan 31 19:11:03 [localhost] ct-woodpecker: /usr/local/go/src/database/sql/sql.go:1014 +0xfb
Jan 31 19:11:03 [localhost] ct-woodpecker: created by database/sql.OpenDB
Jan 31 19:11:03 [localhost] ct-woodpecker: /usr/local/go/src/database/sql/sql.go:672 +0x193
Jan 31 19:11:03 [localhost] ct-woodpecker: goroutine 41 [IO wait]:
Jan 31 19:11:03 [localhost] ct-woodpecker: internal/poll.runtime_pollWait(0x7f3ef742ef00, 0x72, 0x0)
Jan 31 19:11:03 [localhost] ct-woodpecker: /usr/local/go/src/runtime/netpoll.go:173 +0x66
Jan 31 19:11:03 [localhost] ct-woodpecker: internal/poll.(*pollDesc).wait(0xc00010b618, 0x72, 0xc000030100, 0x0, 0x0)
Jan 31 19:11:03 [localhost] ct-woodpecker: /usr/local/go/src/internal/poll/fd_poll_runtime.go:85 +0x9a
Jan 31 19:11:03 [localhost] ct-woodpecker: internal/poll.(*pollDesc).waitRead(0xc00010b618, 0xffffffffffffff00, 0x0, 0x0)
Jan 31 19:11:03 [localhost] ct-woodpecker: /usr/local/go/src/internal/poll/fd_poll_runtime.go:90 +0x3d
Jan 31 19:11:03 [localhost] ct-woodpecker: internal/poll.(*FD).Accept(0xc00010b600, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
Jan 31 19:11:03 [localhost] ct-woodpecker: /usr/local/go/src/internal/poll/fd_unix.go:384 +0x1a0
Jan 31 19:11:03 [localhost] ct-woodpecker: net.(*netFD).accept(0xc00010b600, 0x416b9f, 0xc00eb900a0, 0xa0)
Jan 31 19:11:03 [localhost] ct-woodpecker: /usr/local/go/src/net/fd_unix.go:238 +0x42
Jan 31 19:11:03 [localhost] ct-woodpecker: net.(*TCPListener).accept(0xc00000e4a0, 0xc00005fe00, 0xee85a2a9, 0x2383f89cbcc220f4)
Jan 31 19:11:03 [localhost] ct-woodpecker: /usr/local/go/src/net/tcpsock_posix.go:139 +0x2e
Jan 31 19:11:03 [localhost] ct-woodpecker: net.(*TCPListener).AcceptTCP(0xc00000e4a0, 0xc00005fe28, 0x4933b6, 0x5c534838)
Jan 31 19:11:03 [localhost] ct-woodpecker: /usr/local/go/src/net/tcpsock.go:247 +0x47
Jan 31 19:11:03 [localhost] ct-woodpecker: net/http.tcpKeepAliveListener.Accept(0xc00000e4a0, 0xc00005fe78, 0x18, 0xc0001eb800, 0x6bd135)
Jan 31 19:11:03 [localhost] ct-woodpecker: /usr/local/go/src/net/http/server.go:3232 +0x2f
Jan 31 19:11:03 [localhost] ct-woodpecker: net/http.(*Server).Serve(0xc0001615f0, 0xc08ba0, 0xc00000e4a0, 0x0, 0x0)
Jan 31 19:11:03 [localhost] ct-woodpecker: /usr/local/go/src/net/http/server.go:2826 +0x22f
Jan 31 19:11:03 [localhost] ct-woodpecker: net/http.(*Server).ListenAndServe(0xc0001615f0, 0x0, 0xc0002a2f90)
Jan 31 19:11:03 [localhost] ct-woodpecker: /usr/local/go/src/net/http/server.go:2764 +0xb6
Jan 31 19:11:03 [localhost] ct-woodpecker: github.com/letsencrypt/ct-woodpecker/woodpecker.(*Woodpecker).Run.func1(0xc0001f32c0)
Jan 31 19:11:03 [localhost] ct-woodpecker: /go/src/github.com/letsencrypt/ct-woodpecker/woodpecker/woodpecker.go:396 +0x33
Jan 31 19:11:03 [localhost] ct-woodpecker: created by github.com/letsencrypt/ct-woodpecker/woodpecker.(*Woodpecker).Run
Jan 31 19:11:03 [localhost] ct-woodpecker: /go/src/github.com/letsencrypt/ct-woodpecker/woodpecker/woodpecker.go:395 +0x3f
Jan 31 19:11:03 [localhost] ct-woodpecker: goroutine 42 [select, 1 minutes]:
Jan 31 19:11:03 [localhost] ct-woodpecker: github.com/letsencrypt/ct-woodpecker/monitor.(*sthFetcher).run.func1(0xc000148fa0)
Jan 31 19:11:03 [localhost] ct-woodpecker: /go/src/github.com/letsencrypt/ct-woodpecker/monitor/sth_fetcher.go:136 +0x192
Jan 31 19:11:03 [localhost] ct-woodpecker: created by github.com/letsencrypt/ct-woodpecker/monitor.(*sthFetcher).run
Jan 31 19:11:03 [localhost] ct-woodpecker: /go/src/github.com/letsencrypt/ct-woodpecker/monitor/sth_fetcher.go:132 +0x3f
Jan 31 19:11:03 [localhost] ct-woodpecker: goroutine 43 [select, 9 minutes]:
Jan 31 19:11:03 [localhost] ct-woodpecker: github.com/letsencrypt/ct-woodpecker/monitor.(*certSubmitter).run.func1(0xc00010c240)
Jan 31 19:11:03 [localhost] ct-woodpecker: /go/src/github.com/letsencrypt/ct-woodpecker/monitor/cert_submitter.go:177 +0x184
Jan 31 19:11:03 [localhost] ct-woodpecker: created by github.com/letsencrypt/ct-woodpecker/monitor.(*certSubmitter).run
Jan 31 19:11:03 [localhost] ct-woodpecker: /go/src/github.com/letsencrypt/ct-woodpecker/monitor/cert_submitter.go:172 +0x3f
Jan 31 19:11:03 [localhost] ct-woodpecker: goroutine 45 [select, 1 minutes]:
Jan 31 19:11:03 [localhost] ct-woodpecker: github.com/letsencrypt/ct-woodpecker/monitor.(*sthFetcher).run.func1(0xc0001492c0)
Jan 31 19:11:03 [localhost] ct-woodpecker: /go/src/github.com/letsencrypt/ct-woodpecker/monitor/sth_fetcher.go:136 +0x192
Jan 31 19:11:03 [localhost] ct-woodpecker: created by github.com/letsencrypt/ct-woodpecker/monitor.(*sthFetcher).run
Jan 31 19:11:03 [localhost] ct-woodpecker: /go/src/github.com/letsencrypt/ct-woodpecker/monitor/sth_fetcher.go:132 +0x3f
Jan 31 19:11:03 [localhost] ct-woodpecker: goroutine 46 [select, 9 minutes]:
Jan 31 19:11:03 [localhost] ct-woodpecker: github.com/letsencrypt/ct-woodpecker/monitor.(*certSubmitter).run.func1(0xc00010c3c0)
Jan 31 19:11:03 [localhost] ct-woodpecker: /go/src/github.com/letsencrypt/ct-woodpecker/monitor/cert_submitter.go:177 +0x184
Jan 31 19:11:03 [localhost] ct-woodpecker: created by github.com/letsencrypt/ct-woodpecker/monitor.(*certSubmitter).run
Jan 31 19:11:03 [localhost] ct-woodpecker: /go/src/github.com/letsencrypt/ct-woodpecker/monitor/cert_submitter.go:172 +0x3f
Jan 31 19:11:03 [localhost] ct-woodpecker: goroutine 47 [select]:
Jan 31 19:11:03 [localhost] ct-woodpecker: github.com/letsencrypt/ct-woodpecker/monitor.(*inclusionChecker).run.func1(0xc0001494a0)
Jan 31 19:11:03 [localhost] ct-woodpecker: /go/src/github.com/letsencrypt/ct-woodpecker/monitor/inclusion_checker.go:112 +0xe0
Jan 31 19:11:03 [localhost] ct-woodpecker: created by github.com/letsencrypt/ct-woodpecker/monitor.(*inclusionChecker).run
Jan 31 19:11:03 [localhost] ct-woodpecker: /go/src/github.com/letsencrypt/ct-woodpecker/monitor/inclusion_checker.go:109 +0x3f
Jan 31 19:11:03 [localhost] ct-woodpecker: goroutine 48 [select, 1 minutes]:
Jan 31 19:11:03 [localhost] ct-woodpecker: github.com/letsencrypt/ct-woodpecker/monitor.(*sthFetcher).run.func1(0xc0001495e0)
Jan 31 19:11:03 [localhost] ct-woodpecker: /go/src/github.com/letsencrypt/ct-woodpecker/monitor/sth_fetcher.go:136 +0x192
Jan 31 19:11:03 [localhost] ct-woodpecker: created by github.com/letsencrypt/ct-woodpecker/monitor.(*sthFetcher).run
Jan 31 19:11:03 [localhost] ct-woodpecker: /go/src/github.com/letsencrypt/ct-woodpecker/monitor/sth_fetcher.go:132 +0x3f
Jan 31 19:11:03 [localhost] ct-woodpecker: goroutine 49 [select, 9 minutes]:
Jan 31 19:11:03 [localhost] ct-woodpecker: github.com/letsencrypt/ct-woodpecker/monitor.(*certSubmitter).run.func1(0xc00010c540)
Jan 31 19:11:03 [localhost] ct-woodpecker: /go/src/github.com/letsencrypt/ct-woodpecker/monitor/cert_submitter.go:177 +0x184
Jan 31 19:11:03 [localhost] ct-woodpecker: created by github.com/letsencrypt/ct-woodpecker/monitor.(*certSubmitter).run
Jan 31 19:11:03 [localhost] ct-woodpecker: /go/src/github.com/letsencrypt/ct-woodpecker/monitor/cert_submitter.go:172 +0x3f
Jan 31 19:11:03 [localhost] ct-woodpecker: goroutine 50 [select]:
Jan 31 19:11:03 [localhost] ct-woodpecker: github.com/letsencrypt/ct-woodpecker/monitor.(*inclusionChecker).run.func1(0xc0001497c0)
Jan 31 19:11:03 [localhost] ct-woodpecker: /go/src/github.com/letsencrypt/ct-woodpecker/monitor/inclusion_checker.go:112 +0xe0
Jan 31 19:11:03 [localhost] ct-woodpecker: created by github.com/letsencrypt/ct-woodpecker/monitor.(*inclusionChecker).run
Jan 31 19:11:03 [localhost] ct-woodpecker: /go/src/github.com/letsencrypt/ct-woodpecker/monitor/inclusion_checker.go:109 +0x3f
Jan 31 19:11:03 [localhost] ct-woodpecker: goroutine 51 [select, 1 minutes]:
Jan 31 19:11:03 [localhost] ct-woodpecker: github.com/letsencrypt/ct-woodpecker/monitor.(*sthFetcher).run.func1(0xc000149900)
Jan 31 19:11:03 [localhost] ct-woodpecker: /go/src/github.com/letsencrypt/ct-woodpecker/monitor/sth_fetcher.go:136 +0x192
Jan 31 19:11:03 [localhost] ct-woodpecker: created by github.com/letsencrypt/ct-woodpecker/monitor.(*sthFetcher).run
Jan 31 19:11:03 [localhost] ct-woodpecker: /go/src/github.com/letsencrypt/ct-woodpecker/monitor/sth_fetcher.go:132 +0x3f
Jan 31 19:11:03 [localhost] ct-woodpecker: goroutine 52 [select, 9 minutes]:
Jan 31 19:11:03 [localhost] ct-woodpecker: github.com/letsencrypt/ct-woodpecker/monitor.(*certSubmitter).run.func1(0xc00010c6c0)
Jan 31 19:11:03 [localhost] ct-woodpecker: /go/src/github.com/letsencrypt/ct-woodpecker/monitor/cert_submitter.go:177 +0x184
Jan 31 19:11:03 [localhost] ct-woodpecker: created by github.com/letsencrypt/ct-woodpecker/monitor.(*certSubmitter).run
Jan 31 19:11:03 [localhost] ct-woodpecker: /go/src/github.com/letsencrypt/ct-woodpecker/monitor/cert_submitter.go:172 +0x3f
Jan 31 19:11:03 [localhost] ct-woodpecker: goroutine 53 [select]:
Jan 31 19:11:03 [localhost] ct-woodpecker: github.com/letsencrypt/ct-woodpecker/monitor.(*inclusionChecker).run.func1(0xc000149ae0)
Jan 31 19:11:03 [localhost] ct-woodpecker: /go/src/github.com/letsencrypt/ct-woodpecker/monitor/inclusion_checker.go:112 +0xe0
Jan 31 19:11:03 [localhost] ct-woodpecker: created by github.com/letsencrypt/ct-woodpecker/monitor.(*inclusionChecker).run
Jan 31 19:11:03 [localhost] ct-woodpecker: /go/src/github.com/letsencrypt/ct-woodpecker/monitor/inclusion_checker.go:109 +0x3f
Jan 31 19:11:03 [localhost] ct-woodpecker: goroutine 54 [select, 1 minutes]:
Jan 31 19:11:03 [localhost] ct-woodpecker: github.com/letsencrypt/ct-woodpecker/monitor.(*sthFetcher).run.func1(0xc000149c20)
Jan 31 19:11:03 [localhost] ct-woodpecker: /go/src/github.com/letsencrypt/ct-woodpecker/monitor/sth_fetcher.go:136 +0x192
Jan 31 19:11:03 [localhost] ct-woodpecker: created by github.com/letsencrypt/ct-woodpecker/monitor.(*sthFetcher).run
Jan 31 19:11:03 [localhost] ct-woodpecker: /go/src/github.com/letsencrypt/ct-woodpecker/monitor/sth_fetcher.go:132 +0x3f
Jan 31 19:11:03 [localhost] ct-woodpecker: goroutine 55 [select, 9 minutes]:
Jan 31 19:11:03 [localhost] ct-woodpecker: github.com/letsencrypt/ct-woodpecker/monitor.(*certSubmitter).run.func1(0xc00010c840)
Jan 31 19:11:03 [localhost] ct-woodpecker: /go/src/github.com/letsencrypt/ct-woodpecker/monitor/cert_submitter.go:177 +0x184
Jan 31 19:11:03 [localhost] ct-woodpecker: created by github.com/letsencrypt/ct-woodpecker/monitor.(*certSubmitter).run
Jan 31 19:11:03 [localhost] ct-woodpecker: /go/src/github.com/letsencrypt/ct-woodpecker/monitor/cert_submitter.go:172 +0x3f
Jan 31 19:11:03 [localhost] ct-woodpecker: goroutine 56 [select]:
Jan 31 19:11:03 [localhost] ct-woodpecker: github.com/letsencrypt/ct-woodpecker/monitor.(*inclusionChecker).run.func1(0xc000149e00)
Jan 31 19:11:03 [localhost] ct-woodpecker: /go/src/github.com/letsencrypt/ct-woodpecker/monitor/inclusion_checker.go:112 +0xe0
Jan 31 19:11:03 [localhost] ct-woodpecker: created by github.com/letsencrypt/ct-woodpecker/monitor.(*inclusionChecker).run
Jan 31 19:11:03 [localhost] ct-woodpecker: /go/src/github.com/letsencrypt/ct-woodpecker/monitor/inclusion_checker.go:109 +0x3f
Jan 31 19:11:03 [localhost] ct-woodpecker: goroutine 57 [select, 1 minutes]:
Jan 31 19:11:03 [localhost] ct-woodpecker: github.com/letsencrypt/ct-woodpecker/monitor.(*sthFetcher).run.func1(0xc000149f40)
Jan 31 19:11:03 [localhost] ct-woodpecker: /go/src/github.com/letsencrypt/ct-woodpecker/monitor/sth_fetcher.go:136 +0x192
Jan 31 19:11:03 [localhost] ct-woodpecker: created by github.com/letsencrypt/ct-woodpecker/monitor.(*sthFetcher).run
Jan 31 19:11:03 [localhost] ct-woodpecker: /go/src/github.com/letsencrypt/ct-woodpecker/monitor/sth_fetcher.go:132 +0x3f
Jan 31 19:11:03 [localhost] ct-woodpecker: goroutine 58 [select, 9 minutes]:
Jan 31 19:11:03 [localhost] ct-woodpecker: github.com/letsencrypt/ct-woodpecker/monitor.(*certSubmitter).run.func1(0xc00010c9c0)
Jan 31 19:11:03 [localhost] ct-woodpecker: /go/src/github.com/letsencrypt/ct-woodpecker/monitor/cert_submitter.go:177 +0x184
Jan 31 19:11:03 [localhost] ct-woodpecker: created by github.com/letsencrypt/ct-woodpecker/monitor.(*certSubmitter).run
Jan 31 19:11:03 [localhost] ct-woodpecker: /go/src/github.com/letsencrypt/ct-woodpecker/monitor/cert_submitter.go:172 +0x3f
Jan 31 19:11:03 [localhost] ct-woodpecker: goroutine 59 [select]:
Jan 31 19:11:03 [localhost] ct-woodpecker: github.com/letsencrypt/ct-woodpecker/monitor.(*inclusionChecker).run.func1(0xc0002a4140)
Jan 31 19:11:03 [localhost] ct-woodpecker: /go/src/github.com/letsencrypt/ct-woodpecker/monitor/inclusion_checker.go:112 +0xe0
Jan 31 19:11:03 [localhost] ct-woodpecker: created by github.com/letsencrypt/ct-woodpecker/monitor.(*inclusionChecker).run
Jan 31 19:11:03 [localhost] ct-woodpecker: /go/src/github.com/letsencrypt/ct-woodpecker/monitor/inclusion_checker.go:109 +0x3f
Jan 31 19:11:03 [localhost] ct-woodpecker: goroutine 60 [select, 1 minutes]:
Jan 31 19:11:03 [localhost] ct-woodpecker: github.com/letsencrypt/ct-woodpecker/monitor.(*sthFetcher).run.func1(0xc0002a4280)
Jan 31 19:11:03 [localhost] ct-woodpecker: /go/src/github.com/letsencrypt/ct-woodpecker/monitor/sth_fetcher.go:136 +0x192
Jan 31 19:11:03 [localhost] ct-woodpecker: created by github.com/letsencrypt/ct-woodpecker/monitor.(*sthFetcher).run
Jan 31 19:11:03 [localhost] ct-woodpecker: /go/src/github.com/letsencrypt/ct-woodpecker/monitor/sth_fetcher.go:132 +0x3f
Jan 31 19:11:03 [localhost] ct-woodpecker: goroutine 61 [select, 9 minutes]:
Jan 31 19:11:03 [localhost] ct-woodpecker: github.com/letsencrypt/ct-woodpecker/monitor.(*certSubmitter).run.func1(0xc00010cb40)
Jan 31 19:11:03 [localhost] ct-woodpecker: /go/src/github.com/letsencrypt/ct-woodpecker/monitor/cert_submitter.go:177 +0x184
Jan 31 19:11:03 [localhost] ct-woodpecker: created by github.com/letsencrypt/ct-woodpecker/monitor.(*certSubmitter).run
Jan 31 19:11:03 [localhost] ct-woodpecker: /go/src/github.com/letsencrypt/ct-woodpecker/monitor/cert_submitter.go:172 +0x3f
Jan 31 19:11:03 [localhost] ct-woodpecker: goroutine 62 [select]:
Jan 31 19:11:03 [localhost] ct-woodpecker: github.com/letsencrypt/ct-woodpecker/monitor.(*inclusionChecker).run.func1(0xc0002a4460)
Jan 31 19:11:03 [localhost] ct-woodpecker: /go/src/github.com/letsencrypt/ct-woodpecker/monitor/inclusion_checker.go:112 +0xe0
Jan 31 19:11:03 [localhost] ct-woodpecker: created by github.com/letsencrypt/ct-woodpecker/monitor.(*inclusionChecker).run
Jan 31 19:11:03 [localhost] ct-woodpecker: /go/src/github.com/letsencrypt/ct-woodpecker/monitor/inclusion_checker.go:109 +0x3f
Jan 31 19:11:03 [localhost] ct-woodpecker: goroutine 63 [select, 1 minutes]:
Jan 31 19:11:03 [localhost] ct-woodpecker: github.com/letsencrypt/ct-woodpecker/monitor.(*sthFetcher).run.func1(0xc0002a45a0)
Jan 31 19:11:03 [localhost] ct-woodpecker: /go/src/github.com/letsencrypt/ct-woodpecker/monitor/sth_fetcher.go:136 +0x192
Jan 31 19:11:03 [localhost] ct-woodpecker: created by github.com/letsencrypt/ct-woodpecker/monitor.(*sthFetcher).run
Jan 31 19:11:03 [localhost] ct-woodpecker: /go/src/github.com/letsencrypt/ct-woodpecker/monitor/sth_fetcher.go:132 +0x3f
Jan 31 19:11:03 [localhost] ct-woodpecker: goroutine 64 [select, 9 minutes]:
Jan 31 19:11:03 [localhost] ct-woodpecker: github.com/letsencrypt/ct-woodpecker/monitor.(*certSubmitter).run.func1(0xc00010ccc0)
Jan 31 19:11:03 [localhost] ct-woodpecker: /go/src/github.com/letsencrypt/ct-woodpecker/monitor/cert_submitter.go:177 +0x184
Jan 31 19:11:03 [localhost] ct-woodpecker: created by github.com/letsencrypt/ct-woodpecker/monitor.(*certSubmitter).run
Jan 31 19:11:03 [localhost] ct-woodpecker: /go/src/github.com/letsencrypt/ct-woodpecker/monitor/cert_submitter.go:172 +0x3f
Jan 31 19:11:03 [localhost] ct-woodpecker: goroutine 65 [select]:
Jan 31 19:11:03 [localhost] ct-woodpecker: github.com/letsencrypt/ct-woodpecker/monitor.(*inclusionChecker).run.func1(0xc0002a4780)
Jan 31 19:11:03 [localhost] ct-woodpecker: /go/src/github.com/letsencrypt/ct-woodpecker/monitor/inclusion_checker.go:112 +0xe0
Jan 31 19:11:03 [localhost] ct-woodpecker: created by github.com/letsencrypt/ct-woodpecker/monitor.(*inclusionChecker).run
Jan 31 19:11:03 [localhost] ct-woodpecker: /go/src/github.com/letsencrypt/ct-woodpecker/monitor/inclusion_checker.go:109 +0x3f
Jan 31 19:11:03 [localhost] ct-woodpecker: goroutine 694 [IO wait]:
Jan 31 19:11:03 [localhost] ct-woodpecker: internal/poll.runtime_pollWait(0x7f3ef742e610, 0x72, 0xc000185870)
Jan 31 19:11:03 [localhost] ct-woodpecker: /usr/local/go/src/runtime/netpoll.go:173 +0x66
Jan 31 19:11:03 [localhost] ct-woodpecker: internal/poll.(*pollDesc).wait(0xc025104818, 0x72, 0xffffffffffffff00, 0xc015a0, 0x12a9f80)
Jan 31 19:11:03 [localhost] ct-woodpecker: /usr/local/go/src/internal/poll/fd_poll_runtime.go:85 +0x9a
Jan 31 19:11:03 [localhost] ct-woodpecker: internal/poll.(*pollDesc).waitRead(0xc025104818, 0xc035910400, 0x400, 0x400)
Jan 31 19:11:03 [localhost] ct-woodpecker: /usr/local/go/src/internal/poll/fd_poll_runtime.go:90 +0x3d
Jan 31 19:11:03 [localhost] ct-woodpecker: internal/poll.(*FD).Read(0xc025104800, 0xc035910400, 0x400, 0x400, 0x0, 0x0, 0x0)
Jan 31 19:11:03 [localhost] ct-woodpecker: /usr/local/go/src/internal/poll/fd_unix.go:169 +0x179
Jan 31 19:11:03 [localhost] ct-woodpecker: net.(*netFD).Read(0xc025104800, 0xc035910400, 0x400, 0x400, 0x413a8b, 0xc00001e000, 0xa9c200)
Jan 31 19:11:03 [localhost] ct-woodpecker: /usr/local/go/src/net/fd_unix.go:202 +0x4f
Jan 31 19:11:03 [localhost] ct-woodpecker: net.(*conn).Read(0xc00000e8a0, 0xc035910400, 0x400, 0x400, 0x0, 0x0, 0x0)
Jan 31 19:11:03 [localhost] ct-woodpecker: /usr/local/go/src/net/net.go:177 +0x68
Jan 31 19:11:03 [localhost] ct-woodpecker: crypto/tls.(*block).readFromUntil(0xc03a276a50, 0xc001e0, 0xc00000e8a0, 0x5, 0xc00000e8a0, 0x0)
Jan 31 19:11:03 [localhost] ct-woodpecker: /usr/local/go/src/crypto/tls/conn.go:492 +0x89
Jan 31 19:11:03 [localhost] ct-woodpecker: crypto/tls.(*Conn).readRecord(0xc007adf880, 0xb86c17, 0xc007adf9a0, 0x0)
Jan 31 19:11:03 [localhost] ct-woodpecker: /usr/local/go/src/crypto/tls/conn.go:593 +0xdd
Jan 31 19:11:03 [localhost] ct-woodpecker: crypto/tls.(*Conn).Read(0xc007adf880, 0xc031db8000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
Jan 31 19:11:03 [localhost] ct-woodpecker: /usr/local/go/src/crypto/tls/conn.go:1145 +0xf1
Jan 31 19:11:03 [localhost] ct-woodpecker: bufio.(*Reader).Read(0xc02bc53bc0, 0xc03e0aa578, 0x9, 0x9, 0xc000122000, 0xc000185c58, 0xc000185c70)
Jan 31 19:11:03 [localhost] ct-woodpecker: /usr/local/go/src/bufio/bufio.go:216 +0x22f
Jan 31 19:11:03 [localhost] ct-woodpecker: io.ReadAtLeast(0xbff700, 0xc02bc53bc0, 0xc03e0aa578, 0x9, 0x9, 0x9, 0xc000185cf8, 0x6a3f2b, 0xc02bc53ce0)
Jan 31 19:11:03 [localhost] ct-woodpecker: /usr/local/go/src/io/io.go:310 +0x88
Jan 31 19:11:03 [localhost] ct-woodpecker: io.ReadFull(0xbff700, 0xc02bc53bc0, 0xc03e0aa578, 0x9, 0x9, 0xc000185d28, 0x6a3e3d, 0xc000185fb8)
Jan 31 19:11:03 [localhost] ct-woodpecker: /usr/local/go/src/io/io.go:329 +0x58
Jan 31 19:11:03 [localhost] ct-woodpecker: net/http.http2readFrameHeader(0xc03e0aa578, 0x9, 0x9, 0xbff700, 0xc02bc53bc0, 0x0, 0x0, 0xc03a277e00, 0x0)
Jan 31 19:11:03 [localhost] ct-woodpecker: /usr/local/go/src/net/http/h2_bundle.go:1545 +0x7b
Jan 31 19:11:03 [localhost] ct-woodpecker: net/http.(*http2Framer).ReadFrame(0xc03e0aa540, 0xc03a277e00, 0x0, 0x0, 0x0)
Jan 31 19:11:03 [localhost] ct-woodpecker: /usr/local/go/src/net/http/h2_bundle.go:1803 +0xa3
Jan 31 19:11:03 [localhost] ct-woodpecker: net/http.(*http2clientConnReadLoop).run(0xc000185fb8, 0xb865f8, 0xc0002a3fb8)
Jan 31 19:11:03 [localhost] ct-woodpecker: /usr/local/go/src/net/http/h2_bundle.go:8263 +0x9e
Jan 31 19:11:03 [localhost] ct-woodpecker: net/http.(*http2ClientConn).readLoop(0xc0206608c0)
Jan 31 19:11:03 [localhost] ct-woodpecker: /usr/local/go/src/net/http/h2_bundle.go:8191 +0x76
Jan 31 19:11:03 [localhost] ct-woodpecker: created by net/http.(*http2Transport).newClientConn
Jan 31 19:11:03 [localhost] ct-woodpecker: /usr/local/go/src/net/http/h2_bundle.go:7264 +0x636
Jan 31 19:11:03 [localhost] ct-woodpecker: goroutine 194 [IO wait]:
Jan 31 19:11:03 [localhost] ct-woodpecker: internal/poll.runtime_pollWait(0x7f3ef742e880, 0x72, 0xc000325870)
Jan 31 19:11:03 [localhost] ct-woodpecker: /usr/local/go/src/runtime/netpoll.go:173 +0x66
Jan 31 19:11:03 [localhost] ct-woodpecker: internal/poll.(*pollDesc).wait(0xc000366598, 0x72, 0xffffffffffffff00, 0xc015a0, 0x12a9f80)
Jan 31 19:11:03 [localhost] ct-woodpecker: /usr/local/go/src/internal/poll/fd_poll_runtime.go:85 +0x9a
Jan 31 19:11:03 [localhost] ct-woodpecker: internal/poll.(*pollDesc).waitRead(0xc000366598, 0xc0004b8000, 0x2000, 0x2000)
Jan 31 19:11:03 [localhost] ct-woodpecker: /usr/local/go/src/internal/poll/fd_poll_runtime.go:90 +0x3d
Jan 31 19:11:03 [localhost] ct-woodpecker: internal/poll.(*FD).Read(0xc000366580, 0xc0004b8000, 0x2000, 0x2000, 0x0, 0x0, 0x0)
Jan 31 19:11:03 [localhost] ct-woodpecker: /usr/local/go/src/internal/poll/fd_unix.go:169 +0x179
Jan 31 19:11:03 [localhost] ct-woodpecker: net.(*netFD).Read(0xc000366580, 0xc0004b8000, 0x2000, 0x2000, 0x413a8b, 0xc00001e000, 0xa9c200)
Jan 31 19:11:03 [localhost] ct-woodpecker: /usr/local/go/src/net/fd_unix.go:202 +0x4f
Jan 31 19:11:03 [localhost] ct-woodpecker: net.(*conn).Read(0xc00000e6b0, 0xc0004b8000, 0x2000, 0x2000, 0x0, 0x0, 0x0)
Jan 31 19:11:03 [localhost] ct-woodpecker: /usr/local/go/src/net/net.go:177 +0x68
Jan 31 19:11:03 [localhost] ct-woodpecker: crypto/tls.(*block).readFromUntil(0xc0003c4cc0, 0xc001e0, 0xc00000e6b0, 0x5, 0xc00000e6b0, 0x0)
Jan 31 19:11:03 [localhost] ct-woodpecker: /usr/local/go/src/crypto/tls/conn.go:492 +0x89
Jan 31 19:11:03 [localhost] ct-woodpecker: crypto/tls.(*Conn).readRecord(0xc0001ec000, 0xb86c17, 0xc0001ec120, 0x0)
Jan 31 19:11:03 [localhost] ct-woodpecker: /usr/local/go/src/crypto/tls/conn.go:593 +0xdd
Jan 31 19:11:03 [localhost] ct-woodpecker: crypto/tls.(*Conn).Read(0xc0001ec000, 0xc000438000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
Jan 31 19:11:03 [localhost] ct-woodpecker: /usr/local/go/src/crypto/tls/conn.go:1145 +0xf1
Jan 31 19:11:03 [localhost] ct-woodpecker: bufio.(*Reader).Read(0xc0000333e0, 0xc0001f57d8, 0x9, 0x9, 0xc000122000, 0xc000325c58, 0xc000325c70)
Jan 31 19:11:03 [localhost] ct-woodpecker: /usr/local/go/src/bufio/bufio.go:216 +0x22f
Jan 31 19:11:03 [localhost] ct-woodpecker: io.ReadAtLeast(0xbff700, 0xc0000333e0, 0xc0001f57d8, 0x9, 0x9, 0x9, 0xc000325cf8, 0x6a3f2b, 0xc03c4c9800)
Jan 31 19:11:03 [localhost] ct-woodpecker: /usr/local/go/src/io/io.go:310 +0x88
Jan 31 19:11:03 [localhost] ct-woodpecker: io.ReadFull(0xbff700, 0xc0000333e0, 0xc0001f57d8, 0x9, 0x9, 0xc000325d28, 0x6a3e3d, 0xc000325fb8)
Jan 31 19:11:03 [localhost] ct-woodpecker: /usr/local/go/src/io/io.go:329 +0x58
Jan 31 19:11:03 [localhost] ct-woodpecker: net/http.http2readFrameHeader(0xc0001f57d8, 0x9, 0x9, 0xbff700, 0xc0000333e0, 0x0, 0x0, 0x0, 0x0)
Jan 31 19:11:03 [localhost] ct-woodpecker: /usr/local/go/src/net/http/h2_bundle.go:1545 +0x7b
Jan 31 19:11:03 [localhost] ct-woodpecker: net/http.(*http2Framer).ReadFrame(0xc0001f57a0, 0xc0410525a0, 0x0, 0x0, 0x0)
Jan 31 19:11:03 [localhost] ct-woodpecker: /usr/local/go/src/net/http/h2_bundle.go:1803 +0xa3
Jan 31 19:11:03 [localhost] ct-woodpecker: net/http.(*http2clientConnReadLoop).run(0xc000325fb8, 0xb865f8, 0xc0002b0fb8)
Jan 31 19:11:03 [localhost] ct-woodpecker: /usr/local/go/src/net/http/h2_bundle.go:8263 +0x9e
Jan 31 19:11:03 [localhost] ct-woodpecker: net/http.(*http2ClientConn).readLoop(0xc0001096c0)
Jan 31 19:11:03 [localhost] ct-woodpecker: /usr/local/go/src/net/http/h2_bundle.go:8191 +0x76
Jan 31 19:11:03 [localhost] ct-woodpecker: created by net/http.(*http2Transport).newClientConn
Jan 31 19:11:03 [localhost] ct-woodpecker: /usr/local/go/src/net/http/h2_bundle.go:7264 +0x636
cpu commented 5 years ago

As of f0efe26 pprof data is accessible over the configured metrics address under the /debug/pprof/ path prefix.

e.g. with a debug addr of 10.40.50.3:1971 (docker env defaults) a heap profile can be collected by running:

curl http://10.40.50.3:1971/debug/pprof/heap > heap.pprof.gz