dgryski
commented
4 years ago High-volume error logs can be sampled (1% or whatever).
Closed azhiltsov closed 4 years ago
dgryski
commented
4 years ago High-volume error logs can be sampled (1% or whatever).
Amount of generated logs should never corelate with amount of processed metrics. Currently we are logging every metric which we fail to parse. { "level": "info", "ts": 1578536523.913359, "caller": "nanotube/process.go:40", "msg": "Error parsing incoming record", "record": "monitors.java.gauge.min 1.79769313486232e+308 1578536486", "error": "error converting incoming record val 1.79769313486232e+308: strconv.ParseFloat: parsing \"1.79769313486232e+308\": value out of range", "errorVerbose": "strconv.ParseFloat: parsing \"1.79769313486232e+308\": value out of range\nerror converting incoming record val 1.79769313486232e+308\nnanotube/pkg/rec.ParseRec\n\t/usr/local/git_tree/gopath/src/gitlab.booking.com/graphite/nanotube/pkg/rec/rec.go:39\nmain.proc\n\t/usr/local/git_tree/gopath/src/gitlab.booking.com/graphite/nanotube/cmd/nanotube/process.go:38\nmain.worker\n\t/usr/local/git_tree/gopath/src/gitlab.booking.com/graphite/nanotube/cmd/nanotube/process.go:33\nruntime.goexit\n\t/usr/lib64/go/src/runtime/asm_amd64.s:1337" }