search

grafana / carbon-relay-ng

Fast carbon relay+aggregator with admin interfaces for making changes online - production ready
Other
467 stars 151 forks source link

Panic when carbon-relay connects to carbon-cache on ARM/V7 #439

Open rbuffi opened 4 years ago

rbuffi commented 4 years ago

Hello,

My carbon-relay container panics when connecting to carbon-cache container on k3s (arm/v7):

[2020-08-21 11:31:05] check if the port 3003 for 'carbon-cache-service' is available [2020-08-21 11:31:06] ---------------------------------------------------- [2020-08-21 11:31:06] carbon-relay-ng - build: [2020-08-21 11:31:06] ---------------------------------------------------- panic: runtime error: invalid memory address or nil pointer dereference [signal SIGSEGV: segmentation violation code=0x1 addr=0x4 pc=0x11fd0] goroutine 34 [running]: github.com/grafana/carbon-relay-ng/statsmt.(LatencyHistogram15s32).Value(0x2db6000, 0x29936, 0x0) /opt/go/src/github.com/graphite-ng/carbon-relay-ng/statsmt/latencyhistogram15s32.go:26 +0x5c github.com/grafana/carbon-relay-ng/statsmt.(Graphite).writer(0x2d80480) /opt/go/src/github.com/graphite-ng/carbon-relay-ng/statsmt/out_graphite.go:118 +0x29c created by github.com/grafana/carbon-relay-ng/statsmt.NewGraphite /opt/go/src/github.com/graphite-ng/carbon-relay-ng/statsmt/out_graphite.go:52 +0x35c

Maybe you can point me in the right direction. Im running a K3s cluster on RPI4 (ARM/V7). A container with carbon-cache is already running. Now i'm trying to get carbon-relay running by doing the following:

git clone https://github.com/bodsch/docker-carbon-relay-ng cd docker-carbon-relay-ng docker buildx build --platform linux/arm/v7 -t containerregistrydcsl.azurecr.io/metrics/docker-carbon-relay-ng --push .

And creating the deployment:

apiVersion: apps/v1 kind: Deployment metadata: name: carbon-relay-ng labels: app: metrics spec: replicas: 2 selector: matchLabels: app: metrics template: metadata: labels: app: metrics spec: containers:

When I start the carbon-relay container from docker on my Mac (x86) it connect to carbon-cache without problems:

docker run -d -p 2003:2003 -p 2004:2004 --name=docker-carbon-relay -e GRAPHITE_HOST='192.168.0.202' -e GRAPHITE_PORT='3003' -d containerregistrydcsl.azurecr.io/metrics/docker-carbon-relay-ng

Thanks in advance,

Ronald

rbuffi commented 4 years ago

This is the output with verbose logging:

2020-08-22 18:50:19.847 [INFO] ===== carbon-relay-ng instance 'carbon-relay-ng-7b898f7c97-8jlsj' starting. (version 0.13.0-4-gc9044b5) ===== 2020-08-22 18:50:19.847 [DEBUG] setting GOMAXPROCS to 4 2020-08-22 18:50:19.853 [INFO] initializing routing table... 2020-08-22 18:50:19.870 [INFO] =========================== 2020-08-22 18:50:19.870 [INFO] ========== TABLE ========== 2020-08-22 18:50:19.870 [INFO] =========================== 2020-08-22 18:50:19.870 [INFO] 2020-08-22 18:50:19.870 [INFO] ## Rewriters: 2020-08-22 18:50:19.870 [INFO] old new not max 2020-08-22 18:50:19.870 [INFO] ================== 2020-08-22 18:50:19.870 [INFO] /^/ -1 2020-08-22 18:50:19.870 [INFO] 2020-08-22 18:50:19.871 [INFO] ## Blacklist: 2020-08-22 18:50:19.871 [INFO] prefix notPrefix sub notSub regex notRegex 2020-08-22 18:50:19.871 [INFO] =============================================== 2020-08-22 18:50:19.871 [INFO] 2020-08-22 18:50:19.871 [INFO] ## Aggregations: 2020-08-22 18:50:19.871 [INFO] key func regex notRegex prefix notPrefix sub notSub outFmt cache interval wait dropRaw 2020-08-22 18:50:19.871 [INFO] ================================================================================================== 2020-08-22 18:50:19.871 [INFO] 2020-08-22 18:50:19.871 [INFO] ## Routes: 2020-08-22 18:50:19.871 [INFO] type key prefix notPrefix sub notSub regex notRegex 2020-08-22 18:50:19.871 [INFO] ================================================================================================================================================ 2020-08-22 18:50:19.871 [INFO] > sendAllMatch carbon-default 2020-08-22 18:50:19.871 [INFO] prefix notPrefix sub notSub regex notRegex addr spoolDir spool pickle online 2020-08-22 18:50:19.871 [INFO] ---------------------------------------------------------------------------------------------------------------------------------------------- 2020-08-22 18:50:19.871 [INFO] carbon-cache-service:3003 /var/spool/carbon-relay-ng-7b898f7c97-8jlsj false false false 2020-08-22 18:50:19.871 [INFO] 2020-08-22 18:50:19.871 [INFO] 2020-08-22 18:50:19.874 [INFO] listening on 0.0.0.0:2013/tcp 2020-08-22 18:50:19.874 [INFO] listening on 0.0.0.0:2003/udp 2020-08-22 18:50:19.874 [INFO] listening on 0.0.0.0:2003/tcp 2020-08-22 18:50:19.875 [DEBUG] dest carbon-default_carbon-cache-service_3003 (re)connecting to carbon-cache-service:3003 2020-08-22 18:50:19.874 [DEBUG] dest carbon-default_carbon-cache-service_3003 entering select. conn: false spooling: false slowLastloop: false, slowNow: false spoolQueue: false 2020-08-22 18:50:19.875 [DEBUG] dest carbon-default_carbon-cache-service_3003 entering select. conn: false spooling: false slowLastloop: false, slowNow: false spoolQueue: false 2020-08-22 18:50:19.875 [INFO] admin TCP listener starting on 0.0.0.0:2004 2020-08-22 18:50:19.875 [INFO] listening on 0.0.0.0:2013/udp 2020-08-22 18:50:19.880 [INFO] admin HTTP listener starting on 0.0.0.0:8081 2020-08-22 18:50:19.888 [DEBUG] dest carbon-default_carbon-cache-service_3003 connected to carbon-cache-service:3003 2020-08-22 18:50:19.888 [INFO] dest carbon-default_carbon-cache-service_3003 new conn online 2020-08-22 18:50:19.888 [DEBUG] conn carbon-default_carbon-cache-service_3003 .up query responded with true 2020-08-22 18:50:19.888 [DEBUG] dest carbon-default_carbon-cache-service_3003 entering select. conn: true spooling: false slowLastloop: false, slowNow: false spoolQueue: false 2020-08-22 18:50:19.889 [DEBUG] conn carbon-default_carbon-cache-service_3003 .up query responded with true 2020-08-22 18:50:19.889 [DEBUG] dest carbon-default_carbon-cache-service_3003 entering select. conn: true spooling: false slowLastloop: false, slowNow: false spoolQueue: false 2020-08-22 18:50:20.000 [DEBUG] stats flushing for 2020-08-22 18:50:20 +0000 UTC m=+0.176198045 to graphite 2020-08-22 18:50:20.890 [DEBUG] conn carbon-default_carbon-cache-service_3003 HandleData: c.buffered auto-flushing... 2020-08-22 18:50:20.890 [DEBUG] conn carbon-default_carbon-cache-service_3003 HandleData c.buffered auto-flush done without error 2020-08-22 18:50:20.890 [DEBUG] conn carbon-default_carbon-cache-service_3003 HandleData auto-flush 317.774µs (total iter 1.000484875s) (use this to tune your In buffering) 2020-08-22 18:50:21.000 [DEBUG] stats flushing for 2020-08-22 18:50:21 +0000 UTC m=+1.176197712 to graphite 2020-08-22 18:50:21.014 [INFO] stats now connected to carbon-cache-service:3003 panic: runtime error: invalid memory address or nil pointer dereference [signal SIGSEGV: segmentation violation code=0x1 addr=0x4 pc=0x11fd0]

goroutine 8 [running]: github.com/grafana/carbon-relay-ng/statsmt.(LatencyHistogram15s32).Value(0x210c5a0, 0x32df0, 0x0) /opt/go/src/github.com/graphite-ng/carbon-relay-ng/statsmt/latencyhistogram15s32.go:26 +0x5c github.com/grafana/carbon-relay-ng/statsmt.(Graphite).writer(0x200b9a0) /opt/go/src/github.com/graphite-ng/carbon-relay-ng/statsmt/out_graphite.go:118 +0x29c created by github.com/grafana/carbon-relay-ng/statsmt.NewGraphite /opt/go/src/github.com/graphite-ng/carbon-relay-ng/statsmt/out_graphite.go:52 +0x35c