Change graphite service.port results in pod crashloopbackoff

[scottking2]

Is this a request for help?: yes

Is this a BUG REPORT or FEATURE REQUEST? (choose one):

Version of Helm and Kubernetes: helm: v3.1.2 k8s: v1.15.5

Which chart in which version: graphite: 0.5.0

What happened: I changed the service.port from the default 8080 to port 8000 and the pod gets stuck in a crashloopbackoff with the liveness and readiness probes failing to connect.

What you expected to happen: I expected the service.port to be configurable to a different port number.

How to reproduce it (as minimally and precisely as possible): I ran the following on my mac terminal connected with docker desktop and k8s through docker desktop. helm install graphite --set service.port=8000 --set service.type=NodePort kiwigrid/graphite

Anything else we need to know:

Everything works as expected with the service.type=NodePort if I don't change the service.port. Below are the logs when i follow the pod after changing the service.port:

$ kubectl logs graphite-0 --follow ok: run: nginx: (pid 21) 0s 22 Apr 17:07:27 - [40] reading config file: /opt/statsd/config/udp.js 22 Apr 17:07:27 - server is up INFO Starting carbon-aggregator (instance a) 22/04/2020 17:07:27 :: [console] Using sorted write strategy for cache 22/04/2020 17:07:27 :: [aggregator] reading new aggregation rules from /opt/graphite/conf/aggregation-rules.conf 22/04/2020 17:07:27 :: [aggregator] clearing aggregation buffers 22/04/2020 17:07:27 :: [clients] connecting to carbon daemon at 127.0.0.1:2004:None 22/04/2020 17:07:27 :: [console] twistd 18.9.0 (/opt/graphite/bin/python3 3.6.6) starting up. 22/04/2020 17:07:27 :: [console] reactor class: twisted.internet.epollreactor.EPollReactor. 22/04/2020 17:07:27 :: [console] Installing SIG_IGN for SIGHUP 22/04/2020 17:07:27 :: [console] Starting factory CarbonClientFactory(127.0.0.1:2004:None) 22/04/2020 17:07:27 :: [clients] CarbonClientFactory(127.0.0.1:2004:None)::startedConnecting (127.0.0.1:2004) 22/04/2020 17:07:27 :: [console] CarbonReceiverFactory starting on 2023 22/04/2020 17:07:27 :: [console] Starting factory <carbon.protocols.CarbonReceiverFactory object at 0x7fabdbf33c18> 22/04/2020 17:07:27 :: [console] CarbonReceiverFactory starting on 2024 22/04/2020 17:07:27 :: [console] Starting factory <carbon.protocols.CarbonReceiverFactory object at 0x7fabdbf33c88> 22/04/2020 17:07:27 :: [console] <twisted.internet.tcp.Connector object at 0x7fabdbf33630> will retry in 2 seconds 22/04/2020 17:07:27 :: [clients] CarbonClientFactory(127.0.0.1:2004:None)::clientConnectionFailed (127.0.0.1:2004) Connection was refused by other side: 111: Connection refused. 22/04/2020 17:07:27 :: [clients] Destination is down: 127.0.0.1:2004:None (1/5) 22/04/2020 17:07:27 :: [console] Stopping factory CarbonClientFactory(127.0.0.1:2004:None) Starting carbon-cache (instance a) 22/04/2020 17:07:27 :: [console] Using sorted write strategy for cache 22/04/2020 17:07:27 :: [console] Enabling Whisper fallocate support 22/04/2020 17:07:27 :: [console] twistd 18.9.0 (/opt/graphite/bin/python3 3.6.6) starting up. 22/04/2020 17:07:27 :: [console] reactor class: twisted.internet.epollreactor.EPollReactor. 22/04/2020 17:07:27 :: [console] ServerFactory starting on 7002 22/04/2020 17:07:27 :: [console] Starting factory <twisted.internet.protocol.ServerFactory object at 0x7f2a891cc940> 22/04/2020 17:07:27 :: [console] Installing SIG_IGN for SIGHUP 22/04/2020 17:07:27 :: [console] CarbonReceiverFactory starting on 2003 22/04/2020 17:07:27 :: [console] Starting factory <carbon.protocols.CarbonReceiverFactory object at 0x7f2a88efc470> 22/04/2020 17:07:27 :: [console] CarbonReceiverFactory starting on 2004 22/04/2020 17:07:27 :: [console] Starting factory <carbon.protocols.CarbonReceiverFactory object at 0x7f2a88efc4e0> 22/04/2020 17:07:30 :: [console] Starting factory CarbonClientFactory(127.0.0.1:2004:None) 22/04/2020 17:07:30 :: [clients] CarbonClientFactory(127.0.0.1:2004:None)::startedConnecting (127.0.0.1:2004) 22/04/2020 17:07:30 :: [clients] CarbonClientProtocol(127.0.0.1:2004:None)::connectionMade 22/04/2020 17:07:30 :: [clients] CarbonClientFactory(127.0.0.1:2004:None)::connectionMade (CarbonClientProtocol(127.0.0.1:2004:None)) 22/04/2020 17:07:30 :: [clients] Destination is up: 127.0.0.1:2004:None 22/04/2020 17:07:30 :: [listener] MetricPickleReceiver connection with 127.0.0.1:37928 established 22/04/2020 17:07:37 :: [tagdb] Tagging carbon.agents.graphite-0-a.memUsage 22/04/2020 17:07:38 :: [listener] MetricLineReceiver connection with 127.0.0.1:41142 established 22/04/2020 17:07:38 :: [listener] MetricLineReceiver connection with 127.0.0.1:41142 closed cleanly 22/04/2020 17:07:38 :: [tagdb] Tagged carbon.agents.graphite-0-a.memUsage in 0.09088778495788574 22/04/2020 17:07:48 :: [listener] MetricLineReceiver connection with 127.0.0.1:41204 established 22/04/2020 17:07:48 :: [listener] MetricLineReceiver connection with 127.0.0.1:41204 closed cleanly

[monotek]

The value only changes the kubernetes config, not the graphite config. So you have to change the GRAPHITE_URL variable in the graphite config too.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.