go-graphite / go-carbon

Golang implementation of Graphite/Carbon server with classic architecture: Agent -> Cache -> Persister
MIT License
801 stars 126 forks source link

[BUG] If set to "carbon" user+group instead default root, the service start is stuck and no informative logs. #575

Open interfan7 opened 7 months ago

interfan7 commented 7 months ago

Describe the bug By default the systemd unit doesn't define fields User and Group. If I add it, the process start through systemd is stuck and systemd returns timeout.

$ sudo systemctl start go-carbon.service
Job for go-carbon.service failed because a timeout was exceeded.
See "systemctl status go-carbon.service" and "journalctl -xe" for details.

The process however is alive:

$ sudo ps aux | grep go-carbon
carbon    150322 31.5  0.9 1520064 319508 ?      Ssl  17:36   0:14 /usr/bin/go-carbon -config /etc/go-carbon/go-carbon.conf -pidfile /var/run/go-carbon.pid -daemon
leonid.+  150340  0.0  0.0   8168   656 pts/1    S+   17:36   0:00 grep --color=auto go-carbon

Logs This is the log suffix:

$ tail -n 4 /var/log/go-carbon/go-carbon.log
[2023-11-22T17:39:05.860Z] INFO [carbonserver] starting carbonserver {"listen": "127.0.0.1:8080", "whisperData": "/data/graphite/whisper", "maxGlobs": 1000, "scanFrequency": "5m0s"}
[2023-11-22T17:39:05.861Z] INFO [main] started {}
[2023-11-22T17:39:07.683Z] INFO [carbonserver] file list updated {"handler": "fileListUpdated", "file_scan_runtime": 1.01448487, "indexing_runtime": 0.807947054, "rdtime_update_runtime": 0.000000087, "cache_index_runtime": 0.000000141, "total_runtime": 1.82243443, "Files": 388343, "index_size": 16432, "pruned_trigrams": 9, "cache_metric_len_before": 0, "cache_metric_len_after": 0, "metrics_known": 348427, "index_type": "trigram", "read_from_cache": false}
[2023-11-22T17:39:16.390Z] INFO [access] find success {"handler": "find", "url": "/metrics/find/?format=protobuf&query=%2A", "peer": "127.0.0.1:52564", "query": ["*"], "format": "protobuf", "runtime_seconds": 0.000451381, "Files": 84, "find_cache_enabled": true, "from_cache": false, "http_code": 200, "lookups": 0}

Go-carbon Configuration: Almost identical to default really.

Metric retention and aggregation schemas I think it's irrelevant. Nothing special.

Additional context Having this process as root is a bad practice especially for anyone who considers it 3rd party. Mind you the carbonapi project does define non-root user+group.

deniszh commented 7 months ago

Hi @interfan7 We're running go-carbon under carbon user in production without any issues. So, it's some local issue for your system - maybe pid file can't be created? or selinux doing that.