search

lightninglabs / lndmon

🔎lndmon: A drop-in monitoring solution for your lnd node using Prometheus+Grafana
MIT License
149 stars 47 forks source link

Insufficient info in logs #56

Closed krtk6160 closed 2 years ago

krtk6160 commented 3 years ago

I am trying to run lndmon in kubernetes in its own pod, separate from my lnd pod. I am using these arguments "--prometheus.listenaddr=0.0.0.0:9092", "--lnd.host=lnd-service:10009", "--lnd.macaroondir=/root/.lnd", "--lnd.tlspath=/root/.lnd/tls.cert", "--lnd.network=regtest".

lnd-service is the service that is bound to the lnd pod. I have verified that my tls.cert and readonly.macaroon have been mounted correctly by ssh'ing into the lndmon pod.

Looking at the logs, I see only one entry - 

2020-10-16 13:26:07.260 [INF] LNDMON: Starting Prometheus exporter...

It looks like lndmon is unable to communicate with my lnd pod, but there are no errors in the log to indicate as such. When I visit the /metrics endpoint of lndmon, I do not see any lnd related metrics, which suggests that lndmon indeed is unable to talk to lnd.

The logs do not show any info whatsoever that could help in debugging the issue.

Ideally, lndmon should exit if it's unable to talk to lnd and log the errors for easier debugging.

guggero commented 3 years ago

Yeah, we should add a setting to increase the log level...

My assumption is that you need to add --tlsextradomain=lnd-service to lnd's configuration and remove any existing tls.cert and tls.key files. Otherwise the internal domain name lnd-service won't be in the certificate and lndmon won't be able to connect.

krtk6160 commented 3 years ago

My assumption is that you need to add --tlsextradomain=lnd-service to lnd's configuration and remove any existing tls.cert and tls.key files. Otherwise the internal domain name lnd-service won't be in the certificate and lndmon won't be able to connect.

I already have the --tlsextradomain flag set correctly, and I also have a loop pod (separate from the lnd pod) running that works perfectly with this lnd pod using the lnd-service domain name.

guggero commented 3 years ago

So you never see the message Prometheus active! in the log files? Looking at the code it seems like lndmon should exit with an error when it is unable to connect to lnd.

krtk6160 commented 3 years ago

So you never see the message Prometheus active! in the log files?

Nope, I just get the Starting Prometheus exporter..., and that's it, no further logs.

I am using this docker image (v0.1.1)