search

shizunge / endlessh-go

A golang implementation of endlessh exporting Prometheus metrics, visualized by a Grafana dashboard.
GNU General Public License v3.0
951 stars 28 forks source link

NixOS: "Failed to obtain geohash. Lookup ip-api.com on [::1]:53 connection refused" #108

Closed V3ntus closed 1 week ago

V3ntus commented 4 months ago

The systemd service is not able to resolve the geohash API host. Oddly it's pointing to localhost to try and resolve it. resolveconf.service is enabled and active. There are valid nameservers in /etc/resolv.conf. I can dig ip-api.com just fine. Here's the systemd status for endlessh-go:

● endlessh-go.service - SSH tarpit
     Loaded: loaded (/etc/systemd/system/endlessh-go.service; enabled; preset: enabled)
     Active: active (running) since Thu 2024-05-02 15:22:11 UTC; 54min ago
   Main PID: 271204 (endlessh-go)
         IP: 170.9K in, 533.3K out
         IO: 6.3M read, 0B written
      Tasks: 5 (limit: 1152)
     Memory: 11.4M
        CPU: 884ms
     CGroup: /system.slice/endlessh-go.service
             └─271204 /nix/store/i4kqp7h5n5j2avwj6gjjjihhg6axxxqz-endlessh-go-20230625-3/bin/endlessh-go -logtostderr -host=x.x.x.x -port=22 -enable_prometheus -prometheus_host=x.x.x.x -prometheus_port=2112 -geoip_supplier=ip-api

May 02 16:06:01 nixos endlessh-go[271204]: W0502 16:06:01.326615  271204 client.go:60] Failed to obatin the geohash of 218.92.0.97: Get "http://ip-api.com/json/218.92.0.97": dial tcp: lookup ip-api.com on [::1]:53: read udp [::1]:34947->[::1]:53: read: connection refused.
May 02 16:11:30 nixos endlessh-go[271204]: W0502 16:11:30.645957  271204 client.go:60] Failed to obatin the geohash of 180.101.88.196: Get "http://ip-api.com/json/180.101.88.196": dial tcp: lookup ip-api.com on [::1]:53: read udp [::1]:34406->[::1]:53: read: connection refused.
May 02 16:11:57 nixos endlessh-go[271204]: W0502 16:11:57.753493  271204 client.go:60] Failed to obatin the geohash of 218.92.0.96: Get "http://ip-api.com/json/218.92.0.96": dial tcp: lookup ip-api.com on [::1]:53: read udp [::1]:53675->[::1]:53: read: connection refused.
May 02 16:12:19 nixos endlessh-go[271204]: W0502 16:12:19.649602  271204 client.go:60] Failed to obatin the geohash of 218.92.0.113: Get "http://ip-api.com/json/218.92.0.113": dial tcp: lookup ip-api.com on [::1]:53: read udp [::1]:50887->[::1]:53: read: connection refused.
May 02 16:12:50 nixos endlessh-go[271204]: W0502 16:12:50.385315  271204 client.go:60] Failed to obatin the geohash of 94.203.171.157: Get "http://ip-api.com/json/94.203.171.157": dial tcp: lookup ip-api.com on [::1]:53: read udp [::1]:46093->[::1]:53: read: connection refused.
May 02 16:13:07 nixos endlessh-go[271204]: W0502 16:13:07.635677  271204 client.go:60] Failed to obatin the geohash of 183.129.208.82: Get "http://ip-api.com/json/183.129.208.82": dial tcp: lookup ip-api.com on [::1]:53: read udp [::1]:47188->[::1]:53: read: connection refused.
May 02 16:13:12 nixos endlessh-go[271204]: W0502 16:13:12.766532  271204 client.go:60] Failed to obatin the geohash of 121.135.254.129: Get "http://ip-api.com/json/121.135.254.129": dial tcp: lookup ip-api.com on [::1]:53: read udp [::1]:39893->[::1]:53: read: connection refused.
May 02 16:13:22 nixos endlessh-go[271204]: W0502 16:13:22.202812  271204 client.go:60] Failed to obatin the geohash of 218.92.0.96: Get "http://ip-api.com/json/218.92.0.96": dial tcp: lookup ip-api.com on [::1]:53: read udp [::1]:41506->[::1]:53: read: connection refused.
May 02 16:14:44 nixos endlessh-go[271204]: W0502 16:14:44.648870  271204 client.go:60] Failed to obatin the geohash of 218.92.0.96: Get "http://ip-api.com/json/218.92.0.96": dial tcp: lookup ip-api.com on [::1]:53: read udp [::1]:45123->[::1]:53: read: connection refused.
May 02 16:16:06 nixos endlessh-go[271204]: W0502 16:16:06.965790  271204 client.go:60] Failed to obatin the geohash of 218.92.0.96: Get "http://ip-api.com/json/218.92.0.96": dial tcp: lookup ip-api.com on [::1]:53: read udp [::1]:45047->[::1]:53: read: connection refused.

Running on NixOS 23.11. Relevant config:

services.endlessh-go = {
  enable = true;
  listenAddress = "<PUBLIC IP>";
  port = 22;
  prometheus = {
    enable = true;
    listenAddress = "<VPN IP>";
  };
  extraOptions = [
    "-geoip_supplier=ip-api"
  ];
};
V3ntus commented 4 months ago

CC: @azahi listed as the maintainer of endlessh-go on https://github.com/nixos/nixpkgs

azahi commented 4 months ago

I cannot reproduce this. You can try updating to the latest revision from here, I've disabled CGO in it, maybe this would help with DNS resolution in your case.

V3ntus commented 4 months ago

I've altered my config to reference the mentioned PR's changes, but I am still getting the DNS issues. I'll see about setting up resolved in the meantime.

NixOS config:

{
  config,
  lib,
  pkgs,
  ...
}:
let
  endlessh-go = (import (builtins.fetchTarball {
        url = "https://github.com/azahi/nixpkgs/archive/6a67984ae076908dd92c466911aa7ebabcf0f7b3.tar.gz";
      }) {}).endlessh-go;
in {
  systemd.services.endlessh-go.serviceConfig.ExecStart = let cfg = config.services.endlessh-go;
    in lib.mkForce (lib.concatStringsSep " " ([
            "${endlessh-go}/bin/endlessh-go"
            "-logtostderr"
            "-host=${cfg.listenAddress}"
            "-port=${toString cfg.port}"
          ] ++ lib.optionals cfg.prometheus.enable [
            "-enable_prometheus"
            "-prometheus_host=${cfg.prometheus.listenAddress}"
            "-prometheus_port=${toString cfg.prometheus.port}"
          ] ++ cfg.extraOptions));
}

systemctl status:

[ventus@nixos:~]$ sudo systemctl status endlessh-go.service 
● endlessh-go.service - SSH tarpit
     Loaded: loaded (/etc/systemd/system/endlessh-go.service; enabled; preset: enabled)
     Active: active (running) since Mon 2024-05-06 18:59:15 UTC; 3min 0s ago
   Main PID: 290593 (endlessh-go)
         IP: 1.0K in, 1.8K out
         IO: 0B read, 0B written
      Tasks: 3 (limit: 1152)
     Memory: 1.4M
        CPU: 34ms
     CGroup: /system.slice/endlessh-go.service
             └─290593 /nix/store/ciszpmc2959ziyb4zwyq4qyy8j45ygwy-endlessh-go-2024.0119.1/bin/endlessh-go -logtostderr -host=x.x.x.x -port=22 -enable_prometheus -prometheus_host=x.x.x.x -prometheus_port=2112 -geoip_supplier=ip-api

May 06 18:59:15 nixos systemd[1]: Started SSH tarpit.
May 06 18:59:16 nixos endlessh-go[290593]: I0506 18:59:16.086026  290593 main.go:78] Listening on x.x.x.x:22
May 06 18:59:16 nixos endlessh-go[290593]: I0506 18:59:16.086325  290593 metrics.go:90] Starting Prometheus on x.x.x.x:2112, entry point is /metrics
May 06 19:00:42 nixos endlessh-go[290593]: W0506 19:00:42.666077  290593 metrics.go:128] Failed to obatin the geohash of x.x.x.x: Get "http://ip-api.com/json/x.x.x.x": dial tcp: lookup ip-api.com on [::1]:53: read udp [::1]:37016->[::1]:53: read: connection refused.
V3ntus commented 4 months ago

resolved enabled and populated networking.nameservers, but still same results.

{
  networking.nameservers = [ "1.1.1.1#one.one.one.one" "1.0.0.1#one.one.one.one" ];
  services.resolved = {
    enable = true;
    domains = [ "~." ];
    fallbackDns = [ "1.1.1.1#one.one.one.one" "1.0.0.1#one.one.one.one" ];
    dnsovertls = "true";
  };
}
# /etc/resolv.conf
nameserver 127.0.0.53
options edns0 trust-ad
search .

Below is additional networking configuration. networking.useDHCP = false may be another suspect? image

Adding an entry to networking.hosts has no effect either.