ConnectionResetError when using metrics server and crawler requests metrics

[lars-devs]

Hi, I enabled the metrics server to create fancy graphs with Grafana and found out, that Censys scanners tries to access the metrics. This requests lead to an exception.

Fully blocking access from Censys is possible, though.

2024-05-08 00:25:11 INFO       ____  ___  ____  ___    ____   ___   __   __ _  __ _  ____  ____
2024-05-08 00:25:11 INFO      (_  _)/ __)(_  _)/ __)  / ___) / __) / _\ (  ( \(  ( \(  __)(  _ \
2024-05-08 00:25:11 INFO        )( ( (_ \  )( ( (_ \  \___ \( (__ /    \/    //    / ) _)  )   /
2024-05-08 00:25:11 INFO       (__) \___/ (__) \___/  (____/ \___)\_/\_/\_)__)\_)__)(____)(__\_)
2024-05-08 00:25:11 INFO
2024-05-08 00:25:11 INFO     Version 1.20.3
2024-05-08 00:25:11 INFO     ©2022, Henning Merklinger
2024-05-08 00:25:11 INFO     For documentation and support please visit https://github.com/Der-Henning/tgtg/wiki
2024-05-08 00:25:11 INFO
2024-05-08 00:25:11 INFO     Metrics server startet on port 8000
2024-05-08 00:25:11 INFO     Activated notifiers:
2024-05-08 00:25:11 INFO     - SMTP: ['REDACTED']
2024-05-08 00:25:11 INFO     Scanner started ...
2024-05-08 00:25:11 INFO     Active on schedule: Jede Minute, zwischen 06:00 und 23:59, nur am Wednesday und Saturday
2024-05-08 00:25:11 INFO     Scanner disabled by cron schedule.
----------------------------------------
Exception occurred during processing of request from ('167.94.138.113', 48408)
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/socketserver.py", line 691, in process_request_thread
    self.finish_request(request, client_address)
  File "/usr/local/lib/python3.11/socketserver.py", line 361, in finish_request
    self.RequestHandlerClass(request, client_address, self)
  File "/usr/local/lib/python3.11/socketserver.py", line 755, in __init__
    self.handle()
  File "/usr/local/lib/python3.11/wsgiref/simple_server.py", line 118, in handle
    self.raw_requestline = self.rfile.readline(65537)
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/socket.py", line 706, in readinto
    return self._sock.recv_into(b)
           ^^^^^^^^^^^^^^^^^^^^^^^
ConnectionResetError: [Errno 104] Connection reset by peer
----------------------------------------
----------------------------------------
Exception occurred during processing of request from ('104.152.52.102', 47105)
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/socketserver.py", line 691, in process_request_thread
    self.finish_request(request, client_address)
  File "/usr/local/lib/python3.11/socketserver.py", line 361, in finish_request
    self.RequestHandlerClass(request, client_address, self)
  File "/usr/local/lib/python3.11/socketserver.py", line 755, in __init__
    self.handle()
  File "/usr/local/lib/python3.11/wsgiref/simple_server.py", line 118, in handle
    self.raw_requestline = self.rfile.readline(65537)
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/socket.py", line 706, in readinto
    return self._sock.recv_into(b)
           ^^^^^^^^^^^^^^^^^^^^^^^
ConnectionResetError: [Errno 104] Connection reset by peer
----------------------------------------
----------------------------------------
Exception occurred during processing of request from ('199.45.155.45', 47020)
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/socketserver.py", line 691, in process_request_thread
    self.finish_request(request, client_address)
  File "/usr/local/lib/python3.11/socketserver.py", line 361, in finish_request
    self.RequestHandlerClass(request, client_address, self)
  File "/usr/local/lib/python3.11/socketserver.py", line 755, in __init__
    self.handle()
  File "/usr/local/lib/python3.11/wsgiref/simple_server.py", line 118, in handle
    self.raw_requestline = self.rfile.readline(65537)
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/socket.py", line 706, in readinto
    return self._sock.recv_into(b)
           ^^^^^^^^^^^^^^^^^^^^^^^
ConnectionResetError: [Errno 104] Connection reset by peer
----------------------------------------```

[Der-Henning]

Hi @lars-devs. First I strongly recommend to not expose any ports to the public internet. If you want to do so you should use a more random port. Censys only scans ports that are associated with standard services.

[lars-devs]

Hi @Der-Henning,

Regarding the exposed ports, You're absolutely right. The reason why I decided to do so is that I run the scanner on three servers (1 account per server), each one taking care of 2-3 weekdays to prevent rate limit. On one server I run Prometheus to scrape the three servers and Grafana to visualize.

Do You have an idea, how to easily manage to take the metric data from my servers into one Grafana instance?

Regards, Lars

[Der-Henning]

I would recommend creating a vpn with wireguard on one of your servers and connect all your servers to the vpn. This is especially useful for your server management as you do not have to expose your ssh port to the internet (see latest hack https://nvd.nist.gov/vuln/detail/CVE-2024-3094) by connecting to the vpn with your local machine. Also grafana is only available inside the vpn and not exposed to the internet. If you use a fritzbox for your home network the newest fritzos includes a wireguard server. Using this your remote servers can be available as if they are in your local network. Another way (exposing your hopefully random ssh port) is to use port forwarding via ssh to forward the metrics ports to your grafana / prometheus server.

[lars-devs]

Thanks for Your suggestions! I'll take a look at the VPN approach to avoid exposing unnecessarily exposed ports.