[Feature Request] Support Remote NPM Instance

[CrazyWolf13]

Hi I run a proxmox Cluster where everything is a bit seperated.

I have currently a NPM linuxcontainer (LXC) and a VM that runs all my docker containers, so it would be really really awesome, if this app could fetch that over the network.

Or have you got any ideas of possible workarounds?

[smilebasti]

Hi, you could download sendips.sh and Getipinfo.py into the lxc. Just autostart sendips.sh at boot, change the directories in the files and hardcode the influx host, ip and creditials. Host the influx and grafana container on the docker host.

Hope this helps.

[CrazyWolf13]

Hi, I think I've got that alls et up with geolite DB etc.

But when running the sendips.sh script I just get this as response, a live monitor of which domains I access, nothing more. At the very top is my public IP.

My Influx receives no data. I've filled in correct influxdb credentials into the getipinfo file. How could I further debug this?

Internal IP-Source: 192.168.1.xxx called: my.domain.dev Internal IP-Source: 192.168.1.xxx called: my.domain.dev

[CrazyWolf13]

#!/bin/bash
# Get Reverse Proxy-host logs

# To better understand the collection with regular expression
# {1,3} get one to three characters. [0-9] character from to 0 to 9. \ for special characters. () grouping as an expression. | or
internalips="(10([\.][0-9]{1,3}){3})|(192.168([\.][0-9]{1,3}){2})|(172.(1[6-9]|2[0-9]|3[0-1])([\.][0-9]{1,3}){2})"
externalip=`curl -s ifconfig.me/ip`
echo "Your external IP is: $externalip"

# check if monitoringips.txt exists
if [ -f "/monitoringips.txt" ]
then
  monitorfile=true
else
  monitorfile=false
fi
# check if ASN DB exists
if [ -f "/geolite/GeoLite2-ASN.mmdb" ]
then
  asndb=true
else
  asndb=false
fi

# gets all lines including an IP.
# Grep finds the the IP addresses in the access.log
tail -F /data/logs/proxy-host-*_access.log | grep --line-buffered -E "([0-9]{1,3}[\.]){3}[0-9]{1,3}" | while read line;

do
  # Domain or subdomain gets found.
  targetdomain=`echo $line | grep --line-buffered -m 1 -o -E "([a-z0-9\-]*\.){1,3}?[a-z0-9\-]*\.[A-Za-z]{2,6}" | head -1`

  # Get the first ip found = outsideip
  # head -1 because grep finds two (sometimes three) and only the first is needed
  outsideip=`echo $line | grep --line-buffered -o -m 1 -E "(([0-9]{1,3}[\.]){3}[0-9]{1,3}|([0-9a-fA-F]{1,4}:){7,7}[0-9a-fA-F]{1,4}|([0-9a-fA-F]{1,4}:){1,7}:|([0-9a-fA-F]{1,4}:){1,6}:[0-9a-fA-F]{1,4}|([0-9a-fA-F]{1,4}:){1,5}(:[0-9a-fA-F]{1,4}){1,2}|([0-9a-fA-F]{1,4}:){1,4}(:[0-9a-fA-F]{1,4}){1,3}|([0-9a-fA-F]{1,4}:){1,3}(:[0-9a-fA-F]{1,4}){1,4}|([0-9a-fA-F]{1,4}:){1,2}(:[0-9a-fA-F]{1,4}){1,5}|[0-9a-fA-F]{1,4}:((:[0-9a-fA-F]{1,4}){1,6})|:((:[0-9a-fA-F]{1,4}){1,7}|:)|fe80:(:[0-9a-fA-F]{0,4}){0,4}%[0-9a-zA-Z]{1,}|::(ffff(:0{1,4}){0,1}:){0,1}((25[0-5]|(2[0-4]|1{0,1}[0-9]){0,1}[0-9])\.){3,3}(25[0-5]|(2[0-4]|1{0,1}[0-9]){0,1}[0-9])|([0-9a-fA-F]{1,4}:){1,4}:((25[0-5]|(2[0-4]|1{0,1}[0-9]){0,1}[0-9])\.){3,3}(25[0-5]|(2[0-4]|1{0,1}[0-9]){0,1}[0-9]))" | head -1`

  # head -2 and tail -1 because grep finds two (sometimes three) and only the second is needed
  targetip=`echo $line | grep --line-buffered -o -m 1 -E "(([0-9]{1,3}[\.]){3}[0-9]{1,3}|([0-9a-fA-F]{1,4}:){7,7}[0-9a-fA-F]{1,4}|([0-9a-fA-F]{1,4}:){1,7}:|([0-9a-fA-F]{1,4}:){1,6}:[0-9a-fA-F]{1,4}|([0-9a-fA-F]{1,4}:){1,5}(:[0-9a-fA-F]{1,4}){1,2}|([0-9a-fA-F]{1,4}:){1,4}(:[0-9a-fA-F]{1,4}){1,3}|([0-9a-fA-F]{1,4}:){1,3}(:[0-9a-fA-F]{1,4}){1,4}|([0-9a-fA-F]{1,4}:){1,2}(:[0-9a-fA-F]{1,4}){1,5}|[0-9a-fA-F]{1,4}:((:[0-9a-fA-F]{1,4}){1,6})|:((:[0-9a-fA-F]{1,4}){1,7}|:)|fe80:(:[0-9a-fA-F]{0,4}){0,4}%[0-9a-zA-Z]{1,}|::(ffff(:0{1,4}){0,1}:){0,1}((25[0-5]|(2[0-4]|1{0,1}[0-9]){0,1}[0-9])\.){3,3}(25[0-5]|(2[0-4]|1{0,1}[0-9]){0,1}[0-9])|([0-9a-fA-F]{1,4}:){1,4}:((25[0-5]|(2[0-4]|1{0,1}[0-9]){0,1}[0-9])\.){3,3}(25[0-5]|(2[0-4]|1{0,1}[0-9]){0,1}[0-9]))" | head -2| tail -1`

  # What does length say?
  # save from 14 postion after space and only the first digits found to length
  length=`echo $line | awk -F ' ' '{print$14}' | grep --line-buffered -m 1 -o '[[:digit:]]*'`

  # get time from logs
  measurementtime=`echo ${line:1:26} `
  #echo "measurement time: $measurementtime"

  #Idea of getting device
  #device=`echo $line | grep -e ""'('*')'""`

  if [[ $outsideip =~ $internalips ]] || [[ $outsideip =~ $externalip ]]
  then
    echo "Internal IP-Source: $outsideip called: $targetdomain"
    if [ "$INTERNAL_LOGS" = "TRUE" ]
    then
      echo "$outsideip" "$targetdomain" "$length" "$targetip" "InternalRProxyIPs" "$measurementtime"
      python /root/.config/NPMGRAF/Internalipinfo.py "$outsideip" "$targetdomain" "$length" "$targetip" "InternalRProxyIPs" "$measurementtime"
    fi
  elif $monitorfile && grep --line-buffered -qFx $outsideip /monitoringips.txt
  then
    echo "An excluded monitoring service checked: $targetdomain"
    if [ "$MONITORING_LOGS" = "TRUE" ]
    then
      echo "$outsideip" "$targetdomain" "$length" "$targetip" "MonitoringRProxyIPs" "$measurementtime" "$asndb"
      python /root/.config/NPMGRAF/Getipinfo.py "$outsideip" "$targetdomain" "$length" "$targetip" "MonitoringRProxyIPs" "$measurementtime" "$asndb"
    fi
  else
    echo "$outsideip" "$targetdomain" "$length" "$targetip" "ReverseProxyConnections" "$measurementtime" "$asndb"
    python /root/.config/NPMGRAF/Getipinfo.py "$outsideip" "$targetdomain" "$length" "$targetip" "ReverseProxyConnections" "$measurementtime" "$asndb"
  fi
done

None of those test echo statements near the python3 commands get ever readched nor printed.

Edit: I removed that reboot at the very bootom, don't really like that.

[smilebasti]

Good to see.

The logs say that there are only internal/private ip calls. Maybe try calling your domain with an external ip.

Try removing the if statement if [ "$INTERNAL_LOGS" = "TRUE" ] or setting INTERNAL_LOGS to 'TRUE'. Then the internal ip calls will get send to InfluxDB.

[CrazyWolf13]

Thanks, that did the trick!

I saw this line, how exactly can I exclude an ip? Because I got an internal monitoring service.

elif $monitorfile && grep --line-buffered -qFx $outsideip /monitoringips.txt

Also can I test that the geolite DB works, without having an external IP connect to a domain?

As I have not yet opened up anything.

[smilebasti]

Good to hear.

If you write the local monitoring system ip into the monitoringips.txt file, then that should exclude it.

You will need an external ip to test maxminddb. What sense would it make to use private ips? Just look if the file exists.

[CrazyWolf13]

I created this file in the same folder as the scripts, however the IP's still get logged:

Okay I see. Yeah I just don't have NPM open to the public yet, only with local dns records, until I secured everything enough, it will stay local-only.

[smilebasti]

The script looks for monitoringips.txt in the / directory. Either change the location in the script or the file location.